Listening notes #4: The Rise of the AI Engineer with Shawn Wang

DanielApril 3, 2024April 4, 2024

Artificial intelligence engineers are developers who make the tools, systems, and processes that enable artificial intelligence to be applied in the real world. An AI engineer facilitates the integration between an LLM and a traditional software app.

One of the first times when the term AI Engineer was introduced was in the essay The Rise of the AI Engineer.

Listened to a few podcasts on the weekend about AI Engineering and here are some of my main learnings.

• ⏱ 09:45 AI engineers are Pro users of AI. They're not exactly training the models but they can customize it, they can fine-tune it but they cannot train a model. And that's absolutely fine.

• ⏱ 13:10 The workflow that is being unlocked here is fundamentally very different than what you're used to. In traditional ML workflows, you collect a whole bunch of data then you train the model then you make a product. With Foundation Models and AI Engineers, this is all flipped. You make an MVP first and if the MVP is successful then you use that MVP to collect a whole bunch of data and if that's successful then you use your custom data to train a model right. So it's a complete inversion of the time to market.

• ⏱ 14:15 Foundation models are available via API. This means that for once machine learning is now language-agnostic programming language agnostic. It used to be pure Python and now anyone that can effectively call an API can now wield the language model primarily JavaScript. Given that the number of Python developers is roughly equal to the number of JavaScript developers then the number of people wielding AI now has effectively doubled or maybe tripled overnight.

• ⏱ 16:30 I think that AI using code is way way way way way more powerful than AI just using language (prompt engineering). Code generation and using code to wield and orchestrate AI is going to be very important.

• ⏱ 18:00 I'm seeing a big shift when you want to scale things up. You will start to run into the limits of intelligence. You realize that these models are not smart enough to do the thing that you're trying to get them to do. Or at least they doing it too expensively. Or they're doing it too slowly. You may want to take business logic out of the LM layer into the code layer and flip the order. Put code at the core of it and then put the LLMs as like small little sort of intelligence augment. More details here.

• ⏱ 07:44 there are 3 types of AI Engineers: First, the AI-enhanced engineers who use AI products such as Copilot or Cursor. Second AI products engineer where you use you expose AI capabilities to the end user; a software engineer, not an ML researcher or ML Engineer but a dev interacting with Foundation Models and their API's. The third one is the non-human AI engineer and the fully autonomous AI dream coder

• ⏱ 26:00 Gemini 1.5 has a million token contexts and it is in research for 10 million tokens. It's so big that means that you no longer think about what you have to put into context. You can put the entire knowledge base in there. Books or films, anything. A lot of people think that it kills RAG. I think that's not true because of cost reasons. You still pay per token.
  Google is like perfectly happy to let you pay a million tokens every single time you make an API call but good luck having a $100 API call. The other thing is it's going to be slow. Then finally my criticism of long context is that it's also not debuggable. If something goes wrong, you can't do the RAGs decomposition to see the source of error.

• ⏱ 28:45 The ability of LLMs to reason across modalities gives us something more than we could individually by using text as a universal interface. Check out Simon Willson's post on Gemini 1.5 video capability where he shot a video of his bookshelf and he was able to give back a complete JSON list of the books and the authors.

• ⏱ 32:00 favorite AI tools God mode browser chat and the Cursor IDE

You can check out my other listening notes here.