The AI engineer

A role built on somebody else’s model

The AI engineer builds products on top of models they did not train, usually reached through an API. Retrieval systems, assistants, document extraction, classification pipelines, agents that take actions. The job existed in a recognisable form from about 2023 and is now one of the most commonly advertised titles in the field.

It is closer to software engineering than to data science. If you can build and operate a web service and you understand evaluation, you are most of the way there.

The week

A support assistant answers customer questions from your documentation. Week one it demos beautifully. Week two, someone asks about a product you discontinued and it invents a refund policy. Week three you build a test set of 200 real questions with correct answers, and discover the assistant is right 71% of the time, which nobody knew because nobody had counted. Week four you find that half the failures are retrieval (the right document was never fetched) and the model was never the problem.

That split is the single most useful diagnostic in this job. When the output is wrong, was the right context retrieved? If no, fix retrieval: chunking, embeddings, the query itself. If yes, fix generation: the prompt, the model, the output format. Teams that do not separate these two spend months tuning prompts against a retrieval bug.

The four constraints you are always trading

1. Quality, measured on a set you built, not on vibes.
2. Cost, measured per request in currency. A 40,000-token prompt on every call is a business decision, not a technical detail.
3. Latency, measured at the 95th percentile, not the average. Users experience the slow tail.
4. Risk: what happens when it is wrong, and what the system does to bound that.

Almost every design argument in this role is one of these traded against another. A smaller model is cheaper and faster and worse. More retrieved context is better and slower and dearer. Being able to say “this change costs 1.9 cents per request and buys nine points of accuracy” is what makes somebody senior in this work.

Non-determinism changes how you build

The same input can produce different output. That breaks the habits of ordinary software testing and it is why evaluation, not debugging, is the core discipline. You do not step through a failure; you measure a rate over a set of cases, change something, and measure again.

Practical consequences: pin model versions, because a provider’s silent update will move your numbers; log the full prompt and response for every request so a failure can be reproduced; and keep a regression set that every change must pass before it ships.

The stack

• Python or TypeScript, and ordinary backend engineering: queues, caching, retries, rate limits, timeouts.
• A model API: the commercial providers, or open-weight models. Free paths that work: Ollama or llama.cpp running a small open-weight model on a laptop, and Hugging Face for models and datasets.
• A vector store: pgvector inside PostgreSQL is free, ordinary, and sufficient for most systems; the dedicated services matter at scale.
• An evaluation harness. You can and should write your own before adopting a framework; the free ones worth knowing are promptfoo, Ragas and DeepEval.
• Observability. Log every call with its cost and latency, or you will not be able to answer the first question finance asks.

Two routes in

From backend engineering. Short. Add retrieval, evaluation and cost awareness to what you already do.

From analysis, support, teaching or a domain job. Longer but real, and this route has one genuine advantage: you know what a good answer looks like in the domain, which is the scarce half. Build a small assistant over documents you actually understand, then measure it honestly and publish the number, including the failures.

The failure that ends the first year

Shipping on demos. A system that impresses in a meeting and has never been measured is not a product; it is a rehearsal. The person who says “we are at 71% on 200 real questions and here are the four failure categories” is trusted immediately, even when the number is bad. Especially when the number is bad.

The limitation

This is the least settled role in the field. The tooling changes every few months, half the frameworks that were essential two years ago are abandoned, and job titles are still inconsistent. What transfers is not the framework: it is retrieval, evaluation, cost control and system design. Invest there and the churn stops mattering.