{"course":{"slug":"ai-careers","title":"Working in AI","url":"https://addaly.com/learn/ai-careers","level":"beginner","track":"work"},"lesson":{"slug":"ai-engineer-in-depth","title":"The AI engineer","url":"https://addaly.com/learn/ai-careers/ai-engineer-in-depth","position":15,"of":85,"minutes":10,"answer":"An AI engineer's core skill is separating retrieval failures from generation failures and trading quality, cost, latency and risk with actual numbers.","keyIdea":"An AI engineer's core skill is separating retrieval failures from generation failures and trading quality, cost, latency and risk with actual numbers."},"body":{"format":"markdown","text":"## A role built on somebody else's model\n\nThe AI engineer builds products on top of models they did not train, usually reached through an API. Retrieval systems, assistants, document extraction, classification pipelines, agents that take actions. The job existed in a recognisable form from about 2023 and is now one of the most commonly advertised titles in the field.\n\nIt is closer to software engineering than to data science. If you can build and operate a web service and you understand evaluation, you are most of the way there.\n\n## The week\n\nA support assistant answers customer questions from your documentation. Week one it demos beautifully. Week two, someone asks about a product you discontinued and it invents a refund policy. Week three you build a test set of 200 real questions with correct answers, and discover the assistant is right 71% of the time, which nobody knew because nobody had counted. Week four you find that half the failures are retrieval (the right document was never fetched) and the model was never the problem.\n\nThat split is the single most useful diagnostic in this job. **When the output is wrong, was the right context retrieved?** If no, fix retrieval: chunking, embeddings, the query itself. If yes, fix generation: the prompt, the model, the output format. Teams that do not separate these two spend months tuning prompts against a retrieval bug.\n\n> Figure: Two hundred real questions, sorted by where they failed. A two by two grid, rows Right context fetched and Wrong or no context, columns Answer correct and Answer wrong: Right context fetched and Answer correct: 138; Right context fetched and Answer wrong: 29; Wrong or no context and Answer correct: 4; Wrong or no context and Answer wrong: 29. Seventy-one per cent right. Of the 58 failures, half sit in the bottom row: the right document was never fetched, and no amount of prompt tuning could have fixed them. The four lucky answers in the bottom-left are not to be trusted either.\n\n## The four constraints you are always trading\n\n1. **Quality**, measured on a set you built, not on vibes.\n2. **Cost**, measured per request in currency. A 40,000-token prompt on every call is a business decision, not a technical detail.\n3. **Latency**, measured at the 95th percentile, not the average. Users experience the slow tail.\n4. **Risk**: what happens when it is wrong, and what the system does to bound that.\n\nAlmost every design argument in this role is one of these traded against another. A smaller model is cheaper and faster and worse. More retrieved context is better and slower and dearer. Being able to say \"this change costs 1.9 cents per request and buys nine points of accuracy\" is what makes somebody senior in this work.\n\n## Non-determinism changes how you build\n\nThe same input can produce different output. That breaks the habits of ordinary software testing and it is why evaluation, not debugging, is the core discipline. You do not step through a failure; you measure a rate over a set of cases, change something, and measure again.\n\nPractical consequences: pin model versions, because a provider's silent update will move your numbers; log the full prompt and response for every request so a failure can be reproduced; and keep a regression set that every change must pass before it ships.\n\n## The stack\n\n- **Python** or **TypeScript**, and ordinary backend engineering: queues, caching, retries, rate limits, timeouts.\n- **A model API**: the commercial providers, or open-weight models. Free paths that work: Ollama or llama.cpp running a small open-weight model on a laptop, and Hugging Face for models and datasets.\n- **A vector store**: pgvector inside PostgreSQL is free, ordinary, and sufficient for most systems; the dedicated services matter at scale.\n- **An evaluation harness.** You can and should write your own before adopting a framework; the free ones worth knowing are promptfoo, Ragas and DeepEval.\n- **Observability.** Log every call with its cost and latency, or you will not be able to answer the first question finance asks.\n\n## Two routes in\n\n**From backend engineering.** Short. Add retrieval, evaluation and cost awareness to what you already do.\n\n**From analysis, support, teaching or a domain job.** Longer but real, and this route has one genuine advantage: you know what a good answer looks like in the domain, which is the scarce half. Build a small assistant over documents you actually understand, then measure it honestly and publish the number, including the failures.\n\n## The failure that ends the first year\n\nShipping on demos. A system that impresses in a meeting and has never been measured is not a product; it is a rehearsal. The person who says \"we are at 71% on 200 real questions and here are the four failure categories\" is trusted immediately, even when the number is bad. Especially when the number is bad.\n\n## The limitation\n\nThis is the least settled role in the field. The tooling changes every few months, half the frameworks that were essential two years ago are abandoned, and job titles are still inconsistent. What transfers is not the framework: it is retrieval, evaluation, cost control and system design. Invest there and the churn stops mattering."},"check":{"question":"A retrieval-based assistant gives a wrong answer about a discontinued product. The team spends two weeks rewriting the system prompt with no improvement. What did they most likely skip?","options":[{"text":"Checking whether the correct document was retrieved at all before assuming the model's generation was at fault.","correct":true},{"text":"Upgrading to a larger model, which would have handled the ambiguity better.","correct":false,"tempting":"Spends more money on generation for a case where the evidence may never have reached the model."},{"text":"Lowering the temperature so the output is deterministic and reproducible.","correct":false,"tempting":"Reduces variability without changing whether the necessary information was present in the context."},{"text":"Adding a disclaimer so users know answers may be inaccurate.","correct":false,"tempting":"Manages the consequence rather than diagnosing the cause, and leaves the same failure in place for every other query."}]},"prev":{"slug":"ml-engineer-in-depth","title":"The machine learning engineer","url":"https://addaly.com/learn/ai-careers/ml-engineer-in-depth"},"next":{"slug":"platform-and-mlops-in-depth","title":"The platform and MLOps engineer","url":"https://addaly.com/learn/ai-careers/platform-and-mlops-in-depth"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The AI engineer', in Working in AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-careers/ai-engineer-in-depth (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The AI engineer","courseTitle":"Working in AI","module":{"index":2,"slug":"the-roles-in-depth","title":"The roles, one at a time"},"answer":null,"keyIdea":"An AI engineer’s core skill is separating retrieval failures from generation failures and trading quality, cost, latency and risk with actual numbers.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Answer correct","Answer wrong"],"rows":["Right context fetched","Wrong or no context"],"type":"matrix","cells":[["138","29"],["4","29"]],"title":"Two hundred real questions, sorted by where they failed","caption":"Seventy-one per cent right. Of the 58 failures, half sit in the bottom row: the right document was never fetched, and no amount of prompt tuning could have fixed them. The four lucky answers in the bottom-left are not to be trusted either.","cellLabels":[["working","fix generation"],["lucky, not working","fix retrieval"]]},"alt":"Figure: Two hundred real questions, sorted by where they failed. A two by two grid, rows Right context fetched and Wrong or no context, columns Answer correct and Answer wrong: Right context fetched and Answer correct: 138; Right context fetched and Answer wrong: 29; Wrong or no context and Answer correct: 4; Wrong or no context and Answer wrong: 29. Seventy-one per cent right. Of the 58 failures, half sit in the bottom row: the right document was never fetched, and no amount of prompt tuning could have fixed them. The four lucky answers in the bottom-left are not to be trusted either."}},"blocks":[{"t":"h2","id":"a-role-built-on-somebody-else-s-model","runs":[{"text":"A role built on somebody else’s model"}]},{"t":"p","runs":[{"text":"The AI engineer builds products on top of models they did not train, usually reached through an API. Retrieval systems, assistants, document extraction, classification pipelines, agents that take actions. The job existed in a recognisable form from about 2023 and is now one of the most commonly advertised titles in the field."}]},{"t":"p","runs":[{"text":"It is closer to software engineering than to data science. If you can build and operate a web service and you understand evaluation, you are most of the way there."}]},{"t":"h2","id":"the-week","runs":[{"text":"The week"}]},{"t":"p","runs":[{"text":"A support assistant answers customer questions from your documentation. Week one it demos beautifully. Week two, someone asks about a product you discontinued and it invents a refund policy. Week three you build a test set of 200 real questions with correct answers, and discover the assistant is right 71% of the time, which nobody knew because nobody had counted. Week four you find that half the failures are retrieval (the right document was never fetched) and the model was never the problem."}]},{"t":"p","runs":[{"text":"That split is the single most useful diagnostic in this job. "},{"text":"When the output is wrong, was the right context retrieved?","b":true},{"text":" If no, fix retrieval: chunking, embeddings, the query itself. If yes, fix generation: the prompt, the model, the output format. Teams that do not separate these two spend months tuning prompts against a retrieval bug."}]},{"t":"h2","id":"the-four-constraints-you-are-always-trading","runs":[{"text":"The four constraints you are always trading"}]},{"t":"ol","start":1,"items":[[{"text":"Quality","b":true},{"text":", measured on a set you built, not on vibes."}],[{"text":"Cost","b":true},{"text":", measured per request in currency. A 40,000-token prompt on every call is a business decision, not a technical detail."}],[{"text":"Latency","b":true},{"text":", measured at the 95th percentile, not the average. Users experience the slow tail."}],[{"text":"Risk","b":true},{"text":": what happens when it is wrong, and what the system does to bound that."}]]},{"t":"p","runs":[{"text":"Almost every design argument in this role is one of these traded against another. A smaller model is cheaper and faster and worse. More retrieved context is better and slower and dearer. Being able to say “this change costs 1.9 cents per request and buys nine points of accuracy” is what makes somebody senior in this work."}]},{"t":"h2","id":"non-determinism-changes-how-you-build","runs":[{"text":"Non-determinism changes how you build"}]},{"t":"p","runs":[{"text":"The same input can produce different output. That breaks the habits of ordinary software testing and it is why evaluation, not debugging, is the core discipline. You do not step through a failure; you measure a rate over a set of cases, change something, and measure again."}]},{"t":"p","runs":[{"text":"Practical consequences: pin model versions, because a provider’s silent update will move your numbers; log the full prompt and response for every request so a failure can be reproduced; and keep a regression set that every change must pass before it ships."}]},{"t":"h2","id":"the-stack","runs":[{"text":"The stack"}]},{"t":"ul","items":[[{"text":"Python","b":true},{"text":" or "},{"text":"TypeScript","b":true},{"text":", and ordinary backend engineering: queues, caching, retries, rate limits, timeouts."}],[{"text":"A model API","b":true},{"text":": the commercial providers, or open-weight models. Free paths that work: Ollama or llama.cpp running a small open-weight model on a laptop, and Hugging Face for models and datasets."}],[{"text":"A vector store","b":true},{"text":": pgvector inside PostgreSQL is free, ordinary, and sufficient for most systems; the dedicated services matter at scale."}],[{"text":"An evaluation harness.","b":true},{"text":" You can and should write your own before adopting a framework; the free ones worth knowing are promptfoo, Ragas and DeepEval."}],[{"text":"Observability.","b":true},{"text":" Log every call with its cost and latency, or you will not be able to answer the first question finance asks."}]]},{"t":"h2","id":"two-routes-in","runs":[{"text":"Two routes in"}]},{"t":"p","runs":[{"text":"From backend engineering.","b":true},{"text":" Short. Add retrieval, evaluation and cost awareness to what you already do."}]},{"t":"p","runs":[{"text":"From analysis, support, teaching or a domain job.","b":true},{"text":" Longer but real, and this route has one genuine advantage: you know what a good answer looks like in the domain, which is the scarce half. Build a small assistant over documents you actually understand, then measure it honestly and publish the number, including the failures."}]},{"t":"h2","id":"the-failure-that-ends-the-first-year","runs":[{"text":"The failure that ends the first year"}]},{"t":"p","runs":[{"text":"Shipping on demos. A system that impresses in a meeting and has never been measured is not a product; it is a rehearsal. The person who says “we are at 71% on 200 real questions and here are the four failure categories” is trusted immediately, even when the number is bad. Especially when the number is bad."}]},{"t":"h2","id":"the-limitation","runs":[{"text":"The limitation"}],"role":"limit"},{"t":"p","runs":[{"text":"This is the least settled role in the field. The tooling changes every few months, half the frameworks that were essential two years ago are abandoned, and job titles are still inconsistent. What transfers is not the framework: it is retrieval, evaluation, cost control and system design. Invest there and the churn stops mattering."}]}],"check":{"question":[{"text":"A retrieval-based assistant gives a wrong answer about a discontinued product. The team spends two weeks rewriting the system prompt with no improvement. What did they most likely skip?"}],"options":[{"text":[{"text":"Checking whether the correct document was retrieved at all before assuming the model’s generation was at fault."}],"correct":true,"tempting":null},{"text":[{"text":"Upgrading to a larger model, which would have handled the ambiguity better."}],"correct":false,"tempting":[{"text":"Spends more money on generation for a case where the evidence may never have reached the model."}]},{"text":[{"text":"Lowering the temperature so the output is deterministic and reproducible."}],"correct":false,"tempting":[{"text":"Reduces variability without changing whether the necessary information was present in the context."}]},{"text":[{"text":"Adding a disclaimer so users know answers may be inaccurate."}],"correct":false,"tempting":[{"text":"Manages the consequence rather than diagnosing the cause, and leaves the same failure in place for every other query."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"ml-engineer-in-depth","title":"The machine learning engineer","minutes":10},"next":{"slug":"platform-and-mlops-in-depth","title":"The platform and MLOps engineer","minutes":9}}}