{"course":{"slug":"ai-careers","title":"Working in AI","url":"https://addaly.com/learn/ai-careers","level":"beginner","track":"work"},"lesson":{"slug":"ai-product-manager-in-depth","title":"The AI product manager","url":"https://addaly.com/learn/ai-careers/ai-product-manager-in-depth","position":19,"of":85,"minutes":9,"answer":"An AI product manager's job is to state what decision changes, what being wrong costs, and what the fallback is, then insist on a measured number instead of an impression.","keyIdea":"An AI product manager's job is to state what decision changes, what being wrong costs, and what the fallback is, then insist on a measured number instead of an impression."},"body":{"format":"markdown","text":"## Deciding what should be built, and what should not\n\nA product manager owns the problem, not the code. In AI products the job acquires a specific difficulty: the thing you are shipping is probabilistic, so \"does it work\" has no yes-or-no answer, and every feature has a failure rate you must decide is acceptable.\n\nThis changes the craft. An ordinary feature either logs the user in or does not. An AI feature summarises the document correctly 88% of the time, and your job is to decide whether 88% is shippable, what the other 12% costs, and what the product does to make being wrong survivable.\n\n## The week\n\nYou write a one-page problem statement and get three teams to agree it is the right problem. You sit with six support agents and watch them work, and discover the thing they actually hate is not the thing anybody proposed to automate. You spend a morning labelling 100 model outputs yourself, because you will otherwise be arguing about quality with no shared referent. You cut a feature that demos well because its failure mode is a customer receiving a wrong price, and no confidence score makes that acceptable.\n\n## The three questions that define the role\n\n**1. What decision changes?** If the model's output does not alter what somebody does, the feature is decoration however impressive it looks.\n\n**2. What is the cost of being wrong, and who pays it?** Wrong summary of an internal meeting: annoying. Wrong extraction of a dosage from a prescription: dangerous. Wrong refund decision: expensive and unfair. The design follows from this answer: how much human review, how the interface presents uncertainty, whether the system acts or only suggests.\n\n**3. What is the fallback?** Every AI feature needs an answer to \"what happens when it fails\" that is not \"the user notices\". Show sources so the answer is checkable. Offer the manual route beside the automatic one. Fail into a human queue rather than into silence.\n\nA specification that answers those three is more valuable than one containing model choices, and model choices are not your decision anyway.\n\n## The measurement half\n\nYou are the person who insists the number exists. Not \"it feels better since we changed the prompt\", but 200 real cases, labelled, with a number and a date. In practice you will often build the first evaluation set yourself, in a spreadsheet, because nobody else will and the argument cannot be settled without it.\n\nYou also own the second number that engineers rarely raise: cost per interaction. A feature at 4 cents a request used ten times a day by 50,000 users is $730,000 a year. That is a product decision wearing a technical costume, and it is yours.\n\n## The stack\n\n- **Writing**, above everything. The core artefact is a document.\n- **SQL**, enough to answer your own questions without queuing behind an analyst.\n- **A spreadsheet**, for the evaluation set and the unit economics.\n- **Enough technical literacy** to know what is hard: that latency and quality trade against each other, what retrieval is, why a model cannot cite a document it never saw, and why \"just fine-tune it\" is not a small ask.\n- Free paths for prototyping: Streamlit or Gradio let a non-engineer put a working demo in front of a user in an afternoon, and both are free.\n\n## Two routes in\n\n**From a domain job.** Support lead, operations manager, clinician, teacher, underwriter. You know the workflow being changed, which is the half that cannot be learned quickly. Add the measurement discipline.\n\n**From analysis or engineering.** You have the numbers; you learn the customer research and the writing, and unlearn the instinct to answer \"how\" before \"whether\".\n\n## The failure that ends the first year\n\nWriting \"make it smarter\" as a requirement. It is not actionable, it cannot be tested, and it hands the hardest decision (what good means) to whoever writes the code, who has less information than you do.\n\nThe replacement is a sentence with a number and a consequence in it: *\"On the 300-question set, at least 90% of answers must cite a real source document, because the reviewer's whole job is checking the source and an uncited answer costs them more time than doing it manually.\"* Now an engineer can build, and disagree, on the same terms.\n\n## The limitation\n\nProduct management is difficult to enter from outside, because the role is largely judgement and judgement is hard to evidence in a portfolio. The most reliable route in is sideways, inside a company where people have already seen you make decisions. Plan for that rather than for a cold application, and be sceptical of anyone selling a certificate as the entry."},"check":{"question":"A team proposes an assistant that auto-approves customer refunds under a threshold, at 93% accuracy on a labelled set. What should the product manager establish before agreeing to ship?","options":[{"text":"Whether accuracy can be raised to 97% first, since higher is always safer.","correct":false,"tempting":"Chases a number without asking what the remaining errors do, which is the question that decides whether any threshold is acceptable."},{"text":"Which model the team plans to use and whether a larger one would be better.","correct":false,"tempting":"Takes a decision that belongs to engineering while leaving the product question (the cost and shape of the 7%) unanswered."},{"text":"What the 7% of errors actually look like: wrongly approved versus wrongly refused, who bears each cost, and what the fallback and appeal route is for a customer the system gets wrong.","correct":true},{"text":"How much engineering time the feature will take, so it can be prioritised against other work.","correct":false,"tempting":"Prioritises a feature whose acceptability has not been established, so the estimate is being applied to something that may not be shippable at all."}]},"prev":{"slug":"trust-and-safety-in-depth","title":"Trust, safety and the policy side of AI","url":"https://addaly.com/learn/ai-careers/trust-and-safety-in-depth"},"next":{"slug":"forward-deployed-and-solutions","title":"The forward-deployed and solutions engineer","url":"https://addaly.com/learn/ai-careers/forward-deployed-and-solutions"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The AI product manager', in Working in AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-careers/ai-product-manager-in-depth (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The AI product manager","courseTitle":"Working in AI","module":{"index":2,"slug":"the-roles-in-depth","title":"The roles, one at a time"},"answer":null,"keyIdea":"An AI product manager’s job is to state what decision changes, what being wrong costs, and what the fallback is, then insist on a measured number instead of an impression.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"deciding-what-should-be-built-and-what-should-not","runs":[{"text":"Deciding what should be built, and what should not"}]},{"t":"p","runs":[{"text":"A product manager owns the problem, not the code. In AI products the job acquires a specific difficulty: the thing you are shipping is probabilistic, so “does it work” has no yes-or-no answer, and every feature has a failure rate you must decide is acceptable."}]},{"t":"p","runs":[{"text":"This changes the craft. An ordinary feature either logs the user in or does not. An AI feature summarises the document correctly 88% of the time, and your job is to decide whether 88% is shippable, what the other 12% costs, and what the product does to make being wrong survivable."}]},{"t":"h2","id":"the-week","runs":[{"text":"The week"}]},{"t":"p","runs":[{"text":"You write a one-page problem statement and get three teams to agree it is the right problem. You sit with six support agents and watch them work, and discover the thing they actually hate is not the thing anybody proposed to automate. You spend a morning labelling 100 model outputs yourself, because you will otherwise be arguing about quality with no shared referent. You cut a feature that demos well because its failure mode is a customer receiving a wrong price, and no confidence score makes that acceptable."}]},{"t":"h2","id":"the-three-questions-that-define-the-role","runs":[{"text":"The three questions that define the role"}]},{"t":"p","runs":[{"text":"1. What decision changes?","b":true},{"text":" If the model’s output does not alter what somebody does, the feature is decoration however impressive it looks."}]},{"t":"p","runs":[{"text":"2. What is the cost of being wrong, and who pays it?","b":true},{"text":" Wrong summary of an internal meeting: annoying. Wrong extraction of a dosage from a prescription: dangerous. Wrong refund decision: expensive and unfair. The design follows from this answer: how much human review, how the interface presents uncertainty, whether the system acts or only suggests."}]},{"t":"p","runs":[{"text":"3. What is the fallback?","b":true},{"text":" Every AI feature needs an answer to “what happens when it fails” that is not “the user notices”. Show sources so the answer is checkable. Offer the manual route beside the automatic one. Fail into a human queue rather than into silence."}]},{"t":"p","runs":[{"text":"A specification that answers those three is more valuable than one containing model choices, and model choices are not your decision anyway."}]},{"t":"h2","id":"the-measurement-half","runs":[{"text":"The measurement half"}]},{"t":"p","runs":[{"text":"You are the person who insists the number exists. Not “it feels better since we changed the prompt”, but 200 real cases, labelled, with a number and a date. In practice you will often build the first evaluation set yourself, in a spreadsheet, because nobody else will and the argument cannot be settled without it."}]},{"t":"p","runs":[{"text":"You also own the second number that engineers rarely raise: cost per interaction. A feature at 4 cents a request used ten times a day by 50,000 users is $730,000 a year. That is a product decision wearing a technical costume, and it is yours."}]},{"t":"h2","id":"the-stack","runs":[{"text":"The stack"}]},{"t":"ul","items":[[{"text":"Writing","b":true},{"text":", above everything. The core artefact is a document."}],[{"text":"SQL","b":true},{"text":", enough to answer your own questions without queuing behind an analyst."}],[{"text":"A spreadsheet","b":true},{"text":", for the evaluation set and the unit economics."}],[{"text":"Enough technical literacy","b":true},{"text":" to know what is hard: that latency and quality trade against each other, what retrieval is, why a model cannot cite a document it never saw, and why “just fine-tune it” is not a small ask."}],[{"text":"Free paths for prototyping: Streamlit or Gradio let a non-engineer put a working demo in front of a user in an afternoon, and both are free."}]]},{"t":"h2","id":"two-routes-in","runs":[{"text":"Two routes in"}]},{"t":"p","runs":[{"text":"From a domain job.","b":true},{"text":" Support lead, operations manager, clinician, teacher, underwriter. You know the workflow being changed, which is the half that cannot be learned quickly. Add the measurement discipline."}]},{"t":"p","runs":[{"text":"From analysis or engineering.","b":true},{"text":" You have the numbers; you learn the customer research and the writing, and unlearn the instinct to answer “how” before “whether”."}]},{"t":"h2","id":"the-failure-that-ends-the-first-year","runs":[{"text":"The failure that ends the first year"}]},{"t":"p","runs":[{"text":"Writing “make it smarter” as a requirement. It is not actionable, it cannot be tested, and it hands the hardest decision (what good means) to whoever writes the code, who has less information than you do."}]},{"t":"p","runs":[{"text":"The replacement is a sentence with a number and a consequence in it: "},{"text":"”On the 300-question set, at least 90% of answers must cite a real source document, because the reviewer’s whole job is checking the source and an uncited answer costs them more time than doing it manually.”","i":true},{"text":" Now an engineer can build, and disagree, on the same terms."}]},{"t":"h2","id":"the-limitation","runs":[{"text":"The limitation"}],"role":"limit"},{"t":"p","runs":[{"text":"Product management is difficult to enter from outside, because the role is largely judgement and judgement is hard to evidence in a portfolio. The most reliable route in is sideways, inside a company where people have already seen you make decisions. Plan for that rather than for a cold application, and be sceptical of anyone selling a certificate as the entry."}]}],"check":{"question":[{"text":"A team proposes an assistant that auto-approves customer refunds under a threshold, at 93% accuracy on a labelled set. What should the product manager establish before agreeing to ship?"}],"options":[{"text":[{"text":"Whether accuracy can be raised to 97% first, since higher is always safer."}],"correct":false,"tempting":[{"text":"Chases a number without asking what the remaining errors do, which is the question that decides whether any threshold is acceptable."}]},{"text":[{"text":"Which model the team plans to use and whether a larger one would be better."}],"correct":false,"tempting":[{"text":"Takes a decision that belongs to engineering while leaving the product question (the cost and shape of the 7%) unanswered."}]},{"text":[{"text":"What the 7% of errors actually look like: wrongly approved versus wrongly refused, who bears each cost, and what the fallback and appeal route is for a customer the system gets wrong."}],"correct":true,"tempting":null},{"text":[{"text":"How much engineering time the feature will take, so it can be prioritised against other work."}],"correct":false,"tempting":[{"text":"Prioritises a feature whose acceptability has not been established, so the estimate is being applied to something that may not be shippable at all."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"trust-and-safety-in-depth","title":"Trust, safety and the policy side of AI","minutes":9},"next":{"slug":"forward-deployed-and-solutions","title":"The forward-deployed and solutions engineer","minutes":9}}}