The AI product manager

Deciding what should be built, and what should not

A product manager owns the problem, not the code. In AI products the job acquires a specific difficulty: the thing you are shipping is probabilistic, so “does it work” has no yes-or-no answer, and every feature has a failure rate you must decide is acceptable.

This changes the craft. An ordinary feature either logs the user in or does not. An AI feature summarises the document correctly 88% of the time, and your job is to decide whether 88% is shippable, what the other 12% costs, and what the product does to make being wrong survivable.

The week

You write a one-page problem statement and get three teams to agree it is the right problem. You sit with six support agents and watch them work, and discover the thing they actually hate is not the thing anybody proposed to automate. You spend a morning labelling 100 model outputs yourself, because you will otherwise be arguing about quality with no shared referent. You cut a feature that demos well because its failure mode is a customer receiving a wrong price, and no confidence score makes that acceptable.

The three questions that define the role

1. What decision changes? If the model’s output does not alter what somebody does, the feature is decoration however impressive it looks.

2. What is the cost of being wrong, and who pays it? Wrong summary of an internal meeting: annoying. Wrong extraction of a dosage from a prescription: dangerous. Wrong refund decision: expensive and unfair. The design follows from this answer: how much human review, how the interface presents uncertainty, whether the system acts or only suggests.

3. What is the fallback? Every AI feature needs an answer to “what happens when it fails” that is not “the user notices”. Show sources so the answer is checkable. Offer the manual route beside the automatic one. Fail into a human queue rather than into silence.

A specification that answers those three is more valuable than one containing model choices, and model choices are not your decision anyway.

The measurement half

You are the person who insists the number exists. Not “it feels better since we changed the prompt”, but 200 real cases, labelled, with a number and a date. In practice you will often build the first evaluation set yourself, in a spreadsheet, because nobody else will and the argument cannot be settled without it.

You also own the second number that engineers rarely raise: cost per interaction. A feature at 4 cents a request used ten times a day by 50,000 users is $730,000 a year. That is a product decision wearing a technical costume, and it is yours.

The stack

• Writing, above everything. The core artefact is a document.
• SQL, enough to answer your own questions without queuing behind an analyst.
• A spreadsheet, for the evaluation set and the unit economics.
• Enough technical literacy to know what is hard: that latency and quality trade against each other, what retrieval is, why a model cannot cite a document it never saw, and why “just fine-tune it” is not a small ask.
• Free paths for prototyping: Streamlit or Gradio let a non-engineer put a working demo in front of a user in an afternoon, and both are free.

Two routes in

From a domain job. Support lead, operations manager, clinician, teacher, underwriter. You know the workflow being changed, which is the half that cannot be learned quickly. Add the measurement discipline.

From analysis or engineering. You have the numbers; you learn the customer research and the writing, and unlearn the instinct to answer “how” before “whether”.

The failure that ends the first year

Writing “make it smarter” as a requirement. It is not actionable, it cannot be tested, and it hands the hardest decision (what good means) to whoever writes the code, who has less information than you do.

The replacement is a sentence with a number and a consequence in it: ”On the 300-question set, at least 90% of answers must cite a real source document, because the reviewer’s whole job is checking the source and an uncited answer costs them more time than doing it manually.” Now an engineer can build, and disagree, on the same terms.

The limitation

Product management is difficult to enter from outside, because the role is largely judgement and judgement is hard to evidence in a portfolio. The most reliable route in is sideways, inside a company where people have already seen you make decisions. Plan for that rather than for a cold application, and be sceptical of anyone selling a certificate as the entry.