{"course":{"slug":"ai-careers","title":"Working in AI","url":"https://addaly.com/learn/ai-careers","level":"beginner","track":"work"},"lesson":{"slug":"the-metrics-case","title":"\"How would you measure it?\"","url":"https://addaly.com/learn/ai-careers/the-metrics-case","position":55,"of":85,"minutes":10,"answer":"Answer a metrics case by naming the decision, one sensitive primary metric, guardrails and a counter-metric, and when diagnosing a sudden drop, check the instrumentation before the users.","keyIdea":"Answer a metrics case by naming the decision, one sensitive primary metric, guardrails and a counter-metric, and when diagnosing a sudden drop, check the instrumentation before the users."},"body":{"format":"markdown","text":"## A round with three shapes\n\nThe design conversation asks how you would build a system. The metrics case asks how you would know whether it worked. Analyst, data scientist and product-facing roles nearly all have one, and it comes in three forms.\n\n## 1. \"Define success for this feature\"\n\n*\"We are adding saved searches to the app. How would you measure whether it worked?\"*\n\nThe bad answer starts naming metrics. The good answer starts with the decision.\n\n- **Who is the user and what changes for them?** People who search repeatedly stop retyping.\n- **What decision does the measurement inform?** Whether to invest further, or remove it.\n- **One primary metric.** Something close to the behaviour: proportion of returning searchers who reuse a saved search within seven days.\n- **Two or three guardrails**: things that must not get worse. Search latency. Overall search volume. Support tickets.\n- **A counter-metric**: the way this could look good while being bad. If saved searches make people search *less* because they are stuck in narrow saved queries, discovery falls. Name it.\n\nThen say what would make you kill the feature. Interviewers remember candidates who state a failure condition, because most do not.\n\n## The property that separates a good metric from a bad one\n\nThree tests, and being able to name them is most of the round:\n\n- **Sensitive**: it moves when the thing you changed moves. Monthly active users is insensitive to almost everything you can ship in a quarter.\n- **Hard to game**: \"tickets resolved\" rewards closing tickets prematurely; \"tickets resolved without reopening in 14 days\" mostly does not.\n- **Attributable**: you can tell whether your change caused it, which usually means it is close to the change in time and in the causal chain. Revenue is the thing everybody cares about and is almost never a usable feature metric, because forty other things move it.\n\nThe classic failure is a metric that rises when the product gets worse. Time-in-app for a support tool. Number of searches per session for a search engine. Say the words \"this could go up for a bad reason, so I would pair it with…\" and you have signalled more experience than a fluent framework does.\n\n## 2. \"This metric dropped 15% overnight. Diagnose it.\"\n\nThere is a correct order, and the first step is the one that marks experience.\n\n1. **Is it real?** Before anything else: did the logging change, did a release stop firing an event, did the pipeline fail, did a data source stop delivering, is the dashboard's date filter doing something odd. A large share of overnight step-changes are instrumentation, and a candidate who investigates user behaviour before checking whether the number is trustworthy has revealed how much time they have spent doing this.\n2. **Is it everywhere?** Segment: platform, app version, country, browser, new versus returning, paid versus organic. A drop confined to Android 14 is a bug; a drop everywhere is an event.\n3. **Is it us?** Releases, config changes, pricing changes, an experiment ramped to 50%, a marketing campaign that ended.\n4. **Is it the world?** Holiday, competitor launch, outage at a payment provider, a platform policy change, seasonality, and check the same week last year before claiming anything is unprecedented.\n\nSay the order out loud as you go. The round is scored on the search strategy, not on guessing the answer.\n\n> Figure: A metric dropped fifteen per cent overnight. Steps: 1. Is it real? (logging, a release, the pipeline, the date filter); 2. Is it everywhere? (platform, version, country, new vs returning); 3. Is it us? (releases, config, pricing, an experiment at 50%); 4. Is it the world? (holiday, competitor, outage, the same week last year). Check the instrument before the users. A large share of overnight step-changes are a release that stopped firing an event, a failed pipeline or a dashboard filter, and a candidate who investigates user behaviour first has revealed how much of this they have done.\n\n## 3. \"Design an experiment for this\"\n\nSix things, in this order, and none of them is optional:\n\n- **Unit of randomisation**: user, session, device, or something coarser like a city when there are network effects. Getting this wrong invalidates everything after it.\n- **Primary metric and guardrails**, chosen before the test.\n- **Minimum detectable effect and sample size.** Use the arithmetic from the statistics lesson: roughly `16 * p * (1-p) / difference²` per arm. Then say how long that takes at their traffic. This is the single moment in the round where a real number wins it.\n- **Duration**: at least one full weekly cycle, because Tuesday users are not Saturday users.\n- **A stopping rule agreed in advance.** No peeking and stopping at significance, which manufactures false positives.\n- **What you would do with each outcome**, including no difference. If the answer is \"ship it anyway\", the test was theatre and it is better to say so and skip it.\n\n## The limitation\n\nThis round rewards fluency, and fluency is memorisable. Interviewers know that, so a smooth framework recited without a number in it now reads as preparation rather than experience.\n\nThe discriminator is specifics: an actual sample-size calculation, an actual counter-metric for this product, an actual reason to reject the obvious metric. One concrete trade-off stated out loud is worth more than four tidy headings."},"check":{"question":"A candidate is asked why a company's daily active users fell 15% overnight. Which opening move signals the most experience?","options":[{"text":"Segment by platform and country to find where the drop is concentrated.","correct":false,"tempting":"A strong second step that wastes effort if the event was never logged correctly in the first place."},{"text":"Ask whether a release, experiment or pricing change went out the previous evening.","correct":false,"tempting":"Reasonable and specific, but assumes the measurement is sound before that has been established."},{"text":"Check whether the number is real (logging changes, a failed pipeline, a release that stopped firing the event, a broken dashboard filter) before investigating user behaviour at all.","correct":true},{"text":"Compare against the same week last year to rule out seasonality.","correct":false,"tempting":"Useful for gradual movements, but seasonality rarely produces a single-night step change of this size."}]},"prev":{"slug":"the-design-conversation","title":"\"How would you build this?\" answered well","url":"https://addaly.com/learn/ai-careers/the-design-conversation"},"next":{"slug":"talking-about-your-work","title":"Talking about your work without lying or shrinking","url":"https://addaly.com/learn/ai-careers/talking-about-your-work"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) '\"How would you measure it?\"', in Working in AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-careers/the-metrics-case (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"“How would you measure it?”","courseTitle":"Working in AI","module":{"index":6,"slug":"the-interview","title":"The interview, from first screen to signed offer"},"answer":null,"keyIdea":"Answer a metrics case by naming the decision, one sensitive primary metric, guardrails and a counter-metric, and when diagnosing a sudden drop, check the instrumentation before the users.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Is it real?","detail":"logging, a release, the pipeline, the date filter"},{"label":"Is it everywhere?","detail":"platform, version, country, new vs returning"},{"label":"Is it us?","detail":"releases, config, pricing, an experiment at 50%"},{"label":"Is it the world?","detail":"holiday, competitor, outage, the same week last year"}],"title":"A metric dropped fifteen per cent overnight","caption":"Check the instrument before the users. A large share of overnight step-changes are a release that stopped firing an event, a failed pipeline or a dashboard filter, and a candidate who investigates user behaviour first has revealed how much of this they have done."},"alt":"Figure: A metric dropped fifteen per cent overnight. Steps: 1. Is it real? (logging, a release, the pipeline, the date filter); 2. Is it everywhere? (platform, version, country, new vs returning); 3. Is it us? (releases, config, pricing, an experiment at 50%); 4. Is it the world? (holiday, competitor, outage, the same week last year). Check the instrument before the users. A large share of overnight step-changes are a release that stopped firing an event, a failed pipeline or a dashboard filter, and a candidate who investigates user behaviour first has revealed how much of this they have done."}},"blocks":[{"t":"h2","id":"a-round-with-three-shapes","runs":[{"text":"A round with three shapes"}]},{"t":"p","runs":[{"text":"The design conversation asks how you would build a system. The metrics case asks how you would know whether it worked. Analyst, data scientist and product-facing roles nearly all have one, and it comes in three forms."}]},{"t":"h2","id":"1-define-success-for-this-feature","runs":[{"text":"1. “Define success for this feature”"}]},{"t":"p","runs":[{"text":"”We are adding saved searches to the app. How would you measure whether it worked?”","i":true}]},{"t":"p","runs":[{"text":"The bad answer starts naming metrics. The good answer starts with the decision."}]},{"t":"ul","items":[[{"text":"Who is the user and what changes for them?","b":true},{"text":" People who search repeatedly stop retyping."}],[{"text":"What decision does the measurement inform?","b":true},{"text":" Whether to invest further, or remove it."}],[{"text":"One primary metric.","b":true},{"text":" Something close to the behaviour: proportion of returning searchers who reuse a saved search within seven days."}],[{"text":"Two or three guardrails","b":true},{"text":": things that must not get worse. Search latency. Overall search volume. Support tickets."}],[{"text":"A counter-metric","b":true},{"text":": the way this could look good while being bad. If saved searches make people search "},{"text":"less","i":true},{"text":" because they are stuck in narrow saved queries, discovery falls. Name it."}]]},{"t":"p","runs":[{"text":"Then say what would make you kill the feature. Interviewers remember candidates who state a failure condition, because most do not."}]},{"t":"h2","id":"the-property-that-separates-a-good-metric-from-a-bad-one","runs":[{"text":"The property that separates a good metric from a bad one"}]},{"t":"p","runs":[{"text":"Three tests, and being able to name them is most of the round:"}]},{"t":"ul","items":[[{"text":"Sensitive","b":true},{"text":": it moves when the thing you changed moves. Monthly active users is insensitive to almost everything you can ship in a quarter."}],[{"text":"Hard to game","b":true},{"text":": “tickets resolved” rewards closing tickets prematurely; “tickets resolved without reopening in 14 days” mostly does not."}],[{"text":"Attributable","b":true},{"text":": you can tell whether your change caused it, which usually means it is close to the change in time and in the causal chain. Revenue is the thing everybody cares about and is almost never a usable feature metric, because forty other things move it."}]]},{"t":"p","runs":[{"text":"The classic failure is a metric that rises when the product gets worse. Time-in-app for a support tool. Number of searches per session for a search engine. Say the words “this could go up for a bad reason, so I would pair it with…” and you have signalled more experience than a fluent framework does."}]},{"t":"h2","id":"2-this-metric-dropped-15-overnight-diagnose-it","runs":[{"text":"2. “This metric dropped 15% overnight. Diagnose it.”"}]},{"t":"p","runs":[{"text":"There is a correct order, and the first step is the one that marks experience."}]},{"t":"ol","start":1,"items":[[{"text":"Is it real?","b":true},{"text":" Before anything else: did the logging change, did a release stop firing an event, did the pipeline fail, did a data source stop delivering, is the dashboard’s date filter doing something odd. A large share of overnight step-changes are instrumentation, and a candidate who investigates user behaviour before checking whether the number is trustworthy has revealed how much time they have spent doing this."}],[{"text":"Is it everywhere?","b":true},{"text":" Segment: platform, app version, country, browser, new versus returning, paid versus organic. A drop confined to Android 14 is a bug; a drop everywhere is an event."}],[{"text":"Is it us?","b":true},{"text":" Releases, config changes, pricing changes, an experiment ramped to 50%, a marketing campaign that ended."}],[{"text":"Is it the world?","b":true},{"text":" Holiday, competitor launch, outage at a payment provider, a platform policy change, seasonality, and check the same week last year before claiming anything is unprecedented."}]]},{"t":"p","runs":[{"text":"Say the order out loud as you go. The round is scored on the search strategy, not on guessing the answer."}]},{"t":"h2","id":"3-design-an-experiment-for-this","runs":[{"text":"3. “Design an experiment for this”"}]},{"t":"p","runs":[{"text":"Six things, in this order, and none of them is optional:"}]},{"t":"ul","items":[[{"text":"Unit of randomisation","b":true},{"text":": user, session, device, or something coarser like a city when there are network effects. Getting this wrong invalidates everything after it."}],[{"text":"Primary metric and guardrails","b":true},{"text":", chosen before the test."}],[{"text":"Minimum detectable effect and sample size.","b":true},{"text":" Use the arithmetic from the statistics lesson: roughly "},{"text":"16 * p * (1-p) / difference²","code":true},{"text":" per arm. Then say how long that takes at their traffic. This is the single moment in the round where a real number wins it."}],[{"text":"Duration","b":true},{"text":": at least one full weekly cycle, because Tuesday users are not Saturday users."}],[{"text":"A stopping rule agreed in advance.","b":true},{"text":" No peeking and stopping at significance, which manufactures false positives."}],[{"text":"What you would do with each outcome","b":true},{"text":", including no difference. If the answer is “ship it anyway”, the test was theatre and it is better to say so and skip it."}]]},{"t":"h2","id":"the-limitation","runs":[{"text":"The limitation"}],"role":"limit"},{"t":"p","runs":[{"text":"This round rewards fluency, and fluency is memorisable. Interviewers know that, so a smooth framework recited without a number in it now reads as preparation rather than experience."}]},{"t":"p","runs":[{"text":"The discriminator is specifics: an actual sample-size calculation, an actual counter-metric for this product, an actual reason to reject the obvious metric. One concrete trade-off stated out loud is worth more than four tidy headings."}]}],"check":{"question":[{"text":"A candidate is asked why a company’s daily active users fell 15% overnight. Which opening move signals the most experience?"}],"options":[{"text":[{"text":"Segment by platform and country to find where the drop is concentrated."}],"correct":false,"tempting":[{"text":"A strong second step that wastes effort if the event was never logged correctly in the first place."}]},{"text":[{"text":"Ask whether a release, experiment or pricing change went out the previous evening."}],"correct":false,"tempting":[{"text":"Reasonable and specific, but assumes the measurement is sound before that has been established."}]},{"text":[{"text":"Check whether the number is real (logging changes, a failed pipeline, a release that stopped firing the event, a broken dashboard filter) before investigating user behaviour at all."}],"correct":true,"tempting":null},{"text":[{"text":"Compare against the same week last year to rule out seasonality."}],"correct":false,"tempting":[{"text":"Useful for gradual movements, but seasonality rarely produces a single-night step change of this size."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"the-design-conversation","title":"“How would you build this?” answered well","minutes":9},"next":{"slug":"talking-about-your-work","title":"Talking about your work without lying or shrinking","minutes":8}}}