“How would you measure it?”

A round with three shapes

The design conversation asks how you would build a system. The metrics case asks how you would know whether it worked. Analyst, data scientist and product-facing roles nearly all have one, and it comes in three forms.

1. “Define success for this feature”

”We are adding saved searches to the app. How would you measure whether it worked?”

The bad answer starts naming metrics. The good answer starts with the decision.

• Who is the user and what changes for them? People who search repeatedly stop retyping.
• What decision does the measurement inform? Whether to invest further, or remove it.
• One primary metric. Something close to the behaviour: proportion of returning searchers who reuse a saved search within seven days.
• Two or three guardrails: things that must not get worse. Search latency. Overall search volume. Support tickets.
• A counter-metric: the way this could look good while being bad. If saved searches make people search less because they are stuck in narrow saved queries, discovery falls. Name it.

Then say what would make you kill the feature. Interviewers remember candidates who state a failure condition, because most do not.

The property that separates a good metric from a bad one

Three tests, and being able to name them is most of the round:

• Sensitive: it moves when the thing you changed moves. Monthly active users is insensitive to almost everything you can ship in a quarter.
• Hard to game: “tickets resolved” rewards closing tickets prematurely; “tickets resolved without reopening in 14 days” mostly does not.
• Attributable: you can tell whether your change caused it, which usually means it is close to the change in time and in the causal chain. Revenue is the thing everybody cares about and is almost never a usable feature metric, because forty other things move it.

The classic failure is a metric that rises when the product gets worse. Time-in-app for a support tool. Number of searches per session for a search engine. Say the words “this could go up for a bad reason, so I would pair it with…” and you have signalled more experience than a fluent framework does.

2. “This metric dropped 15% overnight. Diagnose it.”

There is a correct order, and the first step is the one that marks experience.

1. Is it real? Before anything else: did the logging change, did a release stop firing an event, did the pipeline fail, did a data source stop delivering, is the dashboard’s date filter doing something odd. A large share of overnight step-changes are instrumentation, and a candidate who investigates user behaviour before checking whether the number is trustworthy has revealed how much time they have spent doing this.
2. Is it everywhere? Segment: platform, app version, country, browser, new versus returning, paid versus organic. A drop confined to Android 14 is a bug; a drop everywhere is an event.
3. Is it us? Releases, config changes, pricing changes, an experiment ramped to 50%, a marketing campaign that ended.
4. Is it the world? Holiday, competitor launch, outage at a payment provider, a platform policy change, seasonality, and check the same week last year before claiming anything is unprecedented.

Say the order out loud as you go. The round is scored on the search strategy, not on guessing the answer.

3. “Design an experiment for this”

Six things, in this order, and none of them is optional:

• Unit of randomisation: user, session, device, or something coarser like a city when there are network effects. Getting this wrong invalidates everything after it.
• Primary metric and guardrails, chosen before the test.
• Minimum detectable effect and sample size. Use the arithmetic from the statistics lesson: roughly 16 * p * (1-p) / difference² per arm. Then say how long that takes at their traffic. This is the single moment in the round where a real number wins it.
• Duration: at least one full weekly cycle, because Tuesday users are not Saturday users.
• A stopping rule agreed in advance. No peeking and stopping at significance, which manufactures false positives.
• What you would do with each outcome, including no difference. If the answer is “ship it anyway”, the test was theatre and it is better to say so and skip it.

The limitation

This round rewards fluency, and fluency is memorisable. Interviewers know that, so a smooth framework recited without a number in it now reads as preparation rather than experience.

The discriminator is specifics: an actual sample-size calculation, an actual counter-metric for this product, an actual reason to reject the obvious metric. One concrete trade-off stated out loud is worth more than four tidy headings.