Operating a system nobody wrote by hand

At 3am, the useful question changed

The old first question was “who wrote this?”, because that person could say “check the cache invalidation, it is always the cache invalidation”, and forty minutes became four.

That question now often has no useful answer. The replacement is mechanical and works regardless of provenance: what changed in the last twenty-four hours?

Deployments, feature flags, configuration values, dependency updates, database migrations, infrastructure changes, and things that changed on somebody else’s side. A single timeline showing all of these, in one place, is the highest-value observability investment a team can make, and most teams have the data spread across five systems and correlate it by hand under pressure.

Build the timeline before you need it. During an incident it is worth an hour of anybody’s time; at the moment you need it you have four minutes.

What to log so the question is answerable

The general rule: log the values at the boundaries, in a structure a machine can filter.

logger.info("charge.attempt", extra={
    "request_id": ctx.request_id,
    "account_id": account.id,
    "amount_minor": amount,
    "currency": currency,
    "idempotency_key": key,
    "provider": "razorpay",
    "deploy_sha": settings.GIT_SHA,
})

deploy_sha on every log line is the detail most often missing and the one that most often resolves an incident, because it lets you filter to the version and answer the question above directly.

The three that matter most: a request identifier that travels across services, the inputs and outputs at every external boundary, and the version. Everything else is a refinement.

Runbooks, written for the person who does not know

A runbook assuming familiarity is useless at 3am to somebody who does not have it, which, increasingly, is whoever is on call.

One page per alert:

# Alert: checkout_error_rate > 2%

What it means: more than 2% of POST /checkout returned 5xx over 5 minutes.
Who it affects: customers cannot complete purchases. Revenue-affecting.

First: check https://dash/deploys — anything in the last hour?
        If yes, roll back first and diagnose afterwards.

Then:  check the provider status page (link).
       Run: `kubectl logs -l app=checkout --since=10m | grep -c 'provider timeout'`
       If provider timeouts dominate, set flag `checkout.fallback_provider=true`.

Escalate to: #payments-oncall. Ravi knows the reconciliation path.
Do NOT: re-run the settlement job. It is not idempotent. See ADR 0031.

The do not section is the one people skip and the one that prevents the second, worse incident.

Roll back first

With more change flowing and less of it deeply understood, the balance shifted further towards reverting before diagnosing. Diagnosis takes as long as it takes; a rollback takes minutes and stops the bleeding.

The practices that make this available: deploy small, deploy often, keep the previous version ready, make migrations backwards compatible for at least one release, and keep flags for anything risky. These are the same practices that make everything else in this course work, which is not a coincidence: reversibility is the property that makes speed survivable at every level.

The review, and the one output that matters

Blameless, because a review that assigns fault produces careful reporting rather than accurate reporting, and you need the accurate kind.

One mechanical change worth making: the standard prompt has always been “what did the person do and why was it reasonable at the time?” Extend it to “what did the system present to the person, and why was that presentation reasonable to act on?” A generated change that looked correct, passed review, and was wrong is a systems failure with several contributing causes, and hunting for the individual who should have caught it finds nothing useful.

And the output that matters is not the document. It is a permanent test.

Every incident produces one, in the same week, before anybody has moved on. A property, a database constraint, a monitor, an assertion, an item on the hostile-inputs corpus. The suite that catches real problems is built almost entirely out of things that actually happened, and a team that does this consistently has, after two years, a test suite worth more than anything anyone could have designed in advance.

An incident review with no test attached is a document. An incident review with a test attached is verification capacity, which is the constrained resource this entire course is about.