The 3am version

Alert on outcomes, not exceptions

“An exception occurred” is noise. Within a fortnight everybody filters the channel and the alerts have become decoration.

Alert on things that mean something:

• The job did not run. More on this below; it is the one people miss.
• The output is outside its normal range. It usually produces 400 rows and produced 3. Or 40,000.
• The error rate over the last hour crossed a threshold, not a single error.
• The queue depth is growing, meaning you are falling behind rather than failing.
• Spend crossed a line, at half the cap and again at the cap.
• The validator rejected more than the usual share of outputs. This is your early warning that a model or an upstream format has changed.

The heartbeat, which is the alert nobody has

A job that dies produces no errors, because it is not running. Every error-based alerting system is blind to the most common serious failure.

The fix is a dead man’s switch: the job pings a URL on success, and if the ping does not arrive within the expected window, you get alerted. Healthchecks.io has a free tier and is self-hostable; Cronitor and others do the same. The homemade version is a last_success_at row and a second, much simpler job that checks it, and yes, you then need a heartbeat on the checker, which is why most people use the hosted one for that outer layer.

A dead letter queue

Items that fail after all retries go into a table with the payload, the error and the timestamp. Not to a log file, which expires and which nobody reads.

Then somebody can look at what accumulated, fix the cause, and replay the queue. Without one, failures are permanent and invisible, and you find out about them from a customer.

Degrade rather than stop

When a dependency is down, an automation has better options than dying:

• queue the work and retry when it returns
• fall back to a smaller or different model
• fall back to the deterministic path, and mark the output so downstream knows it was not enriched
• deliver a partial result, clearly labelled

Each is a decision you make now, in daylight, and write into the code. None of them can be improvised at 3am.

The runbook: six lines, in the repository

Not a wiki page. A RUNBOOK.md beside the code, containing:

1. What it does, in one sentence.
2. What it touches: which systems, which credentials, what it can change.
3. How to stop it, exactly. The command or the flag.
4. How to see the last runs and their outcomes. A command or a link.
5. The three failures it has actually had, and what fixed each.
6. Who owns it, and who deputises.

Point five is the one that matters and the one that is always missing. It turns your incident history into somebody else’s fifteen-minute fix, and it takes two minutes to add each time.

Practise the recovery

Once a quarter: stop it deliberately, or break it in a controlled way, and have somebody who did not build it restore it using only the runbook.

You will find that step 3 is out of date, that the link in step 4 has moved, and that the owner left in June. That is the point. A recovery procedure nobody has walked through is a hypothesis.