Work that does not belong in a request

The test

Would this still be correct if it took thirty seconds and sometimes failed? If yes, it does not belong in the request.

The usual list: sending email, generating thumbnails or PDFs, calling a slow third-party API, syncing to another system, building an export, sending webhooks out, anything that fans out to many recipients.

Leave them in the request and two things happen. The user waits for work they did not ask to wait for. And when the request times out, the work is lost halfway: the order is saved and the confirmation email is not, with nothing recording that.

Your database is a perfectly good queue

Before reaching for infrastructure, know that Postgres does this well:

CREATE TABLE jobs (
  id bigserial PRIMARY KEY,
  kind text NOT NULL,
  payload jsonb NOT NULL,
  run_after timestamptz NOT NULL DEFAULT now(),
  attempts int NOT NULL DEFAULT 0,
  status text NOT NULL DEFAULT 'pending'
);

A worker claims work with one statement:

UPDATE jobs SET status = 'running', attempts = attempts + 1
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'pending' AND run_after <= now()
  ORDER BY id FOR UPDATE SKIP LOCKED LIMIT 1
)
RETURNING *;

FOR UPDATE SKIP LOCKED is the clause that makes this work: each worker locks a row and other workers step over locked rows instead of waiting. Ten workers, no coordination, no duplicates, no extra service to run or pay for. This handles far more load than most projects will ever produce, and it means your jobs are in the same transaction and the same backup as your data.

Move to Redis with BullMQ, RabbitMQ, SQS or Cloudflare Queues when you need throughput a database cannot give you, or when you want the operational features they bring. Not before.

Every job will run twice

Queues deliver at least once. A worker can do the work, then crash before marking the job done, and the job comes back. This is not a bug you can configure away; it is the nature of the thing.

So every job must be idempotent: safe to run twice with the same effect as running once. In practice:

• Give the job an idempotency key and record completed keys. Check before acting.
• Prefer operations that are naturally repeatable: setting a value rather than incrementing it, INSERT ... ON CONFLICT DO NOTHING rather than a bare insert.
• Where the side effect is external, use the provider’s idempotency key: every serious payment and email API has one.

Otherwise the user gets the email twice, or gets charged twice, and the second one is not a small problem.

Retries, backoff and jitter

Retry with exponential backoff: 1s, 2s, 4s, 8s, capped at a few minutes. And add jitter: a random offset on each delay.

Without jitter, a failing dependency that comes back up is hit by every retry from every worker at the same instant, which knocks it over again. This is a real, repeated pattern in outage reports: the recovery is what causes the second outage. Jitter is one line of code and it prevents it.

The dead-letter queue nobody reads

After N attempts, stop retrying and move the job to a dead-letter queue. Otherwise one malformed message is retried forever, consuming a worker and filling your logs.

A dead-letter queue that nobody looks at is a data-loss queue with a nicer name. Alert on its depth, at greater than zero. These are, by definition, things that were supposed to happen and did not.

The number that tells you if you are keeping up

Not throughput. Oldest message age.

Throughput of 500 jobs a minute sounds healthy and tells you nothing about whether you are falling behind. Oldest-message-age says exactly that: if it is climbing, arrivals exceed capacity and the graph is a straight prediction of when it becomes a customer problem. Graph it, alert on it.

Also make the visibility timeout (how long a claimed job is hidden before it is offered again) comfortably longer than the job takes. Set it too short and a slow job is handed to a second worker while the first is still running it, which is the duplicate-execution problem arriving on purpose.

Scheduled jobs need a lock

cron on one machine is a single point of failure. cron on every machine runs the job N times. And during a rolling deploy, both old and new instances may fire the same nightly job.

Take a lock, and let losers do nothing:

SELECT pg_try_advisory_lock(918273);

The one-machine version also has a subtler failure: the machine is replaced, nobody notices, and the nightly export silently stops. So a scheduled job should record that it ran, and something should alert when the record is missing: a check on absence, not on failure. Jobs that stop running are much harder to notice than jobs that fail.