{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"queues-and-background-work","title":"Work that does not belong in a request","url":"https://addaly.com/learn/cloud-and-shipping/queues-and-background-work","position":35,"of":74,"minutes":10,"answer":"Move anything slow or externally dependent out of the request into a queue, and design every job to be safe to run twice, because at some point it will be.","keyIdea":"Move anything slow or externally dependent out of the request into a queue, and design every job to be safe to run twice, because at some point it will be."},"body":{"format":"markdown","text":"## The test\n\nWould this still be correct if it took thirty seconds and sometimes failed? If yes, it does not belong in the request.\n\nThe usual list: sending email, generating thumbnails or PDFs, calling a slow third-party API, syncing to another system, building an export, sending webhooks out, anything that fans out to many recipients.\n\nLeave them in the request and two things happen. The user waits for work they did not ask to wait for. And when the request times out, the work is lost halfway: the order is saved and the confirmation email is not, with nothing recording that.\n\n## Your database is a perfectly good queue\n\nBefore reaching for infrastructure, know that Postgres does this well:\n\n```sql\nCREATE TABLE jobs (\n  id bigserial PRIMARY KEY,\n  kind text NOT NULL,\n  payload jsonb NOT NULL,\n  run_after timestamptz NOT NULL DEFAULT now(),\n  attempts int NOT NULL DEFAULT 0,\n  status text NOT NULL DEFAULT 'pending'\n);\n```\n\nA worker claims work with one statement:\n\n```sql\nUPDATE jobs SET status = 'running', attempts = attempts + 1\nWHERE id = (\n  SELECT id FROM jobs\n  WHERE status = 'pending' AND run_after <= now()\n  ORDER BY id FOR UPDATE SKIP LOCKED LIMIT 1\n)\nRETURNING *;\n```\n\n`FOR UPDATE SKIP LOCKED` is the clause that makes this work: each worker locks a row and other workers step over locked rows instead of waiting. Ten workers, no coordination, no duplicates, no extra service to run or pay for. This handles far more load than most projects will ever produce, and it means your jobs are in the same transaction and the same backup as your data.\n\nMove to Redis with BullMQ, RabbitMQ, SQS or Cloudflare Queues when you need throughput a database cannot give you, or when you want the operational features they bring. Not before.\n\n## Every job will run twice\n\nQueues deliver **at least once**. A worker can do the work, then crash before marking the job done, and the job comes back. This is not a bug you can configure away; it is the nature of the thing.\n\nSo every job must be **idempotent**: safe to run twice with the same effect as running once. In practice:\n\n- Give the job an idempotency key and record completed keys. Check before acting.\n- Prefer operations that are naturally repeatable: setting a value rather than incrementing it, `INSERT ... ON CONFLICT DO NOTHING` rather than a bare insert.\n- Where the side effect is external, use the provider's idempotency key: every serious payment and email API has one.\n\nOtherwise the user gets the email twice, or gets charged twice, and the second one is not a small problem.\n\n## Retries, backoff and jitter\n\nRetry with exponential backoff: 1s, 2s, 4s, 8s, capped at a few minutes. And **add jitter**: a random offset on each delay.\n\nWithout jitter, a failing dependency that comes back up is hit by every retry from every worker at the same instant, which knocks it over again. This is a real, repeated pattern in outage reports: the recovery is what causes the second outage. Jitter is one line of code and it prevents it.\n\n## The dead-letter queue nobody reads\n\nAfter N attempts, stop retrying and move the job to a dead-letter queue. Otherwise one malformed message is retried forever, consuming a worker and filling your logs.\n\nA dead-letter queue that nobody looks at is a data-loss queue with a nicer name. **Alert on its depth**, at greater than zero. These are, by definition, things that were supposed to happen and did not.\n\n> Figure: A job's life, designed for the fact that it will run twice. Steps: 1. Request inserts a job row (and returns to the user immediately); 2. Worker claims one row (FOR UPDATE SKIP LOCKED; other workers step over it); 3. Check the idempotency key (already done? mark complete and stop); 4. Do the work, then mark done (a crash between the two brings the job back); 5. On failure, retry with backoff and jitter (1 s, 2 s, 4 s, 8 s, each plus a random offset); 6. After N attempts, dead-letter it (and alert when that queue is deeper than zero). Postgres with FOR UPDATE SKIP LOCKED is a perfectly good queue for most projects. Whatever the store, delivery is at least once, so every job carries an idempotency key and every retry is backed off with jitter.\n\n## The number that tells you if you are keeping up\n\nNot throughput. **Oldest message age.**\n\nThroughput of 500 jobs a minute sounds healthy and tells you nothing about whether you are falling behind. Oldest-message-age says exactly that: if it is climbing, arrivals exceed capacity and the graph is a straight prediction of when it becomes a customer problem. Graph it, alert on it.\n\nAlso make the **visibility timeout** (how long a claimed job is hidden before it is offered again) comfortably longer than the job takes. Set it too short and a slow job is handed to a second worker while the first is still running it, which is the duplicate-execution problem arriving on purpose.\n\n## Scheduled jobs need a lock\n\n`cron` on one machine is a single point of failure. `cron` on every machine runs the job N times. And during a rolling deploy, both old and new instances may fire the same nightly job.\n\nTake a lock, and let losers do nothing:\n\n```sql\nSELECT pg_try_advisory_lock(918273);\n```\n\nThe one-machine version also has a subtler failure: the machine is replaced, nobody notices, and the nightly export silently stops. So a scheduled job should record that it ran, and something should alert when the record is missing: a check on absence, not on failure. Jobs that stop running are much harder to notice than jobs that fail."},"check":{"question":"Your worker sends a confirmation email, then updates the job row to \"done\". A deploy restarts workers mid-job, and some customers receive two identical emails. What is the correct fix?","options":[{"text":"Mark the job as done before sending the email, so a crash cannot cause a second send","correct":false,"tempting":"Trades duplicates for silent losses: a crash after marking and before sending means the customer never gets the email and nothing records that."},{"text":"Increase the visibility timeout so a restarted worker does not pick the job up again","correct":false,"tempting":"Delays redelivery rather than preventing it, and a long timeout means a genuinely failed job sits unprocessed for that whole period."},{"text":"Disable retries for email jobs, since a duplicate is worse than a missed message","correct":false,"tempting":"Removes the safety net for every transient failure, most of which are recoverable, to work around a problem that has a direct fix."},{"text":"Make the send idempotent, by passing an idempotency key to the mail provider or recording the key before sending and checking it first","correct":true}]},"prev":{"slug":"the-database-is-the-bottleneck","title":"It is almost always the database","url":"https://addaly.com/learn/cloud-and-shipping/the-database-is-the-bottleneck"},"next":{"slug":"rate-limits-and-abuse","title":"The traffic that is not customers","url":"https://addaly.com/learn/cloud-and-shipping/rate-limits-and-abuse"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Work that does not belong in a request', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/queues-and-background-work (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Work that does not belong in a request","courseTitle":"Shipping It","module":{"index":4,"slug":"load-cost-and-abuse","title":"Load, cost and the traffic that is not customers"},"answer":null,"keyIdea":"Move anything slow or externally dependent out of the request into a queue, and design every job to be safe to run twice, because at some point it will be.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Request inserts a job row","detail":"and returns to the user immediately"},{"label":"Worker claims one row","detail":"FOR UPDATE SKIP LOCKED; other workers step over it"},{"label":"Check the idempotency key","detail":"already done? mark complete and stop"},{"label":"Do the work, then mark done","detail":"a crash between the two brings the job back"},{"label":"On failure, retry with backoff and jitter","detail":"1 s, 2 s, 4 s, 8 s, each plus a random offset"},{"label":"After N attempts, dead-letter it","detail":"and alert when that queue is deeper than zero"}],"title":"A job’s life, designed for the fact that it will run twice","caption":"Postgres with FOR UPDATE SKIP LOCKED is a perfectly good queue for most projects. Whatever the store, delivery is at least once, so every job carries an idempotency key and every retry is backed off with jitter."},"alt":"Figure: A job’s life, designed for the fact that it will run twice. Steps: 1. Request inserts a job row (and returns to the user immediately); 2. Worker claims one row (FOR UPDATE SKIP LOCKED; other workers step over it); 3. Check the idempotency key (already done? mark complete and stop); 4. Do the work, then mark done (a crash between the two brings the job back); 5. On failure, retry with backoff and jitter (1 s, 2 s, 4 s, 8 s, each plus a random offset); 6. After N attempts, dead-letter it (and alert when that queue is deeper than zero). Postgres with FOR UPDATE SKIP LOCKED is a perfectly good queue for most projects. Whatever the store, delivery is at least once, so every job carries an idempotency key and every retry is backed off with jitter."}},"blocks":[{"t":"h2","id":"the-test","runs":[{"text":"The test"}]},{"t":"p","runs":[{"text":"Would this still be correct if it took thirty seconds and sometimes failed? If yes, it does not belong in the request."}]},{"t":"p","runs":[{"text":"The usual list: sending email, generating thumbnails or PDFs, calling a slow third-party API, syncing to another system, building an export, sending webhooks out, anything that fans out to many recipients."}]},{"t":"p","runs":[{"text":"Leave them in the request and two things happen. The user waits for work they did not ask to wait for. And when the request times out, the work is lost halfway: the order is saved and the confirmation email is not, with nothing recording that."}]},{"t":"h2","id":"your-database-is-a-perfectly-good-queue","runs":[{"text":"Your database is a perfectly good queue"}]},{"t":"p","runs":[{"text":"Before reaching for infrastructure, know that Postgres does this well:"}]},{"t":"code","lang":"sql","label":"SQL","text":"CREATE TABLE jobs (\n  id bigserial PRIMARY KEY,\n  kind text NOT NULL,\n  payload jsonb NOT NULL,\n  run_after timestamptz NOT NULL DEFAULT now(),\n  attempts int NOT NULL DEFAULT 0,\n  status text NOT NULL DEFAULT 'pending'\n);"},{"t":"p","runs":[{"text":"A worker claims work with one statement:"}]},{"t":"code","lang":"sql","label":"SQL","text":"UPDATE jobs SET status = 'running', attempts = attempts + 1\nWHERE id = (\n  SELECT id FROM jobs\n  WHERE status = 'pending' AND run_after <= now()\n  ORDER BY id FOR UPDATE SKIP LOCKED LIMIT 1\n)\nRETURNING *;"},{"t":"p","runs":[{"text":"FOR UPDATE SKIP LOCKED","code":true},{"text":" is the clause that makes this work: each worker locks a row and other workers step over locked rows instead of waiting. Ten workers, no coordination, no duplicates, no extra service to run or pay for. This handles far more load than most projects will ever produce, and it means your jobs are in the same transaction and the same backup as your data."}]},{"t":"p","runs":[{"text":"Move to Redis with BullMQ, RabbitMQ, SQS or Cloudflare Queues when you need throughput a database cannot give you, or when you want the operational features they bring. Not before."}]},{"t":"h2","id":"every-job-will-run-twice","runs":[{"text":"Every job will run twice"}]},{"t":"p","runs":[{"text":"Queues deliver "},{"text":"at least once","b":true},{"text":". A worker can do the work, then crash before marking the job done, and the job comes back. This is not a bug you can configure away; it is the nature of the thing."}]},{"t":"p","runs":[{"text":"So every job must be "},{"text":"idempotent","b":true},{"text":": safe to run twice with the same effect as running once. In practice:"}]},{"t":"ul","items":[[{"text":"Give the job an idempotency key and record completed keys. Check before acting."}],[{"text":"Prefer operations that are naturally repeatable: setting a value rather than incrementing it, "},{"text":"INSERT ... ON CONFLICT DO NOTHING","code":true},{"text":" rather than a bare insert."}],[{"text":"Where the side effect is external, use the provider’s idempotency key: every serious payment and email API has one."}]]},{"t":"p","runs":[{"text":"Otherwise the user gets the email twice, or gets charged twice, and the second one is not a small problem."}]},{"t":"h2","id":"retries-backoff-and-jitter","runs":[{"text":"Retries, backoff and jitter"}]},{"t":"p","runs":[{"text":"Retry with exponential backoff: 1s, 2s, 4s, 8s, capped at a few minutes. And "},{"text":"add jitter","b":true},{"text":": a random offset on each delay."}]},{"t":"p","runs":[{"text":"Without jitter, a failing dependency that comes back up is hit by every retry from every worker at the same instant, which knocks it over again. This is a real, repeated pattern in outage reports: the recovery is what causes the second outage. Jitter is one line of code and it prevents it."}]},{"t":"h2","id":"the-dead-letter-queue-nobody-reads","runs":[{"text":"The dead-letter queue nobody reads"}]},{"t":"p","runs":[{"text":"After N attempts, stop retrying and move the job to a dead-letter queue. Otherwise one malformed message is retried forever, consuming a worker and filling your logs."}]},{"t":"p","runs":[{"text":"A dead-letter queue that nobody looks at is a data-loss queue with a nicer name. "},{"text":"Alert on its depth","b":true},{"text":", at greater than zero. These are, by definition, things that were supposed to happen and did not."}]},{"t":"h2","id":"the-number-that-tells-you-if-you-are-keeping-up","runs":[{"text":"The number that tells you if you are keeping up"}]},{"t":"p","runs":[{"text":"Not throughput. "},{"text":"Oldest message age.","b":true}]},{"t":"p","runs":[{"text":"Throughput of 500 jobs a minute sounds healthy and tells you nothing about whether you are falling behind. Oldest-message-age says exactly that: if it is climbing, arrivals exceed capacity and the graph is a straight prediction of when it becomes a customer problem. Graph it, alert on it."}]},{"t":"p","runs":[{"text":"Also make the "},{"text":"visibility timeout","b":true},{"text":" (how long a claimed job is hidden before it is offered again) comfortably longer than the job takes. Set it too short and a slow job is handed to a second worker while the first is still running it, which is the duplicate-execution problem arriving on purpose."}]},{"t":"h2","id":"scheduled-jobs-need-a-lock","runs":[{"text":"Scheduled jobs need a lock"}]},{"t":"p","runs":[{"text":"cron","code":true},{"text":" on one machine is a single point of failure. "},{"text":"cron","code":true},{"text":" on every machine runs the job N times. And during a rolling deploy, both old and new instances may fire the same nightly job."}]},{"t":"p","runs":[{"text":"Take a lock, and let losers do nothing:"}]},{"t":"code","lang":"sql","label":"SQL","text":"SELECT pg_try_advisory_lock(918273);"},{"t":"p","runs":[{"text":"The one-machine version also has a subtler failure: the machine is replaced, nobody notices, and the nightly export silently stops. So a scheduled job should record that it ran, and something should alert when the record is missing: a check on absence, not on failure. Jobs that stop running are much harder to notice than jobs that fail."}]}],"check":{"question":[{"text":"Your worker sends a confirmation email, then updates the job row to “done”. A deploy restarts workers mid-job, and some customers receive two identical emails. What is the correct fix?"}],"options":[{"text":[{"text":"Mark the job as done before sending the email, so a crash cannot cause a second send"}],"correct":false,"tempting":[{"text":"Trades duplicates for silent losses: a crash after marking and before sending means the customer never gets the email and nothing records that."}]},{"text":[{"text":"Increase the visibility timeout so a restarted worker does not pick the job up again"}],"correct":false,"tempting":[{"text":"Delays redelivery rather than preventing it, and a long timeout means a genuinely failed job sits unprocessed for that whole period."}]},{"text":[{"text":"Disable retries for email jobs, since a duplicate is worse than a missed message"}],"correct":false,"tempting":[{"text":"Removes the safety net for every transient failure, most of which are recoverable, to work around a problem that has a direct fix."}]},{"text":[{"text":"Make the send idempotent, by passing an idempotency key to the mail provider or recording the key before sending and checking it first"}],"correct":true,"tempting":null}],"order":[0,3,2,1],"why":null},"prev":{"slug":"the-database-is-the-bottleneck","title":"It is almost always the database","minutes":10},"next":{"slug":"rate-limits-and-abuse","title":"The traffic that is not customers","minutes":9}}}