How the thing starts

Five ways work arrives

1. A schedule: cron, a timer, a platform’s scheduler.
2. A webhook: somebody else’s system pushes an event to you.
3. A queue: an item of work waiting to be picked up.
4. Polling: you ask repeatedly, because there is no webhook.
5. A person: a button, a command, an email to a special address.

Most real systems use three of these at once, and each has one thing that goes wrong.

Webhooks: four rules

Answer fast, then work. Return 200 within a second or two and do the actual job asynchronously: put it on a queue. Providers time out and retry, so a slow handler does not merely feel slow; it manufactures duplicate work while the first copy is still running.

Verify the signature. An unauthenticated webhook endpoint is a public API for creating work in your system, and its address is not a secret.

Deduplicate on the provider’s event id. Delivery is at-least-once everywhere. You will receive the same event twice, and a unique index on the event id is the entire fix:

INSERT INTO processed_events (event_id) VALUES ($1)
ON CONFLICT DO NOTHING;     -- zero rows affected means: already handled

Expect them out of order. The “updated” event can arrive before the “created” event. Order by the event’s own timestamp, and make handlers tolerant of arriving early.

Polling: the cursor rule

When there is no webhook, you poll. The mistake is almost universal:

since = last_run_time            # wrong
since = last_seen_updated_at     # right

With the first version, any record written while your job was running falls in the gap between the query and the timestamp you save. It is invisible, it is a small percentage, and it is permanent: those records are never picked up again.

Track a cursor over the data, not the clock. Save the highest updated_at you actually saw, query with >= on the next run, and keep a small set of ids already processed so that equal timestamps do not cause duplicates. Ordering by (updated_at, id) makes this exact.

Queues, between trigger and work

A queue buys you retries, a concurrency limit, and somewhere for permanently failed items to sit. You do not need a product for this.

The free path is a table:

SELECT * FROM jobs
 WHERE status = 'pending'
 ORDER BY created_at
 LIMIT 1
 FOR UPDATE SKIP LOCKED;

SKIP LOCKED is what makes that safe with several workers: each one takes a different row instead of blocking. That is a real queue, in a database you already run. Redis works too. Managed options (the cloud queue services) are worth it when you need cross-region durability, not before.

Two runs on the same record

If your trigger can fire twice for one entity, two runs will act on it simultaneously and both will believe they are alone. Take a lock on the entity, not on the job: a row in a locks table keyed by the order id, with an expiry.

Backfill is a different program

The first run against four years of history is not the nightly run with a bigger date range. It needs rate limiting, checkpointing, and the ability to resume from where it stopped. Write it as its own script, run it deliberately, and watch it.