Joining two automations without joining their failures

Real systems are several patterns in a row

An invoice system is an extractor, then a validator, then a matcher against purchase orders, then a drafting step for the exceptions, then a monitor watching the whole thing. Five patterns from this module, joined.

How they are joined decides whether the result can be operated.

The interface is a queue, not a function call

The tempting design is direct: stage one finishes an invoice and calls stage two.

Now suppose stage one extracts four thousand invoices an hour and stage two validates nine hundred. With a direct call, stage one blocks or times out, and its logs fill with errors. Somebody investigates the extractor, which is working perfectly. The real constraint is invisible, reported in the wrong place, and every failed call loses the extraction work that had already been paid for.

With a queue between them, stage one keeps running at its own speed and the backlog becomes a number on a chart that names the actual bottleneck. Stage two can be retried, paused, scaled or fixed without touching stage one. And a poisonous item can be set aside instead of stopping everything behind it.

You do not need queueing infrastructure for this. A status column is a queue:

-- stage 1 writes
INSERT INTO invoices (id, raw_text, status) VALUES ($1, $2, 'extracted');

-- stage 2 claims work
UPDATE invoices SET status = 'validating', claimed_at = now()
WHERE id IN (SELECT id FROM invoices WHERE status = 'extracted'
             ORDER BY created_at LIMIT 50 FOR UPDATE SKIP LOCKED)
RETURNING *;

FOR UPDATE SKIP LOCKED lets several workers claim different rows without colliding. That is a working job queue in a database you already run, with visibility, history and transactions, and it is the right answer far more often than a message broker.

Stages must not abort on one bad item

One malformed PDF should not stop the other four hundred. Mark the item failed with a reason, move on, and let failures accumulate somewhere a person reviews: a dead-letter queue, or just status = 'failed' with an error column.

Then alert on the rate of failures, not on each one. Three failures an hour is normal; three hundred means the upstream format changed.

Own your columns

Each stage writes only its own status values and its own fields. A stage that reaches into another’s data creates a coupling nobody documented, and the day somebody changes stage two, stage four breaks for reasons that appear unrelated.

Write the state transitions down as a list (extracted → validating → validated | failed) and enforce them in code, refusing a transition that is not on the list. Ten lines, and it turns an entire class of confusing bug into an immediate exception naming the stage that misbehaved.

One id through everything

Generate a correlation id at the trigger and carry it into every stage, every log line, every outbound request and every row written. Tracing one invoice through five stages then becomes a single query instead of a morning.

Do not chain agents

A pipeline where one stage is an agent is debuggable: the other stages are deterministic, so when the output is wrong you know where to look. A pipeline where every stage is an agent has no fixed point to reason from, per-step errors multiply across stages, and the trace is five transcripts that each look locally reasonable.

Prefer deterministic stages with a model call inside the one or two that need judgement. It is cheaper, it is faster, and it is the difference between a system somebody can fix at 3am and a system somebody restarts and hopes.

Make every stage safe to run twice

Once stages are joined by a queue, items get delivered more than once: a worker that dies after doing the work but before marking it done, a claim that times out and is re-offered, a manual replay of yesterday’s backlog. This is normal operation, not an error condition.

So each stage must be idempotent on its own: keyed writes, upserts on a natural key, and a check for “have I already produced an output for this input id” at the top of the handler. When every stage can be run twice harmlessly, recovery becomes trivial: you re-run the range and stop reasoning about exactly where it stopped. When one stage cannot, every incident turns into an archaeology exercise about which items got halfway.

Backpressure

If intake is faster than the slowest stage, the queue grows without limit until something breaks: usually disk, or a rate limit, or the bill. Decide the behaviour deliberately: cap intake, shed low-priority work, or scale the slow stage. Then alert on queue depth and on the age of the oldest unprocessed item, which is the number that tells you whether you are keeping up.