The observation is where runs die

One step can end a run

A run_sql tool with no limit returns four thousand rows. At roughly thirty tokens a row that is 120,000 tokens in a single observation. Depending on the model, the run either errors immediately or spends the rest of its life dragging that block through every subsequent request at full price.

The plan-act-observe loop is trivial. Choosing what goes back in as the observation is the whole job, and it is mostly a question of size.

Four ways to shrink one, best first

1. Ask for less at the source. A LIMIT 20, a projection of four columns instead of forty, a date filter. Costs nothing, loses nothing you asked for, and is the option people skip because it means changing the tool rather than the prompt.

2. Summarise deterministically. Code, not a model: the row count, the min and max, the distinct values of the status column, the first five rows. For most analytical steps this is better than the raw data, because the model was going to compute those numbers itself and get one of them wrong.

3. Summarise with a cheap model call. Genuine option for long prose. It costs money, adds latency, and can drop the one clause that mattered. Use it when the content is unstructured and large, and keep the original addressable by id.

4. Write it to disk and return the path. Best when a later step will need the detail. Saved 3,412 rows to /work/orders.csv (412 KB). Columns: id, status, total, created_at. The agent can then run code against the file instead of reading it.

Say that you cut

If an observation is truncated, the text the model reads must say so. This one line prevents a category of confident wrong answers:

Showing rows 1-20 of 3,412. This is a truncated view.

Without it the model will write “there are 20 pending orders” and be entirely reasonable in doing so, because you told it there were.

Compacting the whole transcript

Once the transcript passes roughly half the window, replace the middle of it with a summary. What you preserve is the craft:

• the goal, verbatim, never summarised
• decisions made and why, one line each
• facts discovered, with the id or path they came from
• things already tried that failed, or the agent will try them again
• the last two turns, unchanged, so the immediate thread is intact

Note that “already tried and failed” is the item everyone forgets, and it is the one that prevents the loop.

The failure this fixes

You have already met the symptom: an agent that works for five steps, then re-runs searches it has done and contradicts something it established earlier.

The mechanism is now visible. The evidence has been squeezed out of the window while the goal remains, so the model does the only thing it can: reconstructs an approach from the goal, which is exactly what it did the first time. The agent is not confused. It is looking at a different document from the one you think it is looking at.

Measure before you send

Every provider offers a way to count tokens before a request, and tiktoken and equivalents run locally. Put a check in the harness: if any single observation exceeds some fraction of the window (five per cent is a reasonable starting point) summarise it before it enters the transcript, and log that you did.