What the model actually sees each turn

There is no session

The most common wrong mental model is that the model “remembers” the conversation and you are sending it updates. It does not. Every API call is stateless, and your harness re-sends the entire history each time.

Here is what step three of a small agent actually looks like on the wire. Details differ slightly between providers; the shape does not.

[
  { "role": "system",
    "content": "You are an operations assistant. Use tools; never guess an order id." },

  { "role": "user",
    "content": "Refund the duplicate charge on ORD-48213." },

  { "role": "assistant",
    "tool_calls": [
      { "id": "call_1",
        "function": { "name": "get_order",
                      "arguments": "{\"order_id\": \"ORD-48213\"}" } }
    ] },

  { "role": "tool",
    "tool_call_id": "call_1",
    "content": "{\"status\":\"shipped\",\"amount\":2499,\"refunded\":false}" }
]

Four messages, and the model reads all four before deciding anything. Three things in there are worth noticing.

The pairing is load-bearing. Every tool_call_id must match a preceding tool call. Drop one while trimming history and the provider returns a 400, not a graceful degradation. Trim in whole pairs.

The model reads its own past decisions as text. That assistant message is not a memory; it is a transcript entry. If you delete it, the model has no idea it ever called get_order.

Some providers hand back reasoning blocks and expect them returned in the next request. If your harness quietly strips them, the model loses its own train of thought between steps, and the symptom is an agent that seems to restart its thinking every turn.

Consequence one: history is editable, and that is your best tool

Since you own the list, you can rewrite it. Legitimately and routinely:

• Drop a failed detour. Three steps of a dead end, replaced by one line: tried the invoices API, it has no refund endpoint.
• Replace a huge result with a summary. The 30,000-token document becomes a 200-token abstract plus a file path.
• Inject a note as a tool result. The user has been waiting 40 seconds; prefer finishing over further searching.
• Re-state the goal near the end, where it will actually be read.

The model cannot tell that any of this happened. This is the single most useful power in a harness, and most beginners never use it because they think of the transcript as a record rather than as working memory they curate.

Consequence two: the transcript is the debug log

If you cannot print the exact array sent at step seven, you cannot debug the agent: you are guessing about a program whose input you never looked at. Log it. Every step. Redact secrets, keep the rest.

Almost every “the model is being stupid” bug turns out, on inspection, to be a tool result that was empty, truncated, or an error string nobody read.

Consequence three: position matters

Long contexts are not read evenly. The well-replicated finding from Lost in the Middle (Liu and colleagues, 2023) is that material placed in the middle of a long context is used less reliably than material at the start or end, and the effect gets worse as the context grows.

Practical rule: the constraint you most need obeyed goes near the end, not buried in a system prompt written six weeks ago.

The thing that unsettles people

The model cannot distinguish a tool result you fetched from one you invented. It is the same text in the same slot.

That cuts two ways. It is why a fabricated tool result is a perfectly good testing technique: you will use it in module three to test a harness without touching a real API. And it is why text arriving from the outside world inside a tool result is dangerous, which is module four.