Agent loops on models that are not clever

The loop, and the five ways it dies

An agent is a loop: the model chooses an action, something executes it, the result goes back into the context, repeat until done. On a frontier model this often works. On an 8B model it fails in five specific ways, and each has a structural fix that does not involve a bigger model.

1. It never decides it is finished. There is no natural stopping signal, so the loop runs to its step limit. Fix: make finishing an explicit action with a schema ({"action":"done","answer":"..."}), and cap the steps. Never rely on the model producing a sentence you pattern-match on.

2. It repeats the same action. The model searches for the same string four times, because each turn it sees a context in which searching seemed reasonable. Fix: keep the history of attempted actions outside the model and refuse duplicates in code, returning already tried, result was X as the observation.

3. It hallucinates the result. Rather than calling the tool, it writes what the tool would probably have returned and proceeds. This is common and dangerous because everything downstream looks fine. Fix: only ever advance the loop on results your code produced. Ignore any observation text the model wrote itself.

4. It loses the goal. By step six the original instruction is buried under observations, and recency wins. Fix: restate the goal in every turn, immediately before the model generates. It costs fifty tokens and it is the highest-value line in the loop.

5. Context grows until it breaks. Each step adds an observation; long tool outputs add a lot. Ten steps can exhaust an 8k window. Fix: truncate observations to what matters, and keep a running structured state (a small JSON object of what has been established) rather than the raw transcript.

The arithmetic that should change your design

Compounding is the reason all of this matters. At 92% per-step reliability, a ten-step loop completes correctly 43% of the time. Getting to 90% overall from ten steps requires 99% per step, which no small model provides on an open-ended decision.

There are only two ways out, and increasing the model size is the weaker one.

Reduce the number of model decisions. Most of what is written as an agent loop is a workflow with a fixed shape: fetch, extract, decide, write. Encode the shape in code and use the model at the two or three points that need judgement. Five deterministic steps with two model calls at 92% is 85% overall, not 43%.

Verify after each step. A check that catches a bad action and retries turns a 92% step into an effective 99% step, and it is usually a schema validation or a lookup rather than another model call.

What a working small-model agent looks like

state = {"goal": goal, "facts": {}, "tried": []}
for step in range(MAX_STEPS):
    prompt = render(state)                        # goal restated last
    action = model_json(prompt, schema=ACTION)    # constrained decoding
    if action["type"] == "done":
        break
    if action in state["tried"]:
        state["facts"]["note"] = "repeated action refused"
        continue
    state["tried"].append(action)
    obs = execute(action)                         # your code, always
    state["facts"][action["type"]] = summarise(obs, max_chars=600)

Everything that matters is outside the model: the state, the deduplication, the truncation, the step cap, the schema. The model contributes one constrained decision per iteration.

Where small models do well

Narrow loops with few options and verifiable results. Retrieve and answer with citations. Fill a form by looking up three fields. Triage an item into one of five queues, calling one lookup. Retry a failing structured extraction with the error message as feedback.

Where they do not: open-ended research, multi-hour tasks, anything where the correct next step depends on judgement about a situation the model cannot verify.

The honest framing for anyone planning work: a well-built pipeline with an 8B model at three decision points is more useful and far cheaper than a free-form agent with a 70B, and it is debuggable. Free-form autonomy is the most expensive way to buy reliability, at any model size.