{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"agents-on-small-models","title":"Agent loops on models that are not clever","url":"https://addaly.com/learn/open-models/agents-on-small-models","position":46,"of":71,"minutes":9,"answer":"Agent loops on small models fail by never stopping, repeating actions, inventing observations, losing the goal and outgrowing the context, so hold the state.","keyIdea":"Agent loops on small models fail by never stopping, repeating actions, inventing observations, losing the goal and outgrowing the context, so hold the state, the deduplication and the step cap in code and reduce the number of model decisions rather than enlarging the model."},"body":{"format":"markdown","text":"## The loop, and the five ways it dies\n\nAn agent is a loop: the model chooses an action, something executes it, the result goes back into the context, repeat until done. On a frontier model this often works. On an 8B model it fails in five specific ways, and each has a structural fix that does not involve a bigger model.\n\n**1. It never decides it is finished.** There is no natural stopping signal, so the loop runs to its step limit. *Fix:* make finishing an explicit action with a schema (`{\"action\":\"done\",\"answer\":\"...\"}`), and cap the steps. Never rely on the model producing a sentence you pattern-match on.\n\n**2. It repeats the same action.** The model searches for the same string four times, because each turn it sees a context in which searching seemed reasonable. *Fix:* keep the history of attempted actions outside the model and refuse duplicates in code, returning `already tried, result was X` as the observation.\n\n**3. It hallucinates the result.** Rather than calling the tool, it writes what the tool would probably have returned and proceeds. This is common and dangerous because everything downstream looks fine. *Fix:* only ever advance the loop on results your code produced. Ignore any observation text the model wrote itself.\n\n**4. It loses the goal.** By step six the original instruction is buried under observations, and recency wins. *Fix:* restate the goal in every turn, immediately before the model generates. It costs fifty tokens and it is the highest-value line in the loop.\n\n**5. Context grows until it breaks.** Each step adds an observation; long tool outputs add a lot. Ten steps can exhaust an 8k window. *Fix:* truncate observations to what matters, and keep a running structured state (a small JSON object of what has been established) rather than the raw transcript.\n\n## The arithmetic that should change your design\n\nCompounding is the reason all of this matters. At 92% per-step reliability, a ten-step loop completes correctly 43% of the time. Getting to 90% overall from ten steps requires 99% per step, which no small model provides on an open-ended decision.\n\n> Figure: A loop at 92 per cent per-step reliability. Bars (% of runs completing correctly): 1 model decision 92; 3 decisions 78; 5 decisions 66; 10 decisions 43. Ten steps complete correctly forty-three times in a hundred. Reaching ninety per cent over ten steps needs ninety-nine per cent per step, which no small model gives on an open-ended decision, so remove decisions rather than enlarge the model, and verify after each one that remains.\n\nThere are only two ways out, and increasing the model size is the weaker one.\n\n**Reduce the number of model decisions.** Most of what is written as an agent loop is a workflow with a fixed shape: fetch, extract, decide, write. Encode the shape in code and use the model at the two or three points that need judgement. Five deterministic steps with two model calls at 92% is 85% overall, not 43%.\n\n**Verify after each step.** A check that catches a bad action and retries turns a 92% step into an effective 99% step, and it is usually a schema validation or a lookup rather than another model call.\n\n## What a working small-model agent looks like\n\n```python\nstate = {\"goal\": goal, \"facts\": {}, \"tried\": []}\nfor step in range(MAX_STEPS):\n    prompt = render(state)                        # goal restated last\n    action = model_json(prompt, schema=ACTION)    # constrained decoding\n    if action[\"type\"] == \"done\":\n        break\n    if action in state[\"tried\"]:\n        state[\"facts\"][\"note\"] = \"repeated action refused\"\n        continue\n    state[\"tried\"].append(action)\n    obs = execute(action)                         # your code, always\n    state[\"facts\"][action[\"type\"]] = summarise(obs, max_chars=600)\n```\n\nEverything that matters is outside the model: the state, the deduplication, the truncation, the step cap, the schema. The model contributes one constrained decision per iteration.\n\n## Where small models do well\n\nNarrow loops with few options and verifiable results. Retrieve and answer with citations. Fill a form by looking up three fields. Triage an item into one of five queues, calling one lookup. Retry a failing structured extraction with the error message as feedback.\n\nWhere they do not: open-ended research, multi-hour tasks, anything where the correct next step depends on judgement about a situation the model cannot verify.\n\n> The honest framing for anyone planning work: a well-built pipeline with an 8B model at three decision points is more useful and far cheaper than a free-form agent with a 70B, and it is debuggable. Free-form autonomy is the most expensive way to buy reliability, at any model size."},"check":{"question":"A five-step agent using an 8B model succeeds on about 60% of tasks. The team considers moving to a 70B, which tests at 97% per step. What does the arithmetic suggest?","options":[{"text":"The 70B would give about 86% overall, so the real gain comes from removing model decisions or verifying each step rather than from the larger model alone.","correct":true},{"text":"The 70B would reach roughly 97% overall, since per-step accuracy is the dominant factor in a short loop.","correct":false,"tempting":"Treats per-step accuracy as the end-to-end rate; five steps at 0.97 is 0.86, not 0.97."},{"text":"Success rates do not compound in agent loops because later steps can correct earlier errors.","correct":false,"tempting":"Self-correction happens occasionally but is not reliable at any size, and assuming it is the reason 60% was observed with a 92% step rate."},{"text":"The 60% figure implies per-step accuracy around 60%, so the model is much weaker than tested.","correct":false,"tempting":"Inverts the arithmetic: 0.92 to the fifth power is about 0.66, which is close to the observed rate."}]},"prev":{"slug":"tool-calling-on-open-weights","title":"Tool calling, and what breaks at 8B","url":"https://addaly.com/learn/open-models/tool-calling-on-open-weights"},"next":{"slug":"multilingual-reality","title":"What these models really do in your language","url":"https://addaly.com/learn/open-models/multilingual-reality"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Agent loops on models that are not clever', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/agents-on-small-models (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Agent loops on models that are not clever","courseTitle":"The Open Model Ecosystem","module":{"index":5,"slug":"getting-good-output","title":"Getting good output"},"answer":null,"keyIdea":"Agent loops on small models fail by never stopping, repeating actions, inventing observations, losing the goal and outgrowing the context, so hold the state, the deduplication and the step cap in code and reduce the number of model decisions rather than enlarging the model.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"% of runs completing correctly","title":"A loop at 92 per cent per-step reliability","series":[{"label":"1 model decision","value":92},{"label":"3 decisions","value":78},{"label":"5 decisions","value":66},{"label":"10 decisions","value":43,"highlight":true}],"caption":"Ten steps complete correctly forty-three times in a hundred. Reaching ninety per cent over ten steps needs ninety-nine per cent per step, which no small model gives on an open-ended decision, so remove decisions rather than enlarge the model, and verify after each one that remains."},"alt":"Figure: A loop at 92 per cent per-step reliability. Bars (% of runs completing correctly): 1 model decision 92; 3 decisions 78; 5 decisions 66; 10 decisions 43. Ten steps complete correctly forty-three times in a hundred. Reaching ninety per cent over ten steps needs ninety-nine per cent per step, which no small model gives on an open-ended decision, so remove decisions rather than enlarge the model, and verify after each one that remains."}},"blocks":[{"t":"h2","id":"the-loop-and-the-five-ways-it-dies","runs":[{"text":"The loop, and the five ways it dies"}]},{"t":"p","runs":[{"text":"An agent is a loop: the model chooses an action, something executes it, the result goes back into the context, repeat until done. On a frontier model this often works. On an 8B model it fails in five specific ways, and each has a structural fix that does not involve a bigger model."}]},{"t":"p","runs":[{"text":"1. It never decides it is finished.","b":true},{"text":" There is no natural stopping signal, so the loop runs to its step limit. "},{"text":"Fix:","i":true},{"text":" make finishing an explicit action with a schema ("},{"text":"{\"action\":\"done\",\"answer\":\"...\"}","code":true},{"text":"), and cap the steps. Never rely on the model producing a sentence you pattern-match on."}]},{"t":"p","runs":[{"text":"2. It repeats the same action.","b":true},{"text":" The model searches for the same string four times, because each turn it sees a context in which searching seemed reasonable. "},{"text":"Fix:","i":true},{"text":" keep the history of attempted actions outside the model and refuse duplicates in code, returning "},{"text":"already tried, result was X","code":true},{"text":" as the observation."}]},{"t":"p","runs":[{"text":"3. It hallucinates the result.","b":true},{"text":" Rather than calling the tool, it writes what the tool would probably have returned and proceeds. This is common and dangerous because everything downstream looks fine. "},{"text":"Fix:","i":true},{"text":" only ever advance the loop on results your code produced. Ignore any observation text the model wrote itself."}]},{"t":"p","runs":[{"text":"4. It loses the goal.","b":true},{"text":" By step six the original instruction is buried under observations, and recency wins. "},{"text":"Fix:","i":true},{"text":" restate the goal in every turn, immediately before the model generates. It costs fifty tokens and it is the highest-value line in the loop."}]},{"t":"p","runs":[{"text":"5. Context grows until it breaks.","b":true},{"text":" Each step adds an observation; long tool outputs add a lot. Ten steps can exhaust an 8k window. "},{"text":"Fix:","i":true},{"text":" truncate observations to what matters, and keep a running structured state (a small JSON object of what has been established) rather than the raw transcript."}]},{"t":"h2","id":"the-arithmetic-that-should-change-your-design","runs":[{"text":"The arithmetic that should change your design"}]},{"t":"p","runs":[{"text":"Compounding is the reason all of this matters. At 92% per-step reliability, a ten-step loop completes correctly 43% of the time. Getting to 90% overall from ten steps requires 99% per step, which no small model provides on an open-ended decision."}]},{"t":"p","runs":[{"text":"There are only two ways out, and increasing the model size is the weaker one."}]},{"t":"p","runs":[{"text":"Reduce the number of model decisions.","b":true},{"text":" Most of what is written as an agent loop is a workflow with a fixed shape: fetch, extract, decide, write. Encode the shape in code and use the model at the two or three points that need judgement. Five deterministic steps with two model calls at 92% is 85% overall, not 43%."}]},{"t":"p","runs":[{"text":"Verify after each step.","b":true},{"text":" A check that catches a bad action and retries turns a 92% step into an effective 99% step, and it is usually a schema validation or a lookup rather than another model call."}]},{"t":"h2","id":"what-a-working-small-model-agent-looks-like","runs":[{"text":"What a working small-model agent looks like"}]},{"t":"code","lang":"python","label":"Python","text":"state = {\"goal\": goal, \"facts\": {}, \"tried\": []}\nfor step in range(MAX_STEPS):\n    prompt = render(state)                        # goal restated last\n    action = model_json(prompt, schema=ACTION)    # constrained decoding\n    if action[\"type\"] == \"done\":\n        break\n    if action in state[\"tried\"]:\n        state[\"facts\"][\"note\"] = \"repeated action refused\"\n        continue\n    state[\"tried\"].append(action)\n    obs = execute(action)                         # your code, always\n    state[\"facts\"][action[\"type\"]] = summarise(obs, max_chars=600)"},{"t":"p","runs":[{"text":"Everything that matters is outside the model: the state, the deduplication, the truncation, the step cap, the schema. The model contributes one constrained decision per iteration."}]},{"t":"h2","id":"where-small-models-do-well","runs":[{"text":"Where small models do well"}]},{"t":"p","runs":[{"text":"Narrow loops with few options and verifiable results. Retrieve and answer with citations. Fill a form by looking up three fields. Triage an item into one of five queues, calling one lookup. Retry a failing structured extraction with the error message as feedback."}]},{"t":"p","runs":[{"text":"Where they do not: open-ended research, multi-hour tasks, anything where the correct next step depends on judgement about a situation the model cannot verify."}]},{"t":"quote","runs":[{"text":"The honest framing for anyone planning work: a well-built pipeline with an 8B model at three decision points is more useful and far cheaper than a free-form agent with a 70B, and it is debuggable. Free-form autonomy is the most expensive way to buy reliability, at any model size."}]}],"check":{"question":[{"text":"A five-step agent using an 8B model succeeds on about 60% of tasks. The team considers moving to a 70B, which tests at 97% per step. What does the arithmetic suggest?"}],"options":[{"text":[{"text":"The 70B would give about 86% overall, so the real gain comes from removing model decisions or verifying each step rather than from the larger model alone."}],"correct":true,"tempting":null},{"text":[{"text":"The 70B would reach roughly 97% overall, since per-step accuracy is the dominant factor in a short loop."}],"correct":false,"tempting":[{"text":"Treats per-step accuracy as the end-to-end rate; five steps at 0.97 is 0.86, not 0.97."}]},{"text":[{"text":"Success rates do not compound in agent loops because later steps can correct earlier errors."}],"correct":false,"tempting":[{"text":"Self-correction happens occasionally but is not reliable at any size, and assuming it is the reason 60% was observed with a 92% step rate."}]},{"text":[{"text":"The 60% figure implies per-step accuracy around 60%, so the model is much weaker than tested."}],"correct":false,"tempting":[{"text":"Inverts the arithmetic: 0.92 to the fifth power is about 0.66, which is close to the observed rate."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"tool-calling-on-open-weights","title":"Tool calling, and what breaks at 8B","minutes":9},"next":{"slug":"multilingual-reality","title":"What these models really do in your language","minutes":9}}}