{"course":{"slug":"python-for-ai","title":"Python, from Zero, for AI","url":"https://addaly.com/learn/python-for-ai","level":"beginner","track":"build"},"lesson":{"slug":"a-chat-loop-with-memory","title":"A chat loop with memory: keeping history, trimming it, and stopping cleanly","url":"https://addaly.com/learn/python-for-ai/a-chat-loop-with-memory","position":80,"of":89,"minutes":9,"answer":"A conversation is a list you send back in full on every turn, so its cost grows with its length; trim from the oldest user turn while keeping the system prompt.","keyIdea":"A conversation is a list you send back in full on every turn, so its cost grows with its length; trim from the oldest user turn while keeping the system prompt, catch KeyboardInterrupt so the last exchange is saved, and never let the loop run without a way out."},"body":{"format":"markdown","text":"## The loop\n\n```python\ndef chat(provider, system):\n    messages = [{\"role\": \"system\", \"content\": system}]\n    while True:\n        try:\n            text = input(\"you> \").strip()\n        except (EOFError, KeyboardInterrupt):\n            print()\n            break\n        if not text:\n            continue\n        if text in (\"/quit\", \"/exit\"):\n            break\n        messages.append({\"role\": \"user\", \"content\": text})\n        reply = provider.complete(messages)\n        messages.append({\"role\": \"assistant\", \"content\": reply})\n        print(f\"bot> {reply}\\n\")\n    return messages\n```\n\nEvery piece of this you have written before: a `while True` with a `break`, `input()`, a list of dicts, a provider with one method. The two lines that make it a *conversation* rather than a series of unrelated questions are the two `append`s. The model sees the full list every turn. That is its memory. There is no other.\n\n## Memory is a list you pay to resend\n\nModel APIs are stateless. The provider does not remember your last call; you remind it by sending the history back. So on turn 40, the request carries 79 earlier messages plus the new one, and you are billed for all of them as input tokens. Module 6 warned about this; here is where it bites. A chat that starts at 200 input tokens per turn is at 8,000 by turn 40 and climbing, and the model is also slower, because it reads everything before writing anything.\n\n> Figure: Input tokens sent on each turn of one conversation. Line chart of Input tokens in the request against Turn number: Full history, nothing trimmed (1, 120) (5, 900) (10, 2000) (20, 4200) (40, 8600); Trimmed to a 4,000-token budget (1, 120) (5, 900) (10, 2000) (20, 3900) (40, 3950). The provider remembers nothing between calls, so the history is re-sent and re-billed every turn: turn forty pays for the whole conversation so far, and the reply is the cheap part. Trimming from the oldest user turn, keeping the system message and never leaving a tool call without its result, holds the line flat.\n\nThe fix is a budget:\n\n```python\ndef trim(messages, max_tokens, count):\n    system, rest = messages[0], messages[1:]\n    while rest and count(system[\"content\"]) + sum(count(m[\"content\"]) for m in rest) > max_tokens:\n        rest = rest[2:]          # drop the oldest user+assistant pair together\n    return [system] + rest\n```\n\nThree rules inside that function. **Keep the system message**: it carries the instructions, and dropping it changes the model's behaviour mid-conversation. **Drop pairs**, so the history never starts with an assistant reply to a question the model can no longer see; some providers reject that, and the rest answer strangely. **Count tokens with a real counter** from module 6 (`tiktoken`, the provider's endpoint, or the tokeniser of a local model) not by characters, for the reasons given there.\n\nCall it before each request: `messages = trim(messages, 4000, count)`. Pick the budget from the model's context window with room for the reply, not from the window itself: a 128k window with a 120k history leaves no room to answer.\n\nSmarter trimming exists (summarising the dropped turns into one message, keeping turns the model referred back to) and it belongs to the context-engineering course. The Python shape is the same: a function from a list to a shorter list, called before every send.\n\n## Stopping cleanly\n\n`Ctrl-C` raises `KeyboardInterrupt` wherever the program is. Caught only around `input()`, as above, it ends the loop politely. Caught nowhere, it kills the program with a traceback and whatever you meant to save is gone. Caught *everywhere* (a bare `except:` inside the loop) it becomes impossible to stop the program at all, which you will discover during a runaway loop.\n\n`Ctrl-D` on macOS and Linux (or `Ctrl-Z` then Enter on Windows) sends end-of-file to `input()`, which raises `EOFError`. Handle both, and a piped input file (`python chat.py < questions.txt`) works too: each line becomes a turn, and the loop ends at the end of the file.\n\n## Saving and resuming\n\n```python\nfrom pathlib import Path\nimport json\n\ndef save(messages, path):\n    Path(path).write_text(json.dumps(messages, ensure_ascii=False, indent=1), encoding=\"utf-8\")\n\ndef load(path):\n    return json.loads(Path(path).read_text(encoding=\"utf-8\"))\n```\n\n`ensure_ascii=False` keeps Hindi, Arabic and emoji as themselves rather than `\\uXXXX` escapes, so the file is readable. Save after every exchange, not only at the end; a crash on turn 30 should cost you one turn, not thirty. A `/save` command and a `--resume chat.json` flag from module 3's `argparse` lesson complete it.\n\n## Commands, and telling them apart from questions\n\nA leading `/` is the usual convention for commands: `/quit`, `/save`, `/clear` to reset history, `/model gpt-4o-mini` to switch provider mid-chat, which composition from module 7 makes a one-line assignment. Check for commands before appending to history, so `/quit` never becomes a message the model has to answer.\n\n## The errors you will meet\n\nA `RateLimitError` or a 429 from the provider should not end the chat. Catch it around `complete`, print a short line, `sleep`, and let the user try again; the history is intact. A `KeyboardInterrupt` during a slow reply should abandon that reply and return to the prompt, not exit; catch it around `complete` separately from the one around `input()`, and pop the unanswered user message so the history stays paired.\n\nAn empty reply (a model that returned nothing because it hit a content filter or `max_tokens` of zero) should be visible as `[no reply]` rather than a blank line the user thinks is a bug in the terminal.\n\n## Where this goes\n\nThis loop is the core of every chat product you have used, minus the interface. Module 9's Gradio lesson replaces `input()` and `print()` with a web page and keeps everything else; the FastAPI lesson replaces them with an HTTP request and a response. The history list, the trim, the provider: those do not change. Build them once, here, as functions you can import.\n\n## Try this now\n\nWrite the loop against Ollama, add `trim` with a 1,000-token budget and a counter, and watch the printed token count stop growing after a few turns. Then hit `Ctrl-C` mid-reply and confirm the loop survives, and `Ctrl-D` at the prompt and confirm it saves."},"check":{"question":"A chat script appends every user and assistant message to a list and sends the whole list each turn. By turn forty each call costs about twenty times what the first did. Which change addresses the cause?","options":[{"text":"Keep the system message and drop the oldest user–assistant pairs once the history passes a token budget.","correct":true},{"text":"Send only the latest user message on each turn and rely on the model to remember earlier turns from the session ID the provider assigns.","correct":false,"tempting":"Model APIs are stateless; nothing is remembered between calls, so dropping the history removes the model's memory of the conversation entirely."},{"text":"Switch to streaming responses, which are billed only for tokens actually displayed rather than for the full history.","correct":false,"tempting":"Streaming changes when tokens arrive, not what is billed; input tokens are charged in full either way."},{"text":"Compress the messages list with `zlib` before sending so the provider counts fewer tokens for the same content.","correct":false,"tempting":"Tokens are counted on the decoded text the model reads; compressing the transport does nothing to the count."}]},"prev":{"slug":"notebooks-without-hidden-state","title":"Notebooks without hidden state: execution order, restart-and-run-all, and when to leave for a .py file","url":"https://addaly.com/learn/python-for-ai/notebooks-without-hidden-state"},"next":{"slug":"prompt-templates-safely","title":"Prompt templates: building a prompt from parts without letting the parts rewrite it","url":"https://addaly.com/learn/python-for-ai/prompt-templates-safely"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A chat loop with memory: keeping history, trimming it, and stopping cleanly', in Python, from Zero, for AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/python-for-ai/a-chat-loop-with-memory (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A chat loop with memory: keeping history, trimming it, and stopping cleanly","courseTitle":"Python, from Zero, for AI","module":{"index":9,"slug":"building-the-thing","title":"Building the thing: a small AI program, end to end"},"answer":null,"keyIdea":"A conversation is a list you send back in full on every turn, so its cost grows with its length; trim from the oldest user turn while keeping the system prompt, catch KeyboardInterrupt so the last exchange is saved, and never let the loop run without a way out.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 9.1","figure":{"caption":"The provider remembers nothing between calls, so the history is re-sent and re-billed every turn: turn forty pays for the whole conversation so far, and the reply is the cheap part. Trimming from the oldest user turn, keeping the system message and never leaving a tool call without its result, holds the line flat.","series":[{"label":"Full history, nothing trimmed","points":[[1,120],[5,900],[10,2000],[20,4200],[40,8600]]},{"label":"Trimmed to a 4,000-token budget","points":[[1,120],[5,900],[10,2000],[20,3900],[40,3950]]}],"title":"Input tokens sent on each turn of one conversation","type":"line","xLabel":"Turn number","yLabel":"Input tokens in the request"},"alt":"Figure: Input tokens sent on each turn of one conversation. Line chart of Input tokens in the request against Turn number: Full history, nothing trimmed (1, 120) (5, 900) (10, 2000) (20, 4200) (40, 8600); Trimmed to a 4,000-token budget (1, 120) (5, 900) (10, 2000) (20, 3900) (40, 3950). The provider remembers nothing between calls, so the history is re-sent and re-billed every turn: turn forty pays for the whole conversation so far, and the reply is the cheap part. Trimming from the oldest user turn, keeping the system message and never leaving a tool call without its result, holds the line flat."}},"blocks":[{"t":"h2","id":"the-loop","runs":[{"text":"The loop"}]},{"t":"code","lang":"python","label":"Python","text":"def chat(provider, system):\n    messages = [{\"role\": \"system\", \"content\": system}]\n    while True:\n        try:\n            text = input(\"you> \").strip()\n        except (EOFError, KeyboardInterrupt):\n            print()\n            break\n        if not text:\n            continue\n        if text in (\"/quit\", \"/exit\"):\n            break\n        messages.append({\"role\": \"user\", \"content\": text})\n        reply = provider.complete(messages)\n        messages.append({\"role\": \"assistant\", \"content\": reply})\n        print(f\"bot> {reply}\\n\")\n    return messages"},{"t":"p","runs":[{"text":"Every piece of this you have written before: a "},{"text":"while True","code":true},{"text":" with a "},{"text":"break","code":true},{"text":", "},{"text":"input()","code":true},{"text":", a list of dicts, a provider with one method. The two lines that make it a "},{"text":"conversation","i":true},{"text":" rather than a series of unrelated questions are the two "},{"text":"append","code":true},{"text":"s. The model sees the full list every turn. That is its memory. There is no other."}]},{"t":"h2","id":"memory-is-a-list-you-pay-to-resend","runs":[{"text":"Memory is a list you pay to resend"}]},{"t":"p","runs":[{"text":"Model APIs are stateless. The provider does not remember your last call; you remind it by sending the history back. So on turn 40, the request carries 79 earlier messages plus the new one, and you are billed for all of them as input tokens. Module 6 warned about this; here is where it bites. A chat that starts at 200 input tokens per turn is at 8,000 by turn 40 and climbing, and the model is also slower, because it reads everything before writing anything."}]},{"t":"p","runs":[{"text":"The fix is a budget:"}]},{"t":"code","lang":"python","label":"Python","text":"def trim(messages, max_tokens, count):\n    system, rest = messages[0], messages[1:]\n    while rest and count(system[\"content\"]) + sum(count(m[\"content\"]) for m in rest) > max_tokens:\n        rest = rest[2:]          # drop the oldest user+assistant pair together\n    return [system] + rest"},{"t":"p","runs":[{"text":"Three rules inside that function. "},{"text":"Keep the system message","b":true},{"text":": it carries the instructions, and dropping it changes the model’s behaviour mid-conversation. "},{"text":"Drop pairs","b":true},{"text":", so the history never starts with an assistant reply to a question the model can no longer see; some providers reject that, and the rest answer strangely. "},{"text":"Count tokens with a real counter","b":true},{"text":" from module 6 ("},{"text":"tiktoken","code":true},{"text":", the provider’s endpoint, or the tokeniser of a local model) not by characters, for the reasons given there."}]},{"t":"p","runs":[{"text":"Call it before each request: "},{"text":"messages = trim(messages, 4000, count)","code":true},{"text":". Pick the budget from the model’s context window with room for the reply, not from the window itself: a 128k window with a 120k history leaves no room to answer."}]},{"t":"p","runs":[{"text":"Smarter trimming exists (summarising the dropped turns into one message, keeping turns the model referred back to) and it belongs to the context-engineering course. The Python shape is the same: a function from a list to a shorter list, called before every send."}]},{"t":"h2","id":"stopping-cleanly","runs":[{"text":"Stopping cleanly"}]},{"t":"p","runs":[{"text":"Ctrl-C","code":true},{"text":" raises "},{"text":"KeyboardInterrupt","code":true},{"text":" wherever the program is. Caught only around "},{"text":"input()","code":true},{"text":", as above, it ends the loop politely. Caught nowhere, it kills the program with a traceback and whatever you meant to save is gone. Caught "},{"text":"everywhere","i":true},{"text":" (a bare "},{"text":"except:","code":true},{"text":" inside the loop) it becomes impossible to stop the program at all, which you will discover during a runaway loop."}]},{"t":"p","runs":[{"text":"Ctrl-D","code":true},{"text":" on macOS and Linux (or "},{"text":"Ctrl-Z","code":true},{"text":" then Enter on Windows) sends end-of-file to "},{"text":"input()","code":true},{"text":", which raises "},{"text":"EOFError","code":true},{"text":". Handle both, and a piped input file ("},{"text":"python chat.py < questions.txt","code":true},{"text":") works too: each line becomes a turn, and the loop ends at the end of the file."}]},{"t":"h2","id":"saving-and-resuming","runs":[{"text":"Saving and resuming"}]},{"t":"code","lang":"python","label":"Python","text":"from pathlib import Path\nimport json\n\ndef save(messages, path):\n    Path(path).write_text(json.dumps(messages, ensure_ascii=False, indent=1), encoding=\"utf-8\")\n\ndef load(path):\n    return json.loads(Path(path).read_text(encoding=\"utf-8\"))"},{"t":"p","runs":[{"text":"ensure_ascii=False","code":true},{"text":" keeps Hindi, Arabic and emoji as themselves rather than "},{"text":"\\uXXXX","code":true},{"text":" escapes, so the file is readable. Save after every exchange, not only at the end; a crash on turn 30 should cost you one turn, not thirty. A "},{"text":"/save","code":true},{"text":" command and a "},{"text":"--resume chat.json","code":true},{"text":" flag from module 3’s "},{"text":"argparse","code":true},{"text":" lesson complete it."}]},{"t":"h2","id":"commands-and-telling-them-apart-from-questions","runs":[{"text":"Commands, and telling them apart from questions"}]},{"t":"p","runs":[{"text":"A leading "},{"text":"/","code":true},{"text":" is the usual convention for commands: "},{"text":"/quit","code":true},{"text":", "},{"text":"/save","code":true},{"text":", "},{"text":"/clear","code":true},{"text":" to reset history, "},{"text":"/model gpt-4o-mini","code":true},{"text":" to switch provider mid-chat, which composition from module 7 makes a one-line assignment. Check for commands before appending to history, so "},{"text":"/quit","code":true},{"text":" never becomes a message the model has to answer."}]},{"t":"h2","id":"the-errors-you-will-meet","runs":[{"text":"The errors you will meet"}]},{"t":"p","runs":[{"text":"A "},{"text":"RateLimitError","code":true},{"text":" or a 429 from the provider should not end the chat. Catch it around "},{"text":"complete","code":true},{"text":", print a short line, "},{"text":"sleep","code":true},{"text":", and let the user try again; the history is intact. A "},{"text":"KeyboardInterrupt","code":true},{"text":" during a slow reply should abandon that reply and return to the prompt, not exit; catch it around "},{"text":"complete","code":true},{"text":" separately from the one around "},{"text":"input()","code":true},{"text":", and pop the unanswered user message so the history stays paired."}]},{"t":"p","runs":[{"text":"An empty reply (a model that returned nothing because it hit a content filter or "},{"text":"max_tokens","code":true},{"text":" of zero) should be visible as "},{"text":"[no reply]","code":true},{"text":" rather than a blank line the user thinks is a bug in the terminal."}]},{"t":"h2","id":"where-this-goes","runs":[{"text":"Where this goes"}]},{"t":"p","runs":[{"text":"This loop is the core of every chat product you have used, minus the interface. Module 9’s Gradio lesson replaces "},{"text":"input()","code":true},{"text":" and "},{"text":"print()","code":true},{"text":" with a web page and keeps everything else; the FastAPI lesson replaces them with an HTTP request and a response. The history list, the trim, the provider: those do not change. Build them once, here, as functions you can import."}]},{"t":"h2","id":"try-this-now","runs":[{"text":"Try this now"}],"role":"try"},{"t":"p","runs":[{"text":"Write the loop against Ollama, add "},{"text":"trim","code":true},{"text":" with a 1,000-token budget and a counter, and watch the printed token count stop growing after a few turns. Then hit "},{"text":"Ctrl-C","code":true},{"text":" mid-reply and confirm the loop survives, and "},{"text":"Ctrl-D","code":true},{"text":" at the prompt and confirm it saves."}]}],"check":{"question":[{"text":"A chat script appends every user and assistant message to a list and sends the whole list each turn. By turn forty each call costs about twenty times what the first did. Which change addresses the cause?"}],"options":[{"text":[{"text":"Keep the system message and drop the oldest user–assistant pairs once the history passes a token budget."}],"correct":true,"tempting":null},{"text":[{"text":"Send only the latest user message on each turn and rely on the model to remember earlier turns from the session ID the provider assigns."}],"correct":false,"tempting":[{"text":"Model APIs are stateless; nothing is remembered between calls, so dropping the history removes the model’s memory of the conversation entirely."}]},{"text":[{"text":"Switch to streaming responses, which are billed only for tokens actually displayed rather than for the full history."}],"correct":false,"tempting":[{"text":"Streaming changes when tokens arrive, not what is billed; input tokens are charged in full either way."}]},{"text":[{"text":"Compress the messages list with "},{"text":"zlib","code":true},{"text":" before sending so the provider counts fewer tokens for the same content."}],"correct":false,"tempting":[{"text":"Tokens are counted on the decoded text the model reads; compressing the transport does nothing to the count."}]}],"order":[2,3,0,1],"why":null},"prev":{"slug":"notebooks-without-hidden-state","title":"Notebooks without hidden state: execution order, restart-and-run-all, and when to leave for a .py file","minutes":8},"next":{"slug":"prompt-templates-safely","title":"Prompt templates: building a prompt from parts without letting the parts rewrite it","minutes":8}}}