A chat loop with memory: keeping history, trimming it, and stopping cleanly

The loop

def chat(provider, system):
    messages = [{"role": "system", "content": system}]
    while True:
        try:
            text = input("you> ").strip()
        except (EOFError, KeyboardInterrupt):
            print()
            break
        if not text:
            continue
        if text in ("/quit", "/exit"):
            break
        messages.append({"role": "user", "content": text})
        reply = provider.complete(messages)
        messages.append({"role": "assistant", "content": reply})
        print(f"bot> {reply}\n")
    return messages

Every piece of this you have written before: a while True with a break, input(), a list of dicts, a provider with one method. The two lines that make it a conversation rather than a series of unrelated questions are the two appends. The model sees the full list every turn. That is its memory. There is no other.

Memory is a list you pay to resend

Model APIs are stateless. The provider does not remember your last call; you remind it by sending the history back. So on turn 40, the request carries 79 earlier messages plus the new one, and you are billed for all of them as input tokens. Module 6 warned about this; here is where it bites. A chat that starts at 200 input tokens per turn is at 8,000 by turn 40 and climbing, and the model is also slower, because it reads everything before writing anything.

The fix is a budget:

def trim(messages, max_tokens, count):
    system, rest = messages[0], messages[1:]
    while rest and count(system["content"]) + sum(count(m["content"]) for m in rest) > max_tokens:
        rest = rest[2:]          # drop the oldest user+assistant pair together
    return [system] + rest

Three rules inside that function. Keep the system message — it carries the instructions, and dropping it changes the model's behaviour mid-conversation. Drop pairs, so the history never starts with an assistant reply to a question the model can no longer see; some providers reject that, and the rest answer strangely. Count tokens with a real counter from module 6 — tiktoken, the provider's endpoint, or the tokeniser of a local model — not by characters, for the reasons given there.

Call it before each request: messages = trim(messages, 4000, count). Pick the budget from the model's context window with room for the reply, not from the window itself: a 128k window with a 120k history leaves no room to answer.

Smarter trimming exists — summarising the dropped turns into one message, keeping turns the model referred back to — and it belongs to the context-engineering course. The Python shape is the same: a function from a list to a shorter list, called before every send.

Stopping cleanly

Ctrl-C raises KeyboardInterrupt wherever the program is. Caught only around input(), as above, it ends the loop politely. Caught nowhere, it kills the program with a traceback and whatever you meant to save is gone. Caught everywhere — a bare except: inside the loop — it becomes impossible to stop the program at all, which you will discover during a runaway loop.

Ctrl-D on macOS and Linux (or Ctrl-Z then Enter on Windows) sends end-of-file to input(), which raises EOFError. Handle both, and a piped input file — python chat.py < questions.txt — works too: each line becomes a turn, and the loop ends at the end of the file.

Saving and resuming

from pathlib import Path
import json

def save(messages, path):
    Path(path).write_text(json.dumps(messages, ensure_ascii=False, indent=1))

def load(path):
    return json.loads(Path(path).read_text())

ensure_ascii=False keeps Hindi, Arabic and emoji as themselves rather than \uXXXX escapes, so the file is readable. Save after every exchange, not only at the end; a crash on turn 30 should cost you one turn, not thirty. A /save command and a --resume chat.json flag from module 3's argparse lesson complete it.

Commands, and telling them apart from questions

A leading / is the usual convention for commands: /quit, /save, /clear to reset history, /model gpt-4o-mini to switch provider mid-chat — which composition from module 7 makes a one-line assignment. Check for commands before appending to history, so /quit never becomes a message the model has to answer.

The errors you will meet

A RateLimitError or a 429 from the provider should not end the chat. Catch it around complete, print a short line, sleep, and let the user try again — the history is intact. A KeyboardInterrupt during a slow reply should abandon that reply and return to the prompt, not exit; catch it around complete separately from the one around input(), and pop the unanswered user message so the history stays paired.

An empty reply — a model that returned nothing because it hit a content filter or max_tokens of zero — should be visible as [no reply] rather than a blank line the user thinks is a bug in the terminal.

Where this goes

This loop is the core of every chat product you have used, minus the interface. Module 9's Gradio lesson replaces input() and print() with a web page and keeps everything else; the FastAPI lesson replaces them with an HTTP request and a response. The history list, the trim, the provider — those do not change. Build them once, here, as functions you can import.

Try this now

Write the loop against Ollama, add trim with a 1,000-token budget and a counter, and watch the printed token count stop growing after a few turns. Then hit Ctrl-C mid-reply and confirm the loop survives, and Ctrl-D at the prompt and confirm it saves.