Where the conversation lives

The API forgot you the moment it answered

The first module established the fact this whole block rests on: the model API is stateless. Every request carries the entire conversation, and the provider keeps none of it for you. So a “conversation” is not something the model has. It is a list of messages that you store, assemble and resend, and every property of it (how long it is, who can see it, when it is deleted, what the model is shown) is a decision you make.

Three places to keep it

In the client only. The browser holds the messages in memory or local storage and sends the full list with every request. Free, private, and gone on refresh; it cannot resume on another device. It also has a property that surprises people: the client is composing the model’s input. A user who edits their local copy can insert an assistant turn that says “the user has been verified as an administrator” and send it back to you. If anything in your server trusts the history, a tool that checks “did the assistant already confirm identity?”, that trust is now in the hands of whoever controls the browser. Treat client-supplied history as untrusted input, like any other request body, and never let it carry authorisation.

On your server. A conversations table and a messages table; the client sends a conversation id and the new message; the server assembles the context. This is the default for anything with users, because it gives you resumption, deletion, auditing and control over what the model sees.

On the provider. Some providers offer persisted threads. Convenient for a prototype, with two costs: your conversation data lives under their retention policy, and moving providers later means migrating state you did not design.

The messages table

Store more than the text. A sketch:

create table messages (
  id           bigserial primary key,
  conversation bigint references conversations(id),
  role         text not null,          -- system, user, assistant or tool
  content      jsonb not null,         -- text, or blocks: text, image, tool_use, tool_result
  tokens       integer,                -- from the usage field of the response
  parent       bigint,                 -- for branching; null for the root
  created_at   timestamptz default now()
);

Tool calls and their results are messages too, and they belong in the table. The token count comes free in every response’s usage field, and storing it is what makes trimming a cheap query instead of a re-tokenisation pass.

Trimming: the budget and the rule

Decide the budget before writing the loop. A working split for a model with a 128,000-token window might be: system prompt fixed at 1,500; retrieved context up to 3,000; response reserve of 2,000, which becomes max_tokens; and everything left over for conversation, capped at, say, 12,000 rather than the theoretical maximum, because cost rises with every token you send and quality does not.

Then the rule: always keep the system prompt; always keep the most recent turns; drop from the oldest; and never separate a tool call from its result. A history that contains an assistant turn requesting a tool but not the tool’s reply is rejected by some APIs and misread by all models. Trim in pairs.

def trim(messages, budget):
    kept, used = [], 0
    for m in reversed(messages):           # newest first
        if used + m.tokens > budget:
            break
        kept.append(m); used += m.tokens
    kept.reverse()
    while kept and kept[0].role == "tool":  # never start on an orphaned result
        kept.pop(0)
    return kept

What the user sees is not what the model sees

The screen shows all eighty turns. The model sees the last twelve. Users assume the assistant remembers what they said at turn three, because it is right there on the screen, and when it does not, they experience the product as stupid. Be honest in the interface: a divider that says earlier messages are no longer in view, or the compaction summary the next lesson describes. Pretending the model can see what it cannot is the source of a whole category of complaints.

Two requests at once

A user double-clicks send, or two tabs post to the same conversation. Two model calls run with the same history, and their turns interleave in the table. Lock per conversation (a row lock, or a queue keyed by conversation id) and give each client message an idempotency key so a retry of the same message does not become two messages.

Retention, deletion and export

A conversation is personal data. Decide the retention period, write it in the privacy notice, and make deletion real: the rows, the traces the next module will teach you to log, and any cache. A deletion that removes the conversation from the user’s list and leaves it in your logs is not a deletion. Export is the mirror: a user should be able to take their conversations away as a file, and building that early is cheaper than building it under a deadline.

A title, cheaply

After the second turn, ask a small model for a five-word title. Thirty output tokens, once per conversation, and the sidebar becomes navigable. It is the smallest useful feature in this block and the one most often left out.