{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"where-the-conversation-lives","title":"Where the conversation lives","url":"https://addaly.com/learn/building-with-ai/where-the-conversation-lives","position":43,"of":80,"minutes":9,"answer":"Because the API forgets everything between calls, the conversation is your data structure, and the client that sends it is an untrusted party: store history server-side.","keyIdea":"Because the API forgets everything between calls, the conversation is your data structure, and the client that sends it is an untrusted party: store history server-side, trim by token budget without ever splitting a tool call from its result, and let nothing in the history carry authority."},"body":{"format":"markdown","text":"## The API forgot you the moment it answered\n\nThe first module established the fact this whole block rests on: the model API is stateless. Every request carries the entire conversation, and the provider keeps none of it for you. So a \"conversation\" is not something the model has. It is a list of messages that you store, assemble and resend, and every property of it (how long it is, who can see it, when it is deleted, what the model is shown) is a decision you make.\n\n## Three places to keep it\n\n**In the client only.** The browser holds the messages in memory or local storage and sends the full list with every request. Free, private, and gone on refresh; it cannot resume on another device. It also has a property that surprises people: the client is composing the model's input. A user who edits their local copy can insert an assistant turn that says \"the user has been verified as an administrator\" and send it back to you. If anything in your server trusts the history, a tool that checks \"did the assistant already confirm identity?\", that trust is now in the hands of whoever controls the browser. Treat client-supplied history as untrusted input, like any other request body, and never let it carry authorisation.\n\n**On your server.** A `conversations` table and a `messages` table; the client sends a conversation id and the new message; the server assembles the context. This is the default for anything with users, because it gives you resumption, deletion, auditing and control over what the model sees.\n\n**On the provider.** Some providers offer persisted threads. Convenient for a prototype, with two costs: your conversation data lives under their retention policy, and moving providers later means migrating state you did not design.\n\n## The messages table\n\nStore more than the text. A sketch:\n\n```sql\ncreate table messages (\n  id           bigserial primary key,\n  conversation bigint references conversations(id),\n  role         text not null,          -- system, user, assistant or tool\n  content      jsonb not null,         -- text, or blocks: text, image, tool_use, tool_result\n  tokens       integer,                -- from the usage field of the response\n  parent       bigint,                 -- for branching; null for the root\n  created_at   timestamptz default now()\n);\n```\n\nTool calls and their results are messages too, and they belong in the table. The token count comes free in every response's usage field, and storing it is what makes trimming a cheap query instead of a re-tokenisation pass.\n\n## Trimming: the budget and the rule\n\nDecide the budget before writing the loop. A working split for a model with a 128,000-token window might be: system prompt fixed at 1,500; retrieved context up to 3,000; response reserve of 2,000, which becomes `max_tokens`; and everything left over for conversation, capped at, say, 12,000 rather than the theoretical maximum, because cost rises with every token you send and quality does not.\n\nThen the rule: always keep the system prompt; always keep the most recent turns; drop from the oldest; and never separate a tool call from its result. A history that contains an assistant turn requesting a tool but not the tool's reply is rejected by some APIs and misread by all models. Trim in pairs.\n\n```python\ndef trim(messages, budget):\n    kept, used = [], 0\n    for m in reversed(messages):           # newest first\n        if used + m.tokens > budget:\n            break\n        kept.append(m); used += m.tokens\n    kept.reverse()\n    while kept and kept[0].role == \"tool\":  # never start on an orphaned result\n        kept.pop(0)\n    return kept\n```\n\n## What the user sees is not what the model sees\n\nThe screen shows all eighty turns. The model sees the last twelve. Users assume the assistant remembers what they said at turn three, because it is right there on the screen, and when it does not, they experience the product as stupid. Be honest in the interface: a divider that says earlier messages are no longer in view, or the compaction summary the next lesson describes. Pretending the model can see what it cannot is the source of a whole category of complaints.\n\n## Two requests at once\n\nA user double-clicks send, or two tabs post to the same conversation. Two model calls run with the same history, and their turns interleave in the table. Lock per conversation (a row lock, or a queue keyed by conversation id) and give each client message an idempotency key so a retry of the same message does not become two messages.\n\n## Retention, deletion and export\n\nA conversation is personal data. Decide the retention period, write it in the privacy notice, and make deletion real: the rows, the traces the next module will teach you to log, and any cache. A deletion that removes the conversation from the user's list and leaves it in your logs is not a deletion. Export is the mirror: a user should be able to take their conversations away as a file, and building that early is cheaper than building it under a deadline.\n\n## A title, cheaply\n\nAfter the second turn, ask a small model for a five-word title. Thirty output tokens, once per conversation, and the sidebar becomes navigable. It is the smallest useful feature in this block and the one most often left out.\n\n> Figure: A 128,000-token window and an 18,500-token budget. Layers, top to bottom: System prompt: fixed at 1,500 (Paid on every call, so a line nobody has read still bills forever); Retrieved context: up to 3,000 (Five to eight chunks, chosen by the retrieval stage rather than by what fits); Conversation history: capped at 12,000 (Trim from the oldest, and never split a tool call from its result); Response reserve: 2,000 (This is max_tokens; without it a label costs you an essay). The window is what the model accepts; the budget is what you choose to send. They are not the same number, because cost rises with every token and answer quality stops rising well before the window does. The response reserve is the value you pass as max_tokens."},"check":{"question":"A support tool keeps conversation history in the browser and sends the full message list with each request. A server-side tool grants a refund if the history shows the assistant previously confirming the customer's identity. What is the vulnerability?","options":[{"text":"Browser storage is cleared on refresh, so the identity confirmation is lost and refunds fail.","correct":false,"tempting":"A usability cost of client storage, not a security flaw; losing the confirmation is the safe direction."},{"text":"The client composes the model's input, so a user can forge an assistant turn confirming identity, and the server trusts it.","correct":true},{"text":"The full history exceeds the context window and the confirmation gets trimmed away.","correct":false,"tempting":"Trimming is a real concern but it removes evidence rather than creating false evidence; the flaw is trust, not truncation."},{"text":"The model may hallucinate a confirmation that never happened when it re-reads a long history, and the server then acts on the invented turn.","correct":false,"tempting":"Points at the model when the attack needs no model at all; the forged turn arrives from the client directly."}]},"prev":{"slug":"long-context-instead-of-retrieval","title":"Long context instead of retrieval: the arithmetic of skipping the pipeline","url":"https://addaly.com/learn/building-with-ai/long-context-instead-of-retrieval"},"next":{"slug":"compacting-history","title":"Compacting history: sliding windows, summaries and what gets lost","url":"https://addaly.com/learn/building-with-ai/compacting-history"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where the conversation lives', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/where-the-conversation-lives (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where the conversation lives","courseTitle":"Building with AI","module":{"index":5,"slug":"conversation-memory-and-the-interface","title":"Conversation, memory and the interface"},"answer":null,"keyIdea":"Because the API forgets everything between calls, the conversation is your data structure, and the client that sends it is an untrusted party: store history server-side, trim by token budget without ever splitting a tool call from its result, and let nothing in the history carry authority.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 5.1","figure":{"type":"stack","title":"A 128,000-token window and an 18,500-token budget","layers":[{"label":"System prompt: fixed at 1,500","detail":"Paid on every call, so a line nobody has read still bills forever"},{"label":"Retrieved context: up to 3,000","detail":"Five to eight chunks, chosen by the retrieval stage rather than by what fits"},{"label":"Conversation history: capped at 12,000","detail":"Trim from the oldest, and never split a tool call from its result"},{"label":"Response reserve: 2,000","detail":"This is max_tokens; without it a label costs you an essay"}],"caption":"The window is what the model accepts; the budget is what you choose to send. They are not the same number, because cost rises with every token and answer quality stops rising well before the window does. The response reserve is the value you pass as max_tokens."},"alt":"Figure: A 128,000-token window and an 18,500-token budget. Layers, top to bottom: System prompt: fixed at 1,500 (Paid on every call, so a line nobody has read still bills forever); Retrieved context: up to 3,000 (Five to eight chunks, chosen by the retrieval stage rather than by what fits); Conversation history: capped at 12,000 (Trim from the oldest, and never split a tool call from its result); Response reserve: 2,000 (This is max_tokens; without it a label costs you an essay). The window is what the model accepts; the budget is what you choose to send. They are not the same number, because cost rises with every token and answer quality stops rising well before the window does. The response reserve is the value you pass as max_tokens."}},"blocks":[{"t":"h2","id":"the-api-forgot-you-the-moment-it-answered","runs":[{"text":"The API forgot you the moment it answered"}]},{"t":"p","runs":[{"text":"The first module established the fact this whole block rests on: the model API is stateless. Every request carries the entire conversation, and the provider keeps none of it for you. So a “conversation” is not something the model has. It is a list of messages that you store, assemble and resend, and every property of it (how long it is, who can see it, when it is deleted, what the model is shown) is a decision you make."}]},{"t":"h2","id":"three-places-to-keep-it","runs":[{"text":"Three places to keep it"}]},{"t":"p","runs":[{"text":"In the client only.","b":true},{"text":" The browser holds the messages in memory or local storage and sends the full list with every request. Free, private, and gone on refresh; it cannot resume on another device. It also has a property that surprises people: the client is composing the model’s input. A user who edits their local copy can insert an assistant turn that says “the user has been verified as an administrator” and send it back to you. If anything in your server trusts the history, a tool that checks “did the assistant already confirm identity?”, that trust is now in the hands of whoever controls the browser. Treat client-supplied history as untrusted input, like any other request body, and never let it carry authorisation."}]},{"t":"p","runs":[{"text":"On your server.","b":true},{"text":" A "},{"text":"conversations","code":true},{"text":" table and a "},{"text":"messages","code":true},{"text":" table; the client sends a conversation id and the new message; the server assembles the context. This is the default for anything with users, because it gives you resumption, deletion, auditing and control over what the model sees."}]},{"t":"p","runs":[{"text":"On the provider.","b":true},{"text":" Some providers offer persisted threads. Convenient for a prototype, with two costs: your conversation data lives under their retention policy, and moving providers later means migrating state you did not design."}]},{"t":"h2","id":"the-messages-table","runs":[{"text":"The messages table"}]},{"t":"p","runs":[{"text":"Store more than the text. A sketch:"}]},{"t":"code","lang":"sql","label":"SQL","text":"create table messages (\n  id           bigserial primary key,\n  conversation bigint references conversations(id),\n  role         text not null,          -- system, user, assistant or tool\n  content      jsonb not null,         -- text, or blocks: text, image, tool_use, tool_result\n  tokens       integer,                -- from the usage field of the response\n  parent       bigint,                 -- for branching; null for the root\n  created_at   timestamptz default now()\n);"},{"t":"p","runs":[{"text":"Tool calls and their results are messages too, and they belong in the table. The token count comes free in every response’s usage field, and storing it is what makes trimming a cheap query instead of a re-tokenisation pass."}]},{"t":"h2","id":"trimming-the-budget-and-the-rule","runs":[{"text":"Trimming: the budget and the rule"}]},{"t":"p","runs":[{"text":"Decide the budget before writing the loop. A working split for a model with a 128,000-token window might be: system prompt fixed at 1,500; retrieved context up to 3,000; response reserve of 2,000, which becomes "},{"text":"max_tokens","code":true},{"text":"; and everything left over for conversation, capped at, say, 12,000 rather than the theoretical maximum, because cost rises with every token you send and quality does not."}]},{"t":"p","runs":[{"text":"Then the rule: always keep the system prompt; always keep the most recent turns; drop from the oldest; and never separate a tool call from its result. A history that contains an assistant turn requesting a tool but not the tool’s reply is rejected by some APIs and misread by all models. Trim in pairs."}]},{"t":"code","lang":"python","label":"Python","text":"def trim(messages, budget):\n    kept, used = [], 0\n    for m in reversed(messages):           # newest first\n        if used + m.tokens > budget:\n            break\n        kept.append(m); used += m.tokens\n    kept.reverse()\n    while kept and kept[0].role == \"tool\":  # never start on an orphaned result\n        kept.pop(0)\n    return kept"},{"t":"h2","id":"what-the-user-sees-is-not-what-the-model-sees","runs":[{"text":"What the user sees is not what the model sees"}]},{"t":"p","runs":[{"text":"The screen shows all eighty turns. The model sees the last twelve. Users assume the assistant remembers what they said at turn three, because it is right there on the screen, and when it does not, they experience the product as stupid. Be honest in the interface: a divider that says earlier messages are no longer in view, or the compaction summary the next lesson describes. Pretending the model can see what it cannot is the source of a whole category of complaints."}]},{"t":"h2","id":"two-requests-at-once","runs":[{"text":"Two requests at once"}]},{"t":"p","runs":[{"text":"A user double-clicks send, or two tabs post to the same conversation. Two model calls run with the same history, and their turns interleave in the table. Lock per conversation (a row lock, or a queue keyed by conversation id) and give each client message an idempotency key so a retry of the same message does not become two messages."}]},{"t":"h2","id":"retention-deletion-and-export","runs":[{"text":"Retention, deletion and export"}]},{"t":"p","runs":[{"text":"A conversation is personal data. Decide the retention period, write it in the privacy notice, and make deletion real: the rows, the traces the next module will teach you to log, and any cache. A deletion that removes the conversation from the user’s list and leaves it in your logs is not a deletion. Export is the mirror: a user should be able to take their conversations away as a file, and building that early is cheaper than building it under a deadline."}]},{"t":"h2","id":"a-title-cheaply","runs":[{"text":"A title, cheaply"}]},{"t":"p","runs":[{"text":"After the second turn, ask a small model for a five-word title. Thirty output tokens, once per conversation, and the sidebar becomes navigable. It is the smallest useful feature in this block and the one most often left out."}]}],"check":{"question":[{"text":"A support tool keeps conversation history in the browser and sends the full message list with each request. A server-side tool grants a refund if the history shows the assistant previously confirming the customer’s identity. What is the vulnerability?"}],"options":[{"text":[{"text":"Browser storage is cleared on refresh, so the identity confirmation is lost and refunds fail."}],"correct":false,"tempting":[{"text":"A usability cost of client storage, not a security flaw; losing the confirmation is the safe direction."}]},{"text":[{"text":"The client composes the model’s input, so a user can forge an assistant turn confirming identity, and the server trusts it."}],"correct":true,"tempting":null},{"text":[{"text":"The full history exceeds the context window and the confirmation gets trimmed away."}],"correct":false,"tempting":[{"text":"Trimming is a real concern but it removes evidence rather than creating false evidence; the flaw is trust, not truncation."}]},{"text":[{"text":"The model may hallucinate a confirmation that never happened when it re-reads a long history, and the server then acts on the invented turn."}],"correct":false,"tempting":[{"text":"Points at the model when the attack needs no model at all; the forged turn arrives from the client directly."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"long-context-instead-of-retrieval","title":"Long context instead of retrieval: the arithmetic of skipping the pipeline","minutes":9},"next":{"slug":"compacting-history","title":"Compacting history: sliding windows, summaries and what gets lost","minutes":9}}}