Making the assembler testable

The shape that makes everything else possible

Most of the debugging pain in a context-heavy system comes from one design decision: the code that builds the request also fetches the material, calls the model, and parses the answer, all in one function. Nothing can be tested without a network, nothing can be reproduced, and the only way to see what the model received is to add a print statement and wait for the bug to happen again.

The fix is unglamorous and worth a lesson because it changes what is possible later.

@dataclass(frozen=True)
class ContextInputs:
    question: str
    passages: list[Passage]     # already retrieved
    history: list[Turn]         # already loaded
    user: UserProfile
    now: datetime               # injected, never called inside

def build_context(inp: ContextInputs, cfg: AssemblyConfig) -> Request:
    """Pure. No IO, no clock, no randomness. Same inputs, same bytes."""

Retrieval happens outside. The clock is passed in. The function returns the request object it would send, and sends nothing.

What purity buys you

Snapshot tests. You can assert on the exact assembled request in a test that runs in eight milliseconds and needs no API key.

def test_assembly_snapshot(snapshot):
    req = build_context(FIXTURE, CFG)
    assert render(req) == snapshot("support_v3.txt")

When somebody changes a template, the diff appears in code review as text a human can read. This catches the class of bug that is otherwise invisible: a stray newline that multiplies across turns, a heading that stopped rendering, a passage separator that silently changed and broke the model’s ability to tell one document from the next.

Reproducing a production failure. Log the ContextInputs (or a reference to them) and you can rebuild the exact request locally from a trace. Without the separation you can log the final string, which helps, but you cannot re-run the assembler with a changed configuration to see whether your fix would have helped.

Cheap experiments. Assembly variants become configuration rather than branches. Ordering, chunk count, whether the instruction is repeated at the end, how history is compacted, all of these become fields in AssemblyConfig, and an experiment is a different config value rather than a code change.

Honest token accounting. The counter runs on the output of the assembler, which is the thing actually sent, rather than on an estimate built somewhere else that drifts out of sync within a fortnight.

The three things that break purity

In practice three impurities creep in, all with the same fix.

1. datetime.now() inside the builder. Common, because you want to tell the model today’s date. Inject it. Otherwise your snapshot test fails at midnight and somebody deletes the test.
2. A retrieval call inside the builder. Convenient, and it makes every test require a network and a live index. Fetch above, pass the results down.
3. Randomness: sampling which exemplars to include, shuffling passages to combat position bias. Legitimate techniques; pass the seed in.

The pattern is the same each time: move the non-determinism up to the caller, where it can be controlled by a test.

Version the configuration, not just the prompt

Assembly is more than a template. Two requests can use the same words and behave differently because one retrieved eight passages and the other five. So the version identifier has to cover the whole configuration:

AV = hashlib.sha256(
    json.dumps(asdict(cfg), sort_keys=True).encode()
).hexdigest()[:12]

Put AV on every log line and every stored evaluation result. When a metric moves next month, the first question (what changed in assembly) becomes answerable in one query instead of a git archaeology session.

Render for humans as well as models

Add one function nobody regrets writing:

def render(req: Request) -> str:
    """The assembled request as readable text, with a token count per block."""

Wire it to a debug route, a CLI flag, or a file dropped next to a failed test. The first time you read your own assembled context end to end you will find something wrong with it. Everyone does. Common finds: a passage included twice because two chunks overlapped, a header repeated eleven times, a tool description that says “TODO: describe this”, and history entries containing the model’s own previous errors being carried forward as though they were facts.

The boundary this creates

Once assembly is a pure function, the system divides cleanly into three parts you can reason about separately: gathering (retrieval, database reads, tool calls: all IO, all mockable), assembly (pure, tested, versioned), and calling (the model, retries, streaming, parsing).

Each of the remaining modules in this course lives mostly in one of those. Selection and retrieval are gathering. Ordering, caching and trust boundaries are assembly. Measurement spans all three, and it only works because the middle one is deterministic.