Proving the change helped

Four numbers, not one

Every decision in this module (dtype, quantization, offloading, runtime, model size) trades along four axes at once. Report all four or you have not measured anything:

1. Peak memory. The constraint that decides whether it runs at all.
2. Time to first token. What an interactive user perceives.
3. Throughput, in tokens or items per second. What a batch job costs.
4. Quality, on your own examples. The one people skip, and the one that invalidates the rest.

A change that halves memory and costs three points of accuracy is a good trade or a disaster depending on the application, and you cannot tell which without the fourth number.

A harness small enough to actually use

import time, torch

def measure(pipe, prompts, max_new_tokens=128):
    _ = pipe(prompts[0], max_new_tokens=8)           # warm-up
    if torch.cuda.is_available():
        torch.cuda.synchronize(); torch.cuda.reset_peak_memory_stats()

    t0 = time.perf_counter()
    outs = [pipe(p, max_new_tokens=max_new_tokens) for p in prompts]
    if torch.cuda.is_available():
        torch.cuda.synchronize()
    elapsed = time.perf_counter() - t0

    peak = torch.cuda.max_memory_allocated()/1e9 if torch.cuda.is_available() else None
    return {"seconds": round(elapsed, 2),
            "per_prompt": round(elapsed/len(prompts), 3),
            "peak_gb": round(peak, 2) if peak else None}

Three things this gets right that ad-hoc timing gets wrong:

• Warm-up excluded. The first call allocates, compiles kernels and pages weights in. It is two to ten times slower and it is not representative.
• Synchronised. CUDA work is queued asynchronously; without synchronize() you are timing Python.
• Peak, not current. Out-of-memory happens at the peak, which may last milliseconds.

Measure time to first token separately, because it is a different quantity:

t0 = time.perf_counter()
for chunk in streamer:
    ttft = time.perf_counter() - t0
    break

Report the tail

A mean of 300 ms with a 99th percentile of 4 seconds describes a system that one user in a hundred experiences as broken. Collect at least fifty runs and report the median and the 95th percentile. On shared free hardware the spread is wide and the mean is close to meaningless.

Measuring the quality side

This is twenty minutes of work and it is the part that gets skipped.

1. Fix thirty examples from your real inputs, with expected outputs where an expected output exists.
2. Run them through configuration A and configuration B.
3. Score whatever is objective: exact match, valid JSON, correct number, correct label.
4. Read the disagreements yourself. Thirty is few enough to read.

Thirty examples will not detect a two-point difference: it is too small a sample for that, and pretending otherwise is how people convince themselves of improvements that are noise. It will reliably detect the failure that matters here: a configuration that has stopped producing valid structured output, or lost a language, or started truncating. That is a large, visible effect and thirty items find it.

Keep the examples in a file in the repository, not in a notebook cell. They will outlive the notebook.

Change one thing

The universal discipline, and the one people abandon when they are in a hurry. If you switch from bfloat16 on a T4 to 4-bit AWQ on an A10 with a new batch size, and it is faster, you have learned nothing transferable.

Write the configurations down before you run them:

A: bf16, batch 1, T4        → 8.1 tok/s, 14.2 GB, 30/30 valid JSON
B: 4-bit nf4, batch 1, T4   → 6.4 tok/s,  5.1 GB, 27/30 valid JSON
C: Q4_K_M GGUF, CPU laptop  → 7.9 tok/s,  4.4 GB, 29/30 valid JSON

That table (three rows, four columns, an afternoon) is a decision. Note the result in row B that surprises people: 4-bit bitsandbytes is slower than bfloat16, because it buys memory rather than speed.

Do this now

Run your current setup through the harness and write down the four numbers. Even with nothing to compare against, you now have a baseline, and every later claim that something got faster becomes checkable rather than remembered.