Measuring your own position curve

The experiment, on your material, in an afternoon

The position curve is model-specific, task-specific and length-specific, which means the only trustworthy number is one you produced. The measurement is small enough to do properly.

You need:

1. Forty to a hundred questions whose answer is contained in exactly one known passage. If you built the retrieval gold set from the previous module, you already have this.
2. A pool of distractor passages from your own corpus: real ones, retrieved for the same query, not filler. Distractors matter enormously; a probe padded with unrelated text measures nothing you care about.
3. A grader that can score an answer without a human. Exact match, or containment of a key string, is usually enough. Keep it dumb and deterministic.

POSITIONS = [0, 4, 9, 14, 19]     # in a 20-passage context

results = {p: [] for p in POSITIONS}
for q, gold, distractors in probe_set:
    for p in POSITIONS:
        ctx = distractors[:p] + [gold] + distractors[p:19]
        out = call_model(assemble(q, ctx))     # temperature 0
        results[p].append(grade(out, q))

for p in POSITIONS:
    print(p, round(mean(results[p]), 3))

Hold everything else fixed. Same questions, same distractors, same instructions, same sampling settings, same model version. The only variable is p.

Reading the output honestly

With fifty questions per position, the standard error on an accuracy near 0.8 is roughly six percentage points. Two positions differing by four points are indistinguishable. A U with a twenty-point dip is real; a jagged line with five-point wiggles is noise wearing a shape.

Use a bootstrap if you want an interval rather than a feeling:

def ci(xs, n=2000):
    ms = [mean(random.choices(xs, k=len(xs))) for _ in range(n)]
    return quantile(ms, 0.025), quantile(ms, 0.975)

And resist the temptation to interpret the shape before checking whether it survives resampling. This is the single most common error in prompt experiments: a difference of a few points on forty examples, adopted as a design principle, defended for a year.

Run it at the lengths you actually use

A curve measured at 20 passages does not predict behaviour at 60, and it certainly does not predict behaviour at 300,000 tokens. Run the probe at each context size your system produces (typically your median and your 95th percentile) and note that the curve usually flattens at short lengths and deepens at long ones.

If your median request has five passages, you may find the effect is negligible and the whole subject is not worth optimising for. That is a legitimate and valuable finding, and it takes an afternoon to establish rather than a quarter of arguing.

Do it cheaply first

Run the probe against a local model before spending anything. llama.cpp or Ollama with an 8B instruct model gives you the shape for free, and the shape is what you are after. Then confirm the direction on your production model with a few hundred calls, which typically costs less than a coffee.

The local run is not a substitute (smaller models generally show a stronger position effect than larger ones) but it de-risks the design of the harness, which is where the bugs live.

What to do with the curve

Three possible outcomes and three responses.

Flat. Order by whatever else is convenient: chronology, document structure, cache stability. You have one fewer thing to worry about and you now know it rather than assuming it.

A clear U. Put the strongest evidence at the ends. In practice this means best-first and second-best-last, or best-last if your instruction sits at the top. Retest after the change rather than trusting the reordering to have worked.

A monotone decline towards the end. Rarer, and it usually means your prompt is structured so the model reads for the question early. Check where your question is; moving it can change the curve entirely.

Add it to CI, cheaply

Once the harness exists, a reduced version (twenty questions, three positions) runs in a couple of minutes and costs cents. Run it when the model version changes, because the curve is a property of the model and vendors update models underneath you.

That is the practical payoff of building the probe: not the one-off answer, but the ability to detect that the answer has changed without waiting for a user to notice.