Retrieval: what it fixes and what it does not

The idea in one line

Retrieval-augmented generation means: before you ask the model a question, go and find the relevant text yourself, paste it into the prompt, and tell the model to answer from it. That is all. There is no special model and no special API.

def answer(question):
    chunks = search(question, k=6)                 # your search, whatever it is
    context = "\n\n".join(
        f"[{i+1}] {c.title} (updated {c.updated})\n{c.text}"
        for i, c in enumerate(chunks)
    )
    prompt = f"""Answer using only the passages below. Cite sources like [2].
If the passages do not contain the answer, say "I don't know" and name what is missing.
If two passages disagree, prefer the most recently updated one and say that you did.

Passages:
{context}

Question: {question}"""
    return call_model(prompt)

What it genuinely fixes

Knowledge the model never had. Your company's leave policy, a school's fee structure, last week's incident report. No amount of model quality helps here; the information is simply not in the weights.

Staleness. A model trained a year ago has a year-old view of the world. Retrieval reads today's document.

Scale. You cannot paste 40,000 support articles into a prompt, and you would not want to pay for it if you could.

Citations. This is underrated. When the answer names its sources, a user can check it, and a wrong answer becomes a correctable one rather than a trust problem.

What it does not fix

It does not stop hallucination. It reduces it when the right passage is found. It can make it worse when a confident, wrong or outdated passage is found, because now the model has evidence for the wrong answer.

It does not make the model reason better. If the question needs a chain of inference across five documents, retrieval hands over five documents and the reasoning is still the hard part.

It does not answer aggregate questions. "How many refunds did we issue in Q3?" is not a retrieval question. No single passage contains the answer, and pulling six passages that each mention a refund produces a confident, invented number. That question is SQL. Route it to SQL.

It does not fix a bad search index. This is the big one.

Most RAG failures are search failures

If the passage containing the answer is not in the top k, nothing downstream can save you. Not a better model, not a cleverer prompt, not a longer context window. The answer was never in the room.

So measure the two halves separately. Take 50 real questions, note which document should answer each, and compute how often it appears in your top k. If retrieval recall is 60%, your ceiling is roughly 60% and prompt work is wasted effort.

hits = sum(1 for q in golden if q.expected_doc in {c.doc_id for c in search(q.text, k=6)})
print(f"recall@6: {hits}/{len(golden)}")

A related and unglamorous truth: plain keyword search often beats a naive embedding index on the things users actually type. Product codes, error codes, invoice numbers, people's names, drug names. ERR_4021 and ERR_4012 sit close together in embedding space and are completely different problems. Run both and combine them.

Freshness is your job, not the model's

A support bot in Jakarta answers policy questions well for months, then starts confidently stating a rule that changed last year. The logs show both the old and the new policy page were retrieved. The model had no way to tell which one was current — they both look like policy.

Nothing in the model fixes that. The index has to. Delete or archive superseded documents, stamp every chunk with an updated date, filter to current versions before searching, and put the date in the passage text so the model can see it and so your prompt rule about preferring recent sources has something to act on.

Retrieval decides what the model is allowed to know. Spend your time there.