Where it is reliable, and where it is not

One question to ask before you ask

The first lesson gave you the mechanism: the model guesses the next word from what people have written. That gives you a tool for predicting its reliability before you type a single prompt.

Ask yourself: how many times has this exact thing been written down?

The boiling point of water at sea level has been written down millions of times, in every language, in every textbook, and it is always the same. The model has seen it so often that its guess is effectively a fact. The population of Kisumu at the 2019 census has been written down a few hundred times, inconsistently, in documents that also contain other numbers. The model’s guess is a plausible-looking number in the right range. Sometimes it is right. You cannot tell from the answer which time this is.

That is the whole rule. Everything below is that rule applied.

Reliable ground

These are the areas where the model’s answers are nearly always sound, because the writing it learned from is vast and agrees with itself:

• Core school science and maths. Newton’s laws, photosynthesis, the water cycle, quadratic equations, the periodic table. Explanations of these are excellent and the errors are rare.
• Grammar and writing in major languages. English, Spanish, French, Hindi, Mandarin, Arabic, Portuguese. It has read more of these than any human.
• Well-known history. Dates and causes of major events. The more written about, the safer.
• Explaining a concept several ways. This is generation, not recall, and it is what the mechanism is best at.

Unreliable ground

And here the answers look identical but should be treated as drafts to check:

• Local specifics. The founding date of your school, the name of your district’s first commissioner, which crops grow in your division. Written down rarely; often wrong.
• Precise numbers. Populations, distances, percentages from a specific report. It produces a number in the right range because the right range is what appears in text. The exact figure is a guess.
• Citations. Book titles, page numbers, journal articles, quotations attributed to a named person. It has learned what citations look like and will generate one that looks perfect and does not exist. Every citation gets checked. No exceptions.
• Recent events. Anything after its training cutoff, which is typically several months to a year before the date you are using it. Unless it searched the web (the next lesson) it is guessing from older text.
• Small languages and regional literature. A poet who wrote in Bhojpuri or Tigrinya has been written about far less than Shakespeare, and the model will confidently give you a biography assembled from fragments.
• Anything about your specific syllabus. Which chapters are in the Class 9 board paper this year, what the WAEC marking scheme allocates for a question. It has seen some of this and much of it is out of date.

The trap in the middle

The dangerous cases are the ones that sit between the two lists: a well-known topic with a specific detail inside it.

Ask about the French Revolution and the overview will be sound. Ask what Robespierre said on a particular date and you have crossed into a rarely-written-down specific, and the model will supply a quotation that sounds exactly right. The topic was reliable; the detail is not. You have to notice the moment the question changes shape.

A useful habit: when reading an answer, underline every proper noun, number, date and quotation. Those are the specifics. Everything underlined is on the second list until you have checked it. Everything else is probably fine.

Why confidence tells you nothing

The tone does not change between the lists. The mechanism produces the most likely next word whether the underlying knowledge is a million textbooks or three blog posts, and the most likely next word after “The population of Kisumu is” is a number, stated plainly. There is no internal signal of uncertainty reaching the sentence. Some tools now add hedges like “approximately”, but these are trained habits, not measurements.

So you cannot use the answer to judge the answer. You use the question. That is why the habit above works and why “it sounded sure” is never a reason.

Practice for your subject

Write down five questions from your subject: two you expect to be on reliable ground, two on unreliable ground, one in the middle. Ask them. Check the answers against a source you trust. Most teachers find their predictions are right four times in five; and the one miss teaches them where their subject’s specifics hide.