A student project that teaches the mechanism

Why build one

Everything in the offline lesson can be taught in a period. This project takes two weeks and it does something the activities cannot: students build the thing, run it, watch it produce fluent nonsense, and understand from the inside why a machine that predicts the next word can write a paragraph and still not know anything. It is the single most effective AI lesson for students aged 14 and over, and the paper version needs only a page of text and a tally sheet.

Week one: the tally

Take one page of a story, about 300 words. Each student, or pair, takes a paragraph. For every word, they record which word came next. “The” was followed by “old” twice, “market” once, “sun” once. By the end of the paragraph they have a table: each word, and a tally of every word that followed it.

Combine the paragraphs into one class table on the wall, or in a shared spreadsheet. Three hundred words produce a table of perhaps two hundred entries. It took a class an hour, and they have built a bigram model (a model of which word follows which) from data.

Week one, continued: generating

Now run it. Pick a starting word. Look up its row. Choose the next word by the tallies: roll a die, or pick the most common, or draw from a bag with one slip per tally mark. Write it down. Look up that word’s row. Repeat for twenty words.

Read the sentences aloud. They start well (“The old woman walked slowly”) because those pairs were in the text. Then they drift: “slowly into the market was full of the sun was old woman walked”. Grammatical in pieces, meaningless as a whole. The class will laugh. Then ask why.

The answers come fast, and they are the whole of the course’s first block, discovered rather than told. It only looks one word back. It has only seen one page. It has no idea what it is saying; it is following tallies. And a student will ask: “So the real one is just this, but bigger?” Yes. Vastly bigger (trillions of words, thousands of words of context instead of one, and a much cleverer way of storing the tallies) and the same idea. The check question’s drift is the mechanism made visible.

Week two: making it better, and seeing why that helps

Give them the levers and let them pull.

More text. Add three more pages. The table grows; the sentences hold together for longer. They see that data is what it eats.

Longer context. Tally pairs of words and what follows the pair: “the old” is followed by “woman”. Generation from pairs is markedly more fluent, and the table is far larger. They see the trade: context costs memory.

Temperature. Instead of drawing from the bag, always pick the most common next word. The output becomes repetitive and loops. Draw more randomly and it becomes wilder. They have discovered the dial from the “same question, different answer” lesson by turning it.

Bias. Build the table from a page where every doctor is “he” and every nurse is “she”. Generate from “the doctor”. Ask what happened and why. The retelling activity from the offline lesson taught this in ten minutes; here they have built the mechanism that does it.

The free path with a laptop

If any Python is available (Google Colab in a browser, or an online Python environment on a phone) the whole tally can be built in a dozen lines and the class can feed it a whole book from Project Gutenberg. The code is short enough to read:

import random
words = open("story.txt").read().split()
table = {}
for a, b in zip(words, words[1:]):
    table.setdefault(a, []).append(b)
w = "The"
out = [w]
for _ in range(30):
    w = random.choice(table.get(w, words))
    out.append(w)
print(" ".join(out))

Students who have done the paper version read this and recognise every line. That recognition (the code is the tally) is worth more than a term of programming without the paper version first.

Assessment

Not the model. The explanation. Each student writes a page: how the model works, why the sentences drift, what changed when the context grew, and one thing the real model has that theirs does not. A student who can write that page has an understanding of language models that would pass an interview, and it came from a tally sheet.

The honest limit

A bigram model is a real model and a very old one; the modern systems differ in ways that matter (attention over thousands of words, learned representations rather than literal tables, training that adjusts billions of numbers) and a student should be told that the project shows the principle, not the architecture. For students who want the next step, the course on how language models work on this site picks up exactly there. For the rest, the principle is the thing they needed, and they have it in their hands.