{"course":{"slug":"ai-for-teachers","title":"AI for Teachers","url":"https://addaly.com/learn/ai-for-teachers","level":"beginner","track":"work"},"lesson":{"slug":"a-question-bank-you-can-trust","title":"A question bank you can trust","url":"https://addaly.com/learn/ai-for-teachers/a-question-bank-you-can-trust","position":21,"of":57,"minutes":9,"answer":"Build questions into a spreadsheet tagged by topic, type and difficulty, generate the answer key in a separate pass and reconcile the two.","keyIdea":"Build questions into a spreadsheet tagged by topic, type and difficulty, generate the answer key in a separate pass and reconcile the two, and let a term of student results tell you which questions are actually hard."},"body":{"format":"markdown","text":"## From worksheets to a bank\n\nA worksheet is used once. A bank is used for years. The difference is structure: a question in a spreadsheet, with its answer, its topic, its type and its difficulty, can be pulled into a starter, a homework, a test or a revision pack in seconds. Departments that build one stop generating the same questions every term.\n\nThe model makes the bank fast to fill. The procedure in this lesson makes it safe to use.\n\n## The columns\n\nSix columns are enough:\n\n- **id**: a number, never reused.\n- **topic**: from your syllabus list, exact wording.\n- **type**: recall, application, inference, multi-step, and so on. Your own labels, kept short.\n- **difficulty**: 1 to 3, your initial guess, corrected later by data.\n- **question**: the text.\n- **answer**: the text, or the mark scheme for longer questions.\n\nAsk for output as CSV with exactly those columns, and the formats lesson explained why: it pastes straight into Google Sheets, Excel or LibreOffice Calc.\n\n## Generate the answers separately\n\nThis is the procedure that matters. Do not generate question and answer in one pass.\n\nWhen the model writes a question and its answer in the same stream, the answer is predicted from the question it just wrote, and any error in the question's setup flows straight into the answer with total consistency. A word problem whose numbers do not resolve gets an answer that pretends they do.\n\nInstead: generate the questions. Paste the question column, alone, into a new chat. Ask for answers. Now you have two columns produced independently, and the disagreements are where the errors are. Eleven wrong answers in three hundred is a typical rate for a one-pass bank; a two-pass reconciliation catches most of them, because a bad question tends to produce different wrong answers on different runs, and the disagreement flags it.\n\nFor numerical questions, the calculator rule applies on top: the numbers get checked by something that computes.\n\n## Tag by what the question demands\n\nAsk the model to label each question's type as it writes it, and then disbelieve the labels. Its idea of \"inference\" is loose, and around a third of what it labels application is recall in a costume. A quick read of ten questions per type recalibrates you, and the second message fixes the rest: \"Questions 12 to 30 labelled application only require the student to recall the formula. Rewrite them so the student has to decide which formula applies.\"\n\nThe type column is what makes a bank useful. A test with twelve recall questions and none of anything else is a test of memory. Being able to filter by type is how you avoid setting one by accident.\n\n## Difficulty comes from the students, not the model\n\nThe model's difficulty rating is a guess about an average class somewhere. Yours is a guess about your class. Both are wrong until students have answered the questions.\n\n> Figure: The two-pass procedure. Steps: 1. Generate the questions (As CSV: id, topic, type, difficulty, question. Nothing else.); 2. Open a fresh chat (So it has no memory of writing the questions.); 3. Paste the question column alone (Ask for answers, and nothing but answers.); 4. Reconcile the two columns (Every disagreement is a question to read.); 5. Check the numbers separately (A calculator, or code that actually computes.); 6. Record what the class got right (The seventh column. After a year it beats any generated difficulty rating.). Written in one stream, the answer is predicted from the question the model has just written, so a broken question gets an answer that pretends it works. Generated from the questions alone, in a fresh chat, the two columns disagree wherever the question is bad. About eleven wrong answers in three hundred is a typical one-pass rate; the reconciliation catches most of them.\n\nSo the seventh column, added over the term, is the proportion of the class that got each question right the last time it was used. A question you rated 3 that 90 percent got right is a 1. A question rated 1 that 40 percent missed is telling you something about a misconception, and belongs in next week's starter. After a year the bank knows your students better than any generated rating ever will. This is the single most valuable thing a question bank does and it costs a minute per test to record.\n\n## Parallel versions\n\nFor resits, for two classes that share a corridor, or for a student who missed the test, ask for a parallel version: \"Same structure, same difficulty, same type, different numbers and a different context. Change the surface, keep the demand.\" Then run the two-pass answer check on the new version too. Parallel versions are where the arithmetic errors hide, because the model changes the numbers and carries the old answer.\n\n## Sharing across a department\n\nA bank in a shared spreadsheet, with the two-pass procedure written at the top as a rule, is a department resource that compounds. Add a column for who checked each question and when. Questions nobody has checked are drafts, and the sheet should say so, in a colour, so a colleague in a hurry does not print them.\n\n## The honest limit\n\nA bank drifts from the syllabus. Topics get renamed, chapters move between years, a board changes its command words. Once a year, filter by topic and check the list against the current syllabus document, and retire what no longer fits. And a bank makes it easy to set questions without thinking about what the test is for. The type and difficulty columns are there to make the thinking faster, not to replace it."},"check":{"question":"A department generates 300 questions into a spreadsheet with an answer column, all in one pass. A term later they find eleven questions whose stored answer is wrong. Which change to the process would most reliably have caught those eleven before use?","options":[{"text":"Generating the questions in batches of fifty rather than three hundred, so the model could concentrate.","correct":false,"tempting":"Batch size changes little; the wrong answers come from the model producing question and answer in one predictive stream, not from the size of the batch."},{"text":"Asking the model to double-check its answers at the end of the same conversation.","correct":false,"tempting":"A self-check in the same context tends to confirm its own output; it catches some errors but misses the ones it is most confident about."},{"text":"Generating the answer key in a separate pass and comparing the two columns for disagreement.","correct":true},{"text":"Using a paid tier, since free tiers produce more errors in answer keys.","correct":false,"tempting":"Paid tiers are the same models with fewer limits; the error mechanism is identical, and the fix is procedural."}]},"prev":{"slug":"passages-with-the-words-you-are-teaching","title":"Passages with the words you are teaching","url":"https://addaly.com/learn/ai-for-teachers/passages-with-the-words-you-are-teaching"},"next":{"slug":"diagrams-and-visuals-without-a-designer","title":"Diagrams and visuals without a designer","url":"https://addaly.com/learn/ai-for-teachers/diagrams-and-visuals-without-a-designer"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A question bank you can trust', in AI for Teachers. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-for-teachers/a-question-bank-you-can-trust (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A question bank you can trust","courseTitle":"AI for Teachers","module":{"index":3,"slug":"planning-and-materials","title":"Planning and making materials"},"answer":null,"keyIdea":"Build questions into a spreadsheet tagged by topic, type and difficulty, generate the answer key in a separate pass and reconcile the two, and let a term of student results tell you which questions are actually hard.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Generate the questions","detail":"As CSV: id, topic, type, difficulty, question. Nothing else."},{"label":"Open a fresh chat","detail":"So it has no memory of writing the questions."},{"label":"Paste the question column alone","detail":"Ask for answers, and nothing but answers."},{"label":"Reconcile the two columns","detail":"Every disagreement is a question to read."},{"label":"Check the numbers separately","detail":"A calculator, or code that actually computes."},{"label":"Record what the class got right","detail":"The seventh column. After a year it beats any generated difficulty rating."}],"title":"The two-pass procedure","caption":"Written in one stream, the answer is predicted from the question the model has just written, so a broken question gets an answer that pretends it works. Generated from the questions alone, in a fresh chat, the two columns disagree wherever the question is bad. About eleven wrong answers in three hundred is a typical one-pass rate; the reconciliation catches most of them."},"alt":"Figure: The two-pass procedure. Steps: 1. Generate the questions (As CSV: id, topic, type, difficulty, question. Nothing else.); 2. Open a fresh chat (So it has no memory of writing the questions.); 3. Paste the question column alone (Ask for answers, and nothing but answers.); 4. Reconcile the two columns (Every disagreement is a question to read.); 5. Check the numbers separately (A calculator, or code that actually computes.); 6. Record what the class got right (The seventh column. After a year it beats any generated difficulty rating.). Written in one stream, the answer is predicted from the question the model has just written, so a broken question gets an answer that pretends it works. Generated from the questions alone, in a fresh chat, the two columns disagree wherever the question is bad. About eleven wrong answers in three hundred is a typical one-pass rate; the reconciliation catches most of them."}},"blocks":[{"t":"h2","id":"from-worksheets-to-a-bank","runs":[{"text":"From worksheets to a bank"}]},{"t":"p","runs":[{"text":"A worksheet is used once. A bank is used for years. The difference is structure: a question in a spreadsheet, with its answer, its topic, its type and its difficulty, can be pulled into a starter, a homework, a test or a revision pack in seconds. Departments that build one stop generating the same questions every term."}]},{"t":"p","runs":[{"text":"The model makes the bank fast to fill. The procedure in this lesson makes it safe to use."}]},{"t":"h2","id":"the-columns","runs":[{"text":"The columns"}]},{"t":"p","runs":[{"text":"Six columns are enough:"}]},{"t":"ul","items":[[{"text":"id","b":true},{"text":": a number, never reused."}],[{"text":"topic","b":true},{"text":": from your syllabus list, exact wording."}],[{"text":"type","b":true},{"text":": recall, application, inference, multi-step, and so on. Your own labels, kept short."}],[{"text":"difficulty","b":true},{"text":": 1 to 3, your initial guess, corrected later by data."}],[{"text":"question","b":true},{"text":": the text."}],[{"text":"answer","b":true},{"text":": the text, or the mark scheme for longer questions."}]]},{"t":"p","runs":[{"text":"Ask for output as CSV with exactly those columns, and the formats lesson explained why: it pastes straight into Google Sheets, Excel or LibreOffice Calc."}]},{"t":"h2","id":"generate-the-answers-separately","runs":[{"text":"Generate the answers separately"}]},{"t":"p","runs":[{"text":"This is the procedure that matters. Do not generate question and answer in one pass."}]},{"t":"p","runs":[{"text":"When the model writes a question and its answer in the same stream, the answer is predicted from the question it just wrote, and any error in the question’s setup flows straight into the answer with total consistency. A word problem whose numbers do not resolve gets an answer that pretends they do."}]},{"t":"p","runs":[{"text":"Instead: generate the questions. Paste the question column, alone, into a new chat. Ask for answers. Now you have two columns produced independently, and the disagreements are where the errors are. Eleven wrong answers in three hundred is a typical rate for a one-pass bank; a two-pass reconciliation catches most of them, because a bad question tends to produce different wrong answers on different runs, and the disagreement flags it."}]},{"t":"p","runs":[{"text":"For numerical questions, the calculator rule applies on top: the numbers get checked by something that computes."}]},{"t":"h2","id":"tag-by-what-the-question-demands","runs":[{"text":"Tag by what the question demands"}]},{"t":"p","runs":[{"text":"Ask the model to label each question’s type as it writes it, and then disbelieve the labels. Its idea of “inference” is loose, and around a third of what it labels application is recall in a costume. A quick read of ten questions per type recalibrates you, and the second message fixes the rest: “Questions 12 to 30 labelled application only require the student to recall the formula. Rewrite them so the student has to decide which formula applies.”"}]},{"t":"p","runs":[{"text":"The type column is what makes a bank useful. A test with twelve recall questions and none of anything else is a test of memory. Being able to filter by type is how you avoid setting one by accident."}]},{"t":"h2","id":"difficulty-comes-from-the-students-not-the-model","runs":[{"text":"Difficulty comes from the students, not the model"}]},{"t":"p","runs":[{"text":"The model’s difficulty rating is a guess about an average class somewhere. Yours is a guess about your class. Both are wrong until students have answered the questions."}]},{"t":"p","runs":[{"text":"So the seventh column, added over the term, is the proportion of the class that got each question right the last time it was used. A question you rated 3 that 90 percent got right is a 1. A question rated 1 that 40 percent missed is telling you something about a misconception, and belongs in next week’s starter. After a year the bank knows your students better than any generated rating ever will. This is the single most valuable thing a question bank does and it costs a minute per test to record."}]},{"t":"h2","id":"parallel-versions","runs":[{"text":"Parallel versions"}]},{"t":"p","runs":[{"text":"For resits, for two classes that share a corridor, or for a student who missed the test, ask for a parallel version: “Same structure, same difficulty, same type, different numbers and a different context. Change the surface, keep the demand.” Then run the two-pass answer check on the new version too. Parallel versions are where the arithmetic errors hide, because the model changes the numbers and carries the old answer."}]},{"t":"h2","id":"sharing-across-a-department","runs":[{"text":"Sharing across a department"}]},{"t":"p","runs":[{"text":"A bank in a shared spreadsheet, with the two-pass procedure written at the top as a rule, is a department resource that compounds. Add a column for who checked each question and when. Questions nobody has checked are drafts, and the sheet should say so, in a colour, so a colleague in a hurry does not print them."}]},{"t":"h2","id":"the-honest-limit","runs":[{"text":"The honest limit"}],"role":"limit"},{"t":"p","runs":[{"text":"A bank drifts from the syllabus. Topics get renamed, chapters move between years, a board changes its command words. Once a year, filter by topic and check the list against the current syllabus document, and retire what no longer fits. And a bank makes it easy to set questions without thinking about what the test is for. The type and difficulty columns are there to make the thinking faster, not to replace it."}]}],"check":{"question":[{"text":"A department generates 300 questions into a spreadsheet with an answer column, all in one pass. A term later they find eleven questions whose stored answer is wrong. Which change to the process would most reliably have caught those eleven before use?"}],"options":[{"text":[{"text":"Generating the questions in batches of fifty rather than three hundred, so the model could concentrate."}],"correct":false,"tempting":[{"text":"Batch size changes little; the wrong answers come from the model producing question and answer in one predictive stream, not from the size of the batch."}]},{"text":[{"text":"Asking the model to double-check its answers at the end of the same conversation."}],"correct":false,"tempting":[{"text":"A self-check in the same context tends to confirm its own output; it catches some errors but misses the ones it is most confident about."}]},{"text":[{"text":"Generating the answer key in a separate pass and comparing the two columns for disagreement."}],"correct":true,"tempting":null},{"text":[{"text":"Using a paid tier, since free tiers produce more errors in answer keys."}],"correct":false,"tempting":[{"text":"Paid tiers are the same models with fewer limits; the error mechanism is identical, and the fix is procedural."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"passages-with-the-words-you-are-teaching","title":"Passages with the words you are teaching","minutes":8},"next":{"slug":"diagrams-and-visuals-without-a-designer","title":"Diagrams and visuals without a designer","minutes":9}}}