{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"keeping-your-own-judgement","title":"Keeping your own judgement","url":"https://addaly.com/learn/ai-actually-explained/keeping-your-own-judgement","position":71,"of":92,"minutes":9,"answer":"Fluent answers are trusted more than they deserve and a skill you stop practising decays, so the habit that keeps you competent is to draft first, review its draft second.","keyIdea":"Fluent answers are trusted more than they deserve and a skill you stop practising decays, so the habit that keeps you competent is to draft first, review its draft second, and sometimes work without it at all."},"body":{"format":"markdown","text":"## The oldest failure in automation\n\nLong before chatbots, aviation had a name for it: automation bias. A pilot with a reliable autopilot stops monitoring; when the autopilot fails, the pilot's skills have thinned and the handover comes at the worst moment. Drivers following a navigation app into a river are the same failure in cheaper form. The pattern is not stupidity. A tool that is usually right trains you to stop checking, and the training is the tool's reliability itself.\n\nLanguage models add a second ingredient the autopilot did not have: they explain themselves, fluently, in the register of someone who knows. And people trust fluency. Studies since the 1990s have shown that a claim stated smoothly is rated more likely to be true than the same claim stated awkwardly, and that this holds even when people are warned. A model's answers are optimised, by the second training, to be exactly that smooth. The confidence is a property of the output, not of the fact, and you have known that since the fourth module; the difficulty is that knowing it does not switch the effect off.\n\n## What the studies found\n\nThree results are worth carrying.\n\nIn 2025 researchers at a large software company surveyed several hundred knowledge workers about how they used these tools. The more confident a person was in the tool for a task, the less critical thinking they reported doing on it; the more confident in themselves, the more. The authors' concern was not the individual answer but the long run: a habit of not examining the work.\n\nIn 2024 an experiment in Turkish secondary schools gave around a thousand students a chatbot for maths practice. Students with unrestricted access solved more practice problems than the control group. On the exam that followed, without the tool, they scored seventeen per cent *worse*. A second group had a version of the tool that gave hints but not answers, and their exam scores matched the control. The tool that did the work removed the practice that would have built the skill; the tool that helped without doing did not.\n\nAnd the module on work in this course will show you a 2025 study in which experienced programmers believed a tool had sped them up by a fifth when it had in fact slowed them down by about the same. The feeling of help and the fact of help came apart, and self-report followed the feeling.\n\n## Why this is mechanical\n\nSkill is built by doing the task, including doing it badly. A first draft you struggle over teaches you what the piece is about; a first draft you are handed teaches you to edit. Both are skills, but they are not the same skill, and the person who has only ever edited cannot draft when the tool is down, or wrong, or unavailable in the exam room. The module on work returns to this as a problem for whole professions. Here it is a problem for you.\n\n## Habits that hold the line\n\n**Draft first, then ask.** Write your version, however rough. Then ask the model for its version and compare. You keep the skill, you catch its errors because you have already thought about the problem, and you learn something from the difference. The order matters: model first, and your draft becomes an edit of its framing.\n\n**Review it as a colleague's work, not a boss's.** A colleague's draft you read critically and mark up. A boss's you accept. The model's register invites the second; the fourth module tells you it deserves the first.\n\n**Check the specific, every time.** Not the argument: the number, the name, the date, the citation. It takes a minute and it is where the errors live.\n\n**Ask it to argue against itself.** \"What is the strongest case that this is wrong?\" is a useful prompt, because the model is as fluent against a position as for it, and the two answers side by side leave you somewhere in the middle, which is usually where the truth was.\n\n**Work without it sometimes.** Deliberately. The maths students who had the hint-giving version did not lose the skill; the ones with the answer-giving version did. Choose which one you are using, on purpose, for the tasks you want to stay good at.\n\n## The frame to keep\n\nThe tool is a very fast, very fluent, unaccountable colleague with a wide but jagged knowledge and no idea when it is wrong. That colleague is enormously useful and you would not let them sign anything. The whole of this module reduces to that sentence, and the rest of this course is about what it means when everyone has one."},"check":{"question":"Students given unrestricted chatbot access solved more practice problems but scored worse on the exam without it, while students given a hint-only version matched the control group. What explains the difference?","options":[{"text":"The unrestricted chatbot gave wrong answers to the practice problems, which the students learned and then reproduced in the exam.","correct":false,"tempting":"Blames errors; the practice answers were largely right, and it was the students' own practice that was missing."},{"text":"The hint-only version was a better model with more training data.","correct":false,"tempting":"Puts the difference in the model; both groups used the same underlying model with different instructions."},{"text":"The unrestricted version did the work that builds the skill; the hint-only version left the students doing it.","correct":true},{"text":"The students with unrestricted access spent less time on the practice overall.","correct":false,"tempting":"Contradicts the finding; they solved more problems, not fewer, and still learned less."}]},"prev":{"slug":"the-things-to-never-hand-it","title":"What not to hand it","url":"https://addaly.com/learn/ai-actually-explained/the-things-to-never-hand-it"},"next":{"slug":"ai-and-work","title":"AI and jobs, without panic or dismissal","url":"https://addaly.com/learn/ai-actually-explained/ai-and-work"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Keeping your own judgement', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/keeping-your-own-judgement (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Keeping your own judgement","courseTitle":"AI, Actually Explained","module":{"index":7,"slug":"putting-it-to-work","title":"Putting it to work without getting burned"},"answer":null,"keyIdea":"Fluent answers are trusted more than they deserve and a skill you stop practising decays, so the habit that keeps you competent is to draft first, review its draft second, and sometimes work without it at all.","hero":{"kind":"plate","image":"https://addaly.com/art/modules/ai-actually-explained/putting-it-to-work.webp","alt":"A coaching-centre desk with a stack of books, a phone, an open folder of notes, a steel tumbler and a plain wall calendar above.","caption":"Using these tools well is mostly a matter of what you put on the desk beside them and what you decide never to hand over."},"blocks":[{"t":"h2","id":"the-oldest-failure-in-automation","runs":[{"text":"The oldest failure in automation"}]},{"t":"p","runs":[{"text":"Long before chatbots, aviation had a name for it: automation bias. A pilot with a reliable autopilot stops monitoring; when the autopilot fails, the pilot’s skills have thinned and the handover comes at the worst moment. Drivers following a navigation app into a river are the same failure in cheaper form. The pattern is not stupidity. A tool that is usually right trains you to stop checking, and the training is the tool’s reliability itself."}]},{"t":"p","runs":[{"text":"Language models add a second ingredient the autopilot did not have: they explain themselves, fluently, in the register of someone who knows. And people trust fluency. Studies since the 1990s have shown that a claim stated smoothly is rated more likely to be true than the same claim stated awkwardly, and that this holds even when people are warned. A model’s answers are optimised, by the second training, to be exactly that smooth. The confidence is a property of the output, not of the fact, and you have known that since the fourth module; the difficulty is that knowing it does not switch the effect off."}]},{"t":"h2","id":"what-the-studies-found","runs":[{"text":"What the studies found"}]},{"t":"p","runs":[{"text":"Three results are worth carrying."}]},{"t":"p","runs":[{"text":"In 2025 researchers at a large software company surveyed several hundred knowledge workers about how they used these tools. The more confident a person was in the tool for a task, the less critical thinking they reported doing on it; the more confident in themselves, the more. The authors’ concern was not the individual answer but the long run: a habit of not examining the work."}]},{"t":"p","runs":[{"text":"In 2024 an experiment in Turkish secondary schools gave around a thousand students a chatbot for maths practice. Students with unrestricted access solved more practice problems than the control group. On the exam that followed, without the tool, they scored seventeen per cent "},{"text":"worse","i":true},{"text":". A second group had a version of the tool that gave hints but not answers, and their exam scores matched the control. The tool that did the work removed the practice that would have built the skill; the tool that helped without doing did not."}]},{"t":"p","runs":[{"text":"And the module on work in this course will show you a 2025 study in which experienced programmers believed a tool had sped them up by a fifth when it had in fact slowed them down by about the same. The feeling of help and the fact of help came apart, and self-report followed the feeling."}]},{"t":"h2","id":"why-this-is-mechanical","runs":[{"text":"Why this is mechanical"}]},{"t":"p","runs":[{"text":"Skill is built by doing the task, including doing it badly. A first draft you struggle over teaches you what the piece is about; a first draft you are handed teaches you to edit. Both are skills, but they are not the same skill, and the person who has only ever edited cannot draft when the tool is down, or wrong, or unavailable in the exam room. The module on work returns to this as a problem for whole professions. Here it is a problem for you."}]},{"t":"h2","id":"habits-that-hold-the-line","runs":[{"text":"Habits that hold the line"}]},{"t":"p","runs":[{"text":"Draft first, then ask.","b":true},{"text":" Write your version, however rough. Then ask the model for its version and compare. You keep the skill, you catch its errors because you have already thought about the problem, and you learn something from the difference. The order matters: model first, and your draft becomes an edit of its framing."}]},{"t":"p","runs":[{"text":"Review it as a colleague’s work, not a boss’s.","b":true},{"text":" A colleague’s draft you read critically and mark up. A boss’s you accept. The model’s register invites the second; the fourth module tells you it deserves the first."}]},{"t":"p","runs":[{"text":"Check the specific, every time.","b":true},{"text":" Not the argument: the number, the name, the date, the citation. It takes a minute and it is where the errors live."}]},{"t":"p","runs":[{"text":"Ask it to argue against itself.","b":true},{"text":" “What is the strongest case that this is wrong?” is a useful prompt, because the model is as fluent against a position as for it, and the two answers side by side leave you somewhere in the middle, which is usually where the truth was."}]},{"t":"p","runs":[{"text":"Work without it sometimes.","b":true},{"text":" Deliberately. The maths students who had the hint-giving version did not lose the skill; the ones with the answer-giving version did. Choose which one you are using, on purpose, for the tasks you want to stay good at."}]},{"t":"h2","id":"the-frame-to-keep","runs":[{"text":"The frame to keep"}]},{"t":"p","runs":[{"text":"The tool is a very fast, very fluent, unaccountable colleague with a wide but jagged knowledge and no idea when it is wrong. That colleague is enormously useful and you would not let them sign anything. The whole of this module reduces to that sentence, and the rest of this course is about what it means when everyone has one."}]}],"check":{"question":[{"text":"Students given unrestricted chatbot access solved more practice problems but scored worse on the exam without it, while students given a hint-only version matched the control group. What explains the difference?"}],"options":[{"text":[{"text":"The unrestricted chatbot gave wrong answers to the practice problems, which the students learned and then reproduced in the exam."}],"correct":false,"tempting":[{"text":"Blames errors; the practice answers were largely right, and it was the students’ own practice that was missing."}]},{"text":[{"text":"The hint-only version was a better model with more training data."}],"correct":false,"tempting":[{"text":"Puts the difference in the model; both groups used the same underlying model with different instructions."}]},{"text":[{"text":"The unrestricted version did the work that builds the skill; the hint-only version left the students doing it."}],"correct":true,"tempting":null},{"text":[{"text":"The students with unrestricted access spent less time on the practice overall."}],"correct":false,"tempting":[{"text":"Contradicts the finding; they solved more problems, not fewer, and still learned less."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"the-things-to-never-hand-it","title":"What not to hand it","minutes":9},"next":{"slug":"ai-and-work","title":"AI and jobs, without panic or dismissal","minutes":8}}}