{"course":{"slug":"ai-and-you","title":"AI, Safety and What Goes Wrong","url":"https://addaly.com/learn/ai-and-you","level":"beginner","track":"start"},"lesson":{"slug":"when-you-are-the-expert","title":"The competence inversion","url":"https://addaly.com/learn/ai-and-you/when-you-are-the-expert","position":25,"of":73,"minutes":7,"answer":"AI output looks best in fields you cannot evaluate, so convert questions outside your competence into vocabulary and handles you can take to a real source.","keyIdea":"AI output looks best in fields you cannot evaluate, so convert questions outside your competence into vocabulary and handles you can take to a real source, and reserve conclusions for domains where you could catch the error yourself."},"body":{"format":"markdown","text":"## The uncomfortable pattern\n\nIn the subject you know best, the model's output is decent but shallow, and you can see the errors. In every other subject, it seems excellent.\n\nBoth impressions cannot be trusted, and the second one is the dangerous half.\n\nThe journalist and novelist Michael Crichton described the same effect in newspapers: you read the article about your own field, see it is riddled with mistakes, turn the page, and read the article about the Middle East with full confidence. He called it Gell-Mann amnesia. It applies to AI output with more force, because the output is fluent by construction and produced on demand in exactly the volume you asked for.\n\nSo the inversion: **the model is most useful where you are competent, and most trusted where you are not.** Its usefulness and your ability to check it point in the same direction, and your inclination to check points the other way.\n\n## Two different jobs\n\n**Inside your field.** You are using it for speed. Draft the standard section, restructure the argument, generate the boilerplate, list the cases you should consider, catch the thing you forgot. You will spot errors automatically because you have a model of the domain to notice a disagreement against. The main risk here is not error, it is homogenisation: your work drifting towards the average of what has been written before, losing the judgement that made you worth consulting.\n\n**Outside your field.** You are using it for orientation, and you should be explicit that this is all it is. It is excellent at vocabulary: what is this called, what are the standard approaches, what would a professional ask me, what am I not seeing. Those outputs are checkable in a way that answers are not: you can take a term to a search engine or a professional and get somewhere.\n\nThe move that goes wrong is asking for a conclusion in a field you cannot evaluate and then acting on it. Medical, legal, tax, structural, safety. Not because the model is always wrong there, but because you have no way to tell which time it is.\n\n## Practical translations\n\nInstead of \"is this contract clause normal?\", ask \"what is this type of clause called, what does it typically do, and what should I ask a lawyer about it?\" You have converted an unverifiable answer into a set of verifiable handles.\n\nInstead of \"what is wrong with this X-ray?\", ask nothing at all, and take it to a radiologist.\n\nInstead of \"write the SQL and I will run it in production\", ask for the query and read it, and if you cannot read SQL then run it against a copy and check the row counts by hand.\n\nInstead of \"summarise this 90-page report\", read the report's own summary first, then use the model to interrogate the parts you did not understand, with the text in front of it. You keep the judgement about what matters.\n\n## The half-expert\n\nOne group is at unusual risk: people who know a field slightly. Enough to follow the output and find it plausible, not enough to catch the error. This is the junior professional, the enthusiastic beginner, and the manager reviewing a specialist's work.\n\nIf you are in that position, the useful question is not \"does this look right?\" but \"who would know, and what would I show them?\" A specific claim, extracted, taken to someone competent, takes them ninety seconds to assess.\n\n## Keeping your own field sharp\n\nThe last risk is the slowest. If you always use the model for the parts of your work you find tedious, you may find in three years that the tedious part was where your fluency was maintained. This is the subject of a later lesson on deskilling, and the countermeasure begins here: keep a portion of the work unaided, deliberately, on purpose, and notice if it has become harder than it used to be.\n\nYour confidence in an AI answer should be set by your ability to check it, not by how good the answer sounds, and those two are usually in opposite proportion."},"check":{"question":"Someone with no legal training asks a model whether a clause in their lease is enforceable and gets a clear, confident answer. What is the underlying problem, independent of whether the answer is right?","options":[{"text":"The asker cannot check the answer, so its fluency is doing all of the persuading","correct":true},{"text":"Leases depend on jurisdiction, so the model would first need to know the local law","correct":false,"tempting":"A real limit, but supplying the jurisdiction would not fix the deeper issue: the asker still cannot evaluate the answer."},{"text":"The model learned mostly from US legal material, so it will apply the wrong rules","correct":false,"tempting":"Names a real data skew, yet a perfectly localised model would leave the asker in exactly the same position."},{"text":"Models are weaker at legal reasoning than at general knowledge, so it will be wrong","correct":false,"tempting":"Asserts a capability ranking that varies by task and model, and implies the problem vanishes as legal performance improves."}],"why":"An answer you cannot verify carries its confidence as its only evidence, and confidence is what a model produces most easily."},"prev":{"slug":"numbers-that-do-not-add-up","title":"Checking a number without looking it up","url":"https://addaly.com/learn/ai-and-you/numbers-that-do-not-add-up"},"next":{"slug":"what-you-hand-over","title":"What you hand over when you paste","url":"https://addaly.com/learn/ai-and-you/what-you-hand-over"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The competence inversion', in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/when-you-are-the-expert (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The competence inversion","courseTitle":"AI, Safety and What Goes Wrong","module":{"index":3,"slug":"truth-and-verification","title":"Truth, sources and verification"},"answer":null,"keyIdea":"AI output looks best in fields you cannot evaluate, so convert questions outside your competence into vocabulary and handles you can take to a real source, and reserve conclusions for domains where you could catch the error yourself.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-uncomfortable-pattern","runs":[{"text":"The uncomfortable pattern"}]},{"t":"p","runs":[{"text":"In the subject you know best, the model’s output is decent but shallow, and you can see the errors. In every other subject, it seems excellent."}]},{"t":"p","runs":[{"text":"Both impressions cannot be trusted, and the second one is the dangerous half."}]},{"t":"p","runs":[{"text":"The journalist and novelist Michael Crichton described the same effect in newspapers: you read the article about your own field, see it is riddled with mistakes, turn the page, and read the article about the Middle East with full confidence. He called it Gell-Mann amnesia. It applies to AI output with more force, because the output is fluent by construction and produced on demand in exactly the volume you asked for."}]},{"t":"p","runs":[{"text":"So the inversion: "},{"text":"the model is most useful where you are competent, and most trusted where you are not.","b":true},{"text":" Its usefulness and your ability to check it point in the same direction, and your inclination to check points the other way."}]},{"t":"h2","id":"two-different-jobs","runs":[{"text":"Two different jobs"}]},{"t":"p","runs":[{"text":"Inside your field.","b":true},{"text":" You are using it for speed. Draft the standard section, restructure the argument, generate the boilerplate, list the cases you should consider, catch the thing you forgot. You will spot errors automatically because you have a model of the domain to notice a disagreement against. The main risk here is not error, it is homogenisation: your work drifting towards the average of what has been written before, losing the judgement that made you worth consulting."}]},{"t":"p","runs":[{"text":"Outside your field.","b":true},{"text":" You are using it for orientation, and you should be explicit that this is all it is. It is excellent at vocabulary: what is this called, what are the standard approaches, what would a professional ask me, what am I not seeing. Those outputs are checkable in a way that answers are not: you can take a term to a search engine or a professional and get somewhere."}]},{"t":"p","runs":[{"text":"The move that goes wrong is asking for a conclusion in a field you cannot evaluate and then acting on it. Medical, legal, tax, structural, safety. Not because the model is always wrong there, but because you have no way to tell which time it is."}]},{"t":"h2","id":"practical-translations","runs":[{"text":"Practical translations"}]},{"t":"p","runs":[{"text":"Instead of “is this contract clause normal?”, ask “what is this type of clause called, what does it typically do, and what should I ask a lawyer about it?” You have converted an unverifiable answer into a set of verifiable handles."}]},{"t":"p","runs":[{"text":"Instead of “what is wrong with this X-ray?”, ask nothing at all, and take it to a radiologist."}]},{"t":"p","runs":[{"text":"Instead of “write the SQL and I will run it in production”, ask for the query and read it, and if you cannot read SQL then run it against a copy and check the row counts by hand."}]},{"t":"p","runs":[{"text":"Instead of “summarise this 90-page report”, read the report’s own summary first, then use the model to interrogate the parts you did not understand, with the text in front of it. You keep the judgement about what matters."}]},{"t":"h2","id":"the-half-expert","runs":[{"text":"The half-expert"}]},{"t":"p","runs":[{"text":"One group is at unusual risk: people who know a field slightly. Enough to follow the output and find it plausible, not enough to catch the error. This is the junior professional, the enthusiastic beginner, and the manager reviewing a specialist’s work."}]},{"t":"p","runs":[{"text":"If you are in that position, the useful question is not “does this look right?” but “who would know, and what would I show them?” A specific claim, extracted, taken to someone competent, takes them ninety seconds to assess."}]},{"t":"h2","id":"keeping-your-own-field-sharp","runs":[{"text":"Keeping your own field sharp"}]},{"t":"p","runs":[{"text":"The last risk is the slowest. If you always use the model for the parts of your work you find tedious, you may find in three years that the tedious part was where your fluency was maintained. This is the subject of a later lesson on deskilling, and the countermeasure begins here: keep a portion of the work unaided, deliberately, on purpose, and notice if it has become harder than it used to be."}]},{"t":"p","runs":[{"text":"Your confidence in an AI answer should be set by your ability to check it, not by how good the answer sounds, and those two are usually in opposite proportion."}]}],"check":{"question":[{"text":"Someone with no legal training asks a model whether a clause in their lease is enforceable and gets a clear, confident answer. What is the underlying problem, independent of whether the answer is right?"}],"options":[{"text":[{"text":"The asker cannot check the answer, so its fluency is doing all of the persuading"}],"correct":true,"tempting":null},{"text":[{"text":"Leases depend on jurisdiction, so the model would first need to know the local law"}],"correct":false,"tempting":[{"text":"A real limit, but supplying the jurisdiction would not fix the deeper issue: the asker still cannot evaluate the answer."}]},{"text":[{"text":"The model learned mostly from US legal material, so it will apply the wrong rules"}],"correct":false,"tempting":[{"text":"Names a real data skew, yet a perfectly localised model would leave the asker in exactly the same position."}]},{"text":[{"text":"Models are weaker at legal reasoning than at general knowledge, so it will be wrong"}],"correct":false,"tempting":[{"text":"Asserts a capability ranking that varies by task and model, and implies the problem vanishes as legal performance improves."}]}],"order":[2,1,3,0],"why":[{"text":"An answer you cannot verify carries its confidence as its only evidence, and confidence is what a model produces most easily."}]},"prev":{"slug":"numbers-that-do-not-add-up","title":"Checking a number without looking it up","minutes":8},"next":{"slug":"what-you-hand-over","title":"What you hand over when you paste","minutes":7}}}