{"course":{"slug":"prompting","title":"Getting Real Work out of a Model","url":"https://addaly.com/learn/prompting","level":"beginner","track":"start"},"lesson":{"slug":"the-folklore-audit","title":"Magic words, and what the studies actually found","url":"https://addaly.com/learn/prompting/the-folklore-audit","position":7,"of":72,"minutes":9,"answer":"The famous prompt tricks came from narrow studies on older models, and their effects are usually smaller than the ordinary variation between two runs of the same prompt.","keyIdea":"The famous prompt tricks came from narrow studies on older models, and their effects are usually smaller than the ordinary variation between two runs of the same prompt."},"body":{"format":"markdown","text":"## Where the incantations came from\n\nYou have seen the advice. Tell it to take a deep breath. Offer it a $200 tip. Say this is very important to your career. Threaten it. Tell it that it is a world-class expert with twenty years of experience.\n\nThese are not internet inventions. Most trace back to real papers, and knowing what those papers actually did is the fastest cure for treating them as spells.\n\n**\"Take a deep breath and work on this problem step by step.\"** This came out of a 2023 Google DeepMind paper on using a language model to *search* for good prompt phrasings automatically. The method generated candidate instructions and scored them on a grade-school maths benchmark. That sentence scored best on one model, on one benchmark. It was found by search, not designed by insight, and it was never claimed to be a general law. It got copied into a thousand posts as one.\n\n**Emotional stakes.** A 2023 paper added phrases like \"this is very important to my career\" and reported improvements on several tasks. Effects were real in that setup, modest in size, and inconsistent across models.\n\n**The tip.** This one began as an experiment somebody ran and posted about, with small samples and no control for the fact that mentioning money changes the topic of the text. It spread because it is a good story.\n\n**Threats.** In 2025 a senior figure at Google said in an interview that threatening models tends to improve results, which travelled widely. Independent testing since has found no consistent benefit, and it is worth noticing how easy that sentence was to repeat without anyone checking.\n\n## What they have in common\n\nThree things.\n\nThey were measured on **the models of their moment**, mostly 2022 to 2023, when instruction-following was weaker and prompt phrasing had more room to move the result. Every one of these tricks has smaller effects on current models, because the models improved in exactly the dimension the trick was compensating for.\n\nThey were measured on **narrow benchmarks**, usually maths word problems, where a small nudge towards careful step-by-step output shows up as a percentage. Your email is not a maths word problem.\n\nAnd they are **fragile in a specific way**: the effect is often smaller than the run-to-run variation you saw in the sampling lesson. If a trick gives you three points on a benchmark and your own five runs of the same prompt vary by five points, you cannot detect the trick without careful measurement, and neither can the person who told you about it.\n\n> Figure: What moves a ten-item test, and by how much. Bars (points out of ten, three runs each): Run-to-run wobble, prompt unchanged 2; Add 'take a deep breath' 0.5; Offer a tip 0.3; Add 'this matters to my career' 0.6; Paste the material itself 3; State the finished state 2. Typical of what the test in this lesson produces: the magic phrases sit inside the run-to-run wobble of the unchanged prompt, and the two content changes stand outside it. A trick you cannot detect without careful measurement is a trick the person recommending it could not detect either.\n\n## What has held up\n\nNot everything is folklore. The following survive across models and tasks, because they change what information is available rather than trying to change the model's mood:\n\n- **Giving the material** rather than describing it.\n- **Stating the finished state** rather than the topic.\n- **Showing examples** where a rule is hard to state.\n- **Asking for intermediate steps** on multi-stage tasks, which is what \"step by step\" was always doing, minus the breathing.\n- **Naming the output shape** and what to omit.\n- **Providing an escape hatch** so \"I cannot\" is available.\n\nNotice these are all about content. None depends on a phrase.\n\n## How to test one for yourself\n\nYou do not need a lab. Take a task you do repeatedly and where you can tell right from wrong: extracting three fields from a document, classifying a message, converting a format. Build ten test items with known answers.\n\nRun your prompt over the ten. Run the prompt with the magic phrase added over the same ten. Do each three times, because of sampling. Count.\n\nAlmost always you will find a difference smaller than the noise, and you will stop wondering. Occasionally you will find something real for your task and your model, and then you have earned it, with the honest note that it may stop being true at the next model update.\n\n## The reason this matters beyond tidiness\n\nEvery token of folklore is a token you did not spend on context. A prompt that opens with three sentences of role-play and motivational framing, and then asks a vague question, is worse than the same prompt with those sentences replaced by the actual document. The cost of the spell is not that it fails. It is that it occupies the place where the useful information should have been.\n\nThere is no secret vocabulary. There is a habit of under-describing your own problem, and that is the thing to fix."},"check":{"question":"Someone reports that adding \"I will tip you $200\" made a model's answers noticeably better on their task. What is the most likely explanation, and the right response?","options":[{"text":"The claim is impossible, because models have no concept of money and cannot be motivated.","correct":false,"tempting":"Overcorrects into a different error: the phrase does change the text in context and can measurably shift outputs; the problem is reliability, not impossibility."},{"text":"They compared a small number of runs, and the difference is probably within the variation the sampler produces anyway; the fix is a fixed test set run several times each way.","correct":true},{"text":"It works because the phrase resembles high-effort professional writing in the training data, so the effect will transfer to other tasks.","correct":false,"tempting":"Offers a plausible story as if it settled the question; a mechanism that sounds right is still not evidence for this task or this model."},{"text":"It worked once and will keep working, since prompt effects are stable properties of a model.","correct":false,"tempting":"Prompt effects shift between model versions, and several published tricks stopped replicating within a year."}]},"prev":{"slug":"anatomy-of-a-prompt","title":"The six things a working prompt contains","url":"https://addaly.com/learn/prompting/anatomy-of-a-prompt"},"next":{"slug":"specificity-and-its-limit","title":"How much detail is too much","url":"https://addaly.com/learn/prompting/specificity-and-its-limit"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Magic words, and what the studies actually found', in Getting Real Work out of a Model. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/prompting/the-folklore-audit (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Magic words, and what the studies actually found","courseTitle":"Getting Real Work out of a Model","module":{"index":1,"slug":"what-a-prompt-is","title":"What a prompt actually is"},"answer":null,"keyIdea":"The famous prompt tricks came from narrow studies on older models, and their effects are usually smaller than the ordinary variation between two runs of the same prompt.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"points out of ten, three runs each","title":"What moves a ten-item test, and by how much","series":[{"label":"Run-to-run wobble, prompt unchanged","value":2},{"label":"Add ‘take a deep breath’","value":0.5},{"label":"Offer a tip","value":0.3},{"label":"Add ‘this matters to my career’","value":0.6},{"label":"Paste the material itself","value":3,"highlight":true},{"label":"State the finished state","value":2}],"caption":"Typical of what the test in this lesson produces: the magic phrases sit inside the run-to-run wobble of the unchanged prompt, and the two content changes stand outside it. A trick you cannot detect without careful measurement is a trick the person recommending it could not detect either."},"alt":"Figure: What moves a ten-item test, and by how much. Bars (points out of ten, three runs each): Run-to-run wobble, prompt unchanged 2; Add ‘take a deep breath’ 0.5; Offer a tip 0.3; Add ‘this matters to my career’ 0.6; Paste the material itself 3; State the finished state 2. Typical of what the test in this lesson produces: the magic phrases sit inside the run-to-run wobble of the unchanged prompt, and the two content changes stand outside it. A trick you cannot detect without careful measurement is a trick the person recommending it could not detect either."}},"blocks":[{"t":"h2","id":"where-the-incantations-came-from","runs":[{"text":"Where the incantations came from"}]},{"t":"p","runs":[{"text":"You have seen the advice. Tell it to take a deep breath. Offer it a $200 tip. Say this is very important to your career. Threaten it. Tell it that it is a world-class expert with twenty years of experience."}]},{"t":"p","runs":[{"text":"These are not internet inventions. Most trace back to real papers, and knowing what those papers actually did is the fastest cure for treating them as spells."}]},{"t":"p","runs":[{"text":"”Take a deep breath and work on this problem step by step.”","b":true},{"text":" This came out of a 2023 Google DeepMind paper on using a language model to "},{"text":"search","i":true},{"text":" for good prompt phrasings automatically. The method generated candidate instructions and scored them on a grade-school maths benchmark. That sentence scored best on one model, on one benchmark. It was found by search, not designed by insight, and it was never claimed to be a general law. It got copied into a thousand posts as one."}]},{"t":"p","runs":[{"text":"Emotional stakes.","b":true},{"text":" A 2023 paper added phrases like “this is very important to my career” and reported improvements on several tasks. Effects were real in that setup, modest in size, and inconsistent across models."}]},{"t":"p","runs":[{"text":"The tip.","b":true},{"text":" This one began as an experiment somebody ran and posted about, with small samples and no control for the fact that mentioning money changes the topic of the text. It spread because it is a good story."}]},{"t":"p","runs":[{"text":"Threats.","b":true},{"text":" In 2025 a senior figure at Google said in an interview that threatening models tends to improve results, which travelled widely. Independent testing since has found no consistent benefit, and it is worth noticing how easy that sentence was to repeat without anyone checking."}]},{"t":"h2","id":"what-they-have-in-common","runs":[{"text":"What they have in common"}]},{"t":"p","runs":[{"text":"Three things."}]},{"t":"p","runs":[{"text":"They were measured on "},{"text":"the models of their moment","b":true},{"text":", mostly 2022 to 2023, when instruction-following was weaker and prompt phrasing had more room to move the result. Every one of these tricks has smaller effects on current models, because the models improved in exactly the dimension the trick was compensating for."}]},{"t":"p","runs":[{"text":"They were measured on "},{"text":"narrow benchmarks","b":true},{"text":", usually maths word problems, where a small nudge towards careful step-by-step output shows up as a percentage. Your email is not a maths word problem."}]},{"t":"p","runs":[{"text":"And they are "},{"text":"fragile in a specific way","b":true},{"text":": the effect is often smaller than the run-to-run variation you saw in the sampling lesson. If a trick gives you three points on a benchmark and your own five runs of the same prompt vary by five points, you cannot detect the trick without careful measurement, and neither can the person who told you about it."}]},{"t":"h2","id":"what-has-held-up","runs":[{"text":"What has held up"}]},{"t":"p","runs":[{"text":"Not everything is folklore. The following survive across models and tasks, because they change what information is available rather than trying to change the model’s mood:"}]},{"t":"ul","items":[[{"text":"Giving the material","b":true},{"text":" rather than describing it."}],[{"text":"Stating the finished state","b":true},{"text":" rather than the topic."}],[{"text":"Showing examples","b":true},{"text":" where a rule is hard to state."}],[{"text":"Asking for intermediate steps","b":true},{"text":" on multi-stage tasks, which is what “step by step” was always doing, minus the breathing."}],[{"text":"Naming the output shape","b":true},{"text":" and what to omit."}],[{"text":"Providing an escape hatch","b":true},{"text":" so “I cannot” is available."}]]},{"t":"p","runs":[{"text":"Notice these are all about content. None depends on a phrase."}]},{"t":"h2","id":"how-to-test-one-for-yourself","runs":[{"text":"How to test one for yourself"}]},{"t":"p","runs":[{"text":"You do not need a lab. Take a task you do repeatedly and where you can tell right from wrong: extracting three fields from a document, classifying a message, converting a format. Build ten test items with known answers."}]},{"t":"p","runs":[{"text":"Run your prompt over the ten. Run the prompt with the magic phrase added over the same ten. Do each three times, because of sampling. Count."}]},{"t":"p","runs":[{"text":"Almost always you will find a difference smaller than the noise, and you will stop wondering. Occasionally you will find something real for your task and your model, and then you have earned it, with the honest note that it may stop being true at the next model update."}]},{"t":"h2","id":"the-reason-this-matters-beyond-tidiness","runs":[{"text":"The reason this matters beyond tidiness"}]},{"t":"p","runs":[{"text":"Every token of folklore is a token you did not spend on context. A prompt that opens with three sentences of role-play and motivational framing, and then asks a vague question, is worse than the same prompt with those sentences replaced by the actual document. The cost of the spell is not that it fails. It is that it occupies the place where the useful information should have been."}]},{"t":"p","runs":[{"text":"There is no secret vocabulary. There is a habit of under-describing your own problem, and that is the thing to fix."}]}],"check":{"question":[{"text":"Someone reports that adding “I will tip you $200” made a model’s answers noticeably better on their task. What is the most likely explanation, and the right response?"}],"options":[{"text":[{"text":"The claim is impossible, because models have no concept of money and cannot be motivated."}],"correct":false,"tempting":[{"text":"Overcorrects into a different error: the phrase does change the text in context and can measurably shift outputs; the problem is reliability, not impossibility."}]},{"text":[{"text":"They compared a small number of runs, and the difference is probably within the variation the sampler produces anyway; the fix is a fixed test set run several times each way."}],"correct":true,"tempting":null},{"text":[{"text":"It works because the phrase resembles high-effort professional writing in the training data, so the effect will transfer to other tasks."}],"correct":false,"tempting":[{"text":"Offers a plausible story as if it settled the question; a mechanism that sounds right is still not evidence for this task or this model."}]},{"text":[{"text":"It worked once and will keep working, since prompt effects are stable properties of a model."}],"correct":false,"tempting":[{"text":"Prompt effects shift between model versions, and several published tricks stopped replicating within a year."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"anatomy-of-a-prompt","title":"The six things a working prompt contains","minutes":8},"next":{"slug":"specificity-and-its-limit","title":"How much detail is too much","minutes":8}}}