{"course":{"slug":"ai-for-students","title":"AI for Students","url":"https://addaly.com/learn/ai-for-students","level":"beginner","track":"work"},"lesson":{"slug":"what-counts-as-evidence","title":"What counts as evidence","url":"https://addaly.com/learn/ai-for-students/what-counts-as-evidence","position":42,"of":73,"minutes":8,"answer":"Match the claim to the design: causal language requires randomisation, and comparing response rates across two separate trials is invalid however natural it reads.","keyIdea":"Match the claim to the design: causal language requires randomisation, and comparing response rates across two separate trials is invalid however natural it reads."},"body":{"format":"markdown","text":"## Not all sources are the same weight\n\nStudents lose marks by citing a blog post as though it were a trial, and by citing a trial as though it settled a question. Knowing what a source can support is a skill markers reward heavily, and it takes about one lesson to acquire the outline of.\n\nRoughly ascending, for a claim about whether something works:\n\n- **Anecdote.** One person's report. Establishes that something happened once, and nothing else.\n- **Case report or case series.** Documented individual cases. Valuable for the rare and the new; cannot establish frequency or cause.\n- **Cross-sectional survey.** A snapshot. Shows association. Cannot establish direction: did the exercise cause the mood, or the mood cause the exercise.\n- **Cohort study.** Followed over time. Establishes temporal order, which is a real gain. Still vulnerable to confounding: the people who chose the exposure differ from those who did not, in ways you may not have measured.\n- **Randomised controlled trial.** Allocation by chance, so on average the groups differ only in the intervention. This is what licenses causal language. Its weaknesses are its own: often small, often short, often on unrepresentative participants.\n- **Systematic review and meta-analysis.** All the eligible studies located by a stated method and combined. The strongest form, and only as good as the studies in it. A meta-analysis of twelve small biased trials is a precise estimate of a biased effect.\n\nThe hierarchy is a guide, not a law. A well-run cohort study of 100,000 people beats a badly-run trial of 30. What matters is being able to say why.\n\n> Figure: What a source can carry, strongest at the top. Layers, top to bottom: Systematic review and meta-analysis (All eligible studies, found by a stated method. Only as good as the studies in it.); Randomised controlled trial (Allocation by chance, so the groups differ on average only in the intervention. This licenses causal language.); Cohort study (Followed over time, so temporal order is established. Still open to confounding you never measured.); Cross-sectional survey (A snapshot. Shows association and cannot tell you which way it runs.); Case report or case series (Valuable for the rare and the new. Cannot establish frequency or cause.); Anecdote (Establishes that something happened once, and nothing else.). A guide rather than a law: a well-run cohort of a hundred thousand beats a badly-run trial of thirty, and the mark is for being able to say why. Match the claim to the design. Causes needs randomisation. Is common needs a representative sample. Works better than needs a direct comparison, not two separate studies read side by side, however natural that reads.\n\n## Peer review: what it does and does not do\n\nPeer review means two or three people in the field read the manuscript and recommended publication, usually unpaid, usually in a few hours of work. It filters out a lot of obvious rubbish. It does not verify the data, does not usually recompute the analysis, and does not guarantee replication. Plenty of peer-reviewed findings have failed to replicate: the replication crisis in psychology and several other fields is exactly this.\n\nSo *peer-reviewed* is a floor, not a certificate.\n\n**Preprints** have not been reviewed. They are legitimate to cite in many fields, especially fast-moving ones, and you should say so: *(preprint, not peer reviewed)*. This is normal practice, not an admission.\n\n**Predatory journals** publish anything for a fee and exist to look like real journals. Check whether the journal is in the **Directory of Open Access Journals**, whether the editorial board is real, and whether anyone in your field has heard of it. A model will happily cite a predatory journal without comment.\n\n## The specific claims that need specific evidence\n\nMatch the claim to the design:\n\n- *X causes Y* needs randomisation, or an extremely well-argued natural experiment.\n- *X is associated with Y* needs an observational study, and the sentence must not drift into causal language in the next paragraph, which is where most essays lose the mark.\n- *X is common* needs a representative sample, which is a much harder thing to obtain than people assume.\n- *X works better than Y* needs a direct comparison, not two separate studies of each.\n\nThat last one is worth dwelling on. Comparing across studies (this drug had 60 per cent response in one trial, that one 45 per cent in another) is invalid, because the populations, measures and conditions differed. A model will do it happily if asked, and it reads perfectly.\n\n## Where the model helps\n\n- **Explaining a design** and why it supports what it supports.\n- **Naming what would be needed** to establish a claim you want to make. Excellent for planning research.\n- **Spotting causal language over correlational evidence** in your own draft: *find every sentence where I claim causation, and tell me what design each cited source used.* A good self-check.\n\n## The honest state of things\n\nSome of what you will cite is contested, and saying so is a strength rather than a hedge. Fields disagree; results reverse; large literatures rest on findings that later failed to replicate. Writing *the evidence here is mixed, with X finding one thing in a large cohort and Y finding the opposite in a smaller trial* is more accurate and better rewarded than picking one and asserting it.\n\nA model will usually give you the consensus position smoothly and flatten the disagreement, because smooth consensus is what most text about a topic looks like. Ask specifically: *what is contested about this, and who disagrees?* Then go and read both sides."},"check":{"question":"An essay cites two separate trials, one reporting 60 per cent response to drug A and another 45 per cent to drug B, and concludes A is more effective. What is wrong?","options":[{"text":"Neither figure means anything without confidence intervals, so no comparison of any kind can be made.","correct":false,"tempting":"Correctly demands uncertainty but overreaches: intervals would still not license comparing across trials with different populations and measures."},{"text":"The comparison is across studies with different populations, measures and conditions, so the percentages are not commensurable.","correct":true},{"text":"The trials were not randomised against each other, so this is observational evidence rather than causal evidence.","correct":false,"tempting":"Misapplies the randomisation concept; each trial may be perfectly randomised internally, and the flaw is in comparing across them."},{"text":"Response rate is the wrong outcome for a comparison of effectiveness, which requires a head-to-head measure of clinical benefit.","correct":false,"tempting":"Raises a legitimate outcome-selection question that would still leave the cross-study comparison invalid even with an ideal outcome."}]},"prev":{"slug":"reading-a-paper-fast","title":"Reading a paper without reading all of it","url":"https://addaly.com/learn/ai-for-students/reading-a-paper-fast"},"next":{"slug":"search-grounded-answers","title":"When it searches the web","url":"https://addaly.com/learn/ai-for-students/search-grounded-answers"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What counts as evidence', in AI for Students. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-for-students/what-counts-as-evidence (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What counts as evidence","courseTitle":"AI for Students","module":{"index":5,"slug":"research-and-evidence","title":"Research, sources and evidence"},"answer":null,"keyIdea":"Match the claim to the design: causal language requires randomisation, and comparing response rates across two separate trials is invalid however natural it reads.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"What a source can carry, strongest at the top","layers":[{"label":"Systematic review and meta-analysis","detail":"All eligible studies, found by a stated method. Only as good as the studies in it."},{"label":"Randomised controlled trial","detail":"Allocation by chance, so the groups differ on average only in the intervention. This licenses causal language."},{"label":"Cohort study","detail":"Followed over time, so temporal order is established. Still open to confounding you never measured."},{"label":"Cross-sectional survey","detail":"A snapshot. Shows association and cannot tell you which way it runs."},{"label":"Case report or case series","detail":"Valuable for the rare and the new. Cannot establish frequency or cause."},{"label":"Anecdote","detail":"Establishes that something happened once, and nothing else."}],"caption":"A guide rather than a law: a well-run cohort of a hundred thousand beats a badly-run trial of thirty, and the mark is for being able to say why. Match the claim to the design. Causes needs randomisation. Is common needs a representative sample. Works better than needs a direct comparison, not two separate studies read side by side, however natural that reads."},"alt":"Figure: What a source can carry, strongest at the top. Layers, top to bottom: Systematic review and meta-analysis (All eligible studies, found by a stated method. Only as good as the studies in it.); Randomised controlled trial (Allocation by chance, so the groups differ on average only in the intervention. This licenses causal language.); Cohort study (Followed over time, so temporal order is established. Still open to confounding you never measured.); Cross-sectional survey (A snapshot. Shows association and cannot tell you which way it runs.); Case report or case series (Valuable for the rare and the new. Cannot establish frequency or cause.); Anecdote (Establishes that something happened once, and nothing else.). A guide rather than a law: a well-run cohort of a hundred thousand beats a badly-run trial of thirty, and the mark is for being able to say why. Match the claim to the design. Causes needs randomisation. Is common needs a representative sample. Works better than needs a direct comparison, not two separate studies read side by side, however natural that reads."}},"blocks":[{"t":"h2","id":"not-all-sources-are-the-same-weight","runs":[{"text":"Not all sources are the same weight"}]},{"t":"p","runs":[{"text":"Students lose marks by citing a blog post as though it were a trial, and by citing a trial as though it settled a question. Knowing what a source can support is a skill markers reward heavily, and it takes about one lesson to acquire the outline of."}]},{"t":"p","runs":[{"text":"Roughly ascending, for a claim about whether something works:"}]},{"t":"ul","items":[[{"text":"Anecdote.","b":true},{"text":" One person’s report. Establishes that something happened once, and nothing else."}],[{"text":"Case report or case series.","b":true},{"text":" Documented individual cases. Valuable for the rare and the new; cannot establish frequency or cause."}],[{"text":"Cross-sectional survey.","b":true},{"text":" A snapshot. Shows association. Cannot establish direction: did the exercise cause the mood, or the mood cause the exercise."}],[{"text":"Cohort study.","b":true},{"text":" Followed over time. Establishes temporal order, which is a real gain. Still vulnerable to confounding: the people who chose the exposure differ from those who did not, in ways you may not have measured."}],[{"text":"Randomised controlled trial.","b":true},{"text":" Allocation by chance, so on average the groups differ only in the intervention. This is what licenses causal language. Its weaknesses are its own: often small, often short, often on unrepresentative participants."}],[{"text":"Systematic review and meta-analysis.","b":true},{"text":" All the eligible studies located by a stated method and combined. The strongest form, and only as good as the studies in it. A meta-analysis of twelve small biased trials is a precise estimate of a biased effect."}]]},{"t":"p","runs":[{"text":"The hierarchy is a guide, not a law. A well-run cohort study of 100,000 people beats a badly-run trial of 30. What matters is being able to say why."}]},{"t":"h2","id":"peer-review-what-it-does-and-does-not-do","runs":[{"text":"Peer review: what it does and does not do"}]},{"t":"p","runs":[{"text":"Peer review means two or three people in the field read the manuscript and recommended publication, usually unpaid, usually in a few hours of work. It filters out a lot of obvious rubbish. It does not verify the data, does not usually recompute the analysis, and does not guarantee replication. Plenty of peer-reviewed findings have failed to replicate: the replication crisis in psychology and several other fields is exactly this."}]},{"t":"p","runs":[{"text":"So "},{"text":"peer-reviewed","i":true},{"text":" is a floor, not a certificate."}]},{"t":"p","runs":[{"text":"Preprints","b":true},{"text":" have not been reviewed. They are legitimate to cite in many fields, especially fast-moving ones, and you should say so: "},{"text":"(preprint, not peer reviewed)","i":true},{"text":". This is normal practice, not an admission."}]},{"t":"p","runs":[{"text":"Predatory journals","b":true},{"text":" publish anything for a fee and exist to look like real journals. Check whether the journal is in the "},{"text":"Directory of Open Access Journals","b":true},{"text":", whether the editorial board is real, and whether anyone in your field has heard of it. A model will happily cite a predatory journal without comment."}]},{"t":"h2","id":"the-specific-claims-that-need-specific-evidence","runs":[{"text":"The specific claims that need specific evidence"}]},{"t":"p","runs":[{"text":"Match the claim to the design:"}]},{"t":"ul","items":[[{"text":"X causes Y","i":true},{"text":" needs randomisation, or an extremely well-argued natural experiment."}],[{"text":"X is associated with Y","i":true},{"text":" needs an observational study, and the sentence must not drift into causal language in the next paragraph, which is where most essays lose the mark."}],[{"text":"X is common","i":true},{"text":" needs a representative sample, which is a much harder thing to obtain than people assume."}],[{"text":"X works better than Y","i":true},{"text":" needs a direct comparison, not two separate studies of each."}]]},{"t":"p","runs":[{"text":"That last one is worth dwelling on. Comparing across studies (this drug had 60 per cent response in one trial, that one 45 per cent in another) is invalid, because the populations, measures and conditions differed. A model will do it happily if asked, and it reads perfectly."}]},{"t":"h2","id":"where-the-model-helps","runs":[{"text":"Where the model helps"}]},{"t":"ul","items":[[{"text":"Explaining a design","b":true},{"text":" and why it supports what it supports."}],[{"text":"Naming what would be needed","b":true},{"text":" to establish a claim you want to make. Excellent for planning research."}],[{"text":"Spotting causal language over correlational evidence","b":true},{"text":" in your own draft: "},{"text":"find every sentence where I claim causation, and tell me what design each cited source used.","i":true},{"text":" A good self-check."}]]},{"t":"h2","id":"the-honest-state-of-things","runs":[{"text":"The honest state of things"}]},{"t":"p","runs":[{"text":"Some of what you will cite is contested, and saying so is a strength rather than a hedge. Fields disagree; results reverse; large literatures rest on findings that later failed to replicate. Writing "},{"text":"the evidence here is mixed, with X finding one thing in a large cohort and Y finding the opposite in a smaller trial","i":true},{"text":" is more accurate and better rewarded than picking one and asserting it."}]},{"t":"p","runs":[{"text":"A model will usually give you the consensus position smoothly and flatten the disagreement, because smooth consensus is what most text about a topic looks like. Ask specifically: "},{"text":"what is contested about this, and who disagrees?","i":true},{"text":" Then go and read both sides."}]}],"check":{"question":[{"text":"An essay cites two separate trials, one reporting 60 per cent response to drug A and another 45 per cent to drug B, and concludes A is more effective. What is wrong?"}],"options":[{"text":[{"text":"Neither figure means anything without confidence intervals, so no comparison of any kind can be made."}],"correct":false,"tempting":[{"text":"Correctly demands uncertainty but overreaches: intervals would still not license comparing across trials with different populations and measures."}]},{"text":[{"text":"The comparison is across studies with different populations, measures and conditions, so the percentages are not commensurable."}],"correct":true,"tempting":null},{"text":[{"text":"The trials were not randomised against each other, so this is observational evidence rather than causal evidence."}],"correct":false,"tempting":[{"text":"Misapplies the randomisation concept; each trial may be perfectly randomised internally, and the flaw is in comparing across them."}]},{"text":[{"text":"Response rate is the wrong outcome for a comparison of effectiveness, which requires a head-to-head measure of clinical benefit."}],"correct":false,"tempting":[{"text":"Raises a legitimate outcome-selection question that would still leave the cross-study comparison invalid even with an ideal outcome."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"reading-a-paper-fast","title":"Reading a paper without reading all of it","minutes":8},"next":{"slug":"search-grounded-answers","title":"When it searches the web","minutes":8}}}