The evidence on AI tutors

Why a teacher should read the studies

Every vendor of an AI tutor has a chart showing learning gains. Very few of those charts come from a study that a researcher would trust. A teacher who can read the two or three good studies that exist, and say what they found and on whom, is in a stronger position than one who has read forty brochures. This lesson is those studies, and how to read the next one.

The Turkish maths study

The most careful study so far was run in Turkish high schools in 2023 and published in 2024, with roughly a thousand students in Grades 9 to 11. Students did maths practice sessions under one of three conditions, assigned at random: no AI; unrestricted access to a chatbot built on GPT-4; or access to a version of the same chatbot designed as a tutor: it gave hints and guidance, and was instructed not to give the answer.

On the practice problems, both AI groups did much better than the no-AI group: about 48% better with the unrestricted chatbot, and about 127% better with the tutor version.

Then everybody sat an exam without any AI. The unrestricted group scored about 17% lower than the students who had never had AI at all. The tutor group scored about the same as the no-AI group: no harm, and no measurable gain.

The check question is this result, and the reading the authors give is the one that fits: when the answer is available, students take it, and the effortful work of producing it, which is what builds the memory the exam tested, does not happen. Practice looked better because it was not practice. The tutor version, by withholding the answer, kept the effort in place, and the students learned as much as they would have alone.

The Harvard physics study

A second study, at Harvard in 2024, taught two groups of undergraduates the same physics material: one in a well-run active-learning class, the other with a purpose-built AI tutor working through the same material with carefully designed prompts, hints and checks. The AI-tutored students learned more (roughly twice the gain on the test) in less time, and reported higher engagement.

The two studies look contradictory and are not. The Harvard tutor was designed by the course’s own teachers, with the pedagogy built in: it did not hand over answers, it sequenced the material, it asked students to explain. It was the “tutor” condition of the Turkish study, done with more care and compared against a class rather than against practice alone.

What the two together say

Three things, and they are about design, not about the technology.

1. Answers on demand harm learning. Not sometimes: in the best study available, measurably, across a thousand students.
2. A tutor that withholds answers and gives hints does not harm, and can help. How much depends on how well the tutoring is designed.
3. The effect is invisible during use. Both groups felt they were doing well. The harm showed only on an exam the tool could not sit.

For your classroom that translates directly into the policy blocks earlier in this course. Level 1 use (explanation, hints, “help me understand”) is the tutor condition. “Give me the answer” is the unrestricted condition. The prompt card from the feedback lesson, which forbids rewriting, is a design choice with evidence behind it.

How to read the next study

There will be many more, and most will be worse than these two. Five questions to ask of any study or vendor claim:

• Was assignment random? If students chose to use the tool, the ones who chose it differ from the ones who did not, and the result tells you about the students, not the tool.
• What was the comparison? Against nothing? Against a good teacher? Against a textbook? “Better than nothing” is a low bar.
• Was the outcome measured without the tool? Practice scores with the tool are not learning. The Turkish study’s exam is the model.
• How long after? A test the same day measures something different from a test a month later.
• Who ran it, and who paid? A study run by the vendor is a claim. An independent study is evidence.

A teacher who asks those five of the next vendor will find that most of the charts do not survive the second question.

Where AI tutoring probably helps most

The honest reading of the evidence, and of what tutoring has always been, is that the gain is largest where there was no help at all: the student with no one at home who can explain quadratic equations, the school with one maths teacher for three hundred students, the evening before the exam. A tutor that withholds answers and explains patiently, available at eleven at night on a phone, is a genuine thing for that student, and the studies suggest it does not harm and may help. That is a smaller claim than the brochures make and a real one.

The honest limit

Two good studies is two. They were in maths and physics, with secondary and university students, in Turkey and the United States, over a term or less. What happens in history, in a primary classroom, in a second language, over three years, with the tools students actually use rather than research versions, is not known. A teacher should hold the design finding (hints, not answers) as the best available guidance, and hold everything else as open.