How do we improve education if the results are moving in the wrong direction?
That was the starting point for our October meetup. Look at the PISA trends in reading, mathematics and science across OECD countries. The recent direction is not encouraging. It makes the question of how to improve education more urgent, not less.
England looks less bleak in the comparison below. That does not mean the problem is solved, but it is a reminder that the picture is not identical everywhere. PISA 2025 results ↗
Our focus is the technical side of AI in education — almost better described as machine learning in education. How do we build systems that teach, model what students know, choose useful activities and assess the results?
Before discussing our own approach, I want to go through nine papers published since the previous meetup, roughly April to September 2026. My colleagues and I briefly looked through about a hundred papers, read around twenty in more detail, and chose these nine.
A very subjective selection. These are the papers we wanted to discuss, rather than an exhaustive survey of the field.
1. What happened when schools got Khanmigo?
I like the first paper because it is a proper randomised controlled trial. Education is a science: whatever we think ought to work, eventually we have to measure what happens.
Philip Oreopoulos and Nina Low studied Khan Academy with its Khanmigo AI tutor over two years in 18 schools. Students used the programme in supplementary maths sessions, typically 25–40 minutes. This was additional support for children who needed it.
The striking result was how little the students used the chatbot. Many tried it, but sustained conversations about mathematics were scarce — roughly one meaningful tutoring exchange a week.
The programme improved maths achievement by about 0.06–0.08 standard deviations per year. That measures the gain relative to the spread of students’ scores. But it resembled the improvement seen with Khan Academy practice without an AI chatbot.
The chatbot was there. Mostly, the children did not use it. We therefore cannot attribute the programme’s gains to AI tutoring simply because the product included an AI tutor. Read the Khanmigo study ↗
2. Does choosing the right next problem help?
The second paper caught my attention because it describes its approach as reinforcement learning, an area I am particularly interested in. The practical question is choosing problems at the right difficulty for each student.
Angel Tsai-Hsuan Chung, Hamsa Bastani and colleagues studied 770 high-school students in a five-month Python course in Taipei. Everyone had the same AI chatbot. One group followed a fixed sequence from easy to hard; the other received an adaptive sequence.
The adaptive group gained 0.15 standard deviations on an exam taken without AI. Beginners gained about 0.22; experienced students gained about 0.01.
But how much of that gain comes from reinforcement learning itself? The decision rule is close to greedy difficulty matching — choosing the problem that looks best given the student’s current level. Without a simple adaptive baseline, we cannot tell how much extra the more elaborate method adds.
Personalising difficulty helps. We might have expected that, but it is useful to see it measured. It is a separate question whether this particular reinforcement-learning method is necessary. Read the adaptive-sequencing study ↗
3. Why did homework suddenly get faster?
The design of this study is clever. Sina Rismanchian, Eyad Kurd-Misto and colleagues used ten years of ALEKS data — 3.2 million learning interactions — to compare two kinds of maths problems.
Some were word problems that students could copy into ChatGPT. Others involved graphs or interaction with the learning platform, which were much harder to transfer to a chatbot at the time.
After ChatGPT’s release, time spent on the susceptible word problems fell by roughly 27–31% among college and high-school students. The same change did not appear for graph problems.
Doing homework faster sounds useful. But under proctoring, the speed difference disappeared. In the retention analysis, the odds of a correct answer on word problems fell by about 25% relative to graph problems — an odds change, not a 25-point fall in scores.
We cannot prove from these logs that ChatGPT caused every change. Still, the pattern is an alarm bell: getting the homework finished faster can come with less learning. Read the ALEKS study ↗
4. Can an LLM act like a ten-year-old?
Now we come to a group of papers about simulating students. Several papers at our previous meetup raised the same problem: asking an LLM to “behave like a ten-year-old” does not make it behave like a real child.
Krisztian Balog and Arild Michel Bakken tested this against reading-test answers from 2,359 Norwegian pupils in grades 4–6. The children scored about 69%. Models prompted to act like pupils scored around 96–99%.
Even random guessing came closer to the real spread of scores than the prompted models. That comparison is about the score distribution; it does not mean random answers are better answers.
What if we force the model to make more mistakes? Limiting its working memory brings the average closer, but it still struggles on different questions from the children.
Matching the number of mistakes is not enough. We need to match which mistakes the learner makes. A prompt alone did not solve that problem here. Read “Act Like a 5th Grader” ↗
5. What if we train on real learners?
The next papers have more success, but they do more than ask a general model to play a role. StudentSim, by Ke Yang, Jianfeng Gao and colleagues, trains a separate simulator from each learner’s actual records.
The study covers 60 learners across chess, English writing and mathematics. In chess, the simulator matches the learner’s next move about 51% of the time, compared with 45% for Maia2 and 23% for GPT-5.4.
That is better than prompting a general LLM. The model has examples of how this particular person behaves, including where their knowledge falls short.
The researchers also used the simulator to help train a chess tutor through reinforcement learning. Experts rated the resulting guidance more favourably. But no real learners were taught by that tutor in this experiment, so we do not yet have a measured improvement in their learning.
Training on student data helps. How far that takes us towards a useful virtual student is the next question. Read StudentSim ↗
6. How well can we reproduce a student’s mistake?
Suppose we know the student’s previous answers and tutoring conversations. Can we predict what they will say next?
In Who Am I?, Zhangqi Duan, Andrew Lan and colleagues use that history to build a student profile, then use the profile to guide a simulator. The profile-building method is trained with reinforcement learning.
Their Eedi data contain 670 students and 1,775 real tutoring chats. The trained approach beats GPT-5.4 given the raw history.
The result is still modest: on incorrect turns, it reproduces the student’s actual misconception only about 15% of the time.
Think about that. We have a particular student’s history, and we train the system to use it. It improves, but it still usually fails to reproduce the particular misunderstanding behind that student’s answer. Better than the simpler approach, but still a long way to go. Read “Who Am I?” ↗
7. Could we predict ability and difficulty instead?
Knowledge tracing asks what a student knows, based on their learning history. It is an established area of machine learning in education. Can LLMs help with it?
Shuyan Huang, Andrew Lan and colleagues take an approach I like: instead of asking the model to predict the next answer directly, ask it for two things — the student’s ability and the question’s difficulty.
The model reads the tutoring conversation to estimate ability. It also estimates how hard the question is. Item response theory combines the two: take ability minus difficulty and turn the difference into a probability of answering correctly. Then train the system on that prediction.
On the Eedi conversations highlighted here, the approach reaches about 0.65 AUC, compared with 0.56 for the best classic knowledge-tracing comparison. AUC measures how well the predictions rank correct answers above incorrect ones.
The gain over the previous LLM method is much smaller, around 0.004 AUC. But the estimates are more readable: we can inspect ability and difficulty. This combination of LLMs and knowledge tracing worked. Read the dialogue knowledge-tracing study ↗
8. What if we use a much simpler model?
There are many elaborate knowledge-tracing models. But what if we give a boosted decision tree a useful description of the student’s history and ask for the probability of a correct answer?
Praveena Padi, Pradeep Kumar Dolabehera Kakitapelli and colleagues use XGBoost — a model combining many decision trees — with 13 features related to learning. These include practice history, spacing between attempts and response time.
With matched information, the simple model essentially ties the deep models: 0.717 versus 0.720 AUC. With richer features, it reaches 0.786. That last number includes the current answer’s response time; if we exclude that information and predict before the answer, the score is 0.775.
The paper also finds duplicated records in the standard ASSISTments 2009 data. Those duplicates can leak an answer into the history used to predict it.
The appeal is practical: a simple model, useful features, and explanations we can inspect. Before building a more complicated model, it is worth checking what this kind of baseline can do. Read “Stable and Faithful Explanations” ↗
9. Can we learn a teaching strategy from old logs?
The last paper returns to reinforcement learning, which, as you may have noticed, I like a lot.
Usually, an agent tries an action, sees the result and adjusts its strategy. A chess-playing agent can play another game. A robot can try another movement in a simulator.
In education, the feedback may arrive six months later, when the students sit an exam. Change the teaching strategy, wait six months, change it again. We may need millions of iterations. That does not work on a human calendar.
One alternative is offline reinforcement learning: learn from records of decisions and outcomes that already happened. We need enough variation in those records to learn something about different choices.
Badmavasan Kirouchenassamy, Vanda Luengo and colleagues use AlgoPython logs with about 25,000 submissions from 2,672 learners. The old system chose feedback at random from 123 expert-written messages.
That randomisation is useful. The data contain different feedback choices, rather than only one teacher’s preferred response. The researchers use conservative Q-learning, an offline method that avoids putting too much confidence in poorly supported actions, to learn which feedback to select.
Intuitively, we want to use the historical episodes to estimate what would happen under a different strategy. Finding a complete learning journey that follows exactly that strategy is unlikely; shorter episodes give us more evidence to work with. The model learns from those recorded choices and outcomes.
The estimated results are promising, but they are still offline estimates. No students had yet used the new policy in a live deployment. Read the adaptive-feedback study ↗
This is why I wanted to finish with this paper. It gives us a way to work on teaching decisions without waiting another half-year after every change. In Pavel’s article, that question becomes part of a larger architecture: how do we use a learner’s history to choose the next task?
