Back to Articles
What makes a good tutor? How TutorMoments works Preliminary results Limitations and next steps 📄 Tech Report: https://tutormoments.allen.ai/static/paper/tutormoments-preview.pdf | 📊 Data: https://huggingface.co/datasets/allenai/tutormoments-preview | 💻 Code: https://github.com/allenai/tutormoments
Today we're introducing a preview of TutorMoments, a framework to measure whether cutting-edge LLMs can balance one of the hardest trade-offs in education: when to step in and help a student and when to hold back and let the student do more of the work.
TutorMoments is a replay-based evaluation built off real one-on-one math tutoring sessions. Experienced math teachers go through transcripts collected from a U.S. tutoring program and flag the moments where a tutor had to choose between making a problem easier to get started on and pushing the student to do more of the reasoning themselves. TutorMoments then takes the transcript up to that decision point, hands it to a language model, and has the model take over as the tutor in a simulated session – with the student played by another language model – to see what the LLM tutor does.
Told only to "tutor well," we find that models tend to over-help by giving too much support and rarely pushing students to do deeper thinking. Spelling out the trade-off (when to help versus when to hold back) in the tutor's prompt improves performance, but it doesn't close the gap to human tutoring that consistently fits the moment, and LLMs still differ widely in how reliably they make that call.
