AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting

Here we report a randomized, controlled trial measuring college students’ learning and their perceptions when content is presented through an AI-powered tutor compared with an active learning class. The novel design of the custom AI tutor is informed by the same pedagogical best practices as employed in the in-class lessons. We find that students learn significantly more in less time when using the AI tutor, compared with the in-class active learning. They also feel more engaged and more motivated. These findings offer empirical evidence for the efficacy of a widely accessible AI-powered pedagogy in significantly enhancing learning outcomes, presenting a compelling case for its broad adoption in learning environments.

Active learning pedagogies,4 such as peer instruction, small-group activities, or a flipped classroom structure, have demonstrated significant improvements over passive lectures9–14. However, any approach that involves one teacher working with many students will suffer, at least in part, from the same three problems that plague passive lectures.

Working with an expert personal tutor is generally regarded as the most efficient form of education15. A tutor can guide the student while providing personalized feedback and answering questions as they arise. Expert tutors will adapt their approach to a student’s individual ability, pace, and specific needs. They offer a more focused and efficient learning experience, managing the student’s cognitive load. In addition, personalized instruction can foster a growth mindset, which has been shown to promote students’ persistence in the face of difficulties16,17. While the advantages of personalized instruction are clear, this model of education cannot scale to meet the needs of a large number of students15. What if an AI tutor could mimic the learning experience one would get from an expert (human) tutor? It could address the unique needs of each individual through timely feedback while adopting what is known from the science of how students learn best. This is the focus of our work. Through a design that involves targeted, content-rich prompt engineering, we developed an online tutor that uses GAI and best practices from pedagogy and educational psychology to promote learning in undergraduate science education.

Results

In this study, students were divided into two groups, each experiencing two lessons, each with distinct teaching methodologies, in consecutive weeks. During, the first week, group 1 engaged with an AI-supported lesson at home while group 2 participated in an active learning lesson in class. The conditions were reversed the following week. To establish baseline knowledge, students from both groups completed a pre-test prior to each lesson, focusing on surface tension in the first week and fluid flow in the second. Following each lesson, students completed post-tests to measure content mastery and answered four questions aimed at gauging their learning experience, including engagement, enjoyment, motivation, and growth mindset.

Learning gains: post-test scores Learning gains were measured by comparing the post-test scores of the AI group and the in-class active learning group to the pre-test scores of the two groups combined. Students in the AI group exhibited a higher median (M) post-score (M = 4.5, N = 142) compared to those in the in-class active learning group (M = 3.5, N = 174). The median learning gains for students, relative to the pre-test baseline (M = 2.75, N = 316), in the AI-tutored group were over double5 those for students in the in-class active learning group.

Table S1 shows that, controlling for all these factors, the students in the AI group performed substantially better on the post-test compared with those in the in-class active learning group. We show this to be a highly significant (p < 10–8) result with a large effect size.

Discussion

We have found that when students interact with our AI tutor, at home, on their own, they learn significantly more than when they engage with the same content during an in-class active learning lesson, while spending less time on task. This finding underscores the transformative potential of AI tutors in authentic educational settings. In order to realize this potential for improving STEM outcomes, student-AI interactions must be carefully designed to follow research-based best practices.

The extensive pedagogical literature supports a set of best practices that foster students’ learning, applicable to both human instructors and digital learning platforms. Key practices include (i) facilitating active learning11,19, (ii) managing cognitive load4, (iii) promoting a growth mindset15,16, (iv) scaffolding content20, (v) ensuring accuracy of information and feedback, (vi) delivering such feedback and information in a targeted and timely fashion21, and (vii) allowing for self-pacing22. We aimed to design an AI system that conforms to these practices to the fullest extent current technology allows, thus establishing a model for future educational AI applications.