The AI Learning Paradox: What 14 Studies Reveal About Generative AI in Education
A meta-report synthesizing peer-reviewed research on how generative AI affects learning, academic achievement, and critical thinking. The evidence is not a simple yes or no. It depends on one critical variable: how the tool is used.
The Question
Does generative AI help students learn, or does it undermine the cognitive processes that learning depends on? This is arguably the most consequential educational question of the decade. The answer emerging from the research is neither the utopian vision promoted by AI companies nor the dystopian warning issued by critics. It is more interesting than either.
This report synthesizes 14 peer-reviewed studies and meta-analyses from journals including Computers and Education, Scientific Reports, British Journal of Educational Technology, Humanities and Social Sciences Communications, Societies, Acta Psychologica, Smart Learning Environments, Journal of Educational Computing Research, Journal of Computer Assisted Learning, and the proceedings of CHI 2025. The studies range from large-scale meta-analyses of 69 experiments to controlled trials at Harvard and EEG studies at MIT. Together, they reveal a consistent pattern: AI’s effect on learning is conditional, not absolute.
The Positive Evidence
Multiple meta-analyses converge on a clear finding: when used appropriately, generative AI improves academic performance.
A meta-analysis of 69 experimental studies published in Computers and Education found that ChatGPT improves academic performance, boosts motivation and engagement, and enhances higher-order thinking propensities including critical thinking and problem-solving. Notably, it also reduces mental effort, suggesting AI’s cognitive offloading function can free working memory for higher-order reasoning rather than replacing it.[1]
A separate meta-analysis of 35 experimental studies involving 4,193 participants, published in Humanities and Social Sciences Communications, found a moderate positive effect (g = 0.670) on both cognitive and non-cognitive skills. The benefit generalized across K-12 and higher education and across declarative and procedural knowledge types.[2]
A third meta-analysis in the Journal of Educational Computing Research confirmed a significant positive effect of generative AI on college students’ academic achievement across disciplines.[3]
The most striking single study comes from Harvard. A randomized controlled trial published in Scientific Reports compared a custom-designed AI tutor (powered by GPT-4) against in-class active learning, itself an established best practice. Students using the AI tutor achieved more than double the learning gains in less time, with higher engagement. The effect size was estimated at 0.73 to 1.3 standard deviations.[4]
Even the claim that AI “kills creativity” is challenged by the evidence. A preregistered experiment in Computers and Education with 145 university students found that ChatGPT use significantly improved the quality (d = 0.69), elaboration (d = 0.61), and originality (d = 0.55) of creative solutions on a complex problem-solving task.[5]
The Negative Evidence
But the same research literature documents a parallel set of findings that complicate the optimistic picture.
A systematic review in Smart Learning Environments synthesized 14 studies and found that over-reliance on AI dialogue systems significantly impairs students’ decision-making, critical thinking, and analytical reasoning. The mechanism involves a shift toward fast, heuristic cognition and a failure to calibrate trust in AI outputs, compounded by hallucinations and algorithmic bias that students struggle to detect.[6]
A mixed-methods study of 666 UK participants published in Societies found a significant negative correlation between frequent AI tool usage and critical thinking abilities, as measured by the Halpern Critical Thinking Assessment. Cognitive offloading mediated the relationship: AI usage led to increased offloading, which in turn reduced independent analytical reasoning. Younger users were most affected.[7]
A study of 580 university students in Acta Psychologica identified a previously underexamined mechanism: cognitive fatigue. Greater AI dependence was associated with lower critical thinking, with cognitive fatigue partially mediating the relationship. The more students relied on AI, the more cognitively fatigued they became, and the less capable of critical thinking.[8]
Perhaps the most alarming evidence comes from a working paper by researchers at Wharton and Stanford. In a randomized experiment with approximately 1,000 students, standard ChatGPT-style tutoring boosted practice performance by 48%. But when the AI was removed, these students scored 17% worse than peers who never had AI at all, a phenomenon the researchers call the “crutch effect.”[9]
An EEG study from the MIT Media Lab provided neural evidence for this concern. Participants using LLMs showed the weakest brain connectivity during writing tasks. Even after AI was removed, their brains remained under-engaged, suggesting the accumulation of what the researchers call “cognitive debt” that persists beyond the period of AI use.[10]
The Critical Thinking Shift
A survey of 319 knowledge workers presented at CHI 2025, the premier human-computer interaction conference, provides a bridge between the positive and negative findings. The study found that higher confidence in generative AI is associated with less critical thinking, while higher self-confidence is associated with more. But the nature of critical thinking itself shifts: from task execution to information verification, response integration, and task stewardship. Workers do not stop thinking critically; they think critically about different things.[11]
This shift has a dual character. On one hand, AI automates information gathering and content creation, freeing workers to focus on higher-order evaluation. On the other hand, in routine or lower-stakes tasks, workers simply rely on AI, raising concerns about long-term reliance and diminished independent problem-solving. The study reveals a trade-off: efficiency gains come at the cost of reduced hands-on cognitive engagement, and workers may not recognize when they have crossed the line from productive oversight to unreflective acceptance.[11]
The Decisive Variable: How AI Is Used
The studies converge on a single, critical insight: generative AI’s effect on learning is not a property of the technology itself. It is determined by the pedagogical design wrapped around it.
The Harvard RCT makes this explicit. The AI tutor that produced doubled learning gains was deliberately engineered to manage cognitive load, scaffold reasoning, promote metacognition, and provide personalized immediate feedback while never simply giving the answer. The authors caution that unstructured AI interactions, lacking this design, have been shown to harm student performance. The gain came from the pedagogy, not from GPT-4.[4]
A study published in the Journal of Computer Assisted Learning tested this directly. Using the same GPT tool, researchers compared a Socratic-scaffolded configuration (asking questions, guiding reasoning) against a direct-answer configuration (supplying solutions). The results were qualitatively opposite. Socratic-scaffolded students engaged in cyclical, reflective practices with sustained positive attitudes. Direct-answer students fell into superficial trial-and-error mimicry, over-reliance, and declining attitudes. The same model, different configuration, opposite outcomes.[12]
The original study provided by Fan et al. in the British Journal of Educational Technology confirms this pattern from another angle. In a randomized experiment with 117 university students, ChatGPT improved short-term essay scores but did not improve knowledge gain or knowledge transfer. The ChatGPT group exhibited different, less self-regulated patterns of orientation, monitoring, and evaluation. The authors coined the term “metacognitive laziness” to describe what happens when learners offload the metacognitive work of judging, monitoring, and revising onto the AI rather than doing it themselves. The performance gain on the task was real; the learning gain was not.[13]
Synthesis: The Two-Path Model
Across all 14 studies, a consistent two-path model emerges:
- Path 1: AI as Socratic tutor. When AI is configured to ask questions, scaffold reasoning, provide feedback without supplying answers, and force the learner to do the cognitive work, it produces significant learning gains, higher engagement, improved creative output, and boosted motivation. The Harvard RCT, the Socratic-vs-direct study, and multiple meta-analyses all support this path. The mechanism is productive struggle: AI handles lower-level tasks (information retrieval, formatting, basic generation) while the learner handles higher-order reasoning, evaluation, and integration.
- Path 2: AI as answer machine. When AI is used to generate completed work, supply direct answers, or replace the learner’s own cognitive effort, it produces short-term performance gains that mask long-term learning deficits. The crutch effect, metacognitive laziness, cognitive offloading, cognitive fatigue, and neural under-engagement all point to the same mechanism: bypassing the productive struggle that builds durable knowledge and transferable skills. The performance looks good while AI is present. It collapses when AI is removed.
The critical implication for educators and policymakers is that the question is not whether AI should be used in education. The question is how. A tool that doubles learning gains when configured as a Socratic tutor can also produce a 17% performance collapse when used as an answer machine. The same technology, different design, opposite outcomes.
As Fan et al. conclude, the field must move beyond “does AI help?” to “under what conditions and through what processes does AI help versus harm?” The answer, increasingly clear from the evidence, is that AI helps when it demands cognitive effort from the learner and hurts when it substitutes for it.[13]
References
[1] **[SECONDARY]** Deng, R., Jiang, M., Yu, X., Lu, Y., & Liu, S. “Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies.” Computers and Education 227 (2025): Article 105224.
[2] **[SECONDARY]** Wu, X., Zhu, P., Zhang, J., Yin, M., & Wang, Y. “ChatGPT’s impact on student learning outcomes: a meta-analysis of 35 experimental studies.” Humanities and Social Sciences Communications 13 (2026): Article 7019.
[3] **[SECONDARY]** Sun, L., & Zhou, L. “Does Generative Artificial Intelligence Improve the Academic Achievement of College Students? A Meta-Analysis.” Journal of Educational Computing Research 62, no. 7 (2024): 1896-1933.
[4] **[SECONDARY]** Kestin, G., Miller, K., Klales, A., Milbourne, T., & Ponti, G. “AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting.” Scientific Reports 15 (2025): Article 17458.
[5] **[SECONDARY]** Urban, M., Dectherenko, F., Lukavsky, J., Hrabalova, V., Svacha, F., Brom, C., & Urban, K. “ChatGPT improves creative problem-solving performance in university students: An experimental study.” Computers and Education 215 (2024): Article 105031.
[6] **[SECONDARY]** Zhai, C., Wibowo, S., & Li, L.D. “The effects of over-reliance on AI dialogue systems on students’ cognitive abilities: a systematic review.” Smart Learning Environments 11 (2024): Article 28.
[7] **[SECONDARY]** Gerlich, M. “AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking.” Societies 15, no. 1 (2025): Article 6.
[8] **[SECONDARY]** Tian, J., & Zhang, R. “Learners’ AI dependence and critical thinking: The psychological mechanism of fatigue and the social buffering role of AI literacy.” Acta Psychologica 260 (2025): Article 105725.
[9] **[SECONDARY]** Bastani, H., Osadchiy, N., Valavi, E., et al. “Generative AI Can Harm Learning.” Working paper, Stanford Graduate School of Business / Wharton (2024).
[10] **[SECONDARY]** Kosmyna, N., et al. “Your Brain on ChatGPT: Accumulation of Cognitive Debt When Using an AI Assistant for Essay Writing Tasks.” arXiv:2506.08872, MIT Media Lab (2025).
[11] **[SECONDARY]** Lee, H.-P. (H.), Sarkar, A., Drosos, I., Tankelevitch, L., Rintel, S., Banks, R., & Wilson, N. “The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers.” In CHI Conference on Human Factors in Computing Systems (CHI ’25), 2025.
[12] **[SECONDARY]** Sun, D., Zheng, Y., Xu, J., & Yang, Z. “When generative AI meets Socratic method: Investigating programming learning dynamics through behaviours, interaction qualities and perceptions.” Journal of Computer Assisted Learning 42, no. 2 (2026): e70210.
[13] **[SECONDARY]** Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Gasevic, D. “Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance.” British Journal of Educational Technology 56, no. 2 (2025): 489-530.
AI automated and generated content. Human reviewed.
Listen to the Podcast

Episode 19 of AI Policy for the Bears — two bears discuss the AI Learning Paradox in education.
The Weekly AI Policy Brief
AI policy moves fast. Get the week’s developments, new trackers, and podcast episodes in one short email — free, ad-free, no spam.
Leave a Reply