With a standard chatbot, practice scores jumped 48 percent. Once it was gone, those students scored 17 percent below peers who never had it.
A study by Bastani and colleagues, published in the Proceedings of the National Academy of Sciences in 2025, gave nearly 1,000 high school students AI math tutors. Practice scores rose sharply: 48 percent with a standard chatbot, 127 percent with a version built to give hints rather than answers. Then the researchers took the AI away. Students who had used the standard chatbot scored 17 percent lower than classmates who never had a tutor at all. The hint-giving version largely prevented that drop.¹
Talent Takeaway: When you roll out AI, decide in the same breath where people will keep working without it — and set the tool up to coach, not to answer. Otherwise you buy speed today and pay for it in skill tomorrow.
¹ Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26). https://doi.org/10.1073/pnas.2422633122
Further reading: Lee, H., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025). The impact of generative AI on critical thinking. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 1–22. https://doi.org/10.1145/3706598.3713778