← Back to Research
When to Help with AI? Designing Student–AI Collaboration for Deeper Learning
Learners tend to turn to AI the moment they hit a problem,but immediate AI help can actually hinder learning. Across two field studies with data science learners, we found that when AI assistance is provided matters as much as whether it is provided at all. Requiring students to struggle productively before receiving AI support leads to significantly higher-quality work and more active engagement.
Study 1: To Use AI Assistance or Not? 🏆 Best Short Paper (Learning Analaytics & Knowledge '24)
Methods
A randomized crossover experiment comparing independent work with revision of AI-generated outputs, with learners randomly assigned to the order of conditions and completing each condition once. N = 72 adult learners (mean age = 32.50 years, SD = 7.99)
Evaluation of learner output based on accuracy, specificity, and communication efficacy + Investigating impact on student experience via surveys
Key Findings
- Quality of AI outputs influenced the quality of learner outputs.
- Learners showed signs of over-reliance, unable to substantially improve upon poor quality AI outputs.
- Trust in AI outputs mediated preference for working with vs without AI assistance.
Study 2: Fostering Productive Struggle with Deferred AI Asisstance
Methods
Between-subjects RCT in University of Michigan online course comparing: (i) independent work, (ii) work done with on-demand AI assistance, and (iii) work done with deferred-AI assistance, where learners were required to work independently before getting AI assistance N = 97 adult learners (mean age = 30.94, SD = 8.60)
Evaluation of learning outcomes, quality of learner output, time spent on the task, and perceived learning benefits via validated rubrics, pre/post-tests, and surveys
Key Findings
Deferring AI assistance by requiring students to work independently before obtaining AI assistance led to significantly higher-quality learner outputs, compared to both no assistance and on-demand AI assistance.
Impact
- Demonstrated that timing of AI assistance is a critical and underexplored lever in learning design and, more broadly, in enhancing human-AI collaboration processes and outcomes.