Research Summary: When the Scaffold Stays On: AI, Practice Style, and Screening in Elite Skill Formation

Generative AI can boost short-term productivity by assisting learners, but its long-term impact on 'frontier skill' development depends on usage patterns. 'Substitute-users' who let AI replace practice risk skill erosion, while 'complement-users' leverage AI for accelerated learning. Both modes may produce similar outputs, making it difficult for organizations to discern actual skill development through traditional assessment methods. Prohibiting AI during evaluation, as seen in competitive programming, might differentiate these usage styles.

Why it matters

The widespread adoption of generative AI necessitates a re-evaluation of skill development and assessment methodologies. Understanding how AI impacts learning processes is crucial for maintaining and advancing high-level expertise across various domains, preventing a decline in core capabilities within the workforce.

Key insights

• Generative AI can increase short-term productivity by performing tasks learners would typically practice.
• The effect of AI on the development of 'frontier skills' varies based on how it is used.
• 'Substitute-users' may fail to develop skills because AI replaces their practice.
• 'Complement-users' can learn faster by integrating AI into their practice routines.
• AI-aided outputs from both user types can appear similar, complicating skill assessment for organizations.
• AI-prohibited evaluation gates, such as those in competitive programming contests, may help differentiate between effective and ineffective AI usage modes.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2606.06253