Analysis: AI learns from "traces" of thinking, not from the process, and risks eroding human skills

The authors (including researchers from Macquarie University), writing for The Conversation, argue that everything AI models are trained on – text, images, code – are merely "traces" of finished thinking, not the actual process of its creation with its errors, trials, and dead ends. Even reasoning models, which print out the steps of their reasoning, according to the authors produce just another trace: the published steps often don't match the actual computation that led to the answer. Agentic systems, according to the text, come somewhat closer to the process, because the action-result-revision loop introduces real trial and correction, but the core formed by the large language model remains a trace.

The second part of the argument concerns people. The authors point to older instances of automation dependency – pilots losing manual skills after the introduction of autopilot ("children of the magenta line," a warning from 1997), doctors whose ability to detect polyps without software assistance declined after the introduction of AI-assisted colonoscopy, and students who performed better with unlimited AI support but scored worse after it was removed. According to the cited study, a decline in performance without AI can occur after as little as about ten minutes of using the assistance.

As a solution, the authors propose bringing more process back into AI systems – for example, having the assistant respond to challenges to the task instead of automatically agreeing, or having the agent learn from its own mistakes. For people, they recommend tools designed to preserve the user's own thinking process: they cite a Harvard experiment in which students using a well-designed AI physics tutor learned faster and better than peers in active instruction without AI, because the tutor forced them to do their own generative work – offering another question along the path to the answer instead of a direct solution.