AI & TECHNOLOGY / RESEARCH INTO PRACTICE
AI and Learning: Can You Explain What It Just Did?
The code worked. The quiz told a different story. A 2026 experiment asks an uncomfortable question about AI and learning: when the assistant solves a new problem, what can you still do when the chat closes?
Your practical takeaway: Close the assistant, explain the rule, solve a changed example and catch a deliberate error. A finished answer alone cannot show what you learned.
In this guide
The result hidden behind a working answer
Anthropic’s January 29, 2026 report describes a randomized experiment with 52 participants learning the unfamiliar Python library Trio. The AI group scored lower on an immediate comprehension quiz. That is a finding about learning this library in this task, not a measurement of intelligence. Task completion was about two minutes faster with AI, but that difference was not statistically significant. Read the study team’s report.
A numerical wrinkle: Anthropic’s summary gives quiz averages of 50% and 67%. The paper reports a 4.15-point difference on a 27-point quiz, which works out to about 15.4 percentage points. Those descriptions do not reconcile, so we use the direction of the result rather than presenting one exact gap as settled. Check the paper’s results.
The distinction matters if your aim is to learn. A working file is evidence about the file. A closed-chat explanation is evidence about what you can do yourself.
The same assistant, different kinds of work
The researchers also examined recordings of how participants used AI. Asking conceptual questions or following generated code with comprehension questions was associated with stronger quiz performance. Those patterns were observed within the experiment; they were not separately randomized, so they do not establish that one prompting style caused the improvement. Interaction patterns and limitations.
Our practical response is to separate two jobs before opening the chat. For a task you already understand, you may want production help. For a skill you need to acquire, you need practice that leaves you able to explain and repair the result. Label today’s task “deliver” or “learn” in your notes. If it is both, give each job its own check.
A worked example: the formula looks right
Invented learning exercise: you ask an assistant to calculate a trading position from a $2,000 account, a 0.5% risk budget, a $25 entry and a $24.50 planned stop. It returns 20 shares. Before treating that as understanding, hide the answer.
Explain the chain yourself: $2,000 × 0.005 gives a $10 planned risk budget. The entry-to-stop distance is $0.50 per share. Dividing $10 by $0.50 gives 20 shares. Now move the stop to $24. If you understand the relationship, you can work out 10 shares and explain why a wider stop means a smaller position for the same budget.
Finally, inspect a deliberately wrong answer: “The distance is $0.50, so buy 40 shares.” Forty shares would put $20 at risk under those simplified assumptions, twice the budget. Being able to identify that error is more useful than recognizing that the original answer looked familiar. This exercise excludes fees and gaps; the full position-sizing guide explains those limits.
Use this three-part teach-back test
| Check | What to do with the chat closed | What a miss tells you |
|---|---|---|
| Explain | Write the rule in your own words, including its units or assumptions. | You may recognize the output without understanding the reasoning. |
| Transfer | Change one input and solve the new version. | You may have memorized one example. |
| Repair | Find and explain a planted mistake. | You may struggle to supervise a plausible wrong answer. |
This is our practice method, not a validated score from the study. Record which check you missed rather than converting it into a claim about your ability. If you fail transfer, ask for a hint about the changed variable. If you fail repair, compare your reasoning with the source explanation before asking for a complete replacement.
A prompt that keeps you doing the thinking
“Give me one hint, then wait for my attempt. Ask me to explain my reasoning. After I solve it, change one input and ask me to catch an incorrect answer. Do not give the final solution until I have tried.”
The assistant can still mark an answer incorrectly. Check the underlying formula, documentation or original source when the judgement matters. Our AI verification routine is the next step.
What this study does not settle
This was a small experiment on an unfamiliar coding library, with a quiz shortly after the task. The paper identifies GPT-4o as the assistant’s base model. It does not establish permanent skill loss, results in every subject, or the effect of every current AI product. The research paper and revision history let you inspect the design rather than treating the headline as a universal verdict.
On your next learning task, save the generated answer in one column and your closed-chat attempt in another. The useful question is precise: which part could you explain, transfer or repair without help?
Primary sources and research dates
- Anthropic, January 29, 2026: experiment, results and limitations
- Shen and Tamkin: How AI Impacts Skill Formation, arXiv v2, February 1, 2026
Research summaries above are linked to their original sources. Worked examples and practice methods are our own and are labelled separately from study results. This article was written with AI assistance; source claims and calculations were checked before publication.