🔍 Read the full analysis: Consultants’ Data Science Skills Rose With ChatGPT, Then Slipped on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A randomized trial at Boston Consulting Group found that consultants using ChatGPT performed better on coding, statistics and prediction tasks than a group given conventional online resources. The advantage did not show up in a later technical test without ChatGPT, and the study’s authors describe the task improvement as temporary.
A randomized trial of Boston Consulting Group consultants found that ChatGPT substantially improved their results on data science tasks outside their usual skills, but the advantage did not carry over to a follow-up test completed without the chatbot. Published October 26 in Nature Human Behaviour, the study offers evidence that generative AI can help workers perform unfamiliar technical work while leaving open whether that use builds lasting skills.
In the experiment, the treatment group received 15 to 20 minutes of training on using ChatGPT for technical problems and then worked with GPT-4. A control group received instruction on using Google and resources including Stack Overflow and Khan Academy. Participants completed two of three 90-minute tasks involving Python coding, statistics or prediction. The study was preregistered before data collection, and the tasks were designed so ChatGPT could guide users but could not independently provide all the correct answers.
Among the 487 consultants who submitted both tasks included in the main analysis, the ChatGPT group scored 49 percentage points higher than the control group on coding, 20 points higher on statistics and 17 points higher on prediction. The researchers compared performance with that of 44 BCG data scientists who completed the tasks without AI. The consultants using ChatGPT still averaged below those experts on all three tasks. The study also reports that the ChatGPT group finished coding and prediction faster on average, while time spent on statistics did not differ significantly.
After the work tasks, participants answered technical questions on related topics without access to ChatGPT. The authors report that they “find no evidence” the ChatGPT group performed better than the control group on this test. The study also found that ChatGPT users were more confident that the system could solve problems it could not; the authors characterize that finding as exploratory.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Performance Gains Without the Chatbot
The results distinguish between doing a task with AI and learning to do it independently. For employers, that difference affects how they interpret productivity, training and expertise: a worker may produce stronger work with a chatbot without having acquired the knowledge to repeat the task when the tool is unavailable.
The findings do not show that AI cannot support learning in other settings, or that the same pattern applies across jobs and tools. They do indicate that task performance alone is not proof of durable skill development. Organizations using AI-assisted work may need separate ways to check employees’ understanding and to review outputs, particularly when a task requires judgment beyond what the system can reliably supply.
Products Worth Considering
Bose QuietComfort Ultra Earbuds (2nd Gen), Wireless Noise-Cancelling Earbuds with Immersive Audio and Wireless Charging Case, Black
Logitech MX Keys S Combo Wireless Keyboard and Mouse, Graphite
Samsung 32-Inch Odyssey G5 QHD 180Hz Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Set Up
The trial took place at BCG in late March and early April 2024. Of 986 consultants allocated to the study, 573 began the survey and 487 submitted work on both assigned tasks. All participants had at least a bachelor’s degree; the authors say most reported no coding experience. The experiment’s expert benchmark consisted of 44 BCG data scientists working without AI.
The task design matters to interpreting the results. BCG data scientists and OpenAI’s Economic Impacts Research team developed problems for which ChatGPT could be useful but could not correctly answer every question by itself. The assessment therefore measured how participants worked with the tool, not simply whether they could copy a complete answer. In the later test, participants answered five questions on probability, machine learning and coding without ChatGPT.
The paper’s title refers to “temporary upskilling”. Its authors compare AI assistance to an exoskeleton: it can extend what a person can do while in use, without necessarily leaving the person with the same capability afterward. That is the authors’ interpretation of this trial’s findings, not a conclusion that every kind of AI training has the same effect.
““We find no evidence” that the ChatGPT group did better on the technical questions completed without the tool.”
— The study’s authors
Limits of the Skill Test
The follow-up assessment was a short test of five technical questions, not a long-term measure of whether participants learned or retained skills. The reported lack of evidence for a group advantage should not be read as proof that no individual learned anything, or that repeated practice with AI could never support learning.
The study was conducted at one consulting firm, with one model and a particular training session and set of tasks. It is not clear from these results whether the same performance pattern would hold in other industries, for different kinds of work, or after longer training. The authors also describe the finding about users’ confidence in ChatGPT’s abilities as exploratory, so its scope and implications remain uncertain.
Questions for Workplace Training
The study reports results from an experiment already completed and does not announce a new trial or a forthcoming policy change. A key next step for researchers is to test whether different forms of instruction, repeated practice or follow-up feedback help workers retain technical skills when AI is not available.
For employers, the immediate practical question is how to distinguish AI-assisted output from independent competence. Organizations can assess both the quality of work produced with AI and whether employees can explain or perform relevant parts of a task without it. Whether such approaches improve lasting skill remains to be tested, rather than established by this study.
Key Questions
What did the study find about ChatGPT and consultants’ performance?
In a randomized BCG trial, the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than the group using Google and other online resources.
Did the consultants become better at the tasks without ChatGPT?
The study found no evidence of an advantage for the ChatGPT group on a later technical test completed without the tool. The authors describe the measured performance improvement as temporary.
Did ChatGPT users perform as well as BCG data scientists?
No. On average, consultants using ChatGPT scored below the 44 data scientists who completed the tasks without AI, although the gap was smaller for the ChatGPT group than for the control group.
How many consultants were included in the main analysis?
The main analysis included 487 consultants who submitted work on both assigned tasks. A total of 986 consultants were allocated to the study, and 573 began the survey.
Does the trial prove AI cannot teach lasting skills?
No. The results concern one firm, one model, a brief training session and a short follow-up test. They show no detected group advantage on that test, but do not settle whether other training methods or longer practice can produce lasting learning.
Source: ThorstenMeyerAI.com
