📊 Full opportunity report: Engineering Is Automated. Research Is the Residual. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
AI systems are now capable of automating large parts of engineering tasks involved in AI development, with benchmarks nearing saturation. Research, however, still involves residual manual effort, though this may change soon. The development could reshape AI R&D workflows.
Recent empirical evidence confirms that AI systems can now automate a significant portion of AI engineering tasks, with several key benchmarks approaching saturation. This development suggests that engineering work in AI R&D may soon be fully automated, while research remains partly manual but potentially on the cusp of automation, according to experts.
Analysis of six core benchmarks measuring AI capabilities in research reproduction, Kaggle competition performance, and kernel design shows rapid progress. For example, the CORE-Bench, which tests the ability to reproduce research papers, has improved from 21.5% in September 2024 to 95.5% in December 2025, with the benchmark author declaring it ‘solved.’ Similarly, the MLE-Bench, assessing performance on Kaggle competitions, has advanced from 16.9% in October 2024 to 64.4% in February 2026, reaching a level comparable to mid-tier human practitioners.
Experts such as Thorsten Meyer interpret these trends as evidence that much of the engineering involved in AI development—installing dependencies, running experiments, optimizing kernels—has been effectively automated. The remaining challenge, according to Meyer, is understanding how much of the research process itself can be automated, as some aspects of research may be fundamentally different from engineering tasks.
Engineering is automated.
Research is the residual.
Six skill benchmarks. Edison’s framing. The question Clark leaves open is whether research is just engineering at scale.
Jack Clark’s Import AI #455 catalogs six benchmarks measuring AI capability on AI R&D tasks and concludes “AI can today automate vast swatches, perhaps the entirety, of AI engineering.” The residual question is research. The structural read on the residual: it may not be a permanent moat.
Six skills. One trajectory.
Clark catalogs six benchmarks measuring AI capability on AI R&D-relevant tasks. Each individual benchmark could be noise. Six benchmarks moving together is a curve. The pattern is the cascade observed across the broader Clark series — visible here in the specific R&D-skill domain.
![Claude AI for Beginners Bible: [5 in 1] The Ultimate Guide to Automate Your Work, Save Hours Every Week, and Use AI for Real-World Results](https://m.media-amazon.com/images/I/415+fSJacsL._SL500_.jpg)
Claude AI for Beginners Bible: [5 in 1] The Ultimate Guide to Automate Your Work, Save Hours Every Week, and Use AI for Real-World Results
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three data points. Mixed signal.
Clark provides three data points on the creative-spark question. Yes-evidence: Erdős-1051, centaur math discovery, sporadic Move-37-style moments. No-evidence: low yield, framing dependence, absence of acceleration. The mixed signal is the honest read.
The data supports two readings. Pessimistic: rare moments suggest creative insight is qualitatively distinct from engineering work. Optimistic: rare moments are an artifact of low-volume exploration; more shots on goal yields more discoveries. Both readings are consistent with Clark’s “vast swatches, perhaps the entirety” claim. They differ on the residual.
AI Tools for Finance and Accounting Professionals: Automate Tasks, Save Hours, Work Smarter
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five dimensions Clark gestures at but leaves underdeveloped.
Clark’s section is rigorous on the empirical evidence. Five strategic dimensions matter for the institutional response that the Clark series synthesis argues is structurally inadequate.
AI Business Automation: The Practical Guide to Building Self-Running Workflows and Scaling Fast
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Two readings. Different equilibria.
The structural question Clark leaves open: is research a permanent moat that bounds automated AI R&D, or is it engineering at scale that dissolves with more shots on goal? Both readings are consistent with the current data. They differ by orders of magnitude in consequences.
Productivity multiplier years
Recursive loop operational
Recent Advances in Artificial Intelligence in Cost Estimation in Project Management (Artificial Intelligence-Enhanced Software and Systems Engineering, 6)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five audiences. Asymmetric cost of being wrong.
The institutional response should not bet on inspiration being a permanent moat. If the distinction holds, capacity built is still useful. If it closes, capacity is necessary. Asymmetric cost-of-being-wrong points toward building now.
IN INDUSTRY
IN ACADEMIA
POLICYMAKERS
INVESTORS
EVERYONE ELSE
Engineering is automated. The residual is the question. The institutional response should not bet on inspiration being a permanent moat.
Implications for AI Development and Research Processes
This rapid automation of engineering tasks could dramatically reduce the time and cost required to develop new AI models, shifting the focus toward research and innovation. If research activities also become automatable, the pace of AI advancement could accelerate further, potentially leading to a new era of recursive self-improvement. However, the current limits of automation in research mean that human insight and creativity remain relevant, at least for now.
Progress Patterns in AI Capabilities and Benchmarks
Since early 2024, multiple independent benchmarks have shown consistent progress toward saturation, indicating that AI systems are approaching or have reached human-level competence in core engineering tasks. The CORE-Bench, MLE-Bench, and kernel design research demonstrate a clear pattern: capabilities are improving rapidly and nearing the measurement limits of existing benchmarks. These developments are part of a broader trend of AI systems becoming more capable of handling complex, research-relevant tasks without human intervention.
“The bottleneck on reproducing existing research papers is no longer ‘can it be reproduced.’ It’s ‘should it be reproduced.’ When an AI agent can take a paper and run its experiments at 95.5% reliability, the marginal cost of reproduction drops dramatically.”
— Thorsten Meyer
Unresolved Questions About AI Research Automation
It remains unclear how much of the research process—beyond engineering tasks—can be automated, especially aspects involving hypothesis generation, creative problem framing, and theoretical insights. The structural question left open by Clark and Meyer is whether research itself is simply scaled engineering, which could close the residual gap faster than expected. Details of how AI might autonomously conduct novel research are still emerging and subject to debate.
Next Steps for AI Capability Development and Benchmarking
Within the next 12 to 24 months, researchers expect further improvements in automation benchmarks, potentially reaching full automation of engineering tasks. Efforts will focus on refining benchmarks, understanding the limits of AI in research activities, and exploring the implications for AI development cycles. Monitoring these developments will be critical to assess whether research automation follows engineering automation’s rapid progress.
Key Questions
What does automation of engineering tasks mean for AI development?
It means that much of the manual, repetitive work involved in building and testing AI models can be handled by AI systems, potentially reducing costs and development time significantly.
Will AI fully automate research activities soon?
It is not yet clear. While engineering tasks are approaching full automation, research involves creative and theoretical components that may require human input for the foreseeable future.
What are the risks of automating AI research?
The main concerns include loss of human oversight, potential biases in automated hypotheses, and the challenge of ensuring AI-generated research is valid and safe.
How reliable are current benchmarks in measuring AI progress?
They are approaching saturation, indicating that AI capabilities are nearing human-level performance in these specific tasks, but they may not capture all aspects of research or engineering complexity.
Source: ThorstenMeyerAI.com
