Every Benchmark Launched 2023-2024 Has Fallen — The METR / SWE-Bench / CORE-Bench / MLE-Bench / PostTrainBench Sequence

📊 Full opportunity report: Every Benchmark Launched 2023-2024 Has Fallen — The METR / SWE-Bench / CORE-Bench / MLE-Bench / PostTrainBench Sequence on ThorstenMeyerAI.com — validation score, market gap, and execution plan.

All six major benchmarks designed to measure AI research and development capability, launched between 2023 and 2024, have now saturated or are approaching saturation within months, according to recent analysis by Thorsten Meyer.

These benchmarks include metrics for software engineering, model training speed, research reproduction, and AI fine-tuning, among others. Notably, the SWE-Bench, which measures software engineering skills, has reached 93.9% accuracy after 30 months, up from 2%, and is now considered saturated, according to the benchmark’s authors.

Similarly, the METR time horizon benchmark, which assesses the duration of AI tasks, has expanded from 30 seconds in 2022 to 12 hours in 2026, a 1,440-fold increase, with the trajectory indicating it will reach approximately 100 hours by the end of 2026. The CORE-Bench, used for research reproduction tasks, was declared solved by its authors after reaching 95.5% accuracy in December 2025, just 15 months after initial measurement.

Additional benchmarks tracking end-to-end ML engineering and AI fine-tuning show similar rapid progress, with improvements occurring over 16-18 months. These patterns suggest that the capabilities measured are reaching their upper limits on current evaluation methods, implying a saturation point in AI research benchmarks.

Implications of Rapid Benchmark Saturation for AI Progress

The saturation of all six major AI benchmarks launched recently indicates that AI systems are rapidly approaching or have already achieved the highest levels of performance these tests can measure. This pattern suggests that AI research capabilities are advancing faster than many industry observers predicted, raising questions about the trajectory of AI development and the potential for near-term breakthroughs in autonomous research, software engineering, and model training.

For policymakers, investors, and AI developers, these findings imply that the AI landscape may experience a phase of diminishing returns on traditional benchmarks, prompting a need to develop new evaluation metrics. It also underscores the urgency of considering regulatory and safety frameworks as AI systems approach human-level or beyond capabilities in specific tasks.

Recent Benchmark Developments and Their Significance

The six benchmarks analyzed were launched from late 2023 through 2024, each designed to challenge different facets of AI research and development, including software engineering, model training, research reproduction, and AI fine-tuning. Their rapid saturation over the past two years reflects a consistent pattern of exponential progress, driven by improvements in model architectures, hardware acceleration, and data availability.

Prior to these developments, AI progress was often measured by slower, incremental improvements. The current pattern suggests a structural shift, with capabilities reaching upper limits on existing evaluation frameworks in a matter of months rather than years.

Experts like Jack Clark have argued that this pattern supports a forecast of AI capability reaching 60% of human-level research competence by 2028, challenging previous assumptions about the pace of AI development.

“The pattern of saturation across all six benchmarks within a short timeframe indicates a fundamental shift in AI development speed, suggesting we’re approaching the limits of current evaluation methods.”

— Thorsten Meyer

Uncertainties About Long-Term AI Capability Trajectory

While the benchmarks show rapid progress and saturation, it remains unclear whether these results fully translate to real-world AI capabilities beyond these specific tests. Some experts question whether current benchmarks can continue to measure meaningful improvements or if saturation indicates a ceiling in current evaluation metrics rather than true AI limits.

Additionally, the potential for future breakthroughs or shifts in AI research methodology that could disrupt these patterns is still unknown. The impact of hardware limitations, data constraints, or novel architectures on future progress has not yet been fully assessed.

Next Steps and Future Benchmark Developments

Researchers and industry leaders are expected to develop new benchmarks to evaluate AI capabilities beyond current saturation points. Monitoring how new tests are designed and how AI systems perform on emerging metrics will be crucial to understanding whether these saturation patterns continue or if new frontiers emerge.

Further analysis is also anticipated to determine whether the observed saturation reflects genuine limits or artifacts of existing evaluation frameworks. Policymakers and stakeholders should prepare for potential rapid shifts in AI capabilities as these new benchmarks are introduced and validated.

Key Questions

What does benchmark saturation mean for AI development?

It suggests that AI systems are reaching the upper limits of current evaluation metrics, indicating rapid progress but also potential plateaus in measured capabilities.

Are these benchmarks indicative of real-world AI abilities?

Not necessarily. While they measure specific skills effectively, it’s still uncertain if saturation in these tests equates to true AI competence in broader, real-world scenarios.

Will new benchmarks be developed to replace these?

Yes, experts anticipate the creation of new, more challenging benchmarks to continue assessing AI progress beyond current saturation levels.

How soon might AI systems surpass human-level research capabilities?

Based on current trends, some forecasts suggest AI could reach significant research capability levels by 2028, but this depends on future developments and evaluation methods.

What are the implications for AI safety and regulation?

The rapid advancement highlights the need for proactive safety and regulatory measures as AI systems approach or surpass human-level performance in key tasks.

Source: ThorstenMeyerAI.com