🔍 Read the full analysis: Inside My Test Of 71 AI Configurations For A News Pipeline on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Thorsten Meyer says he tested 71 AI configurations on a news-writing pipeline between September 8 and October 13, 2026. GPT-6 Astra scored highest, but he kept the much cheaper GPT-6 Luna in production; the test also led him to remove two fallback models that fabricated quotes or failed output checks.
Thorsten Meyer says a test of 71 AI configurations for his news-writing pipeline found that GPT-6 Astra scored highest, but he kept the far cheaper GPT-6 Luna as his production model. The test also identified two backup models that, according to Meyer, invented quotes or failed to produce usable articles; both were removed from the news route on October 7.
Meyer conducted the test from September 8 to October 13, 2026, using the production prompt and real news sources. The configurations included API and subscription models, free-tier options accessed through OpenRouter, and open-weight models run on his own hardware. Each source was turned into an article in English and German, then scored by two AI judges: GPT-6.1 Sol and Claude Opus 5.5. They rated source fidelity, fabrication, epistemic clarity, news value, structure and polish, along with German language quality.
In the standard test, GPT-6 Astra scored 4.00 out of 5 in both languages and received a 5.00 source-fidelity score. Across a larger robustness run of 12 articles, it scored 3.93, with a reported 95% bootstrap interval of 3.83–4.02. Meyer put Astra’s cost at about $0.12 per article and its generation time at around 50 seconds. GPT-6 Luna scored 3.67 in English and 3.68 in German in the standard test; its robustness-run score was 3.67. Meyer reported a cost of about $0.0012 per article in the test and about $0.0013 in production, with a median generation time of 17 seconds. Every Luna article in the robustness run was valid under the pipeline’s output checks.
The test also found that more reasoning effort did not produce a clear quality gain for Luna. At the highest setting, an article cost about 12 times more and took about 12 times longer than at “none,” while scores remained within the reported range of variation. Meyer said the small apparent advantage at “high” effort was not sustained as a meaningful difference in the robustness run.
Infographic 01 · Who writes the best news article?
The best writer is not the one in production
71 model configurations, one real job: turn a news source into an article for an English and a German edition. GPT-6 Astra writes the best articles. GPT-6 Luna writes them for about a hundredth of the price.
Robustness run: 12 real news sources, English edition
Dot = mean score (1–5), bar = 95% bootstrap interval. German edition mean below each score. Overlapping bars are not a real difference.
Two judges, two caveats: GPT-6.1 Sol and Claude Opus 5.5 grade every article and were also tested as writers, so they partly grade themselves. Claude models ran with effort medium unless noted.
Source: Thorsten Meyer AI model-scout benchmark, 8 Sep – 13 Oct 2026. Task: write a news article from a real source with the production prompt, English and German editions. Judges: GPT-6.1 Sol (reasoning high) and Claude Opus 5.5, both with the source in context; score 1–5, mean of both judges. Robustness run: 2 test articles plus 10 more real sources, mostly short like production; mean with 95% bootstrap interval.
Powered by Thorsten Meyer AI
Cost and Accuracy in the News Pipeline
The results illustrate why a model leaderboard may not answer the practical question of which system to use for a specific publishing task. Meyer’s test suggests that the top-scoring model and the production choice can differ when quality, cost and speed are weighed together. Astra performed best in the reported evaluation, but its price was about 100 times Luna’s, while Luna produced consistently usable articles at a much lower reported cost.
The fallback findings have a direct quality-control implication. Meyer said the two removed models produced invented quotations or invalid output, problems that can pass beyond a model evaluation if a pipeline relies on backups during outages. The report supports testing the entire route, including failover models, against the same sources and validation rules used in production. It does not establish how other publishers’ systems would perform.
AI news article generation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How the Models Were Tested
The test used two source texts: a longer product filing of 6,340 characters and a short political news item of 619 characters. Meyer said most of his news begins with short sources, making the shorter item a more representative stress test for his workflow. The strongest configurations then completed a robustness run using 10 additional articles, mostly based on short sources.
Meyer counted an article only if it produced valid output after the same repair step used in production. He also cautioned that, with just two articles in the initial comparison, score differences below about 0.3 points should be treated as noise; he regarded the robustness run as the more reliable ranking. The report is a test of one publisher’s prompt, sources and pipeline, rather than an independent, broad benchmark of all news-writing tasks.
Model performance did not map neatly from one task to another. Meyer reported that Claude Opus 5.5, which he describes as the leading model on a coding benchmark, scored 3.46 in English and 3.39 in German as a news writer, with source fidelity of 3.75. Claude Sonnet 5.5 scored 3.67 and 3.61 respectively, but its API-equivalent cost was about $0.036 per article, around 30 times Luna’s reported cost.
“Extra thinking tokens don’t buy discipline.”
— Thorsten Meyer, describing the reasoning-effort results
AI language model API subscription
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Reported Results
The findings come from one publisher’s workflow and use a limited set of source material: two articles in the standard comparison and 10 additional articles in the robustness run for the strongest configurations. The report does not provide enough evidence to conclude that the same model ranking would hold for other prompts, languages, news beats or publishing systems. Meyer reports 95% bootstrap intervals for the robustness results, but the test remains specific to the material and scoring approach described.
The judges were AI models, and one of them, Claude Opus 5.5, was also evaluated as a writer. The report does not specify whether independent human editors assessed the articles or how the two judges’ scores were combined. It also does not provide a full configuration-by-configuration results table in the supplied material. Meyer’s claims about fabricated quotations and failed output are attributed to his evaluation; the source examples and independent verification are not included here.
The supplied account says GPT-6 Luna now runs first in the news route, but its description of the replacement fallback is incomplete: it ends with “GPT-6 Luna as its”. The identity and performance of the current secondary fallback are therefore not established by the supplied report.
As an affiliate, we earn on qualifying purchases.
Production Checks After the Test
Meyer says his production route now uses GPT-6 Luna as the primary model, following the removal of GPT-4.1 nano and gpt-oss-120b on October 7. The next practical test is whether Luna continues to meet the pipeline’s quality and validity requirements on new, real-world sources, particularly the short items Meyer says are common in his work.
The report does not announce a date for another benchmark or name the current backup model. Further testing could clarify whether the cost and quality balance holds over a larger and more varied set of articles, and whether the revised failover route avoids the fabrication and formatting issues Meyer reported. Until then, the published results should be read as a specific production evaluation, not a general ranking of AI systems for journalism.
As an affiliate, we earn on qualifying purchases.
Key Questions
Which configuration scored highest in Meyer’s test?
GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test, with a source-fidelity score of 5.00. It scored 3.93 across 12 articles in the robustness run.
Why did Meyer keep GPT-6 Luna in production?
Meyer reported that Luna scored 3.67 in the robustness run and that all its articles in that run were valid. At about $0.0012–$0.0013 per article, it was far cheaper than Astra, which he priced at about $0.12 per article.
Did higher reasoning effort improve the articles?
Not clearly in the reported test. Luna’s scores across reasoning settings were within the range Meyer treated as noise; the maximum setting cost and took about 12 times more than the “none” setting without a reported quality gain.
What happened to the two fallback models?
Meyer said GPT-4.1 nano received a source-fidelity score of 1.00 and produced invented quotes, while local gpt-oss-120b produced no valid articles out of four. He said both were removed from the news route on October 7.
Do the results show which AI model is best for every newsroom?
No. The test used Meyer’s own prompt, sources, languages and output checks. It provides evidence about his pipeline, but does not establish that the same rankings apply to other newsrooms or assignments.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
