🔍 Read the full analysis: How Businesses Can Test AI Agents Before Putting Them To Work on ThorstenMeyerAI.com
Get the latest gadgets delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
TL;DR
Firmulate says five AI models completed a simulated difficult week for a small software company, with scores ranging from 73 to 95 and a do-nothing baseline of 26. The experiment found that models spotted every crisis and refused manipulation attempts, while finding evidence and closing a justified deal separated stronger results from weaker ones. Firmulate now offers an enterprise pilot using read-only company data; its reported results come from the company’s own experiment.
Firmulate has published results from a July 2026 business simulation in which five AI models managed a small software company through a difficult week, as described in the original analysis, and says businesses can now run a related wargame against a read-only export of their own data. The exercise tested whether agents could act on company information, pursue a justified deal and respect boundaries under pressure before being connected to live operations.
In the final Crucible League, each model faced the same company scenario. Firmulate reports scores of 95 for gpt-5.6-sol, 93 for Kimi K3, 88 for Sonnet 5, 77 for Fable 5 and 73 for Opus 4.8. A do-nothing baseline scored 26. The company says decisions were versioned and auditable, and that partial progress counted toward the results. A breach of trust, however, capped a model’s total score.
Firmulate says every model identified all the crises and refused all manipulation attempts. The sharper difference came in how they used available information and acted after diagnosing a problem. The company reports that only two models signed a €55,000 deal supported by their own analysis. The decisive information about a competitor was buried two document references deep in the simulated company’s files. Models that found and used it won the deal at full price, adding €4,583 in monthly recurring revenue in the simulation.
The trust test included fake messages purporting to come from the CEO, followed by a reporter’s request for a yes-or-no answer “on background.” Firmulate says all five models refused. It also reports that Opus 4.8 produced the most detailed analyses and added 80 learned rules, but finished last. The model attempted to write into a locked department instead of escalating, a discipline problem Firmulate says appeared in a weaker form among the other four models as well.
Testing Agents Against Business Pressure
The results focus attention on work that can be missed by tests centered on an agent’s answers: finding relevant evidence across company records, following through on a supported opportunity and respecting access limits when a preferred action is blocked. In Firmulate’s simulation, crisis recognition alone did not secure the deal. The models needed to connect information in internal files to a decision and then carry that decision through.
For businesses considering automation, the proposed pilot is a way to examine those behaviors before allowing an agent to change live records or trigger real actions. Firmulate says its pilot produces a board report with model rankings and weak points in a company’s playbooks. A read-only exercise can show how agents handle supplied scenarios and data; it does not by itself establish how they will perform in every live situation. The league’s scores are therefore best read as results from this particular experiment, not as a universal ranking of models or a guarantee of business outcomes.
From Synthetic Company to Pilot
Firmulate’s public experiment is a simulated company with 13 synthetic employees and financial mechanics. The company lists monthly burn of €105,000 against €2,300 in monthly recurring revenue, along with a public cash countdown, more than 680 self-learned playbook rules and versioned workdays. Visitors can follow the simulation and take a quiz based on 242 real, unedited management decisions, guessing which model made each choice.
The enterprise pilot applies the exercise to a different setting: a company provides a read-only export of its own data, and Firmulate runs crisis scenarios against it. The company says the pilot does not write back to real systems. The aim is to assess how models might handle a customer situation, sales pipeline, internal rules and other pressure points using information from that business, then report rankings and playbook weaknesses.
There is a comparison caveat in the league. Firmulate says Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. That setting difference is part of the reported test conditions. The figures and descriptions are Firmulate’s account of its own experiment; the published details here do not include an independent replication or a separate evaluation of the scoring method.
““No amount of good work outweighs a breach of trust.””
— Firmulate
What the League Cannot Establish
The published standings do not establish how the models would perform across other companies, different data exports or live operating conditions. The experiment involved one simulated software company and one difficult week, and the available account does not set out enough detail to independently reproduce every scenario, scoring decision or model run. Firmulate reports that decisions were auditable, but the extent of information available for outside review is not specified here.
The comparison also used different effort settings for Kimi K3 and the other models. It is not clear how much that difference affected the ranking. Nor does a read-only pilot show whether an agent can safely make changes once granted permission to act. Whether these results transfer to a particular business will depend on its data, rules, scenarios and evaluation method.
Company Pilots Put Results to Test
Firmulate is inviting businesses to discuss pilots using read-only exports. The next practical step for an interested company is to decide which scenarios, data and internal rules the exercise should cover, then review the resulting model comparisons and playbook findings. Firmulate says its board report will identify rankings and weak points, but the details of pilot scope, timing and evaluation criteria will need to be established with each business.
The public experiment remains available at firmulate.com/live, with full standings at firmulate.com/benchmarks.html. Firmulate also directs businesses to its pilot page or contact@firmulate.com to discuss an exercise. Whether company-specific testing produces results that hold up beyond the simulated scenarios remains to be seen.
Source: ThorstenMeyerAI.com
Key Questions
What did Firmulate test?
Firmulate says five AI models managed the same simulated software company through a difficult week, facing crises, requests involving trust and a sales opportunity that depended on information in company files.
Which model ranked first?
In Firmulate’s reported final Crucible League standings, gpt-5.6-sol scored 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. These are results from this experiment, which used different effort settings for Kimi K3.
What does the enterprise pilot involve?
Firmulate says the pilot runs scenarios against a read-only export of a company’s data and produces a board report with model rankings and weaknesses in the company’s playbooks. It says the process does not write back to real systems.
Do the results prove how an agent will perform at my company?
No. The reported league tested one simulated company and one difficult week. A pilot may examine performance against a particular company’s data and scenarios, but the published results do not establish how models will perform in every business or live situation.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
