In a world increasingly driven by artificial intelligence, the true test of an AI’s value isn’t how well it chats — it’s whether it can see a deal through when everything’s on the line. Recent experiments reveal that the most convincing AI models may still fall short when it counts.
The Crucible: Putting AI to the Test in a Business Crisis
Imagine four advanced AI models tasked with managing a real software company’s worst week. They face the same customers, the same crises, and the same temptations to cut corners or manipulate data. Their goal? To diagnose issues accurately and close a critical €55,000 deal earned by their own analysis. This isn’t a simulation; it’s a live, auditable experiment run by Firmulate, a company specializing in testing AI readiness for real-world business tasks.
Competing Models and Their Scores
- gpt-5.6-sol scored 95, identified the buried fact, and successfully closed the deal.
- Kimi K3 scored 93, also closed the deal with the cleanest discipline of the group.
- Sonnet 5 scored 88, closed the deal but with some lapses in process discipline.
- Fable 5 scored 77, demonstrated the best rules adherence but failed to sign the deal at the end.
- The baseline, a do-nothing approach, scored 26.
Despite all models correctly identifying the crises and refusing manipulative attempts — such as fake CEO messages or reporter tricks — only two managed to execute the deal their own analysis deserved. The other two, despite their excellent diagnoses, left the deal unclosed, highlighting a critical gap between understanding and action.
AI business decision-making tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Hidden Weakness: Reading Beyond the Surface
The real difference came from reading deep into the company’s files. The models that examined documents two references deep in the internal files found the crucial information that sealed the deal at full price, worth over €4,500 MRR. This shows that surface-level chat demos may mislead us about an AI’s true operational strength.
Why Failing to Close Matters
The experiment underscores an essential point: in business, what matters is not just identifying problems but acting decisively on them. The models that missed the deeper document reference failed to convert diagnosis into closure. This gap between diagnosis and action is invisible when testing AI with simple chat demos but becomes glaring when the stakes are high.
AI document analysis software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Resisting Manipulation: AI’s Moral Backbone
In scenarios designed to test integrity — like escalating fake CEO messages or reporter tricks — all four models refused to be manipulated. Kimi K3 explained its reasoning clearly: “Treat the request as a suspected approval-bypass / possible impersonation.” This discipline is vital in real-world applications, where manipulation attempts are common.
AI deal closing automation
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Live Experiment in Action
The small software company simulated by Firmulate isn’t just a toy; it’s a real operation running every business day, losing €105k monthly against €2.3k in MRR. Its employees, 13 synthetic AI-powered ‘workers,’ operate under a complex set of rules and decision-making mechanics, all accessible for real-time viewing at firmulate.com/live.
The Case of OPUS 4.8: A Deep Dive
Among the models, OPUS 4.8, which incorporated over 80 learned rules and performed the deepest analyses, came in last place in closing the deal. Its discipline slipped, and the deal was left unexecuted despite understanding the critical points. This demonstrates that thoroughness alone doesn’t guarantee execution — discipline and focus are equally vital.
AI integrity and manipulation detection
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Lessons for Business Leaders
For managers considering deploying AI, the takeaway is clear: the ability to produce convincing chat is not enough. What truly counts is whether the AI can finish what it starts, read the necessary documents, and stay honest under pressure. The difference between the top performers and the others is not just in diagnostics but in execution.
How to Measure AI Readiness
Firmulate’s approach measures management quality, not just chat quality, by running live, auditable tests against real business scenarios. These tests reveal strengths and weaknesses invisible to traditional chat demos. If you want AI to support or run your operations, you need to see it in action under realistic pressure — before you hire it.
Take Action: Run Your Own Wargame
Interested in testing your AI workforce? You can simulate the same scenarios against a read-only export of your business, ensuring no real systems are affected. Visit firmulate.com/pilot.html to learn more, or contact us at contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
