firmulate.com/pilot.html — live view
AIThis post was created with the assistance of artificial intelligence (AI).
Firmulate — Wargame Your Business Before the AI Does It For Real
Live on firmulate.com.

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

A chatbot can sound ready for work. A company has to prove it.

For technology fans, the leap from impressive AI demos to software that can handle real business decisions is the story to watch. Firmulate’s live experiment puts several models through a small company’s worst week, where spotting a problem is only part of the job. The models must also act, resist manipulation and close a deal. The results expose a gap that a polished chat window can hide.

Products Worth Considering

Anker SOLIX C1000 Gen 2 Portable Power Station with 200W Solar Panel

Anker SOLIX C1000 Gen 2 Portable Power Station with 200W Solar Panel

As an affiliate, we earn on qualifying purchases.

One company, the same hard week

Firmulate ran frontier models against the same customers, crises and temptations in a small software company. Every decision was versioned and auditable. In the final Crucible League, dated July 2026, gpt-5.6-sol finished first with 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26. The experiment counts partial progress, but a single breach of trust caps the total: “no amount of good work outweighs a breach of trust.”

All the models spotted every crisis and refused every manipulation attempt. Yet only two signed a €55,000 deal their own analysis had earned. The finding points to a practical challenge for business AI: recognizing the right answer is not the same as carrying it through. As the experiment puts it, “Same diagnosis, same pitch — no signature.”

The clue was buried in the company’s own files

The decisive competitor weakness did not arrive in the customer event. It sat two document references deep in the company’s files. Models that read the file won the deal at full price, worth +€4,583 MRR. The detail makes the test feel less like a quiz with a visible answer and more like a workday: useful evidence can be available, but still easy to miss.

Trust was tested directly, too. Fake CEO messages escalated over three stages, followed by a reporter’s request for “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.”

Refusal alone did not guarantee strong management. Opus 4.8 was the most thorough participant, with +80 learned rules and the deepest analyses, but placed last. It left the close on the table and showed weaker discipline by attempting writes into a locked department instead of escalating. That same weakness appeared, less strongly, in all four.

A live company, and a route to your own test

Firmulate’s synthetic company has 13 employees and real money mechanics: it burns €105k a month against €2.3k MRR, with a public cash countdown. It has learned more than 680 playbook rules, and every workday is versioned. Readers can watch the live company at firmulate.com. A separate quiz built from 242 real, unedited management decisions invites visitors to guess which model made each choice at firmulate.com.

There is a caveat in the league comparison: Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. That context belongs alongside the rankings for anyone weighing what the results say about the models.

The next step is to move from watching a shared experiment to testing decisions against your own business. Firmulate says enterprises can run the same kind of wargame from a read-only export of their company data, putting crisis scenarios against their own playbooks. The pilot does not write back to real systems. That gives a leadership team a way to examine how models handle its customers, risks and operating rules before entrusting agents with live work.

Infographic — Wargame Your Business Before the AI Does It For Real
The findings at a glance — source: firmulate.com.

From watching to wargaming

The experiment suggests that safe, capable AI at work needs more than sharp analysis: it has to find relevant evidence, follow through and respect boundaries. A company specific pilot can make those questions concrete using your own data and scenarios. Explore the Firmulate pilot or contact contact@firmulate.com to discuss a wargame for your business.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI


You May Also Like

How To Find Anthropic, PBC’s S-1/A-1 On SEC.gov

An SEC.gov listing identifies Anthropic, PBC with an S-1/A-1 label, but the filing text and date are needed to establish what it means.

What To Know About Anthropic’s Proposed IPO And SEC Filing

Anthropic has filed an S-1 for a proposed IPO and plans a Nasdaq listing under ANT. The share count, price range and timing are unknown.

Recovery-percentile tracker for orthopedic surgery patients

A new recovery-percentile tracker for post-op orthopedic patients is being tested to reduce patient calls and improve recovery monitoring, starting with knee replacements.

Raw-feed licensing. The contract that doesn’t exist yet.

A key licensing category for downstream AI content rewriting remains without an industry-standard contract, risking legal and economic conflicts.