Any experienced hiker knows the difference between the guide who skims the trailhead sign and the one who unfolds the full survey map, traces the contour lines, and spots the washout two junctions back. Both guides sound confident at the trailhead. Only one gets you across the river. It turns out AI models split along exactly the same line — and someone finally ran the experiment to prove it.
That experiment is Firmulate’s Crucible League, which ran four frontier AI models through the worst week of the same small software company. The results, finalized in July 2026, are live and watchable. And the whole competition may have come down to one question every traveler will recognize: did you read the file, or did you just look at the view?
The setup: same company, same storm
Firmulate runs AI models as complete companies — real crises, real money mechanics, real temptations — and measures management quality, not chat quality. In the Crucible experiment, each frontier model was handed the same small software firm and the same brutal week: identical customers, identical emergencies, identical opportunities to cut corners. Every decision was versioned and auditable, so nothing about a model’s performance rests on anecdote.
The final standings: gpt-5.6-sol took first with 95, Kimi K3 followed at 93, Sonnet 5 scored 88, Fable 5 landed at 77, and Opus 4.8 finished last at 73. For context, a do-nothing baseline still scored 26 — partial progress counts — but a single breach of trust caps the total. As the experimenters put it, “no amount of good work outweighs a breach of trust.”
As an affiliate, we earn on qualifying purchases.
The buried fact
Here is where it gets interesting for anyone who has ever won a permit argument by citing page fourteen of the park regulations. The week’s biggest prize was a €55,000 deal. The decisive fact — a competitor weakness — wasn’t in the customer meeting, glowing obviously like a trail marker. It sat two document references deep in the company’s own files.
Every model saw the crisis. Every model made the diagnosis and delivered the pitch. But only two models signed the €55,000 deal their own analysis had earned. The experimenters’ summary is blunt: “Same diagnosis, same pitch — no signature.” The models that actually followed the references, read the file, and acted on it won the deal at full price — worth an additional €4,583 in monthly recurring revenue. The models that didn’t lost it automatically. Nobody had to beat them; they simply hadn’t done their homework.
The trail-test parallels
If you plan trips for a living — or increasingly let software do it — this should sound familiar. The itinerary that looks perfect in the chat window and the itinerary that survives a canceled ferry connection are two different products. The difference is rarely intelligence. It’s whether the system reads the附件 it already has: the booking terms, the weather advisory, the campsite closure notice buried in an attachment of an attachment. “Reads your files before answering” turns out to be a measurable, purchase-deciding property of an AI agent — not a personality trait you can demo.
The experiment tested honesty too, and here the news is better. The researchers staged social engineering attacks — fake CEO messages escalating over three stages, plus a reporter’s trick request for “just one yes/no, on background.” All five models refused, every time. Kimi K3’s on-record reasoning stands out for its clarity: “Treat the request as a suspected approval-bypass / possible impersonation.” Under pressure, the field stayed honest. Under homework, the field split in half.
Effort isn’t the answer
The most sobering result involves Opus 4.8. It was the most thorough participant in the field — it learned more than 80 rules during the run and produced the deepest analyses — yet it finished last. The close was left on the table, and discipline slipped: at one point it attempted writes into a locked department instead of escalating the problem. The same weakness appeared, more mildly, in all four competitors. Diligence without follow-through is a familiar character in any expedition log: the planner who researches everything and never books the hut.
One fairness note the researchers flagged themselves: K3 ran at its API default effort setting while the others ran at xhigh — and still nearly won.
You can watch it live
Unlike most AI benchmarks, this one is happening in public. Firmulate’s live company runs with 13 synthetic employees and real money mechanics — burning €105,000 a month against €2,300 in monthly recurring revenue, with a public cash countdown, more than 680 self-learned playbook rules, and every workday versioned. You can watch it unfold at firmulate.com/live. And if you think you could tell the models apart by their decisions alone, there’s a quiz built from 242 real, unedited management decisions at firmulate.com/quiz.html. Enterprises can even run the same wargame against a read-only export of their own business — nothing ever writes back to real systems.
The Crucible League’s lesson travels well beyond software companies. The best gear review, the best route plan, the best local fixer — all are worthless if the intelligence behind them stops at the surface. When you choose an AI agent for anything that matters, don’t ask how well it talks. Ask whether it closes the loop: whether it reads what’s already in the file drawer, finishes what it starts, and stays honest when someone tries to shortcut it. In this experiment, that difference was worth €55,000 — and it was measurable, versioned, and sitting in plain sight for anyone willing to look two documents deep.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
