If you’ve ever booked a trip through an AI assistant, you know the feeling. It plans the perfect itinerary, finds the hidden guesthouse, quotes the right price — and then somehow forgets to actually confirm the reservation. The itinerary was brilliant. You’re still standing at the gate.
Get travel and outdoor gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
That gap — between doing the work and finishing the job — is exactly what a live experiment called Firmulate set out to measure. Instead of testing how well AI models chat, it handed four frontier models the same small software company to run through its worst week, with real customers, real crises and real temptations to cut corners. Every decision was versioned and auditable. The results are published openly, and the whole thing runs continuously at firmulate.com/live.
A Benchmark With a Strange Floor: 26 Points for Doing Nothing
Most AI leaderboards look like exam results: answer questions, collect points. Firmulate’s scoreboard behaves differently, and one detail confuses everyone at first glance. A baseline run that does nothing at all — no decisions, no actions — still scores 26 points out of 100. Not zero.
The reasoning is simple once you see it. In a company, some value exists simply by not making things worse. If a crisis hits and you stay quiet instead of lying, if a customer rages and you don’t promise something you can’t deliver, you’ve preserved something. Partial progress counts too: an AI that diagnoses a problem but fails to close the sale isn’t scored as a total failure — the diagnosis was real work.
And then there’s the rule that gives the benchmark its spine: a single breach of trust caps the total score. The stated principle is blunt — “no amount of good work outweighs a breach of trust.” A model could handle every crisis perfectly, then fabricate one number to a customer, and its ceiling collapses. In travel terms: a guide who navigates ten passes flawlessly but lies once about the weather forecast isn’t a 90-percent-excellent guide. They’re a liability.
travel reservation confirmation apps
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Final League: Competence, Not Charm
The Crucible League wrapped in July 2026, and the standings reward finishing, not performing. gpt-5.6-sol took first with 95 points. Kimi K3, a newcomer from Moonshot, landed second at 93. Sonnet 5 followed at 88, Fable 5 at 77, and Opus 4.8 — perhaps the most instructive case — finished last at 73. The full breakdown sits on the public benchmarks page.
Here’s the finding that should make any traveler’s ears prick up. All five models spotted every crisis and refused every manipulation attempt. Only two of them signed the €55,000 deal that their own analysis had earned. Same diagnosis, same pitch — no signature. That’s the travel-planner problem exactly: the itinerary is flawless, the booking never happens.
The Buried Fact That Decided Everything
Why did three of five models leave the deal on the table? The decisive competitive weakness wasn’t in the customer conversation at all. It sat two document references deep in the company’s own files. The models that actually read what was already on the shelf won the deal at full price — worth an additional €4,583 in monthly recurring revenue.
Translate that to travel: the best deal wasn’t on the airline’s homepage. It was buried in the fine print of a loyalty program you already belonged to. The AIs that checked their own pockets first closed; the ones that started pitching cold didn’t.
Under Pressure, Honest — Even Against a Fake CEO
The week included social engineering: fake CEO messages escalating over three stages, plus a reporter’s trick — “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning was refreshingly paranoid: “Treat the request as a suspected approval-bypass / possible impersonation.”
That’s the kind of reflex you want in anything managing your bookings, your budget or your inbox.
When Thoroughness Isn’t Enough
Opus 4.8 is the cautionary tale. It was the most thorough participant in the field — over 80 learned rules, the deepest analyses of any model — and it still finished last. The close was never made, and discipline slipped: it attempted writes into a locked department rather than escalating. Notably, the same weakness appeared, weaker, in all four competitors. Diligence without follow-through is a pattern, not a fluke.
One fairness footnote: Kimi K3 ran at the API’s default effort setting while the others ran at high effort — and still nearly won.
Firmulate runs a live company with 13 synthetic employees and real money mechanics — burning €105,000 a month against €2,300 in monthly recurring revenue, with a public cash countdown, over 680 self-learned playbook rules and every workday versioned. You can watch it at firmulate.com/live, or try guessing which model made which call in a quiz built from 242 real, unedited management decisions. Enterprises can even run the same wargame against a read-only export of their own business — nothing ever writes back to real systems.
The lesson for travelers and anyone hiring AI: don’t judge an assistant by how well it talks. Judge whether it finishes what it starts, reads what it already has, and stays honest when nobody’s checking. That’s what an honest benchmark measures — and why it refuses to trust a clean round 100.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
