Six Frontier LLMs Called Churn on Almost Every Account. Ours Called It on the Right Ones.

Every one of the six frontier models we tested flagged nearly every account in our churn benchmark as at risk. Not most. Nearly all. All six models, independently. That sounds like safety. It isn’t. On this benchmark, a team acting on those flags would open roughly five investigations for every real churner underneath. An alarm that fires on everything isn’t an alarm. It’s noise you eventually learn to ignore, and the one time it’s right, you’ve already stopped listening.

Here’s why we ran this test. We claim our engine predicts churn better than simply asking a frontier AI model. That’s a convenient thing for us to believe, so we built a benchmark designed to prove us wrong: we locked the passing bars in before the test data existed, ran it once, and are publishing everything. The results are below, and the corpus and exact prompts are free to download, so you don’t have to take our word for any of it.

What we pre-registered, before we saw a single score

Before the test data existed, we wrote our success bars into a freeze manifest and committed it to our repository. A verbatim copy, with its commit hash, ships inside the benchmark pack, so you can check the bars against the results yourself. In plain terms, our engine had to:

• Rank real churners near the top of the risk list (AUC of at least 0.85)
• Be right at least 70% of the time when it raised an alarm, while still catching at least half of the real churners (precision ≥ 0.7 at recall ≥ 0.5)
• Beat every model tested on alarm accuracy (precision)

We also pre-registered a prediction about how we’d lose, if we lost: that LLMs would over-flag, trading precision for recall, calling risk on everything to avoid missing anything. That prediction turned out to be right for all six models. We’re not claiming credit for foresight; we’re pointing out that the failure mode was named in writing before the run, which is the only way “we predicted this” means anything.

No goalposts moved after the fact. What you’re reading is the same bar we set before we knew who’d clear it.

The setup

Both sides answered the same question, from the same account histories, on a corpus none of them had seen:

Will this account churn after day 90?

• 200 synthetic B2B SaaS accounts, simulated over six months. Every system predicts at day 90 and sees only the first 90 days of each account’s history (about 3,000 conversations across support, sales, and success touchpoints). The public pack ships exactly that observation window.
• A fixed set of test accounts, generated once and then locked so the test couldn’t be gamed (a “frozen holdout” in ML terms)
• The same underlying account history available to every system: the Resonant IQ engine consumed it through its evidence pipeline; six frontier LLMs (GPT-5, Claude Opus 4.8, Gemini 2.5 Pro, DeepSeek, Kimi K2, and GLM-4.6) were each prompted directly against the raw conversations and events, untruncated (each account’s full 90-day history fits comfortably inside every model’s context window)
• 178 of the 200 accounts were scorable; the other 22 had outcomes already settled by day 90, leaving nothing to predict. 32 of the 178 were real churners: far more than a normal customer base has, on purpose, so the comparison has statistical teeth.

That last point matters for how you read the numbers below. More on it in a minute.

The results

System	Precision	Recall	AUC
Resonant IQ engine	0.94	0.53	0.93
GPT-5	0.21	0.97	0.75
Claude Opus 4.8	0.18	1.00	0.78
Gemini 2.5 Pro	0.18	1.00	0.64
DeepSeek	0.23	1.00	0.70
Kimi K2	0.19	1.00	0.63
GLM-4.6	0.18	1.00	0.61

A note on reading these numbers: this test group deliberately contains far more churners (32 of 178) than a real customer base would have in one quarter. That makes the head-to-head comparison fair and meaningful, but it means the percentages describe this test, not the share of alerts that would be right at your company’s actual churn rate.

If you don’t live in these metrics daily, here’s the whole table in three sentences. Precision answers: when the system raises an alarm, how often is it right? (Ours: 94 times out of 100. The LLMs: about 1 in 5.) Recall answers: of the customers who really did churn, how many did it catch? (Ours: about half. The LLMs: nearly all, but only because they flagged nearly everyone, which is how you get perfect recall for free.) AUC answers: if you sorted every account by the system’s risk score, would the real churners be at the top of the list? 0.5 is a coin flip, 1.0 is perfect. (Ours: 0.93. Best LLM: 0.78.)

Read the two middle columns together and the shape is unmistakable. Every LLM caught almost every churner (recall of 0.97 to 1.00). And every LLM was wrong on roughly four out of every five accounts it flagged (precision of 0.18 to 0.23). Our engine caught about half the churners it was shown, and was right on 94% of the accounts it raised a hand on.

Recall alone would make the LLMs look like the better product. Precision is what reveals the cost: four false alarms for every real one.

A fair question at this point: why is our recall only 0.53? Doesn’t catching half the churners mean the system misses a lot? Here’s the thing to know: precision and recall pull against each other. Any system can catch every churner by flagging everyone, and the LLM rows show exactly what that costs. The real question is what a system does with the trade, and we deliberately tuned ours toward high precision, because an alert that’s right 94% of the time is one a team actually acts on. Two more things soften that 0.53 in practice. This table scores only our strictest alert tier; the product also keeps a watch list one level below it, which catches a larger share of churners at a still-useful confidence bar (that tier wasn’t part of this benchmark, so we’ll save its numbers for a future post rather than quote them here). And our diagnostic work identified exactly where the misses live (slow, gradual declines are the biggest group), which means recall improves by teaching the engine new evidence, not by lowering the bar and flooding the queue.

Why this is the product story, not just the stats story

A team that gets a churn alert on 100 accounts and finds 20 real fires under it stops trusting the alarm. Not immediately; the first few false alarms get worked in good faith. By the twentieth, the alert gets a glance and a shrug. The team is back to running on instinct, which is the exact state a churn tool was supposed to fix.

Recall is worth nothing if nobody acts on the flag it produced. Precision is what makes a flag worth someone’s Tuesday morning. That’s the whole difference between an alarm you can act on and an alarm you learn to ignore, and it’s a difference every one of these six models landed on the wrong side of, independently, without coordinating with each other.

Why we think the gap exists

We don’t think GPT-5 or Claude Opus 4.8 are bad at reasoning. We think one-shot judgment over raw conversation transcripts is a hard way to answer a question that’s really about corroboration over time: whether several independent signals (engagement decline, a champion going quiet, broken commitments, churn-intent language) agree with each other, not whether any single conversation sounds concerning.

Our engine doesn’t read a transcript and guess. It distills months of conversations and behavior into a small number of risk severities, then requires independent evidence families to corroborate before a severity counts. One tense support thread doesn’t move the score. One tense support thread plus a usage decline plus a quiet executive sponsor does.

After the official run was locked, we tried one more experiment to test that theory. We gave GPT-5 the same accounts again, but instead of the raw transcripts, we handed it the distilled evidence summary our own engine works from. The result: its ranking ability (the AUC column above) jumped from 0.75 to 0.93, tying our engine. Same model, same accounts. The only thing that changed was what it got to read. This experiment came after the pre-registered run, so we’re calling it what it is: an after-the-fact diagnostic, not an official result. But it points hard at one conclusion: what makes our engine win isn’t a smarter model. It’s the evidence pipeline in front of the model, the part that turns months of raw customer history into a handful of clear risk signals.

The honest caveat

The customers in this benchmark aren’t real. Every account is synthetic: generated to behave the way real accounts behave, with churn stories that build over months, not random noise with a label attached. That realism has limits. Synthetic data can be easier than real data, or harder, in ways nobody notices until someone checks it against the real thing.

There’s a second limitation we’d rather name ourselves before someone else does. The test accounts were brand new to our engine, but they came from the same simulator we used while tuning it. An engine can get comfortable in the world of its own simulator. Only a test on real customer data can settle that question.

Both limits are exactly why we’re giving the data away instead of asking you to trust our table. Running the benchmark yourself costs about $5 in API credits with any OpenAI-compatible model. If the numbers don’t hold up, we want to be the first to hear about it.

The next chapter is a backtest on real, anonymized customer data, published the same way as this one: success bars committed in advance, before we see a single score.

Get the benchmark pack

The full pack is free and public: the corpus, the exact prompts we used for every model, the ground-truth answer labels, and a one-file runner to reproduce the table above. Grab it directly from GitHub. If you want the follow-up post with the community’s results, leave your email and we’ll send it when it’s out.

The challenge: beat 0.94 precision at recall ≥ 0.5 on this corpus. One rule, since the pack includes the answer key for scoring: your predictions have to be generated without reading the labels file; it exists to score your run, not to write it. Share your run (model, prompt, and numbers) wherever you post: tag us, open an issue on the repo, or email us. We’ll feature the best attempts in the follow-up.