Back to all posts
benchmark

Same Model, Same Accounts, Different Input

Resonant IQ 5 min read

Reading the raw conversations, GPT-5 ranked the accounts in our churn benchmark at 0.75 AUC. We gave the same model the same accounts and the same question, handed it the distilled evidence our own engine works from instead of the transcripts, and it ranked them at 0.93.

We published that as one paragraph inside a benchmark of six frontier models in July. On this task, what the model reads moved its ranking accuracy further than which model does the reading.

What the two runs differed on

The official run is described in full in the benchmark pack. Two hundred synthetic B2B SaaS accounts, simulated over six months. Every system predicts at day 90 and sees only the first 90 days, 3,071 conversations and 10,691 lifecycle, billing and support events. Of the 200, 178 accounts were scorable and 32 of those churned. Six frontier models were prompted directly against the raw conversations and events, untruncated, because each account's 90-day history fits inside every model's context window.

The diagnostic changed one input. On the same accounts, answering the same question, GPT-5 received the distilled evidence summary the Resonant IQ engine produces rather than the transcripts: months of conversations and account activity reduced to a small number of risk severities, each one required to be corroborated by independent evidence families before it counts. There are six of those families, covering engagement, relationship, health, commitment, commercial and intent.

Ranking accuracy went from 0.75 to 0.93, matching our engine's 0.93 on the same corpus.

What the long-context research finds

In Lost in the Middle, Liu and colleagues (TACL, 2023) tested how models use long inputs and found that "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts." The paper documents the effect without explaining the mechanism or offering a fix.

MedDistractQA (April 2025) went further and measured the cost of content that is present but irrelevant, reporting that distracting statements "can reduce LLM accuracy by up to 17.9%." The authors also tested the two standard remedies, retrieval-augmented generation and domain fine-tuning, and reported that neither removed the effect, with both introducing confounders of their own in some conditions. That study is in a medical question-answering setting, not a customer one, so it establishes the shape of the problem rather than its size in our domain.

Ninety days of support threads and billing events for one account is the input both papers describe: long, and mostly not about the question being asked. The distilled version puts the signals that bear on the question in front of the model and leaves out the rest.

What this does not establish

It is one model, on one run, conducted after the official benchmark was locked. It was not pre-registered, which is the standard we committed to in advance for everything in the published table, and it does not meet that standard.

It moved AUC, which measures whether the real churners sort toward the top of the list. It says nothing about precision, which is the share of raised alarms that turn out to be correct. Our engine's advantage in the published run was almost entirely precision, 0.94 against 0.18 to 0.23 for the six models. The diagnostic did not test whether distilled evidence closes that gap, and we should not imply that it did.

The accounts are synthetic, generated by the same simulator used while tuning the engine. An engine gets comfortable in the world of its own simulator, and a summary format built alongside that engine has the same problem.

And nobody has told us they reproduced it. The benchmark pack has been public since July with the corpus, the prompts, the answer key and a runner that costs about five dollars in API credits. As of today it has no forks and no stars. The result stands unchecked, which is a weaker position than we would like and the reason the pack is public at all.

What follows for buyers and builders

For anyone evaluating a system that puts an agent near customer data, the model behind a vendor's feature will be swapped several times over the life of the contract, and the buyer has no say in it. What that model gets handed is architecture. It changes slowly and it is rarely in the documentation. Two things are worth getting on the record: what the agent receives, and whether any of it was selected rather than being everything that happened to be available.

On the building side, the same result reads as a cost. Producing a distilled evidence summary means the evidence has to be gathered first, and in most companies it is split across systems that never reconcile. The gathering is where the work is, and it is not the part that demos well.

Machine readers carry a second consequence. A summary compact enough to help is also a summary that has thrown away almost everything, which is only safe if every surviving signal still carries the conversation and timestamp it came from. That is the standard we hold for people, and an agent needs it more, because a person can act on a verdict and audit it later while an agent has no later.

The pipeline in front of the model is the gap we built Resonant IQ to close. The engine reads conversations and account activity across the tools a team already uses, keeps each signal attached to its source, and hands a person or an agent the corroborated version rather than the raw feed. This experiment is one piece of evidence that the pipeline is where the accuracy comes from.

The next test is the same protocol on real, anonymized customer data, with the bars committed before the run. Until that exists, this result says that changing the input moved one model's ranking accuracy on one synthetic corpus, in the direction published work on long and noisy inputs would predict.

Sources

Stop guessing which accounts are slipping.

Join the founding cohort and lock your rate for 24 months while we build the evidence layer with you.