How to Test an AI QA Scorer Before You Trust Its Scores

More support teams now have a model scoring their conversations. It reads the transcript, checks it against each line of the scorecard, and returns a pass or fail with a reason. It can score every conversation instead of a handful a month.

Before those scores reach a coaching session or a team average, two questions need an answer. Does the scorer give the same verdict when nothing has changed? And when it does, does it agree with a careful human reviewer?

The score itself won't tell you. A 78 looks the same whether every criterion behind it is steady or two of them are flipping between runs.

Test 1: score the same conversations more than once

Pick ten conversations. Score each one five times without changing anything: same scorecard, same wording, same model settings.

Then count, criterion by criterion, how often the verdict changed. Look at criteria, not totals. A total can hold still while two criteria trade places underneath it.

If "resolved the issue" passes on three runs and fails on two for the same transcript, that score says more about the scorer than about the agent. An agent coached on that verdict is being coached on noise.

Sort the criteria into three groups:

• Same verdict on every run. Move on to test 2.
• Flips on one or two conversations. Read those conversations. They are often borderline cases two people would argue about too, and the criterion needs an anchor for that case. How to Design a QA Rubric covers writing anchors.
• Flips across many conversations. Leave this criterion out of reports for now and rewrite its wording first.

If your QA tool doesn't let you score the same conversation twice, ask the vendor how often a criterion's verdict changes on a rerun, criterion by criterion.

Test 2: compare it with people on conversations it hasn't seen

A scorer can be steady and still wrong. The second test is agreement.

Take a set of conversations and have two reviewers score them separately, then settle where they disagreed. That gives you a reference set. Keep it apart from everything else: if you use these conversations to tune the criterion wording, the test ends up measuring how well you tuned, not how well the scorer scores.

Run the scorer on the reference set and compare, one criterion at a time. Look at which way it disagrees, not only how often. A scorer that passes conversations where the identity check was skipped is a different problem from one that fails conversations that were fine, and each one needs its own fix.

Test 3: run both again when the wording changes

If your scorer reads each criterion's written description as its instruction (ours does), editing that description changes how every later conversation is scored. Rewording "acknowledged the impact" to fix one wrong score can be the right call, and it can also move verdicts on conversations nobody was looking at.

Keep the old wording, rerun the repeat test and the reference set, and mark the date of the change on the trend line. Otherwise a jump in the average can read as the team changing when the instruction did.

What a passing scorer still doesn't tell you

A scorer that is steady and agrees with your reviewers applies your scorecard the way your reviewers would. It doesn't tell you the scorecard measures what matters to customers. Weighted QA Scorecards covers that question.

If the conversations being scored come from an AI agent rather than a person, some criteria need changing first. Should Your AI Agent Use the Same QA Scorecard as Your Team? goes through which ones.

What we're doing in Resonant IQ

In Resonant IQ, the written description of each criterion is what the scorer reads. Every change to a scorecard creates a new version, and each review keeps a reference to the version it was scored against, so a trend can be split at the change. Each score comes with a written explanation a reviewer can check against the transcript.

In September we started running the first two tests on our own scorer: each criterion checked on its own against a held-out set of conversations, and each conversation scored repeatedly to see how often the verdict flips. We don't have results to share yet.