Should Your AI Agent Use the Same QA Scorecard as Your Team?

More support teams now have an AI agent answering customers before a person does. Its conversations land in the same help desk as everyone else's, and often in the same QA sample.

So someone has to decide how to score them. The quick answer is to run them through the scorecard the team already uses. Some of that scorecard carries over well. Some of it was written to coach a person and tells you little about a bot. And a few of the ways an AI agent goes wrong aren't on a human scorecard at all.

The reporting question can catch teams out too: whose average the bot's scores land in.

What carries over

Criteria about what happened to the customer apply to anything that answers them:

• Was the answer correct? Accurate information, matched to the customer's actual question.
• Was the problem resolved, or routed to someone who could resolve it?
• Was policy followed? Refunds, account changes, identity checks.
• Did the customer have to repeat themselves?

If your scorecard already weights these heavily, most of its value carries over. How to Design a QA Rubric covers writing criteria like these so two reviewers score them the same way, and that matters even more when one of the things being scored is software.

What doesn't carry over

Some criteria exist because people need coaching on them: using the customer's name, opening with the standard greeting, an empathy statement before the answer, offering further help at the end.

An AI agent can be configured to do all of these every time. It will score well on them, and the score tells you mostly that the configuration works. If those criteria carry real weight, a bot conversation with a wrong answer and a perfect greeting can still pass.

Tone still matters. Rewrite the criterion to ask whether the tone fit this customer in this conversation.

What to add

AI agents go wrong in ways a human scorecard may not ask about:

• Stating something that isn't true. A policy, a feature or a timeline that doesn't exist, stated with the same confidence as a correct answer.
• Answering when it should have handed off. A billing dispute, a cancellation, an angry customer, a question outside what it was set up for.
• The handoff itself. When it did hand off, did the person who picked it up get the context, or did the customer start over?
• Loops. The same answer offered again after the customer said it didn't help.

Each of these can be checked against the transcript, the same way the criteria above can.

Keep its scores out of the team average

If bot conversations count toward the team's QA average, that average can move when the bot's share of conversations changes, even if nobody on the team changed how they work. A rising average might mean the team got better, or that the bot took over more of the easy questions. The report can't tell you which.

Report the AI agent as its own line, with its own trend.

Then decide what happens to conversations that started with the bot and ended with a person. One rule: score them as the team's work, because a person finished the conversation, and score the handoff as its own criterion, because that part belongs to the bot setup. Whatever rule you pick, write it down, so the numbers mean the same thing next quarter.

How to set it up by hand

1. Tag the bot's conversations. Many help desks record who sent each message. Separate conversations the bot handled alone from ones with a handoff.
2. Sample them separately. A bot can answer far more conversations than a person in a week, so a shared random sample can end up mostly bot.
3. Start from your current scorecard. Keep the outcome criteria, rewrite or drop the ones that only check for phrases, and add the four above. What Is a QA Scorecard? covers the structure if you're starting from scratch.
4. Calibrate. Have two reviewers score the same 20 bot conversations and compare where they disagreed. Disagreement often points to a criterion that needs clearer wording.
5. Report it separately. A line for the team, a line for the bot, and the handoff criterion on both.

What this doesn't settle

A scorecard reads the conversations that happened. A customer who gave up on the bot and didn't write again may leave nothing to score. Check what the account did next, through renewals, cancellations or a later complaint, before deciding the bot's scores mean customers are fine.

Twenty conversations is enough to find criteria that need rewording. It's a small basis for comparing the bot with the team.

Where Resonant IQ fits

Resonant IQ marks conversations an AI agent handled on its own (one help desk's AI agent so far), lets a team score them on a scorecard set up for AI agents, and keeps them out of the human team's averages. Conversations with a handoff count as the team's. Each score comes with a written explanation, so a reviewer can check it against what the customer asked.