How to Design a QA Rubric: Criteria, Anchors, Calibration

A QA rubric is the written definition behind each line of a QA scorecard. The scorecard says resolution is worth 40 points. The rubric says what a conversation has to contain to earn 40, what earns 20, and what earns nothing.

Without a rubric, each evaluator answers those questions alone. Two people can score the same conversation, both follow the scorecard, and arrive at different totals. The score then reflects who read the conversation as well as how the agent handled it.

Start from what the score is for

Decide what the rubric measures before writing any criteria. COPC Inc., which publishes the COPC CX Standard for contact centers, separates quality errors into three kinds. Customer-critical errors affect the customer's experience. Business-critical errors cost the business money, revenue or required documentation. Compliance-critical errors break a regulation or a privacy requirement. COPC's guidance is to measure and report each kind separately, because each has a different effect.

That split makes a good first sort for a draft rubric. Put each criterion you are considering into one of the three groups. A criterion that fits none of them, such as using the customer's name twice, is a candidate for removal. If agents nearly always pass it, it adds points to every total and separates no one.

COPC also suggests finding the customer-critical attributes from the customer's side, with a key-driver survey or a regression of overall satisfaction against the attributes, instead of choosing them in a workshop. A team building its first rubric may not have the survey data for that analysis. It can still write down which criteria they believe matter to the customer, and later test that belief against what the accounts did next.

Write each criterion as one observable behavior

A criterion that two evaluators can score the same way describes one thing that either happened in the conversation or did not.

Two problems make criteria hard to score:

• Two behaviors in one line. "Acknowledged the issue and resolved it" asks the evaluator what to do when the agent did one but not the other. Split it into two criteria.
• A judgment with no description. "Showed empathy" can mean something different to each evaluator. Rewrite it as what the agent did: "Acknowledged the impact on the customer before moving to the fix" is something two people can look for and find.

Resolution needs the most care, because it is the criterion closest to the customer and the hardest to see from inside one conversation. "Resolved the issue" asks the evaluator to guess. "Answered the question the customer asked, or stated the next step, who owns it and when" can be checked against the transcript.

Pick the scale for each criterion

A binary criterion (met or not met) is the easiest to score the same way. It suits anything observable: the account was verified, the refund policy was followed, the ticket was tagged.

Use a scale only where you can write a description for every point on it. A one-to-five scale for clarity needs five descriptions. If you can describe only "clear" and "unclear", the scale is binary in practice, and the points in between are where evaluators will disagree.

For judgment criteria, three levels (met, partly met, not met) are easier to describe in writing than five, and each level still gets its own description.

Anchor every level with an example

An anchor is a short description of what a conversation at that level looks like, with an example. For "answered the question the customer asked":

• Met: the customer asked whether their plan includes single sign-on, and the agent said which plans include it and which plan the customer is on.
• Partly met: the agent linked the pricing page without saying which plan the customer is on.
• Not met: the agent explained how to configure single sign-on without saying whether the customer's plan includes it.

Take anchors from your own conversations, and update them when the product or the policies change. An anchor that describes a retired feature or an old refund policy will be misread.

Write the rules that sit outside the criteria

Three rules change scores without being criteria. Write each one on the rubric:

• Not applicable. What happens to a criterion's points when it does not apply to the conversation. The scorecard post shows one conversation scoring 60, 75 or 80 percent depending on this rule alone.
• Auto-fail items. Which failures zero the conversation, and whether they are reported inside the quality score or as a separate count.
• Evidence. What the evaluator may use: only the conversation being scored, or the account's history as well. A promise made in an earlier ticket changes what "resolved" means in this one.

Calibrate before the scores count

Calibration tests whether the rubric says what you meant it to say. Before the rubric goes into coaching or performance reviews, have several evaluators score the same set of conversations independently, without discussing them first, then compare the results criterion by criterion.

The disagreements point at the wording. Suppose three evaluators score the same ten conversations. They agree on verification in all ten, and on "acknowledged the impact" in five. The second criterion needs a rewrite or a clearer anchor. Agreeing on a score for those five conversations leaves the wording as it was, so the next set is likely to split the same way. Rewrite the definition, score a fresh set, and repeat until agreement holds.

Compare agreement per criterion as well as on the total. Two evaluators can reach the same total with different scores on individual criteria, because the differences cancel out.

Calibrate again whenever the rubric, the product or the evaluator team changes. An automated scorer needs the same test: score the same conversations more than once and check how often a criterion's verdict changes.

Date every change

A rubric change breaks the comparison between scores before and after it. If resolution gets a stricter anchor in March, a drop in March scores could come from the rubric or from the agents, and only the date of the change tells them apart. Keep a dated change log, and mark the change on any trend chart that crosses it.

How Resonant IQ uses the rubric

In Resonant IQ, the written description of each criterion is what the automated scorer reads when it scores a conversation, so the rubric's wording is its instruction. QA managers can edit those descriptions, for example after correcting a score the scorer got wrong, and later conversations are scored against the new wording. Every change to a scorecard creates a new version, and each review keeps a reference to the version it was scored against, so a trend can be split at the change.

We are running the repeat-scoring test described above on our own scorer, criterion by criterion. The results are not in yet.

What this does not settle

Agreement between evaluators shows that the criteria are clear. Whether they measure what matters to customers is a separate question. Answering it means comparing criterion scores with what the accounts did next, as set out in the post on scorecard weights, and that comparison needs more scored conversations than a monthly sample of four per agent provides.

Sources

• COPC Inc., "What are the guidelines for QA form design and attributes?", Clearly CX series, undated. The three kinds of critical error, reporting them separately, and the key-driver approach to customer-critical attributes.