Satisfaction by Category, and Carefully by Agent

The quarterly review at Tollgate Foods, a Thursday afternoon in a meeting room that smells faintly of the biscuit line. Femi runs the four-person desk and has brought the satisfaction figures, cut by technician because the operations manager, Lorna, asked for them that way. Ade is at 89 percent good, Beth at 91, Carla at 84, and Tomasz at 71. Lorna looks at the table for about four seconds and asks the question the table was designed to make her ask: "So Tomasz is the problem?" Tomasz is the technician who looks after the production-line systems. He handles the label printers that stop a line when they jam, and the scales that lose their calibration. He handles the terminal in the chiller that nobody else will touch. He gets the tickets where the requester has already lost an hour of output before they pick up the phone. Femi has about a minute to explain why 71 is not what it looks like.

A cut is the satisfaction score for one slice of the tickets rather than for the whole desk. It could be by ticket type, by department, by channel, or by technician. Cuts are how a desk finds out where its low scores come from, and they are also how a desk fools itself fastest. Every slice has fewer answers than the whole and some slices have almost none. The cut by technician is the most useful and the most dangerous, which is why this article's title has the word "carefully" in it. This article is part of our Measuring Satisfaction series. And it is about making the cuts that help, with a minimum sample before any number is read. It includes a rule that per-person numbers are for coaching and never for ranking.

It covers why to cut at all, then the category cut and what it finds. Next come the minimum sample before a number means anything, and the other cuts worth making. We cover the agent cut as coaching rather than ranking, then the agent whose score is low because they get the hard tickets. We cover what a per-agent number can and cannot tell you, then how to talk to a technician about their number. Next come a worked example from Tollgate's quarter and a short war story. We finish with the cut table with its minimum-sample rule, in that order.

Why Cut at All

A desk-wide score hides where the trouble is. Eighty-five percent good across the whole desk could mean that every ticket type is at about eighty-five. Or it could mean that passwords are at ninety-eight and hardware repairs are at sixty. Those are different desks with different problems. The score for the whole desk tells you how the requesters felt on average. A cut tells you which requesters, about which tickets, and that is the information a desk can act on. The pattern review described in Acting on the Feedback does this by reading comments; the cut does it by counting, and the two agree more often than not.

Every cut costs answers. A desk with 128 answers in a quarter has a figure it can read. Cut it six ways by category and the largest slice has forty answers and the smallest has nine. Nine answers cannot carry a percentage, for reasons set out at length in Response Bias and How to Read Around It. Cut it again by technician and the slices are smaller still. The arithmetic does not care how interesting the question is. A cut that leaves a slice with too few answers has not produced a finding. It has produced a number that will be read as one.

So the rule for cutting is two rules. Cut along the lines that lead to action. Cut by ticket type, because types have owners and fixes. Cut by department, because departments have managers you can talk to. Cut by channel, because channels have designs you can change. And never read a slice below the minimum count, whatever it seems to show. The second rule is the one this article keeps returning to, because it is the one that gets broken in meetings.

The Category Cut

The category is the ticket type. It is the label that says whether a ticket was a password reset, a hardware repair, a software request or a line-stoppage. And the category cut is the most useful cut a desk can make, because categories map onto causes. Low satisfaction in hardware repair usually means waiting for parts and a silence while waiting. Low satisfaction in access requests usually means an approval sitting with someone outside the desk. Low satisfaction in "other" means the categories need work, and the series on categories that make reports worth reading is where to start. The cut does not tell you the cause. It tells you which pile of tickets to read for it.

Categories also carry difficulty, and this is the fact that the agent cut depends on. A password reset is a ten-minute ticket for a requester who is mildly inconvenienced. A line-stoppage is a two-hour ticket for a requester who is losing output by the minute and has a supervisor behind them. The same technician, doing equally good work on both, will score differently on them, because the requester's starting mood is different. The outcome the requester wanted, which is "this never happened", is not available on the second. A category's satisfaction score is partly a measure of the desk and partly a measure of how bad the ticket was before the desk saw it.

Read the category cut beside the other category cuts, because they share the explanation. If hardware repairs score low and also have the longest resolution times and the most days at a waiting status, the low score is about time. In that case, the fix is updates and a parts process, not a conversation about manner. The companion cuts live in Resolution Time by Category and Priority and First Contact Resolution by Category. And a category that scores low on all three is a category with a process problem. One that scores low on satisfaction alone is the interesting one.

The Minimum Sample Before a Number Means Anything

Set a minimum number of answers per slice, write it on the definition card, and do not quote a percentage for any slice below it. The figures that follow are the fictional desks' rules of thumb and not standards. And your desk may reasonably choose others; what matters is that the rule exists and holds. For a category, department or channel slice, the fictional desks use about twenty answers per period before they quote a share. Below that they report in words: "twelve of seventeen good". For a technician slice, they use about thirty, because a per-person figure will be read more personally than any other. A number that will be read personally should be held to a higher standard before it is read at all.

The arithmetic behind the rule is one sentence long. In a slice of six answers, one poor answer is seventeen points; in a slice of twenty, it is five points; in a slice of thirty, three. A category at "67 percent" with six answers is two people who were fine and two who were not. Nothing about the desk can be inferred from that except that four people clicked. Printing it as a percentage next to a category with forty answers at 90 percent invites a comparison the data cannot support. The comparison will be made, because that is what tables are for. (I once circulated a per-agent table with six answers in one cell. The technician in that cell has forgiven me, I think.)

When a slice is below the minimum, roll up until it is not. A category with eight answers a month has twenty-four in a quarter and is readable quarterly. A technician with ten answers a month reaches thirty in a quarter; one with five reaches thirty in a half year. And their figure is a half-yearly figure, however much anyone would like it sooner. If a slice never reaches the minimum, it does not get a percentage, ever, and the honest report says so: "too few answers to read; comments reviewed." That sentence is not a failure of reporting. It is reporting.

The Other Cuts Worth Making

Cut by department, because departments are where the silent middle lives and the response rate tells you whose voice the score is. A department that answers one ticket in four is heard; one that answers one in fifty is not. And the department cut will show you a score that is really two departments' opinion of the desk. Report the response rate alongside the score for each department. When one is near-silent, go and talk to its manager, which is the subject of the last article in this series.

Cut by channel, because the way a ticket arrived shapes the requester's expectations. Phone requesters expect speed; e-mail requesters expect a reply; walk-ups expect to be remembered. If one channel scores consistently lower, the design of that channel's intake is usually the cause, not the people answering it. Cut by whether the ticket was reopened, which is the sharpest cut of all. Reopened tickets score low almost everywhere. Look at the share of a technician's or category's low scores that come from reopens. That tells you whether the problem is the fix or the manner.

Do not cut by priority on its own, because priority is set by the desk and a cut by it mostly measures triage. And do not cut by anything you cannot act on. A cut by the requester's building, by the day of the week, or by the first letter of their surname will produce a table. The table will have a lowest row, and someone will ask about it. Categories were invented so that reports could be read. They were not invented so that every possible table could be drawn.

Carefully by Agent: Coaching, Not Ranking

The per-agent score is for coaching, not ranking, and the desk lead says so, in writing, before the first one is produced. Coaching means a lead and a technician looking together at the tickets and comments behind a number. They look to find what the technician could do differently and what the desk could do to help. Ranking means putting the numbers in a column and reading from the top down. The first improves the desk. The second improves the number, by whichever route is quickest. And the quickest routes are closing the hard tickets silently, steering the easy ones your way, and asking nicely for a good score. The general shape of this is in Metric Gaming and Its Cures, and satisfaction is the metric it bites hardest.

Never publish per-agent satisfaction, never rank it, never attach money or appraisal grades to it, and never put it on a wall screen. Each of those turns a coaching tool into a scoreboard, and a scoreboard with four names on it is a seating plan with grievances. The technicians will see a number they cannot control, produced by a handful of requesters they did not choose. People who did not read the tickets will read it as a judgement on them. And the technicians will respond rationally, which is to say they will protect the number. The series on dashboards that motivate instead of surveil covers the general case. Satisfaction is the specific one where surveillance does the most damage, because the number is the least the technician's own.

The per-agent figure, when the minimum is met, is read by the lead, with the technician, with the comments and the tickets open, and nowhere else. The lead's first question is never "why is your number low?" It is "what is in these tickets?", because the answer to the second question is usually the answer to the first, and it is usually not the technician.

The Agent Whose Score Is Low Because They Get the Hard Tickets

Tomasz's 71 percent is 71 percent of the people who had a production line stop. His answers, cut by category, came from fourteen line-stoppages, six hardware repairs and four password resets in the quarter. Beth's 91 came from thirty-one password and access tickets and four software requests. They are not doing the same job. And a table that puts their two numbers in one column is comparing a surgeon with a receptionist on the share of patients who left smiling. The honest per-agent comparison is within category. It compares Tomasz's share of good answers on line-stoppages against the desk's share on line-stoppages. It also compares his share against his own share on line-stoppages last half year. On that comparison, Tomasz was the desk's best technician on its hardest tickets, and very nearly the only one.

Build the comparison as a small table before the review, not during it. For each technician, list their answers by category, and beside each category the desk-wide share for that category. A technician whose mix is heavy on the low-scoring categories will have a low overall figure for reasons that have nothing to do with them. The table shows it in one glance. Tomasz's mix of categories had a desk-wide share in the mid-seventies, and he scored 71 on it. On twenty-four answers, that is a difference of one click. The mix is the proportion of each technician's tickets that fall in each category. It is the single largest cause of per-agent differences at every desk this series has drawn. It is invisible in the one-column table that every manager asks for first.

The hard tickets also go to the people who can handle them, which is a second reason the best technicians often carry the lowest raw scores. Consider the requester who has lost an hour of output, who has been passed from one technician to another, who is on their third ticket for the same fault. That requester lands with the person the desk trusts most. That person inherits the mood. If the raw per-agent score is read as a ranking, the desk punishes exactly the people it relies on, and they will notice. A ranking does not find your worst technician. It finds your hardest tickets and names whoever was brave enough to take them.

What a Per-Agent Number Can Tell You, and What It Cannot

Once the minimum is met and the mix is accounted for, a per-agent figure can tell you a little. The comments beside it can tell you a lot. Suppose a technician's share is low within category, on tickets of the same kind their colleagues score well on. If the comments also say "had to explain twice" or "felt rushed", that is a useful conversation about replies and listening. If the comments say "took ages", that is a conversation about updates and estimates. If the comments say "fixed, then broke again", that is about testing before the close. The number says look here. The comments say at what.

The better per-agent satisfaction figure, in my experience, is not the score at all but the response rate. That is how many of a technician's closed tickets came back with an answer. That figure measures something the technician fully controls: whether they closed with a real reply that the requester bothered to react to. It is not distorted by the mix. A line-stoppage requester is as likely to click as a password requester if the reply is worth reading. A technician with a high response rate and a middling score is closing properly and getting hard tickets. One with a low response rate and a high score is closing the easy ones with a flourish and the rest with a status change. The second is the one to coach, and the one-column table would have shown them at the top.

What the per-agent number cannot tell you is who is good. It cannot tell you who the desk would miss, who trains the others, or who takes the ticket nobody wants. It cannot tell you who the requesters ask for by name when something is really wrong. All of that is in the tickets, the comments, and the corridor, and a lead who knows the desk already knows it. The number is a prompt to go and look. It is not a verdict. A lead who lets it become one has handed their judgement to a handful of requesters who were never asked to exercise it.

How to Talk to a Technician About Their Number

Privately, with the tickets open, starting from the comments. "Here are the twelve comments on your tickets this quarter. Six are thank-yous, and I want to read you two of them. Here are the three that were not. I have read the tickets, and I think two of them are about the parts delay and one is worth us looking at together." That opening tells the technician that the lead has done the reading and that the praise was noticed. It says that the one conversation worth having is about a specific ticket and not about a percentage. Most technicians, offered that, will open the ticket themselves and tell you what they would do differently before you ask. Do the reading on the tickets, not from memory. If you run QuerySolv, every ticket carries its complete event history. So the lead can see each reply and handoff before the conversation rather than relying on anyone's recollection of it.

Never open with the number, never compare them with a named colleague, and never say "the data shows". The data shows that some people clicked. Where the conversation does lead to a change, write it down as a thing to try, with a date to look again, and look again. Where it leads to "that was the parts delay and there is nothing you could have done", say that too, in those words, and mean it. The per-agent cut, used this way, is one of the kindest tools a lead has, because it brings the requesters' thanks to the person who earned them. Used the other way it is one of the cruellest, and the difference is entirely in how the lead holds it.

A Worked Example: Tollgate's Quarter

Here is Femi's quarter at the fictional Tollgate desk, laid out as the cut table at the end of this article asks. The desk had 128 answers from 1,040 tickets closed with a reply. Of the answers, 109 were good, which is 85 percent, well above the twenty-answer minimum for the whole desk. By category, password and access had 41 answers, 40 good. Software requests had 19 answers, 15 good; hardware repair had 22 answers, 15 good. Line-stoppages had 17 answers, 12 good; new starter setups had 9 answers, 8 good. Other had 20 answers, 19 good. Only the three categories with twenty or more answers met the minimum and got a share; the rest were reported in words.

By technician, with the per-person minimum of thirty, Ade had 38 answers, 34 good. Beth had 35 answers, 32 good; Carla had 31 answers, 26 good. Tomasz had 24 answers, 17 good. Tomasz's twenty-four was below the per-person minimum, so his quarter was not read as a figure at all. The comparison that was read was within category. On the fourteen line-stoppage answers he received, eleven were good, against twelve of seventeen for the category as a whole. Of the three line-stoppage answers that were not his, two were poor, and both were on tickets handled by colleagues covering while he was on leave. The table Lorna saw the second time had the mix column in it. And the question she asked was "why are hardware repairs at 68?", which is the right question, and the answer was a parts supplier.

A Short War Story: The Wall Screen at Larkfield Council

At Larkfield Council, a desk lead called Gavin was asked by a new head of service for "transparency on performance". Gavin responded by adding a per-technician satisfaction panel to the wall screen in the open-plan office. It showed five names and five percentages, refreshed monthly. A technician called Sunita was the only person on the desk who understood the payroll and finance system. She therefore received every ticket about it, each one from a payroll officer under deadline. She sat at the bottom of the panel for two months on nineteen and then twenty-two answers, beneath four colleagues whose tickets were laptops and passwords. She asked to be moved off the finance queue and was told the queue needed her. She resigned at the end of the third month for a role elsewhere with, she said in her exit conversation, "a boss who reads the tickets". The council covered the finance system with a contractor for a quarter at considerable cost. The next payroll run went out a day late. And the panel was removed from the wall the week after she left, which was about ten weeks too late to matter. A ranking does not find your worst technician. It finds your hardest tickets, and it names whoever was carrying them.

The Cut Table With a Minimum-Sample Rule

The template below is the cut table as a form, one copy per period, with the minimum-sample rule built into it. That way, the rule is applied on the page rather than remembered in the meeting. Section A sets the minimums once and carries them forward. Section B is the whole-desk line that every slice is read against. Section C is the category cut. Section D is the department and channel cut with the response rate beside it, because that is where the silent departments show. Section E is the per-technician cut, and it is deliberately the last section. It has the highest minimum, and carries the mix column and the rule in its heading. Fill in the answers and the good count for every slice; fill in the share only where the minimum is met; write the words otherwise.

SATISFACTION CUT TABLE — ONE PERIOD
Desk: ______________   Period: ______________   Prepared by: ______________

A. MINIMUM ANSWERS BEFORE A SHARE IS QUOTED (from the definition card)
   Whole desk ______   Category / department / channel slice ______   Per technician ______
   Below the minimum: report as "____ of ____ good", never as a percentage.

B. WHOLE DESK
   Tickets closed ______   Tickets asked ______   Answers ______   Good ______
   Share of good answers ______   Response rate (answers / asked) ______ per 100

C. BY CATEGORY                 Answers   Good   Min met?   Share or words      Points at
   ______________________      ______    ____   [ ]        ______________      ______________
   ______________________      ______    ____   [ ]        ______________      ______________
   ______________________      ______    ____   [ ]        ______________      ______________
   ______________________      ______    ____   [ ]        ______________      ______________
   ______________________      ______    ____   [ ]        ______________      ______________

D. BY DEPARTMENT / CHANNEL     Asked   Answers   Resp. rate   Good   Min met?   Share or words
   ______________________      _____   ______    ________     ____   [ ]        ______________
   ______________________      _____   ______    ________     ____   [ ]        ______________
   ______________________      _____   ______    ________     ____   [ ]        ______________
   Quietest department this period: ______________  (visit, do not weight)

E. BY TECHNICIAN — FOR COACHING, NOT RANKING. NOT PUBLISHED. NOT ON ANY SCREEN.
   Technician   Answers   Good   Min met?   Resp. rate   Main categories (mix)   Within-category view
   __________   ______    ____   [ ]        ________     ____________________    ______________________
   __________   ______    ____   [ ]        ________     ____________________    ______________________
   __________   ______    ____   [ ]        ________     ____________________    ______________________
   __________   ______    ____   [ ]        ________     ____________________    ______________________
   Rules: a share is quoted only when the minimum is met; comparison is within
   category against the desk's share for that category; comments are read with
   the technician, privately, before any number is discussed.

F. ONE LINE FOR THE REPORT
   Where the low scores are, and what the desk will do about it this period:
   ______________________________________________________________________

Femi brought the second version of the table to Lorna the following week, with sections A and E filled in as above. Tomasz's row read "not read this quarter; eleven of fourteen good on line-stoppages, above the category". Lorna read it and asked about the hardware repairs. Then she asked whether Tomasz could train one of the others on the line systems. That way, he would not be the only person who ever saw a furious supervisor. That is what a per-agent cut is for. It took a mix column and a minimum to get there, and without them the same numbers had pointed at the wrong person in four seconds flat.

Frequently Asked Questions