What a Satisfaction Score Actually Measures

Tuesday, 09:15, at Ashgrove Housing. Dana runs the four-person desk. The monthly report has just come back from Malcolm, the operations director, with one line added at the top: "Satisfaction 4.6 out of 5. So basically everyone's happy?" Dana looks at the report again. The desk closed 418 tickets last month. Thirty-seven people answered the one-question survey, and thirty-one of them chose the top score or the one below it. The other 381 people got their printer fixed or their password reset, or did not, and said nothing to anyone. She starts to type "Yes" and stops, because she is not sure the number says what Malcolm thinks it says. She is not sure what she would say it does say.

A satisfaction score is the answer to one question, asked of the requester at one moment, usually when the ticket is resolved. It is summarised across everyone who chose to answer. The industry calls it CSAT, short for customer satisfaction, and treats it as the desk's report card. It is nothing so grand. It is a count of how a self-selected group of people felt about one interaction at the instant they were asked. That is a useful thing to know, provided you know that this is what you know. This article is part of our Measuring Satisfaction series, which is about collecting that number honestly and reading it for exactly what it is.

It covers what the question actually asks. Next come the moment it is asked and why the moment matters; the scale, and why averaging it is a convention rather than arithmetic. Then come the person answering, who is not the business; the three things folded into one number, and who is not in the number at all. Next come the denominator that has to travel with the score; reading the score with its pair. Finally come a worked example from Ashgrove's month; a short war story; and the definition card to copy, in that order.

What the Question Actually Asks

The question on a help desk survey is almost always some form of "how was this?" The first thing to notice is how small a question that is. It is not "is IT good?" It is not "do you trust the desk?" It is not even "was your problem solved?", although people answer as if it were. It is one person's reaction to one ticket, at the end of that ticket. Everything the score can tell you is bounded by that question. The commonest mistake in reading satisfaction is to let the number answer a bigger question than the one that was asked.

Write the question down and read it as a stranger would. "How satisfied are you with the handling of your request?" invites a judgement about the handling, meaning the people and the process. "Did we resolve your issue?" invites a judgement about the outcome. "How would you rate our service?" invites a judgement about everything the requester has ever thought about IT. Those three questions produce three different numbers from the same month of tickets. A desk that switches from one to another has not improved or declined; it has changed the subject. The question is part of the definition of the metric. That is why the question goes on the card at the end of this article, in full, and does not change without a note in the report.

The score is an opinion, not a measurement of the ticket. Two people can receive the same fix, in the same time, in the same words, and rate it differently because one of them had a bad morning and the other had expected worse. That is not a flaw in the survey; it is what the survey is for. The resolution time measures the ticket. The satisfaction score measures the person. The pair together tell you something neither does alone, and the series on metrics that matter is built around reading them that way.

The Moment It Is Asked

The survey is asked at the close, and the close is a particular mood. In most tools the request to rate the service rides on the resolving reply, the message from the technician that says the fix is in. A status change on its own asks nothing. So the requester reads "I've reset the connection and it should be working now; let me know if not", and under it a question about how it went. They answer with the fix fresh, before they have tested it properly, and before they have found out whether it holds. The score is therefore a verdict on the close, not on the week after. A fix that falls over on Thursday produces a reopen, not a lower score. The reopen is where that story gets told, in the reopened tickets series rather than here.

The moment also decides who is asked. A ticket that is closed without a reply is a ticket whose requester never saw the question. A desk that closes a third of its tickets silently has a satisfaction score built from the other two thirds. That is the subject of Avoiding the Silent Close, and it belongs to the definition of the score as much as the question does. The number describes the tickets that were closed properly, rated by the people who answered. If you run QuerySolv, the rating request goes out with the resolving reply. That means the way to get more people into the sample is the same as the way to close tickets well: write the reply.

A survey asked later, a week after the close or once a quarter about the desk in general, is a different instrument with a different answer. It measures memory rather than reaction, and memory is kinder to some tickets and crueller to others. Neither moment is wrong. They are different questions wearing the same name, and the card says which one you are asking. (The one thing you cannot do is ask at both moments and add the answers together, which I have seen done, in a spreadsheet, with confidence.)

The Scale, and Why the Average Is a Convention

The scale is the set of answers the requester can choose from, and desks use many. Examples are a run of numbers, a row of faces, a thumb up or down, three words. Which one your tool offers, and whether you can change it, is a question for its documentation, not for this article. What matters for reading the score is that every scale is a set of ordered labels, not a ruler. The gap between "good" and "great" is not the same size as the gap between "poor" and "okay", and nobody has ever measured either. The numbers printed beside the faces are names for the faces. They are not quantities.

That is why "4.6 out of 5" is a convention rather than arithmetic. When a desk averages its scores, it treats the labels as if they were evenly spaced amounts, adds them up and divides. The result is a familiar-looking number that behaves oddly. One furious answer in a small month moves it a long way. A hundred mildly pleased answers move it almost nowhere. The same average can come from thirty people who all chose the second-best answer or from fifteen who chose the best and fifteen who chose the middle. The average hides the shape. The shape is the information.

The alternative most desks settle on is a share. That is the proportion of answers that landed on the top one or two labels, as in "thirty-one of thirty-seven chose good or very good". A share does not pretend the labels are numbers. It survives a change of scale better than an average does. It arrives with its own count attached, which an average never does. Report the share, keep the count at each label underneath it, and produce the average only when someone above you insists on one decimal place. They will. Give it to them with the count beside it.

The Person Answering

The person answering is the requester, which seems too plain to say until you notice who is not answering. The requester is not the department, not the manager who signed off the purchase, not the organisation, and not the person whose problem it really was if the ticket was raised on their behalf. When the office manager raises a ticket for a colleague's screen, the office manager gets the survey. The office manager's satisfaction is about how easy the desk was to deal with, not about whether the screen now works. That is worth knowing. It is not the same thing.

The person answering is also the person who chose to answer, and that choice is the largest single fact about the score. People answer when they feel something: relief, gratitude, anger, the mild pleasure of being asked. People who were served adequately and got on with their day mostly do not answer, because the survey is one more item in a full inbox and the fix already works. So the sample leans towards the ends of the mood range. The middle, which is where most of your tickets live, is thinly represented. The mechanism, the arithmetic and the ways to read around it have their own article, Response Bias and How to Read Around It. Here the point is only that the score describes the people who answered, and they are not a random draw.

The answer is also written by someone with no stake in the desk's statistics. The requester does not know the score is reported, does not know it feeds a monthly chart, and does not care. They are reacting. This is the score's great virtue, and the reason it is worth collecting at all. It is the only number in the monthly report that was produced by someone outside the desk, with no reason to flatter it and no reason to game it. Every other figure in the report was made by the desk, about the desk. This one was posted through the letterbox.

Three Things Folded Into One Number

When a requester rates a ticket, they are reacting to at least three separate things and reporting one feeling. The first is the outcome: was it fixed, and does the fix hold. The second is the time: how long they waited, and how it felt to wait, which is not the same as how long it was. The third is the manner: were they kept informed, spoken to like an adult, made to repeat themselves, blamed. Those three are produced by different parts of the desk and are improved by different work. The score blends them into one answer with no indication of the recipe.

The table below separates the three, with what each one sounds like in the comment box and which other figure in the report measures it directly. The point of the table is not to decompose every score, which cannot be done from the number alone. It is to show that a falling score has at least three possible causes before you count the ones outside the desk. The comment box, where there is one, is the only place the requester tells you which.

The last row is the one that bites. A requester who waited a week for a part because the supplier was slow, or who was told no because the policy says no, rates the ticket. The ticket is the only thing in front of them. The desk takes the score for the policy. Some of that is fair, because the desk chose how to say no and how to explain the wait. A good explanation earns a better score than a bad one for the same outcome. The craft of that explanation is in the managing expectations series. Some of it is not fair, and the comment is how you tell the difference.

Who Is Not in the Number

The score does not include the people who did not answer, and that is most people. At Ashgrove, 418 tickets closed and 37 were rated, so the score describes about one requester in eleven. The other ten are not neutral, and they are not absent from the desk's reputation; they are absent from the report. Most of them were fine. Some of them were not fine and did not think a survey was the place to say so. You cannot tell which from the number, and the number does not even tell you how many of them there are unless you print the count.

The score does not include the user who gave up. Somebody who raised a ticket, waited, heard nothing, and fixed it themselves or stopped caring usually gets a closing reply weeks later or none at all. Either way, that person is not in a mood to rate anything. Their ticket closes as resolved, abandoned or stale. Their dissatisfaction is real and shows up elsewhere: in the volume figures for their department, which fall; in the walk-ups to the one technician they trust; in the next ticket they raise, which arrives angry. The series on aging and follow-up is about catching them before they give up. The score will not do it for you.

The score does not include the person who never raised a ticket, who is the largest group of all. The staff who have decided the desk is not worth the trouble, who ask a colleague, who live with the fault, who found a workaround and told nobody, have no ticket. So there is no close, so there is no survey. A desk can run a very high score for years while a quarter of the building has quietly stopped using it. The last article in this series, Beyond the Score, is about hearing those people. The honest summary is that no survey can. You have to go and ask.

And the score does not include the tickets closed without a reply, the ones never offered the question. If the desk's habit is to set Resolved and move on, the sample is not just small. In that case, it is selected for the tickets where a technician wrote something, which are the tickets the technician felt good about. That is a sample chosen twice for contentment, and a desk that fixes its silent closes usually watches the response count rise and the score dip. The score was never that high. It had been asking a kind question of a friendly subset.

The Denominator Travels With the Score

Never report the score without the number of responses and the number of closes beside it, in the same sentence, every time. "4.6 out of 5" is not a finding. "Thirty-one of thirty-seven answers were good or very good, from 418 closed tickets" is a finding, and a reader can see at once how much weight it will bear. The count of responses is the score's confidence, and the count of closes is its coverage. A score without either is a number wearing a lab coat it has not earned.

The response rate is the number of answers divided by the number of tickets that were offered the survey. It is worth reporting as a figure in its own right, because it measures something the score cannot. It measures how often the desk closed a ticket in a way that invited an answer. A rising response rate with a steady score means the desk is reaching more people. A falling response rate with a rising score means the sample is shrinking towards the fans. The ranges a small desk can expect, and what a good rate looks like, are in Surveys People Actually Answer. If you run QuerySolv, satisfaction sits in the standard reports next to volume and resolution time. That makes the denominator a lookup rather than a count, and leaves no excuse for leaving it off.

Reading It With Its Pair

A satisfaction score read alone is a mood ring. Read beside the measures of what the desk actually did, it becomes a check on whether the doing was felt. The pairing idea, one metric for the work and one for the person, is the organising principle of Reading Metrics Together, Not One at a Time. Satisfaction is the metric most in need of a partner, because it is the one the desk did not produce.

Pair it with resolution time and you learn whether speed is what the requesters notice. Pair it with the reopen rate and you learn whether the close was real. A high score beside a high reopen rate means the resolving reply reads well and the fix does not hold. Pair it with first contact resolution and you learn whether being passed around is what annoys people, which it usually is. Pair it with the response rate, always, because without that one none of the others can be trusted. A score that rises while resolution time, reopens and the response rate all hold still has probably not risen at all. It has wobbled.

The pair also protects you from the single most common misreading, which is treating a month-to-month movement as news. Satisfaction moves because a dozen people felt differently, and a dozen people feel differently every month for reasons that have nothing to do with the desk. If resolution time, reopens and volume did nothing, and the score moved, the score moved on its own. Note it, and wait for the quarter. The quarterly view, trends rather than months, is the subject of The Quarterly Review. Satisfaction is the metric that needs it most.

A Worked Example: Ashgrove's Month

Here is Dana's month, laid out the way the card at the end asks for it; every figure is from the fictional desk, not a benchmark. The desk closed 418 tickets. Of those, 351 were closed with a resolving reply that carried the one-question survey. The other 67 were closed without one, mostly walk-ups logged after the fact and tickets merged into others. Thirty-seven people answered, which is about one in nine of the people who were asked and one in eleven of all closes. The answers were: 24 chose the top label, 7 the one below, 4 the middle, none the one below that, and 2 the bottom. Eleven of the thirty-seven left a comment.

Dana's sentence for the report, in place of "4.6 out of 5", read: "Thirty-one of thirty-seven survey answers were good or very good, from 351 tickets closed with a reply out of 418 closes; two answers were poor and both were followed up." That is longer. It is also true in every part, it carries its own weight, and it invited the question Malcolm actually needed to ask, which was "what were the two about?" (A laptop that came back from repair with the wrong keyboard, and a password reset that took a day because the requester's manager was away and nobody else could vouch for them. One of those is the desk's; the other is a policy.)

What the month does not say is also worth writing down, because Malcolm will not think of it and should. It does not say the 381 people who did not answer were happy. It does not say anything about the two regional offices, which between them sent three of the thirty-seven answers and raise a fifth of the tickets. And it says nothing at all about whoever stopped raising tickets in the spring. Dana added one line: "Response count is too small to read by team or by technician; quarterly cut to follow." That line is the difference between a report and a rumour.

A Short War Story: The Target at Bellweather Logistics

At Bellweather Logistics, a desk lead called Ola had reported a satisfaction average for two years with no response count beside it. The number had drifted up from the low fours to 4.8 while the desk got quieter and the two warehouse sites, which had always rated hardest, stopped answering altogether. The operations board liked the trend and wrote a target into the desk's annual objectives: hold 4.8 or better. The following quarter a new technician started closing every ticket with a proper reply. The response count tripled, the warehouses came back into the sample, and the average fell to 4.3 in a single month. The board read that as a collapse in service and the technician, briefly, as her own mistake. Ola spent six weeks rebuilding every month of the two years with the counts attached. That showed that the 4.8 had rested on nine to fourteen answers a month, nearly all from head office. The counts also showed that the "collapse" was the first honest number the desk had produced. The target was withdrawn, the technician was thanked, and the board asked, reasonably, why nobody had mentioned the count before. A score reported without its denominator will eventually be believed, and that is the worst thing that can happen to it.

The Definition Card

The card below is one page, kept beside the metric wherever the metric is reported. That way, anyone reading the score, this quarter or in three years, can see exactly what was asked, of whom, when, and how the answers were turned into a number. Fill it in once and change it only with a dated note in the report, because every line on it is part of the definition. Change the question, the moment or the summary method and you have a new metric with the old name. The last section, what the score does not measure, is the part people are most tempted to leave blank and the part a reader most needs. The practice of writing a card like this for every metric is in Defining Each Metric Exactly. This is the card for this one.

SATISFACTION SCORE — DEFINITION CARD
Desk: ____________________   Owner of this metric: ____________________
Card last changed: ____________   Reason: ____________________________

THE QUESTION
  Exact wording shown to the requester: _________________________________
  Comment box offered:  [ ] yes, optional   [ ] no

THE MOMENT
  Asked when:  [ ] with the resolving reply   [ ] on close   [ ] other: ________
  Reminders sent:  [ ] none   [ ] one, after ____ days
  Tickets that never see the question: _________________________________
  (for example: closed without a reply, merged, spam, internal)

THE SCALE
  Labels, in order from worst to best: ______ / ______ / ______ / ______ / ______
  "Good" means the top ____ labels

THE PERSON
  Who receives it:  [ ] the requester on the ticket   [ ] other: ______________
  One answer per ticket:  [ ] yes

THE SUMMARY
  Headline figure: share of answers that are "good" ........... ____ of ____
  Always printed beside it: responses ____   closes ____   asked ____
  Response rate = responses divided by tickets asked .......... ____ per 100
  Count at each label kept underneath the headline:  [ ] yes
  Average quoted only on request, and only with the count:  [ ] yes
  Smallest response count before a figure is quoted at all: ______
  When the count is too small, roll up to:  [ ] quarter   [ ] half year

WHAT IT DOES NOT MEASURE (print with every report)
  [ ] the people who did not answer (____ of ____ this period)
  [ ] tickets closed without a reply (____ this period)
  [ ] people who gave up before the close
  [ ] people who no longer raise tickets
  [ ] whether the fix held after the close (see the reopen rate)

REPORTED
  Where: ______________   How often: ______________   Paired with: ______________

Dana filled the card in that afternoon and pinned it to the first page of the monthly report. The question from Malcolm the following month was not "so everyone's happy?" but "what are the two low ones?" That is a question a desk can answer. The score did not change. What it was allowed to claim did.

Frequently Asked Questions