Measuring Experience Honestly: Satisfaction Plus Effort

Wednesday, 11:15, at Stonebridge Insurance. Jonah runs a desk of five and has just finished the slide with the satisfaction score on it. The score has sat near the top of its range for a year. He shows it at every monthly operations meeting. Rosalind, who manages the claims team, waits for him to finish and says: "Your score is lovely. My team dreads raising a ticket. Both of those are true, and I don't know how." Jonah does not know how either, at that moment. By the end of the week he does. The survey asks one question, at the end, of the people who still answer. It asks whether they got what they needed. It has never asked how hard it was to get it. The claims team gets what it needs. It gets it after two chases, a re-explanation, and a reply they have to ask a colleague to translate.

Measuring experience honestly means asking two questions instead of one, satisfaction and effort. Then it means listening for the people who answer neither. It also means reporting what you hear in ranges rather than as a single flattering figure. Satisfaction is whether the requester got what they needed and how the ending felt. Effort is how hard it was to get there: how many times they had to ask, explain, chase, wait without news or decode a reply. A desk can score well on the first and badly on the second for a year without noticing. That is Jonah's desk, and this article is about not being Jonah's desk. It is part of our Experience Over Metrics series. This article adds effort to the survey the Measuring Satisfaction series already designed. It does not redesign that survey, and it names no instrument, because none is needed.

It covers why satisfaction measures the ending, what effort measures, and why effort predicts whether people come back. Then come the two questions read together, counting effort from the ticket itself without asking anyone, and the follow-up conversation. Next come the silent users, ranges rather than points as a worked example, and what not to do with the numbers. The article ends with a short war story and the two-question experience survey to copy, in that order.

Satisfaction Measures the Ending

The satisfaction question is asked at the end of the ticket, and so it measures the end of the ticket. The request goes out with the resolving reply. The requester has just been told the problem is fixed. The question is some version of "did you get what you needed?" or "how was your experience?". Most people, asked that question at the moment of relief, answer generously. They are not wrong to: they did get what they needed. The score is a true measurement of how the fix felt. It says nothing about the four days before the fix, and the four days are where the dread lives.

The series on satisfaction has already made the other point: the score hears only the people who answer. They are a sample that flatters: requesters who were pleased enough to click, or cross enough. The quiet middle, who got what they needed after a tiring week and have no strong feeling about the ending, click nothing. The quiet departments, who stopped raising tickets, were never asked at all. Rosalind's claims team is both: they answer rarely, and when they do, they answer about the fix. The score has been telling Jonah the truth about the wrong thing.

Keep the score. It is cheap, it is honest about what it measures, and when it falls it is telling you something real. What it needs is a second question beside it that measures the part of the ticket it cannot reach, and that question is about effort.

Effort Measures the Journey

Effort is the amount of work the requester had to do to get help, counted in the things they should not have had to do. Asking twice. Explaining the problem to a second person. Writing "any update?". Looking up what a reply meant. Discovering that the ticket had been closed and raising it again. Walking to the desk because the ticket had gone quiet. Each of those is a cost the requester paid. The sum of them is what they mean when they say a desk is "hard work". That is the phrase that comes before "I usually just ask Nathan". A ticket can end in a perfect fix and have cost the requester an hour of chasing to get there. The hour is what they remember.

Effort predicts whether people come back in a way that satisfaction does not. A requester who got a good fix with no effort raises the next ticket without thinking. A requester who got a good fix after a week of chasing hesitates. The hesitation is where shadow channels are born: the direct message, the hallway, the colleague who "knows about computers", the department's own rota. The satisfaction score cannot see the hesitation, because the hesitating requester answered the survey kindly. The effort question can, because it asks about the week rather than the ending.

The question is one line in plain words, and it needs no instrument, scale or institute behind it: "How easy was it to get this sorted?" It has three answers: easy, took some effort, hard. There is an optional box asking what nearly made you give up. Three answers is enough. More precision than that is precision about a feeling, and the requester will not supply it reliably. The box is where the value is. "Had to ask twice and the second person didn't know what the first had done" is a sentence a desk can act on by Friday. No number says it.

The Two Questions Together

Read together, the two questions put every answered ticket into one of four boxes, and each box means something different about the desk. The table below gives the four, with what each one usually indicates and what to do about it. The boxes are a reading aid, not a scoring system. The point is that a desk full of top-left tickets is doing fine. A desk full of top-right tickets is doing fine by heroics that will not survive a busy month.

Rosalind's team, when Jonah finally looked, sat almost entirely in the second row: yes, they got what they needed; no, it was not easy. The score could never have shown it, because the score only reads the first column. A desk that adds the second column sees, for the first time, the difference between a good desk and a desk that gets there eventually. (The two look identical on a slide. They are not identical on a Tuesday afternoon.)

Counting Effort From the Ticket Itself

Most of the effort a requester paid is visible in the ticket without asking them. Counting it is the most honest effort measure a desk has, because it covers every ticket rather than the ones that answered. A chase is a message from the requester, while the ticket is open, that asks for status rather than adding information: "any update?", "is anyone looking at this?", "hello?". Count them. A re-explanation is a requester message after a handoff that restates the problem. Count those. There may be a reopen inside the confirmation window or a second ticket from the same requester about the same fault. There may be a walk-up logged against an open ticket. Each is a unit of effort the requester paid and the ticket recorded.

The counts turn into one ticket-derived figure that goes beside the survey. It is the share of resolved tickets in the month on which the requester had to chase at least once. Jonah's desk resolved 210 tickets in the month he looked. On 46 of them the first message after the acknowledgement came from the requester asking what was happening. That is an effort measure that heard every requester, including the ones who never click surveys. It took one filter and an hour of reading to produce. Most tools keep the complete event history on a ticket: every message with its time and author. That makes the chase count a filter rather than a reconstruction. Where the tool keeps less, the sent-mail folder is the fallback.

The ticket-derived count and the survey answer do not always agree, and the disagreement is informative. A requester who had to chase once and still said "easy" is telling you the chase was answered quickly and warmly. That is a recovery, and recoveries are worth knowing about. A requester who never chased and said "hard" is telling you about something the ticket did not record. That could be a reply they did not understand, a walk to the desk, or a colleague consulted. Read a few of each. The two measures together see more than either, which is the general rule of this series and most of the library: no number travels alone.

The Follow-Up Conversation

The follow-up conversation is a short message or call to a handful of requesters a week or so after their ticket closed. It asks three questions in plain words: what happened, what nearly made you give up, and what would have made it easier. It is not a survey; it is a person asking. It hears things a survey cannot, because a week later the relief of the fix has faded and the memory of the week remains. Five or ten of these a month, chosen at random from the second row of the table above, cost an hour and produce most of the sentences that change how a desk works.

The conversation has rules, because without them it becomes either a complaint line or a fishing trip for compliments. Choose the requesters at random from tickets that said "took some effort" or "hard", or from tickets with a chase on them, never from the ones you are curious about. Ask the three questions and then stop talking. Write down what they said in their words, in a note on the ticket, marked internal so it stays with the team. Do not defend the desk during the call. The requester is not accusing anyone. A technician who explains why the silence was reasonable has taught the requester not to be honest next time. The habit of acting on what you hear, so that the conversation is not theatre, is the subject of the satisfaction series' article on acting on feedback. The only addition here is that effort comments are nearly always about a moment. So the moments map from The Moments That Matter is where they get filed.

Jonah's first five calls were to the claims team. Three of them mentioned the same thing: the second technician on a handed-off ticket asking what the problem was. None of them had written it on a survey. All of them said it inside two minutes of being asked. The habit that fixed it, an introduction written before every reassignment, took the team a week to adopt. A month later the claims team's effort answers had moved from the second row toward the first. The score, throughout, did not move at all. It had nowhere to go.

The Silent Users

The honest measurement has to account for the people it cannot hear, and there are two groups. The first are requesters who raise tickets and never answer surveys, which on most desks is most of them. The ticket-derived chase count is the answer to that group. It measures effort on every ticket, answered or not. It is the figure to report when someone asks what the survey's silence means. The second group are the people who have stopped raising tickets, or never started. No ticket measure can see them because there is no ticket. They are found by going to look, with the department conversation, the manager round and the silent-user check. The satisfaction series describes those in Beyond the Score. That article owns the method and this one only adds why the method matters more for experience than for satisfaction.

It matters more because effort is why people leave. A requester does not stop using the desk because a fix was poor; they raise another ticket and complain. They stop using it because getting help was hard work. Once they have found an easier route, the desk never hears from them again except in a falling volume line that the monthly report describes as a good month. The silent users are, almost by definition, the high-effort users who gave up. Any measurement of experience that counts only the people still in the queue is measuring the survivors.

Rosalind was Jonah's silent-user check, delivered unasked in a meeting. Most desks are not that lucky. Ask the quiet department's manager, once a quarter, how their team gets help and what it cost them last time. Listen for a name, and listen for the word "eventually". The question takes five minutes and finds more than the survey finds in a year.

Ranges, Not Points

Report experience in ranges and counts, never as a single number with a decimal point. The samples are small and the people who answered are not the people who did not. Here is Jonah's month, as a worked example from a fictional desk and nothing more. 210 tickets resolved; 41 requesters answered the two questions. Of the 41, 34 said they got what they needed, 5 said partly, 2 said no. On the effort question, 27 said easy, 11 said it took some effort, 3 said hard. From the tickets themselves, 46 of 210 carried at least one chase, and 9 had a re-explanation after a handoff.

The honest sentence from those numbers is not "experience is at 83 percent". It is: "Of the 41 who answered, most got what they needed; about one in three said getting help took effort or was hard. Across all 210 tickets, the requester had to chase on 46, roughly one in five, which is the figure that covers everyone." The ranges are there because 41 answers can move several points when three people change their minds. They are also there because the 169 who did not answer are not assumed to agree with the 41. If a single figure is demanded, give the ticket-derived one, the chase share, with its two counts. It was measured on every ticket and nobody chose whether to be in it.

Small desks need this more, not less. A desk resolving sixty tickets a month with twelve survey answers has an experience figure that is mostly weather. A month-on-month comparison of it is noise with a trend line drawn through. Report the quarter, report the counts, and let the comment box carry the meaning. Three verbatim comments tell a manager more than a figure to one decimal place, and they have the additional merit of being true.

What Not to Do With the Numbers

Do not attach the effort figure, or the satisfaction figure, to anyone's pay, bonus or ranking. The moment a number decides a reward, the number is managed rather than measured. Then technicians close tickets before the awkward part, ask favoured requesters to answer, and avoid the hard tickets that produce honest effort scores. The figure then rises, the dread stays, and the desk has lost the only instrument that could have seen it. Per-technician experience figures exist to coach. They are read only from a reasonable sample of comparable tickets, which on a small desk means a quarter's worth at least. The rule, and the minimum sample, are in Satisfaction by Category and, Carefully, by Agent. Effort follows the same rule for the same reasons.

Do not send reminders, and do not change the wording. Reminders raise the answer count by annoying the people who had already decided not to answer, and the answers they produce are about the reminder. The case against them is made in Surveys People Actually Answer, and it applies with equal force to the second question. Changing the wording of either question resets the series, so that this quarter cannot be compared with last. A desk that rewrites its question whenever the answers disappoint has a different measurement every quarter and learns nothing from any of them. Pick the words once. Live with them for a year.

And do not present the effort figure as if it were a fact about the organisation rather than about the desk. "Our users find it easy to get help" is a claim about users, most of whom did not answer. By contrast, "about one in five of last month's tickets needed a chase, down from one in three" is a claim about tickets, every one of which was counted. The second sentence is less impressive and will still be believed in a year. The first will be believed until the first manager says "my team dreads raising a ticket", and then nothing on the slide will be believed again.

A Short War Story: The Easy Score at Meridian Parts

Rafael ran the desk at Meridian Parts with a satisfaction score he was rightly proud of and a survey that asked one question, at the close, of whoever clicked. When the operations director asked him to add "how easy was it?" as a second line, he resisted for a quarter. His grounds were that the first score was already excellent and a second question would lower the response count. Then he added it because she asked twice. The first month's answers put most of the warehouse's tickets in the second row, good fix, hard work. The comment box filled with the same three complaints. Nobody said when they would be back. The second person asked for the problem again. And the close arrived before the question in the ticket had been answered. Rafael pulled the chase count and found that on about a third of the warehouse's tickets the requester had written to ask what was happening before the desk had written anything. The cost was a year in which the warehouse's shift leads had routed most of their problems to a supervisor who "knew the systems". Another cost was a backlog of unlogged faults in the picking scanners that surfaced only when the supervisor left. And the cost included a quarter of Rafael's credibility with the director. She had asked for the second question because she already knew what it would say. The lesson he gave his team was that a desk which measures only whether people got what they needed will never find out what it cost them.

The Two-Question Experience Survey

The survey below is the one-question design from the satisfaction series with the effort question added and the rules kept. It is sent with the resolving reply, with no login and no reminders. The wording is fixed for a year. The comment box is optional and read by a person every month. If you run QuerySolv, the rating request already goes out with the resolving reply and satisfaction appears in the standard reports. Whether a second question can ride on the same request varies by tool. Where it cannot, the effort question goes into the resolving reply as a one-line ask and the answers are read by hand. For a small desk, that is an hour a month and worth it. The second half of the sheet is the ticket-derived count and the reporting line, so that the survey never leaves the desk alone.

THE TWO-QUESTION EXPERIENCE SURVEY — SENT WITH THE RESOLVING REPLY
Desk: ______________________   Wording fixed from: ____________   Reviewed: yearly

THE TEXT (one screen, no login, no reminders)
  Your ticket ________ has been marked resolved. Two quick questions, if you have a moment:

  1. Did you get what you needed?
     [ ] Yes      [ ] Partly      [ ] No

  2. How easy was it to get this sorted?
     [ ] Easy     [ ] Took some effort     [ ] Hard

  Optional: what nearly made you give up, or what would have made it easier?
  ____________________________________________________________________

  If anything is still wrong, reply to this message and the ticket comes back to the same person.

THE RULES
  [ ] Sent once, with the resolving reply, to the requester only
  [ ] No login, no reminder, no second send
  [ ] Wording unchanged for at least a year; changes dated here: ____________
  [ ] Comments read by a person every month; three quoted verbatim in the report
  [ ] Per-technician answers read for coaching only, never ranked, not below ____ tickets

THE TICKET-DERIVED COUNT (every ticket, answered or not)
  Resolved this month: ______
  Tickets with at least one requester message asking for status before any update: ______
  Tickets with a re-explanation after a handoff: ______
  Tickets reopened inside the confirmation window: ______

THE REPORTING LINE
  "____ of ____ answered. Got what they needed: ____. Easy: ____, took effort: ____, hard: ____.
   Across all ____ tickets, the requester had to chase on ____, about ____ in ____."
  Three comments, verbatim, chosen by rule (one good, one hard, one about a moment):
  1. "________________________________________________"
  2. "________________________________________________"
  3. "________________________________________________"
  The moment most comments point at: ______________________   What we are changing: ______________________

Jonah's next slide for the operations meeting had the score on it, unchanged. Beside it was a line he had never shown before: "Across all 210 tickets, the requester had to chase on 46." Rosalind read it and nodded, which was the first time anyone had nodded at one of his slides for a reason he understood. The score has still not moved. The chase count has, and that is the one he watches now.

Frequently Asked Questions