Beyond the Score: Other Ways to Hear Users

Monday, 10:10, at Northgate Retail. Jo runs the five-person desk that supports head office and twenty-two stores. The quarterly satisfaction figure is the best it has been: 88 percent good, from 140 answers, with comments that are mostly thank-yous. She is about to paste it into the report when she notices a line in the volume table underneath. The Harbor Street store raised four tickets this quarter. Last quarter it raised twenty-six. There are no low scores from Harbor Street, no comments, and no complaints. There are no anything. Jo rings the store manager, Carys, and asks whether everything is all right. Carys says, cheerfully, "Oh, we stopped bothering. It takes a day to hear back, so we just work round things now. Dafydd on the tills is quite handy." Jo thanks her, puts the phone down, and looks at the 88 percent for a while.

A satisfaction survey is a microphone pointed at the people who walked up to it. It hears the requesters who raised a ticket, got a resolving reply, and chose to click. The earlier articles in this series, starting with What a Satisfaction Score Actually Measures, have shown this to be a small and self-selected group. The survey cannot hear the person who gave up on a ticket, the department that stopped raising them, or the colleague who has been fixing the tills for a year because the desk "seemed busy". Hearing users beyond the score means going to find those people on purpose. It uses a small set of habits that cost a few hours a quarter and tell the desk things no survey ever will. This article is part of our Measuring Satisfaction series. It is the article about the people the rest of the series could not reach.

It covers why the survey cannot hear everyone. Next come the quarterly conversation with a department; the manager round. Then come the silent-user check, which is the one that found Harbor Street; the walk-around. Next come what you hear and what to do with it; keeping the listening honest; a worked example from Northgate's quarter, with the hours. Finally come a short war story and the listening plan to copy, in that order.

Why the Survey Cannot Hear Everyone

The survey reaches a requester at exactly one moment, the resolving reply, and only if there is one. Everyone outside that moment is outside the survey. That includes the person whose ticket is still open after three weeks and who has stopped expecting an answer. It includes the person whose ticket was closed silently, and the person who raised nothing because last time took a week. The series has already dealt with the first two kinds of silence, in Response Bias and How to Read Around It. This article is about the third, which is the largest and the quietest, because it leaves no ticket to be silent about.

People stop raising tickets for three reasons, and only one of them is good. The good one is that nothing broke, or the thing that used to break was fixed for good, which is what a problem record and a permanent fix are for. The second is that they found a workaround, which means the fault is still there, costing them time every day, and the desk has lost sight of it. The third is that they gave up on the desk and are getting help somewhere else: a handy colleague, a supplier's phone line, a former technician's mobile number. The second and third look identical to the first in every report the desk produces. Volume falls, the score holds, and the desk congratulates itself on a quiet quarter.

This is the mechanism behind the shadow channel and the untracked walk-up, two of the quiet failures named in How Support Quietly Fails. This mechanism is why a desk's satisfaction figure and its real standing can drift apart for years without a single number flagging it. The desk's numbers describe the desk's tickets. The people who have left the desk are not in the desk's tickets. Somebody has to go and ask them, and the four habits below are four ways of asking.

The Quarterly Conversation With a Department

Once a quarter, sit down with one department for half an hour, in their space, and ask them three questions. The quarterly conversation is not a complaints session, a training session or a presentation of the desk's figures. It is the desk going to a group of its users and listening. Rotate through the departments so that each one is visited once or twice a year. Pick the ones with the lowest response rates first, and send one person from the desk, not three. Three people from IT arriving in a department looks like an audit. (Bring biscuits. This is not a joke; it is logistics.)

The three questions are: what slows you down most that IT could change; what do you work around rather than report; and what did we get right. The first question gets the frustrations, which are usually not the things in the tickets. The second gets the workarounds, which are tickets that were never raised, and it is the question that pays for the whole visit. People will describe, cheerfully, a process they have been doing by hand for eighteen months because the proper way broke and nobody mentioned it. The third question matters because it ends the conversation on something the desk can repeat, and because people who have just been asked what went right are more candid about what went wrong.

Take notes by hand, and write them up the same day as a short list of what was heard and what the desk will do about each item, including "nothing, and here is why". Send the list to the department's manager within the week. The write-up is what turns a pleasant conversation into a listening habit the department will take seriously the next time. The items on it feed the same pattern review as the survey comments, described in Acting on the Feedback. Anything that is a fault gets a ticket raised by the desk on the department's behalf that afternoon, so that the workaround finally has a record.

The Manager Round

Once a quarter, speak to every department or site manager for ten minutes, by phone or in person, and ask two questions. The manager round is quicker and shallower than the department conversation. It covers the whole organisation every quarter rather than one department at a time, which is what makes it the earliest warning the desk has. Twenty-five managers at ten minutes each is about four hours, plus the scheduling. Most managers will take the call gladly, because almost nobody from IT has ever rung them to ask a question rather than to announce an outage.

The two questions are: how is your team finding the desk at the moment; and who on your team has stopped asking us for help, and why. The first question will usually get the manager's own experience, which is useful and is not what you asked. Ask about the team twice. The second question is the important one, and it is worth asking in exactly those words. Managers know who on their team has given up on IT long before the desk does. It is the person who asks the manager, or the person the rest of the team asks instead. Carys at Harbor Street would have named Dafydd on the tills in the first thirty seconds.

Keep a one-line note per manager per quarter, in a single document, so that the round builds a history. "Finance: fine; new starter accounts still slow. Harbor Street: stopped raising tickets, using Dafydd; visit booked." Over a year the notes show which departments are drifting away and which are coming back. That is a trend the survey will never draw. Where a manager raises something specific, it gets a ticket or an action the same day. The manager hears what happened, because the round only keeps working if the managers can see that it leads somewhere.

The Silent-User Check

At the start of every quarter, find out who stopped raising tickets. The silent-user check compares the requesters and departments of the last quarter with those of the quarter before, and lists the ones whose tickets fell sharply or stopped altogether. It takes an hour with a ticket export and a spreadsheet, or less where the desk's reporting shows volume by department over time. If you run QuerySolv, the volume reports already show how many tickets each department raised, so the first half of the check is a glance at a report rather than an export. The second half is always the same: a list of names and departments, and someone picking up the phone.

The mechanics are plain. Count tickets by department for each of the last two quarters, and flag any department whose count fell by a third or more. Then count tickets by requester, and flag anyone who raised three or more tickets in the earlier quarter and none in the later one. The two lists together are the silent users. Most of them will have a dull explanation: a reorganisation, a leaver, a system that was replaced, a quiet season. The broader reading of departmental volume is in Per-Department and Per-User Patterns. The ones without a dull explanation are the ones to ring. The question is the same as the manager round's: "we noticed we haven't heard from you lately; is everything working, or have you stopped asking us?"

Ask it as a genuine question, because sometimes the answer is "everything's working", and that is a result worth having too. But listen for the other two answers. "We just work round it" is a fault with no ticket, and the desk raises the ticket there and then. "We ask Dafydd" is a shadow channel. The conversation that follows is not a reprimand of Dafydd, who has been doing the desk's job for free. It is an honest question about why the desk was slower or harder than a colleague on the tills, followed by a fix to whatever the answer is. Most shadow channels exist because the real channel was slow once, and nobody from the desk noticed the silence that followed. The silent-user check is the desk noticing.

The Walk-Around

Every other week, one person from the desk spends an hour walking through part of the building, or one store, or one site, looking at how people are actually working. The walk-around is the oldest listening habit there is and the one most desks have quietly dropped. That is usually because the queue is full and an hour away from it feels like a luxury. It is not a luxury. It is the only one of the four habits that finds things nobody would have thought to mention, because the people living with them no longer see them.

Look for the workarounds, because the workarounds are the unraised tickets made visible. A hand-written sign on a printer saying "use the one upstairs". A monitor with a note taped to the bezel listing six steps to make the scanner work. A screen with a spreadsheet open that is plainly doing the job of a system the organisation already pays for. A phone handset with a cable held in by tape. A person waiting at a terminal that takes four minutes to log in, every morning, who has decided that is just how it is. The workaround taped to the monitor is a ticket that has been printed out and never sent. The walk-around is how the desk collects it.

Talk to people as you go, briefly, and ask the second question from the department conversation: "anything you've been working around?" Then raise a ticket for everything you find, on the spot, from whichever door is nearest. In QuerySolv, as in most tools, a ticket raised by e-mail, web form or chat lands in the same queue as everything else. So the walk-around's finds join the ordinary work rather than living on a notepad. Log them in the requester's name with a note that the desk found it on a walk-around, so the person gets the resolving reply and, for the first time, the survey. The idea of many doors into one queue is set out in One Queue, Many Doors. The walk-around is a door the desk opens from the outside.

What You Hear, and What to Do With It

Everything heard goes in one place, with a date, who said it, and what the desk did. Call it the listening log; a page per quarter in a shared document is enough. It is not a survey and it must not become one: no scores, no scales, no attempt to average the comments of twenty-five managers into a number. It is a record of what people said in their own words. Its value is that a desk lead can read a year of it in half an hour and see what the organisation has been trying to tell the desk. The themes from it go into the same monthly pattern review as the survey comments. The themes usually agree with the comments on the small things and disagree on the large ones. That is because the large ones are the things people stopped raising tickets about.

Close the loop with the department as you would with a requester. Every item heard gets one of three fates, written beside it: fixed, with the ticket number; planned, with a date; or not possible, with a reason. The department hears which, in a short message to the manager within the month. This is the step most desks skip. It is the step that makes the next conversation worth having. A department that told the desk about four workarounds and heard nothing back has learned that listening is a performance. One that heard "two fixed, one planned for next month, one we can't do and here is why" has learned the opposite.

Bring the listening to the quarterly report alongside the score, in a few lines, because it is the context the score lacks. "Satisfaction 88 percent good from 140 answers. Silent-user check found one store that had stopped raising tickets and was using a colleague; visited, three faults logged, response time to stores now being watched. Manager round: new starter accounts raised by four managers; with HR." A reader who sees that beside the score understands what the score is and is not saying. The format for that page is in The Quarterly Review. The score says how the people who answered felt. The listening says what everyone else is doing instead.

Keeping the Listening Honest

Do not turn the conversations into surveys. The moment a department conversation ends with "and on a scale of one to five", it has become the thing it was meant to get around. The people in the room will give you a number to end the meeting. Do not turn the walk-around into an inspection, either. The person who taped the six steps to their monitor is not in trouble. If they think they might be, the next walk-around will find a very tidy office and nothing else. The desk is there to find the faults it missed. The people who have been living with those faults are the desk's allies in finding them, not the subjects of an audit.

Do not promise. A conversation in which the desk says "we'll sort that" to six things and sorts two has done more harm than one in which it said "I'll find out and tell you". Write things down, say when the department will hear, and then make sure they hear, even when the news is that nothing can be done. And do not let the listening become one person's private project. A walk-around that only Jo does stops when Jo is on leave, and the quarter she is away is the quarter Harbor Street drifts. Put it on the desk's calendar with a rota, like any other recurring job.

Count the hours, so that the habit survives its first busy quarter. The four habits together, at the fictional Northgate desk, come to something under twenty hours a quarter, which is about an hour and a half a week across a team of five. The table below shows where the hours went. That total is less time than the desk spent that quarter on the tickets Harbor Street would have raised if it had still been raising them, and much less than it would have spent if Dafydd on the tills had ever got one wrong. Listening is not an addition to the desk's work. It is how the desk finds out what its work is.

A Worked Example: Northgate's Quarter

Here is Jo's quarter at the fictional Northgate desk, the first one run to the listening plan at the end of this article. The silent-user check was done in the first week, from a volume report by store and an export by requester, and took two hours. It flagged Harbor Street, one other store whose volume had halved because its manager had changed, and six individuals, three of whom had left the company. Jo rang the three remaining individuals and the two stores. Harbor Street was the finding. The visit that followed logged three faults, two of them about the till network. The till-network issue turned out to be a cable run that had been degrading for months. The department conversation that quarter was with finance, and it produced the workaround of the quarter. A month-end report was being rebuilt by hand because the scheduled job had failed silently since the spring. Nobody had raised a ticket because "it only takes an hour".

The manager round took six hours including the scheduling, covered twenty-five managers, and produced the new starter account theme from four of them. That theme went to HR with a count attached. The walk-around, six hours across the quarter, found nine small things, all logged as tickets on the spot. These included a store back office whose printer had been "temporarily" unplugged for the better part of a year. Nineteen hours in all. The satisfaction figure for the quarter did not move. The number of stores raising tickets went up by one. Harbor Street's first survey answer in two quarters was "good, thanks for coming out", which Jo read as the best result the desk had that year.

A Short War Story: The Site That Called Pete at Westmoor Dairy

At Westmoor Dairy, a food producer with three sites and a two-person desk led by Anya, a technician called Pete had looked after the Fellside site for years. He had given the site office his mobile number, which they used for everything. So Fellside's tickets had run at two or three a month while the other sites ran at thirty. The desk's satisfaction figure was consistently high and Fellside never appeared in it, which Anya read as a site with no problems. Pete left for another job, kept answering the Fellside calls as a favour for a few months, and then, reasonably, stopped. Fellside did not start raising tickets, because nobody there had done so in years and the number they had was Pete's. The site's backup job, which Pete used to restart by hand each week, failed in the spring and stayed failed. When the server's disk died eleven weeks later the most recent usable backup was nine days of production records old. Anya spent a weekend on the rebuild, the site spent a week re-entering records from paper, and the first ticket Fellside raised in two years was the one asking why the server was off. The user who stopped raising tickets never fills in a survey, and the survey never notices.

The Listening Plan

The plan below is one page per quarter, filled in at the start of the quarter with names and dates and completed as the quarter runs. Section A books the department conversation and fixes its three questions. Section B is the manager round, with one line per manager that accumulates into the history. Section C is the silent-user check, done in the first week, with the two lists and what each call found. Section D is the walk-around rota. Section E is the loop, where everything heard gets one of its three fates. Section F is the hours, so that the habit has a cost attached and can defend itself. Put the dates in the desk's calendar on the day the plan is filled in, with names, because a listening plan without dates is a wish list.

THE LISTENING PLAN — ONE QUARTER
Desk: ______________   Quarter: ______________   Owner: ______________

A. QUARTERLY DEPARTMENT CONVERSATION (one this quarter, half an hour, in their space)
   Department: ______________   Chosen because: ______________________
   Date: ________   From the desk: ______________   Their people: ______________
   The three questions, asked in this order:
     1. What slows you down most that IT could change?
     2. What do you work around rather than report?
     3. What did we get right?
   Write-up sent to their manager by: ________   Tickets raised on their behalf: ____

B. THE MANAGER ROUND (every manager, ten minutes, once a quarter)
   The two questions:
     1. How is your team finding the desk at the moment?  (ask about the team twice)
     2. Who on your team has stopped asking us for help, and why?
   Manager / department        Date      One-line note                          Action
   ______________________      ______    ___________________________________    __________
   ______________________      ______    ___________________________________    __________
   ______________________      ______    ___________________________________    __________
   ______________________      ______    ___________________________________    __________

C. THE SILENT-USER CHECK (first week of the quarter)
   Departments whose tickets fell by a third or more since last quarter:
     ______________________  last ____  this ____   called on ______  found: ______________
     ______________________  last ____  this ____   called on ______  found: ______________
   People with three or more tickets last quarter and none this quarter:
     ______________________  called on ______  found: ______________________________
     ______________________  called on ______  found: ______________________________
   The question: "We noticed we haven't heard from you lately. Is everything
   working, or have you stopped asking us?"

D. THE WALK-AROUND (one hour, every other week, on a rota)
   Date      Who          Route / site                 Faults found and logged as tickets
   ______    __________   ________________________     ____
   ______    __________   ________________________     ____
   ______    __________   ________________________     ____
   Look for: hand-written signs, steps taped to screens, spreadsheets doing a
   system's job, unplugged devices, anyone waiting on a machine.

E. WHAT WE HEARD AND WHAT HAPPENED (every item gets one fate; the department is told)
   Item                                   From          Fixed (ticket) / Planned (date) / Not possible (why)
   ___________________________________    __________    __________________________________________
   ___________________________________    __________    __________________________________________
   ___________________________________    __________    __________________________________________
   Loop closed with each department by: ________

F. HOURS THIS QUARTER
   Conversation ____   Manager round ____   Silent-user check ____   Walk-around ____   Total ____

Jo filled in the first plan on the Monday she rang Carys, with Harbor Street already in section C and a visit in the calendar for the Thursday. The 88 percent went into the report unchanged, with three lines of listening beneath it. The operations director's only question was about the till cable, which was the right question. The score had been true all along. It had simply been describing the people who were still talking to the desk. The plan was how Jo went to find the rest.

Frequently Asked Questions