Vanity Metrics to Ignore, and the Test That Finds Them

Thursday, 14:10, at Stonebridge Insurance, in the quarterly review. Fiona, who runs the four-person support desk, is on her third slide, which is a bar chart titled "Productivity by Agent". Dev's bar says 212. Aisha's bar says 97. Martin, the chief operating officer, leans forward and asks the only question a bar chart like that can produce: "What's going on with Aisha?" Fiona knows what is going on with Aisha. Aisha takes every ticket about the policy administration system. Each one takes two days and a phone call to the supplier, and nobody else on the desk can do them. Dev takes password resets, and is very good at them. The chart knows none of this. The chart has two bars and an opinion, and the opinion is now Martin's.

A vanity metric is a number that looks like information and changes no decision for the better. It goes up and people feel good and nothing has improved; it goes down and people feel bad and nothing was wrong. The number is usually accurate, which is what makes it dangerous. A wrong number gets corrected and a vanity metric gets a slide. This article is part of our Helpdesk Metrics That Matter series. It is about the five vanity metrics that turn up on almost every desk, what each one gets wrong, and what to show instead. It also offers a short test that catches the rest before they reach a page.

It covers the three marks of a vanity metric, then the five usual suspects in turn. Those are tickets closed per agent as a score, averages of everything, and utilisation. They also include the number that only goes up, and the metric that exists because the tool shows it. Then come the honest substitute for each, as a table, a short war story, and the vanity-metric test to copy.

The Three Marks of a Vanity Metric

The first mark is that nobody can name the decision it changes. Ask, of any number on a report, what the desk would do differently if it went up by a fifth. Then ask what it would do if it went down by a fifth. If the honest answer both times is "nothing, but we would feel differently about it", the number is decoration. The six metrics in the handful worth tracking each pass this test on their own. Volume changes staffing, backlog age changes what gets dug out this week, and reopens change what gets reviewed. A vanity metric changes the mood of a meeting.

The second mark is that it can move the right way while the desk gets worse. Tickets closed can rise because technicians closed early. Utilisation can rise because the desk lost its only slack. Average resolution time can fall because two hundred password resets arrived in the same month as a failed system upgrade. A metric that improves when the desk improves and also improves when the desk cuts a corner is not measuring the desk. It is measuring whichever of the two is easier, and the corner is always easier.

The third mark is that it describes the desk's effort rather than the requester's experience. Hours logged, tickets touched, activities recorded, replies sent: these are all things the desk did. The desk doing a great deal is not the same as anyone being helped. The requester never sees utilisation. They see the time until a person replied, the time until it worked, and whether it stayed working. A metric that cannot be explained to the requester as something that happened to them is, at best, an internal number for the lead. Vanity metrics are not lies. They are honest answers to questions nobody should have asked.

Tickets Closed per Agent, as a Score

The count of tickets each technician closed in the period is the first thing most tools show and the most harmful thing most desks report. It is attractive for three reasons: it is easy and it is already on the screen. And it feels like fairness, because surely the person who closed more did more. The trouble is that tickets are not the same size. The count treats a password reset and a two-day investigation of why the policy system rejects renewals for one branch as one each. Aisha's 97 were the second kind. Dev's 212 were the first. The chart said Dev had done twice the work. By any measure a requester would recognise, Aisha had done rather more.

Worse than being wrong, the number teaches. A technician who knows their closed count is on a slide learns, within a month and without any malice, to take the quick tickets first. They learn to resolve at the first reply where a second question would have been wiser. And they learn to leave the ticket about the policy system for Aisha. None of this is cheating. It is people responding sensibly to what they are being measured on. That is the subject of metric gaming and its cures. The point here is only that a count displayed as a score will get you what it counts. It will not get you the hard ticket fixed. The hard ticket does not score well.

The number has one honest use, and it is private. A desk lead balancing workload needs to know who holds how many open tickets right now and who resolved how many last month. Those figures are read beside what kinds of tickets they were. That reading is for coaching and balancing, never ranking. The number is also not read at all below a minimum sample. Stonebridge settled on twenty resolved tickets in the period before any per-person figure was looked at. A desk can pick its own threshold as long as it has one. The honest substitute for the slide is the team's volume in and out. If the question is really "is Aisha struggling?", the honest answer is a conversation with Aisha, which no bar chart has ever been.

Averages of Everything

An average is the sum of all the values divided by how many there were. For ticket times it is the single most misleading number a desk can produce. That is because ticket times are not spread evenly. Most tickets take hours; a few take weeks; the average lands between them, in a place where no ticket actually is. Stonebridge's average resolution time for the quarter was about thirty-one hours. The median, the middle ticket when they are all lined up by duration, was about five. The gap was three tickets that had waited six weeks each for the policy system's supplier to ship a fix. The average described a desk that takes four days to fix things. The median described the desk Stonebridge's requesters actually used. An average is a number that has met every ticket and remembers none of them.

The same fault runs through every average on the default report. Average satisfaction on a five-point scale, 3.9, is a number that cannot be acted on. It hides whether that was mostly fours or a mix of fives and ones. The ones are the only ratings worth reading. Average first response is pulled by the ticket that sat unassigned over a bank holiday. The average of averages, each technician's average resolution time averaged again across the team, is a number so far from any ticket that it has stopped being about tickets at all. Any time a report says "average", ask what the slowest tenth looked like, and watch the room.

The honest substitute is the median and the 90th percentile, the value nine tickets in ten beat, reported together. List the slowest few tickets by number so that someone can go and read them. For satisfaction, the substitute is counts: how many ratings, how many good, how many poor, and the comments. The full argument, with a worked example of twenty tickets and the spreadsheet method in plain words, is in median and percentiles, not averages. If a manager insists on the average, show it beside the median once, and let the gap between them make the case.

Utilisation and the Busy Number

Utilisation is the hours a technician logged against tickets divided by the hours they were available. It arrived at help desks from factories and call centres, where a machine or a phone line standing idle is money leaking. It is attractive to anyone who has to justify a desk's headcount, because a desk at ninety-two percent utilisation is visibly busy and a desk at sixty is visibly not. It is also, for a help desk, close to meaningless, and in one important way it is backwards.

The backwards part is this. Tickets do not arrive evenly; they arrive in lumps, forty before ten on a Monday and six on a Wednesday afternoon. A desk with every technician fully booked has nowhere to put the lump, so it waits. The requester's waiting time rises steeply as the last bit of slack disappears. There is no queueing theory needed to see it: when every chair is taken, the next person stands. A desk run at very high utilisation is a desk whose first response and resolution times are about to get worse. That means the number that looks best to a budget holder is the one that predicts the requester's bad week. Add that logging time against tickets is itself work, is itself inaccurate, and is itself the easiest number on the desk to improve by typing. The metric has very little left.

The honest substitute depends on the question. If the question is whether the desk is overloaded, the answer is volume in against volume out and the growth of the backlog. A desk that is genuinely over capacity leaves a trail of tickets behind it and a desk that is merely busy does not. If the question is whether the desk needs another person, the arithmetic is hours of work arriving against hours available, built from a sample of tickets rather than from logged time. That arithmetic is laid out in the headcount case. Neither needs anyone to log minutes against a ticket, and neither rewards a technician for looking busy. Cousins of utilisation that fail the same way include tickets touched, activities logged, and anything called handle time.

The Number That Only Goes Up

A cumulative count is a total since the beginning of time: tickets closed since the desk opened, articles in the knowledge base, users registered on the portal, replies sent. It only goes up, by construction, and so every month it is a record, and so every month it can be announced. Stonebridge's desk celebrated its fifteen-thousandth closed ticket with cake, which was pleasant and told nobody anything. Compare a desk that had fixed every one of them with a desk that had closed every one of them unread. Both would have reached the number at the same moment. A number with no denominator is a brag.

The quieter version is the count with the wrong period. Tickets closed this year, quoted in November, is a number that cannot be compared with anything until next November. Articles in the knowledge base is a count that goes up when somebody writes an article about the history of the desk's naming conventions. It does not go down when forty articles stop being true. Portal registrations rise whenever a new starter is created and never fall when the starter leaves. So after three years the portal has more registered users than the organisation has staff. That is reported as adoption. The common fault is that nothing on the other side of the division line tells you what the number means.

The honest substitute is a rate: the count per period, with the period the same each time, and a denominator that makes it a share of something. Tickets resolved this month beside tickets created this month. Articles that were read last month beside articles that exist. Portal users who raised a ticket this quarter beside people in the building. And where the point is to show that something changed, the honest form is a before and after against a baseline of at least three ordinary months. That is more work than a cumulative count and is the reason the cumulative count exists.

The Metric That Exists Because the Tool Shows It

Every ticketing tool ships with a default reporting screen. The default screen is designed to look impressive on the day the tool is evaluated, not to run a desk with. It has tiles. The tiles have names. Some of the names are standard and some were invented by a product team on a Tuesday. A new desk lead inherits all of them and assumes that a tile called Throughput Velocity must mean something to somebody, because it is green. It does not need to mean anything. It needs to be green.

The test for a tile is the same as for any metric and faster to apply. Can anyone on the desk define it from memory, and name the last decision it changed? Ask about each tile in turn at a team meeting. The usual result is that two or three are the six in disguise. Four or five are averages of things that should be medians. The rest are met with a silence in which somebody eventually says "I think it's tickets per something". Tiles also drift: a count that was useful during a migration stays on the screen for three years, which is long enough to be mistaken for policy. I have inherited a dashboard with a tile counting tickets from a department that had been dissolved. The tile was at zero, and had been for some time, and was still green.

The honest response is to turn the default screen into your screen. Hide or ignore what fails the test, keep the handful, and let the reporting answer the six questions rather than the forty-one it offered. If you run QuerySolv, the standard reports stop at the handful: volume, backlog by age, response and resolution times, reopen and escalation rates, satisfaction. That is a short list, and the shortness is the useful part. Whatever you run, read its list of reports with the test at the end of this article in your hand. Do that before any of them reaches a page that someone else will read. Where the tool cannot produce one of the six, the gap goes in a spreadsheet, not on the slide as something else that happened to be available.

The Honest Substitute for Each

The table puts the five together with what each one hides and what to show instead. The substitutes are all drawn from the six in the first article of this series. That is not a coincidence: a vanity metric is almost always a real question measured the wrong way, and the fix is to find the question and answer it properly. Read the "read with" column as the thing that stops the substitute going the same way.

Two of the six rows deserve a second look because the substitute is not a number at all. Tickets closed per agent, when the real question was "is this person struggling or coasting", is answered by looking at which tickets they hold and talking to them. And the default tile, when nobody can say what question it answered, is replaced by finding the question first. A table of substitutes is only a start. The habit is asking, of every number, which of the six questions it is trying to answer. If the answer is none of them, let it go.

A Short War Story: The Four-Hour Average at Harbourview Clinic

Harbourview Clinic's two-person IT desk asked for a third person trained on the clinical records system. The practice manager, Gareth, turned the request down in a budget meeting using a number from the desk's own report: average resolution time. It had fallen to just under four hours and was, he said, hardly the picture of a team under strain. The average had fallen because the quarter had brought a wave of password resets after a security change, about two hundred of them at ten minutes each. Under that wave the clinical-system tickets had a median of three days and a slowest tenth that ran past two weeks. One of them, from the nursing team about appointment slots showing on the wrong day, waited nine days. During that wait, reception kept paper rotas for one site. The clinic later put the lost staff time at somewhere between twenty-five and thirty-five hours, not counting the patients who were telephoned twice. Priya, the technician who had made the request, recut the quarter by ticket type with medians. She showed the two columns side by side at the next meeting. The third person was approved before the slide was finished. A vanity metric does not merely fail to inform a decision; it makes one, and it makes the wrong one with great confidence.

The Vanity-Metric Test

The test below is run on every number before it goes onto a report, a dashboard or a slide. It is run again once a year on everything that is already there. Part A is five questions; a "no" on one is a warning and on two is a verdict. Part B is the three marks from the start of this article, as boxes. Part C is the decision, and the second option in it matters. Some numbers are useful to the lead and harmful on a wall. The test is how they get kept in the right place. It takes about five minutes a metric, which is less time than the metric will take to explain in a meeting if it is wrong.

THE VANITY-METRIC TEST — RUN BEFORE ANY NUMBER GOES ON A PAGE
Metric: ______________________   Proposed by: ______________   Date: ________
Where it would appear:  [ ] monthly report  [ ] dashboard  [ ] slide  [ ] lead only

PART A — FIVE QUESTIONS (one "no" is a warning; two is a verdict)
  [ ] 1. Decision: the last decision this number changed, or the one it will
         ____________________________________________________________
  [ ] 2. Direction: can it improve while the desk gets worse? If yes, how?
         ____________________________________________________________
  [ ] 3. Denominator: what is it divided by, and over what period?
         ____________________________________________________________
  [ ] 4. Definition: can two people on the desk produce it and get the same figure?
         ____________________________________________________________
  [ ] 5. Partner: which number is it read with, and what does the pair catch?
         ____________________________________________________________

PART B — THE THREE MARKS (tick any that apply)
  [ ] It measures the desk's effort rather than the requester's experience
  [ ] It is an average of things that are not alike, or a count with no denominator
  [ ] It is here because the tool shows it, not because anyone asked a question

PART C — VERDICT
  [ ] Keep: answers a question someone acts on; partner named; definition written
  [ ] Keep privately: useful to the lead for coaching or balancing; never displayed
  [ ] Replace with: ______________________________ (the honest substitute)
  [ ] Drop
  Owner of this metric: ______________   Review it again on: ________

Fiona ran the test on all eleven numbers in the quarterly deck the week after the review. Four passed, three were kept privately, and four were dropped, among them the bar chart. The next quarter's deck had one slide on the desk. It showed volume in and out, backlog by age, median and slowest-tenth resolution time by ticket type, and reopens. Martin asked what had happened to the productivity chart. Fiona said it had been promoted to a conversation, and that Aisha was fine.

Frequently Asked Questions