Measuring What Got Deflected: Counting Shadows Honestly

First Monday of the month, 09:00, at Acme. Sara runs a desk of five and is in the monthly review with Hal, the finance director. He has a reasonable question: "How many tickets did the knowledge base save us last quarter?" Sara has two numbers in front of her. The portal says the knowledge base had 1,840 article views in the quarter. The ticket report says the desk closed 60 fewer tickets than the quarter before. Hal would like one of those numbers to be the answer. Neither of them is, and Sara knows it. She also knows that "it's complicated" is not a sentence that survives a finance review.

The honest answer is a range. And it takes about four sentences to give: "Somewhere between 40 and 75 tickets a month, by three methods that disagree with each other a bit, and here is the sheet that shows how each was counted." That answer is shorter than it looks once the sheet exists, and this article is about building the sheet.

Measuring deflection is estimating, honestly, how many tickets did not need raising because an answer arrived first. It is an estimate, always, because a deflected ticket leaves no ticket behind to count. What it leaves is shadows, in the article views, the "this solved it" clicks, and the drop in a category's ticket count after its article went live. The three honest methods each read one shadow, and the range between them is the answer. This article is part of our Ticket Deflection Strategy series. It counts; turning the count into hours and money belongs to the return-on-investment series, and it is linked where the hand-off happens.

It covers why deflection cannot be counted directly. It then covers the three methods in turn: views beside ticket counts, "this solved it" clicks, and the decline in a category after an article. A worked example from one fictional desk runs through all three. Then come the counting trap in its six forms and counting the other doors. It ends with moving from ledger to money, what to do with the number once you have it, a short war story, and the deflection ledger to copy, in that order.

Why You Cannot Count It Directly

Every other number on a help desk comes from a ticket: tickets opened, tickets closed, time to first reply, tickets reopened. Deflection is the one number that comes from a ticket that does not exist, and that changes everything about how it can be measured. There is no record of the user who read the card on the printer and went back to work. There is no record of the user who searched, found the article, followed it, and closed the tab. The desk's tool knows nothing about either of them, because its whole purpose is to record the people who did come to the desk.

So the measurement reads shadows instead. A shadow is a thing that happens alongside a deflection and leaves a trace. The article was viewed. The user clicked "yes, this solved it". The category's ticket count fell. None of these is a deflection. Each is evidence that one may have happened, with a different reliability and a different direction of error. Views overcount, because many views are curiosity, technicians, and users who read and then raised the ticket anyway. Clicks undercount, because most users who are helped click nothing. Declines count everything that happened to the category that month, including the things that had nothing to do with the article.

The honest measurement uses all three and reports the range between the one that undercounts and the one that overcounts. A single number, however it was arrived at, is a claim to a precision the method does not have. And a claim like that gets used by somebody with a budget. Count the shadows, state the method next to each, and give Hal a range. Ranges are less satisfying and more true, and in a finance review the second property eventually matters more than the first.

Method One: Article Views Beside Ticket Counts

The first method puts each article's view count beside the ticket count for the question it answers, month by month, and reads the two columns together. Every number in this section and the next two is a worked example from the fictional desk at Acme, invented to show the shape of the method. Sara's remote-access article, "Working from home: connecting to the office network (VPN)", had 310 views last month. The remote-access category had 21 tickets. In the three months before the article existed, the category averaged 46 tickets a month and the article, not existing, had no views at all.

What the two columns say together is more than either says alone. The 310 views say the article is being found: the title works, the search finds it, and people are arriving. The fall from 46 tickets to 21 says something happened to the category in the same months. What the views do not say is that 310 tickets were avoided. The temptation to read them that way is the first and largest counting trap. Views include the technicians who open the article to paste its link, the users who read it and raised the ticket anyway, and the users who read it twice. They also include the users who were only curious. A view is a user standing in front of an answer. It is not a user who walked away satisfied.

Use views, then, for reach rather than for deflection. An article with high views and an unchanged ticket count is being found and is not working, which is a writing problem with a precise address. An article with low views and an unchanged ticket count is not being found, which is a title or placement problem. An article with high views and a falling ticket count is the one you hoped for, and the views still are not the count. The next two methods are. The question of which articles earn their keep, read across the whole knowledge base through views, searches and votes, is the subject of Reading Knowledge Base Usage. This article borrows the view count for one purpose only, which is to sit beside the ticket count and explain it.

Method Two: "This Solved It" Clicks

The second method counts the users who said so. Most knowledge bases can put a one-click question at the bottom of an article, "Did this solve it?" with yes and no. Some portals ask the same question after a suggested article on the request form. A yes is the closest thing to a direct observation of a deflection that exists. It records a user with a problem, an answer, and a statement at the moment it happened that the answer worked. Sara's remote-access article collected 14 yes clicks and 6 no clicks last month, from 310 views.

Fourteen is the floor. It is not the total, because the great majority of users who are helped by an article close it and go back to work without clicking anything. A desk that treats the yes count as the whole of its deflection will undercount by a wide and unknowable margin. But fourteen is also a number nobody can argue with: fourteen people said, in writing, at the time, that the article did the job. Report it as "at least", and let the third method supply the "at most".

The no clicks are worth more than they look. Six people found the article, tried it, and said it did not work. That is six people who would have been deflected if the article had been better. Those six probably raised a ticket in the next five minutes. A high no count on one article is a flag to its owner, with the ticket numbers from that week attached so they can see what the article missed. A yes count with no no count beside it is usually a question nobody is answering, which tells you about the question rather than the article. Keep both, by article, by month, and read them together.

Method Three: The Decline After an Article

The third method compares the ticket count for a question before its article existed with the count after, and treats the difference as the ceiling of what was deflected. Sara's remote-access category had an average of 46 tickets a month over the three months before the article. The category averaged 21 over the three months after. That is a fall of 25 a month. Twenty-five is the most the article could have deflected, because it is everything that happened to the category, and some of what happened was not the article.

What else happened is the question that keeps this method honest. In the same month the article went live, the remote-access client was updated, and the update fixed a fault that had been generating its own tickets. Some of the 25 belongs to the update, which is prevention, not deflection. Sara cannot say how much, and says so: "between 14 and 25 a month, probably nearer the lower end, because the client update landed in the same month." The 14 is the floor from method two; the 25 is the ceiling from method three. And the sentence about the update is the method stated beside the number, which is what makes it usable in a review.

The table below is Sara's quarter for five questions, with the other changes column filled in, because that column is where the honesty lives. Read the last two rows especially. The expenses article had views and no decline, and the failed-search log showed why: users typed "receipt" and the title said "attachment". The new-starter row has no article and is there as a control, a category left alone. That way, a general fall in tickets, from a quiet month or a holiday, shows up somewhere it cannot be credited to the knowledge base.

Add the floors and the ceilings across the rows that are genuinely deflection and you have the quarter's range. It is roughly 21 to 37 a month from these three articles, with the password row set aside as self-service and the expenses row waiting on a retitle. That is the number Sara gave Hal, with the table under it. It is less impressive than 1,840 and it will still be true when somebody checks it.

The Counting Trap

The counting trap has six forms, and every desk that reports deflection falls into at least one of them in its first quarter. The first is calling every view a saved ticket, which has been dealt with above and is still the commonest, because 1,840 is such a satisfying number. The second is counting the blocked. A form with a hoop in it produces a fall in tickets that is not deflection, as the rest of this series explains at length. And a desk that measures only the form count will report the hoop as a success. The third is crediting the knowledge base with prevention or self-service, the client update and the password reset, which were real improvements that had nothing to do with an article.

The fourth is double counting across methods: adding the views to the clicks to the decline and reporting the sum. The three methods measure the same deflections through different shadows; they are compared, never added. The fifth is ignoring the calendar. A category that fell in the month half the organisation was on holiday fell because of the holiday, and the control row exists to catch it. A desk without a control row will credit the knowledge base with August. The sixth is reporting a point instead of a range, which is the trap that makes all the others dangerous. That is because a point gets copied into a slide and a range makes somebody ask how it was counted. A deflection figure with no range attached was invented at a slide, whoever's name is on it.

The defence against all six is the same sheet, with the same columns, filled in the same way each month: before, after, views, clicks, other changes, control, range. It is dull. It takes about half an hour a month once the categories are set up. It is also the only version of the number that will not be used against the desk a year later. And that property is worth considerably more than the half hour.

Counting the Other Doors

A fall in tickets by one door is not a fall in demand until the other doors have been counted for the same period. If the form count fell by 40 and the phone log, the walk-ups and the technicians' inboxes rose by 35 between them, the knowledge base deflected about 5. In that case, something on the form is pushing people to the phone. The other-doors check is a row at the bottom of the ledger, filled in every month. It records tickets by form, by e-mail, by phone, by chat, by walk-up. It also records a note of any change to the form or portal in the month. Deflection shows as a fall across all of them. A hoop shows as a fall in one and a rise in the rest.

The check is easiest when every door lands in the same queue, because then "tickets by door" is a filter on one list rather than a tour of a phone log, a shared mailbox and three people's memories. If you run QuerySolv, requests by web form, e-mail, chat and social messages all become ordinary tickets in the one queue. In that case, a phone call the technician logs sits beside them. That makes the other-doors row a report rather than a reconstruction. In any tool, the walk-ups are the door that goes unrecorded, and the only fix is the habit of logging a ticket for every corridor conversation that would have been one, however short. The share of tickets by channel, and what each share is telling you about the portal, is read in full in Measuring Portal Adoption. Here it is one row, and it is the row that stops a hoop from being reported as a result.

From Ledger to Money

The ledger ends with a range of tickets a month, and that is where this series stops and the return-on-investment series starts. Converting the range into hours uses the desk's own figure for the time a ticket of that kind takes, from first touch to close. The calculation then subtracts the hours spent writing and maintaining the article. That way, the result is a net saving in hours and, if the organisation wants it, in money. The rules for that sum, and in particular the rule that it is done with ranges and never with the ceiling alone, are in Deflection Math Without Fooling Yourself. The ledger's job is to hand that article an honest range with the method attached. Everything after that is arithmetic.

Keep the two documents separate on purpose. The ledger is the desk's working record, filled in monthly, with the notes about client updates and retitles that make it true. The money sum is the quarterly view for people who do not want the notes. It should cite the ledger rather than replace it. That way, when Hal asks how the figure was counted, there is a sheet to show him rather than a memory to consult. A desk that keeps only the money sum has kept the conclusion and thrown away the evidence. And the next finance director will not take the conclusion on trust. The one-page brief that carries the result upward and the before-and-after shape it should take also belong to the return-on-investment series. The ledger is the brief's appendix.

What to Do With the Number

The number is for deciding what to do next, and each pattern in the ledger points somewhere specific. High views, low decline, many no clicks: the article is found and does not work, so it goes back to its owner with the week's ticket numbers. Low views, no decline, the question still in the failed-search log: the article is not being found, so the title or the synonyms are wrong. In that case, the article goes to whoever runs the search tuning. Found, working, and the tickets still arriving from the people who never search: the answer is in the wrong place. In that case, the fix is a card, a banner or a line in the acknowledgment, which is the subject of Where the FAQ Goes. The ledger does not only measure the last article. It chooses the next move.

And when a row works, when the floor and ceiling are both high and the other-changes column is empty, that row tells you what the next article should be. In that case, it is the next question in the queue that looks like this one. The ledger, run for a year, becomes a record of which kinds of question deflect well on this desk, with these users. And that record is worth more than any general advice about what to write, including the advice in this series. Some desks find that device questions deflect and account questions do not. Some find the opposite. The ledger will tell you which desk you are, in about three quarters, if you keep filling it in.

A Short War Story: Nine Hundred Views at Marlow Transit

At Marlow Transit, a bus operator, the desk lead, Fiona, built a knowledge base over a summer. She reported to the operations director, Graham, that it had "deflected over nine hundred tickets" in its first quarter. The figure was taken directly from the portal's article view count because it was the only number the portal offered and nobody had said it was the wrong one. Graham, who had a budget review the following week, used the figure to justify not replacing a technician who was leaving. The arithmetic was that nine hundred fewer tickets a quarter was most of one person's work. The actual fall in tickets over the same quarter, when Fiona went back and counted by category with the before-and-after method, was somewhere between sixty and ninety. Much of it was in the two months after the ticketing machines at the depots were replaced. The desk of four became a desk of three. The backlog grew by about two hundred tickets in the next two months. A depot rollout slipped by three weeks because the technician who would have done it was working the queue. Fiona now reports a range, with the method and the other-changes column attached. Graham, to his credit, now asks for them. A number without its method does not belong to the person who reported it. It belongs to whoever has a budget to balance, and they will use it.

The Deflection Ledger

The ledger is one page a month. It has one row per question that has an article or a placed answer. It has the before and after counts, the two shadows, the other-changes column, a control row, and the other-doors row underneath. Fill in the "before" column once, from the three months before each article went live, and leave it alone. Fill in the rest on the first working day of the month, from the ticket report and the portal's counts. Write the range in the last column with the floor from the yes clicks and the ceiling from the decline. The sheet is dull by design. It is the version of the number that will still be true when someone checks. The quarterly total it feeds goes to the return-on-investment sum as a range, never as the ceiling alone.

THE DEFLECTION LEDGER — ONE PAGE A MONTH
Desk: ______________________   Kept by: ______________   Month: ________

1. PER QUESTION (one row per article or placed answer; "before" is the three-month average before it went live)
   Question / category         Before   This month   Views   Yes / No   Other changes this month        Range (floor = yes; ceiling = before minus this month)
   _________________________   _____    _____        _____   ___ / ___  _____________________________   ____ to ____
   _________________________   _____    _____        _____   ___ / ___  _____________________________   ____ to ____
   _________________________   _____    _____        _____   ___ / ___  _____________________________   ____ to ____
   _________________________   _____    _____        _____   ___ / ___  _____________________________   ____ to ____
   _________________________   _____    _____        _____   ___ / ___  _____________________________   ____ to ____
   Control row (a category with no article, left alone):
   _________________________   _____    _____        -       -          _____________________________   general fall? [ ] yes [ ] no

2. SET ASIDE (not deflection; recorded so they are not credited to the knowledge base)
   Prevention (cause fixed):  ______________________________   tickets fell by ____
   Self-service (tool switched on):  ______________________   tickets fell by ____
   Season / holiday / headcount change affecting the month:  __________________________

3. THE OTHER DOORS (same month; a fall in one door with a rise in others is a hoop, not deflection)
   Form: ____   E-mail: ____   Phone: ____   Chat: ____   Walk-up (logged): ____   Total: ____
   Changes made to the form or portal this month: ______________________________________

4. THE MONTH'S RANGE
   Sum of floors: ____   Sum of ceilings: ____   Stated as: "between ____ and ____ a month"
   Nearer which end, and why: ______________________________________________________

5. NEXT MOVES FROM THE LEDGER
   Found, not working (high views, many no, little decline):  article -> owner: ______________
   Not found (low views, question still in failed-search log):  title -> search tuner: ________
   Working but tickets still arrive from non-searchers:  place the answer at: _______________
   Next article to write (the next question like the best row):  ___________________________

6. TO THE QUARTERLY ROI SUM
   Quarter's range: between ____ and ____ a month   Method and this sheet attached: [ ]

Section four is the sentence Sara read to Hal, and section two is why he believed it. A desk that shows what it has set aside, the reset it did not claim and the client update it did not credit to itself, is a desk whose range gets trusted. And a trusted range does more for the next budget conversation than any number of views. Hal asked one more question at the end of the review, which was whether the expenses article would be retitled before next month. It was. That is what the ledger is for.

Frequently Asked Questions