Measuring the Tickets That Never Arrived: Prevention, Honestly

Wednesday, 14:00, at Meridian Parts. Tomasz, who runs the desk, is stopped in the corridor by Ruth, the finance director, who has his budget paper in her hand. "You say here the Friday sweeps are worth keeping. How many tickets have they actually stopped?" Tomasz has a feeling, a count of sweep tickets, and an honest answer, which is that he does not know. He says "quite a lot", which Ruth writes down. Back at the desk he opens the reports and starts counting disk-space tickets by month. Within an hour he has a number he likes, a number he can defend, and the uncomfortable discovery that they are not the same number.

You cannot count an absence. You can only count what used to be there and is not, and then argue about why. Prevention measurement is comparing the rate of a ticket type before the desk started preventing it with the rate after, from the same machines. It uses a counting rule written down in advance and reports the result as a range. This article is part of our Proactive Remediation series and is its last. That is because every sweep, swap and scheduled job in the five before it produces work that nobody sees. This is how the desk shows it without inflating it. The conversion of hours into money belongs to the self-service ROI series in this library. This article stops at tickets and hours, which is where the honest part is.

It covers why counting prevention is hard and the two ways desks get it wrong. Then come before-and-after by type and the counting rule and why it is written down first. Next come ranges rather than points, with a picture, and turning tickets into hours. After that come the other signals, hardware and patterns, which count differently. Then come the trap of the quiet month, writing it up, and a short war story. Last is the prevention ledger to copy.

Why This Is Hard, and the Two Ways to Get It Wrong

A prevented ticket has no record of its own, so every count of prevention is an inference from something else. Inferences have alternatives. If disk-space tickets fell from twenty a month to seven after the sweep began, the sweep is the obvious explanation and not the only one. The fleet might have shrunk. A department might have moved to a contractor's systems. The new standard build might have shipped with bigger drives. Users might have stopped reporting because the last three tickets took a week. Any of those produces the same drop. The budget paper that attributes all of it to the sweep is making a claim it cannot support if anyone looks. Someone always looks.

The first wrong way is to claim everything: every quiet week is prevention, every drop in volume is the sweep, every hit on the report is a ticket that would have arrived. The second wrong way is to claim nothing, on the grounds that the number cannot be known exactly. That leaves the desk's best work invisible and its budget exposed. Both are failures of the same kind: they treat a quantity that can be bounded as if it had to be either exact or unknowable. It is neither. It sits in a range, and the range can be defended.

I once reported that a sweep had prevented "about forty tickets a month", a figure I had arrived at by feeling confident. It was eleven. I know because someone checked, which is a thing that happens to confident numbers. The eleven would have been a perfectly good result if I had led with it. The rest of this article is the method I use now, and it starts with a decision about what, exactly, is being counted.

Before and After, by Type

The unit of measurement is one ticket type, counted monthly, from the same population of machines, before and after the prevention work began. The type is the one the sweep targets, in the words the desk records at close. That means "disk space" for the disk sweep, "hardware failure" for the hardware sweep, and the pattern's own words for a scheduled job. Count it for three to six months before the start date and the same number of months after, from the same machines. The difference between the two averages is the gross drop: the most the sweep could possibly have prevented, before any honesty is applied.

Same population matters more than it sounds. If the fleet grew from fifty laptops to seventy in the after period, the raw count flatters the sweep. If it shrank, the count hides it. Count per machine per month where the fleet has changed, or restrict the count to the machines that were present throughout. Same type matters too. It depends on the ticket type being recorded truthfully when the ticket closes, which is the discipline in Categorize at Close, Not Just at Open. A desk whose "disk space" tickets were filed as "slow" half the time before the sweep and as "disk space" after will measure a rise, and will conclude something strange.

Where the type was not recorded cleanly in the before period, which is common on a desk that started this work recently, count by the words in the ticket instead. A search for "space", "full", "cannot save" and "slow" in the before period's titles and first lines, read by a person for an hour, produces a before count that is good enough. The after count comes from the proper type. Write down that the before count was done by search and the after by type, because that is a difference in method and belongs in the ledger. The method of the deflection series, in Deflection Math Without Fooling Yourself, is the same shape applied to a knowledge base article rather than a sweep. The two ledgers can sit on the same page.

The Counting Rule, Written Down First

The counting rule is the sentence that says which of the gross drop the desk will claim as prevented. It is written before the counting starts. Before. Not while the number is on the screen looking a little small. The rule produces two figures. The low estimate counts only the sweep hits that clearly match. For a clear match, the machine must have been at the act level. Also, either that machine had produced a ticket of this type in the previous year or the condition was already producing symptoms the user had mentioned. The high estimate counts the whole gross drop. The truth is somewhere between, and the ledger reports both ends rather than picking one.

Hits are not prevented tickets, and the rule has to say so. A sweep that cleans a laptop at nine percent free has found a condition. Whether that condition would have become a ticket depends on how fast the drive was filling, and whether the user would have noticed. It also depends on whether the machine was due for replacement the following month anyway. Some hits would never have become tickets. Some tickets would have arrived from machines the sweep never reached. The low estimate's "clear match" test is the desk's way of counting only the hits where the ticket was plainly on its way. The test is deliberately strict, so that the low end is a figure nobody can argue down.

Here is the worked example this article follows, from a fictional desk of sixty laptops. Disk-space tickets ran at about twenty-two a month for the six months before the sweep began and about seven a month for the six months after, from the same machines. The gross drop is fifteen a month. Of the sweep's hits over the after period, nine a month passed the clear-match test. The remaining six a month of the drop have no hit behind them and may be the sweep, may be the new build, may be chance. The desk claims nine to fifteen prevented tickets a month. It does not claim fifteen.

Ranges, Not Points

A range is reported because a point invites an argument about the point and a range invites a decision about the work. Tell a finance director the sweep prevents fifteen tickets a month and the conversation is about whether it is really fifteen. It is not, and the desk cannot prove it. Tell her it prevents between nine and fifteen, with the nine counted strictly and the fifteen being the whole drop. Then the conversation is about whether nine is worth an hour a week, which it is. The fifteen is a bonus nobody has to defend. The range is not a hedge. It is the shape of what the desk actually knows.

The picture below is the worked example drawn. Six amber bars at about twenty-two a month, the dashed line where the sweep began, six green bars at about seven. Over the after period, two bands show how the drop of fifteen splits. The lower band, nine a month, is the clear matches, the low estimate. The upper band, six a month, is the rest of the drop with no hit behind it. The footer is the sentence that goes in the report.

Both ends of the range carry their rule when they are reported. "Nine a month, counting only hits on machines that had this ticket before or were already showing symptoms; up to fifteen, counting the whole drop" is one sentence, and it is the sentence that survives a finance director with a spreadsheet of her own. That is the moment most prevention claims meet their first serious reader. A range without its rule is two guesses. A range with its rule is a measurement with its error bars, and finance directors are more comfortable with error bars than desks tend to assume.

From Tickets to Hours

Tickets become hours by multiplying the range by the time a ticket of that type used to take. The time comes from the before period's resolved tickets rather than from anyone's memory. Take the disk-space tickets from the six months before the sweep. Read the time each took from open to resolved as recorded on the ticket. Find the middle of the distribution and the spread. On the fictional desk, a disk-space ticket took the technician forty-five minutes to an hour and a half, counting the call, the visit or remote session, the cleanup and the note. So the range is three quarters of an hour to an hour and a half. Nine to fifteen prevented tickets a month at that rate is roughly seven to twenty-two desk hours a month, gross.

The user's time is a separate line, never blended into the desk's. A user with a laptop that froze during a report lost somewhere between half a morning and a day by the time the ticket was raised, waited for, and fixed. The ledger records that as its own range: nine to fifteen tickets at two to six hours each is eighteen to ninety user hours a month. It is a bigger number than the desk's, and it is softer. The people who read budget papers know it is softer, so keep it apart and let it be seen apart. A desk that adds its hours and the users' hours into one total has made a number that nobody can check.

Subtract the cost of the sweep. An hour a week for the pass plus the cleanups is four to six desk hours a month on the fictional desk. The net is therefore the gross desk range less that: roughly one to eighteen desk hours a month, plus the user time. The low end is small and it is honest, and it is still positive, which is the point. The sweep pays even on its strictest count, and the paper says so. The same arithmetic for a scripted fix, where the question is whether the hours spent building the button were paid back by the hours it saved, is in Measuring Whether the Buttons Pay. The two ledgers share their per-ticket figures.

Hardware and Patterns Count Differently

Hardware prevention is measured by the ratio of planned swaps to unplanned failures, not by a ticket count, because a swapped drive is still a ticket. Count the hardware tickets in the before period and sort them into "failed in use" and "replaced by appointment". Do the same after the hardware sweep began. The movement from the first column to the second is the result. It is reported as "of thirty-one hardware events this quarter, twenty-four were planned swaps and seven were failures in use; a year ago the split was nine and nineteen". Beside it goes the count of data-loss incidents, which should fall toward zero and is the figure a director remembers.

Pattern-triggered maintenance has the cleanest count in the series, because a scheduled job that works removes a ticket type entirely. A column going to zero is hard to misread. Count the pattern's tickets by month before the job and after, from the same population, exactly as for a sweep. Apply the control check in the next section before claiming it. Where the job also opened failure tickets of its own, count those too, because a job that replaced four user tickets a month with two of its own has prevented two, not four. Each pattern row on the maintenance table gets its own line in the ledger. The lines are not added together until the control check has been done on each.

The Trap of the Quiet Month

Before any drop is attributed to the desk's work, three alternative explanations are checked and the check is written on the ledger. Did the population change: machines added, removed, rebuilt with a different drive, moved to a contractor? Did something else change in the same period: a new build, a replaced server, a department that left the building? And did users stop reporting? That is the quietest failure of all and the one The Seven Ways Support Quietly Fails describes. In that case, a drop in tickets means a drop in trust rather than in faults. The third is checked by asking: three users whose machines were on the list, "has your laptop been slow lately, and did you tell us?"

The control type is the ticket type the sweep does not target, chosen in advance, whose count should have stayed flat. If disk-space tickets fell by two thirds and password resets, which no sweep touches, also fell by two thirds, something happened to the whole queue. In that case, the sweep cannot claim its share until that something is found. If the control stayed flat while the target fell, the drop belongs to the target's cause, and the sweep is the leading candidate. The control is one line on the ledger and five minutes of counting. It is the single most persuasive thing the desk can show a sceptical reader, because it is the check the sceptical reader was about to do.

Every vendor of prevention tools has a calculator that produces a number ending in several zeros. None of them has a column for the department that moved out. The desk's ledger does. That is the whole difference between a number that gets a budget line and a number that gets a raised eyebrow, and it costs a page.

Writing It Up

The monthly report carries one line, "found before reported", with the count of proactive tickets raised that month and the counting rule in brackets beside it. The quarterly note to management carries three lines: what was swept, what was found, and what did not arrive as a range with its rules. It uses the form Proving It to Management in One Page sets out. The ledger sits behind both as the working, one page per ticket type, and is produced when somebody asks how the range was reached. Three documents, three audiences, one set of numbers.

The proactive tickets themselves are what make the monthly line possible. That is why every article in this series insists on raising one per hit with a first line that says so. A desk whose proactive tickets carry that marker can count them in a minute. A desk whose sweep work lives in a spreadsheet counts nothing. If you run QuerySolv, the volume reports by period give the before-and-after counts without exporting anything. In QuerySolv, a search on the marker in the first line gives the proactive count beside them. In any tool, the test is that the three numbers on the quarterly note can each be traced to a filter someone else could run.

Resist improving the number. The temptation, once the ledger exists, is to adjust the counting rule after a weak month, widen the before period to a worse one, or quietly drop the control. Each of those produces a better figure once and a worse reputation permanently, because the reader who checked last quarter will check this one. The rule that moved is the thing they will find. A prevention figure that is modest and stable for a year is worth more in a budget meeting than one that is impressive and has been restated twice. Keep the rule, report the range, and let the number be what it is.

A Short War Story: The Three Hundred at Lantern Media

Lantern Media's desk lead, Hal, put a figure of three hundred prevented tickets a year into a budget paper asking for a third technician. He had taken the whole year-on-year drop in volume and attributed it to the sweeps and scheduled jobs the desk had introduced. The finance director, who had a spreadsheet of her own, noticed that the editorial department of forty people had moved to a contractor's systems in the same year. They had taken their tickets with them. She also noticed that password-reset tickets, which no sweep touched, had fallen by the same proportion as everything else. The three hundred, recounted with a control type and a clear-match rule, came to somewhere between eighty and a hundred and twenty. That was a good result and would have supported the paper on its own. The paper was declined. The third technician was not hired that year. The desk's numbers were quietly discounted in every meeting for the following two. The sweeps had worked. Hal had simply reported the drop instead of the measurement, and the cost of the difference was a hire.

The Prevention Ledger

The ledger below is one page per ticket type. Every line on it is either a count somebody else could repeat or a rule written before the count. Fill the before and after lines from the ticket reports and the hits line from the sweep tickets. Fill the control line from the type you chose in advance and the hours lines from the before period's resolved tickets. Let the range at the bottom be whatever the lines produce. Keep every quarter's ledger; the stable rule across quarters is what makes the numbers believable.

THE PREVENTION LEDGER — ONE TICKET TYPE, ONE PAGE
Ticket type: ______________________   Prevention work: ______________________
Start date of the work: ________   Population: ______ machines, same throughout [ ]
Prepared by: ______________   Quarter: ________

COUNTING RULE (written before counting; do not change it after)
  Low estimate counts a hit only if: the machine was at the act level AND
  (it had a ticket of this type in the previous year OR symptoms were already
  reported). High estimate counts the whole drop.

 1. Before period: ____ months, from ________ to ________
    Tickets of this type per month (list): ____ ____ ____ ____ ____ ____
    Average before:                                           ______ a month
 2. After period: ____ months, from ________ to ________
    Tickets of this type per month (list): ____ ____ ____ ____ ____ ____
    Average after:                                            ______ a month
 3. Gross drop (line 1 minus line 2):                        ______ a month
 4. Hits in the after period that pass the counting rule:    ______ a month
 5. Control type (chosen in advance): ______________________
    Before average ______   After average ______   Flat? [ ] yes  [ ] no
    If no: what else changed, and how was it allowed for? __________________
 6. Alternatives checked: population change [ ]  other change [ ]
    users stopped reporting [ ]   Notes: __________________________________
 7. Prevented tickets, low (line 4) to high (line 3):   ______ to ______ a month
 8. Desk hours per ticket, from before-period resolved tickets:
    middle ______ h   spread ______ to ______ h
 9. Gross desk hours (line 7 times line 8):              ______ to ______ a month
10. Cost of the prevention work (sweep hours, cleanups, job checks):
                                                          ______ a month
11. Net desk hours (line 9 minus line 10):               ______ to ______ a month
12. User hours, kept separate (line 7 times ______ to ______ h each):
                                                          ______ to ______ a month
13. Reported sentence for the quarterly note:
    "______ to ______ tickets a month, counting ______________________;
     ______ to ______ net desk hours; ______ to ______ user hours."
Ledgers for other types this quarter: ______________________________________

Tomasz went back to Ruth on the Friday with one page: twenty-two a month before, seven after, nine to fifteen prevented, the control type flat, one to eighteen net desk hours a month and a separate line for the users. Ruth read it, asked what the control type was, and nodded when he explained. She wrote "nine to fifteen" in the margin of the budget paper next to where she had written "quite a lot". The sweeps stayed. The next quarter's ledger used the same rule, and so did the one after that. The number has not needed restating since, which is the only kind of number a finance director keeps.

Frequently Asked Questions