Measuring Whether the Buttons Pay: Runs, Failures, Minutes

Monday, 14:10, at Larkfield Academy, a trust of four schools with a desk of three. Ingrid, the business manager, stops by Joel's desk on her way to a budget meeting. "Those scripts you spent three weeks on in the spring. What did we get for them?" Joel knows the answer is "a lot" and knows that "a lot" is not an answer. He has a folder with five scripts in it and a management console that shows a button for each. He has a vague memory that the drive one gets pressed most days, and no number of any kind. Ingrid is not being hostile. She is asking the question anyone paying for three weeks is entitled to ask. And she will ask it again in the autumn, and the honest reply right now is that he does not know.

A remediation ledger is one page per button that records how many times it ran, how many of those runs fixed something, and how many failed. It records how many tickets never had to be raised because of it and how many minutes the desk got back. Those minutes are set against the hours it cost to build and keep. The remediation ledger is the same shape as the ledger a desk keeps for a knowledge base article. It follows the same rules: counts you can show and minutes as a range rather than a point. Every number comes from your own desk and none from anybody's brochure. This article is part of our Scripted Fixes & One-Click Remediations series. And it is about turning a folder of scripts into a page Ingrid can read.

It covers the five numbers to keep per button and where each number comes from. Next come minutes saved, counted honestly as a range, and tickets avoided, with why only some buttons avoid any. Then come the cost side, including failures, and the ledger worked through for one button. The article covers the buttons that do not pay and what to do with them, then the one page for management. It finishes with a short war story and the remediation ledger to copy, in that order. The wider method for turning counts into hours and hours into an argument is in Measuring Self-Service ROI, and this article leans on it rather than repeating it.

Five Numbers Per Button

The first number is runs: how many times the button was pressed, or the rule fired and acted, in the period. It is the easiest to count and the least interesting on its own, because a run is not a saving; it is an event. The second is successes: runs where the script found the problem, acted, and its check afterwards passed. The third is failures, and it has two parts that must be kept apart. One is runs that acted and whose check did not pass. The other is runs that refused to act because something was not as expected. A fourth kind of run, where the script found nothing to do, is neither success nor failure and gets its own small count. A button that often finds nothing to do is being pressed for the wrong symptom.

The fourth number is tickets avoided, and for most buttons it is zero, which surprises people. A button pressed by a technician is pressed on a ticket that already exists; the button saved minutes on that ticket, not the ticket itself. Only a fix that ran before anyone noticed the problem avoids a ticket, and even then only if the problem would otherwise have been reported. That means a self-healing rule or a proactive sweep. The fifth number is minutes saved, which is the one everybody wants and the one most often made up, and which the next sections are about. Five numbers, per button, per quarter. The ledger has room for nothing else, on purpose.

Keep the five numbers per button and never add them across buttons until the last line of the page. Take a desk with six buttons that reports "the buttons saved ninety hours". It has hidden the fact that one button saved seventy of them and two saved nothing. And the two that saved nothing are the ones Ingrid should hear about, because retiring them is also a saving. The ledger is a page per button so that each one can be judged on its own. It has a summary line at the end so that the total exists for whoever needs it.

Where the Numbers Come From

Runs, successes and failures come from the log line the script writes on every run, and from nowhere else. If the script was built the way this series describes, every run wrote one line with an exit code. In that case, the quarter's counts are a matter of counting lines by code: zero for fixed, one for nothing to do, two for failed, three for refused. That is five minutes with the log file and a spreadsheet, and it is accurate to the run. If the script does not write such a line, the ledger cannot be filled in honestly. In that case, the first entry on the cost side is the hour it takes to add the logging.

The ticket side gives you the same counts a second way and a check on the first. Every button press wrote an internal note on a ticket, and every self-healing act did the same. So the number of notes mentioning the button's name in the quarter should match the log's count of runs that acted. If you run QuerySolv, those notes sit in each ticket's event history alongside everything else that happened to the ticket. In that case, the ticket-side count means finding the button's name across the quarter's tickets rather than asking who remembers pressing it. In any tool, when the two counts disagree, something is wrong with the button's record-keeping, and the ledger's first finding for the quarter is that disagreement.

The minutes come from the candidate list, which is where the manual fix was timed before anyone wrote a script, and from timing the button. The candidate list from the first article in this series has a "minutes by hand" column. It was filled by timing three real manual fixes and taking the middle one. That number is the "before". The "after" is the time to press the button, read the record and write the reply to the user. That is usually one to three minutes and should also be timed rather than guessed. If the candidate list was never filled in, time the manual fix now, three times, when the button refuses and someone has to do it by hand. The refusals are not only failures. They are your control group.

Minutes Saved, Honestly

Minutes saved per successful run is the manual time minus the button time, and it is written as two numbers, not one. The low number uses the hands-on manual time only: the minutes the technician spent at the keyboard and at the user's desk doing the fix. That was twelve for a drive remap at Larkfield, which the candidate list recorded. The high number adds what a fix really costs the desk. It adds the walk, the interruption to whatever the technician was doing, the conversation, and the time to get back into the interrupted task. For the same drive remap that might be twenty-five minutes. Neither number is wrong. They answer different questions, and reporting only the high one is how desks lose the trust of people who can multiply.

Subtract the button time from both. Pressing the button, reading the record and writing a two-line reply takes about two minutes at Larkfield. So the drive remap saves ten minutes per run at the low end and twenty-three at the high. Then multiply by successes only. Do not multiply by runs. A run that found nothing to do saved nothing, because the user's problem was something else and still needed a person. A failed run saved less than nothing, because the technician pressed the button, read the failure and then did the fix by hand anyway. The arithmetic for counting only what you can defend, and the reasons the low column exists at all, are set out in Deflection Math Without Fooling Yourself. The same discipline applies here with the word "article" replaced by "button".

Resist the loaded hourly figure unless somebody who owns the budget asks for it. Minutes saved are the desk's minutes, and the desk did not shrink. The minutes went into other tickets, which is the point and is also hard to see. "The drive-remap button gave the desk back between nine and twenty-one hours this quarter" is a sentence Joel can stand behind. "The button saved the trust eight hundred pounds" is a sentence Ingrid will test by looking at the payroll, and it will not be there. How technician hours convert, when they do, is in The Time Saved by Technicians; the ledger stops at hours, and lets the reader convert if they want to.

Tickets Avoided

A ticket is avoided only when a fix ran before anyone raised one. The only buttons that do that are the ones nobody presses: self-healing rules and scheduled sweeps. Larkfield's night-time rule restarts the library catalogue service when it stops. In the quarter it acted nine times, each at a time of night when nobody was using the catalogue. Without the rule, the morning would have brought a ticket from a librarian on perhaps seven of those nine occasions. On the other two, whoever was first in would have done a quiet restart. The count of tickets avoided is therefore not nine. It is somewhere between seven and nine, and the ledger says so.

Each avoided ticket saves the whole ticket, not just the fix: the triage, the reply, the walk, the close. That makes an avoided ticket worth more minutes than a shortened one. It is why self-healing rules, when they are justified, pay back faster than buttons. It is also why the temptation to overcount them is strong. The honest method is to look at the ticket count for that symptom in the quarters before the rule and the quarters after. Attribute to the rule only the fall that matches the rule's acts. That is the method set out in Measuring the Tickets That Never Arrived. The rule's own count of acts is the ceiling; the before-and-after fall is the evidence. The ledger reports the lower of the two in its low column and the acts in its high.

For a button a technician presses, the tickets-avoided line reads zero, and that zero is worth writing down. It stops the quarterly conversation drifting towards "the buttons mean fewer tickets", which they do not. It points at the right question: whether the fix underneath should eventually run itself. A button pressed sixty times a quarter with sixty successes and no cost to the user is a candidate for a self-healing rule card. The ledger is where that candidacy becomes visible.

The Cost Side

The cost side has three lines. And the first is the build: the hours spent writing the procedure, writing and testing the script, writing the specification, and having both reviewed. Count them honestly from the start, in a line on the candidate list. They are the hours Ingrid asked about and they are the easiest to underestimate from memory. Joel's drive-remap button took about seven hours across a fortnight, most of it in the test ladder and the review rather than the script. That is a typical shape. The script is the quick part. The confidence is the slow part.

The second line is upkeep, which is the hours each quarter spent keeping the button true. It includes re-testing after the operating system updates, reading the refusals and adding a branch. It also includes reviewing the specification and re-testing the off switch on a rule. An hour or two a quarter per button is normal; zero is not. And a button whose upkeep line reads zero for a year is a button nobody has looked at, which is a risk, not a saving. The third line is failures. Every failed or refused run cost a technician the minutes to press, read, and then do the fix by hand. There were also some minutes working out why. Twenty minutes per failure is a fair charge at Larkfield. And the ledger multiplies the failure count by it and puts the result against the button.

Failures also carry a cost the ledger cannot count, which is trust. A button that fails one time in three stops being pressed, and once it stops being pressed its successes fall to zero and its upkeep continues. The ledger shows this as a falling run count beside a flat upkeep line. And the right reading of that pattern is "fix the script or retire the button", not "the fix is needed less often". Watch the ratio of failures to runs each quarter. And treat anything above about one in ten as a defect in the script that the next quarter's upkeep line must pay to fix.

The Ledger Worked Through

Here is Larkfield's drive-remap button for one quarter, in the two-column form. The counting rules are written beside the numbers so that a reader can disagree with a rule rather than with the arithmetic. The log counted sixty-one runs: fifty-five fixed, three nothing to do, two failed, one refused. The candidate list said twelve minutes by hand. Joel's own estimate of the loaded cost, with the walk across to the humanities block and the interruption, was twenty-five. The button takes two minutes including the reply. Build was seven hours, upkeep this quarter an hour and a half for a re-test after an update.

Read the range, not either column. The button gave the desk back somewhere between seven and nineteen hours this quarter after its own costs, and it paid for itself somewhere between the end of the first quarter and the start of the second. That is a true sentence Joel can say to Ingrid without flinching, and the two columns are what let him say it. If she thinks the loaded figure is generous, the low column is still there, and it still pays. If the low column did not pay, the right answer would be to say so, which is the subject of the next section.

Notice also what the ledger made visible that the folder of scripts did not. Three runs found nothing to do, which means three times a technician pressed the drive button for a symptom that was not the drive. The specification's "fixes" line might need a word about what else "my drive has gone" can mean. One run refused, and the reason in the log was a mapping pointing at a server that had been retired. That is the same problem the script was taught to spot during testing and a sign that a machine missed a policy update. None of that was going to be found by remembering that the drive button gets pressed most days.

The Buttons That Do Not Pay

Some buttons do not pay, and the ledger's job is to say so plainly. A button that does not pay is costing upkeep every quarter and is sitting in the catalogue pretending to help. The clock-resync button at Larkfield was pressed four times in the quarter, saving six minutes each at the low end. That is twenty-four minutes against an hour of upkeep. It is a good script. It is not a good button. The right move is to retire the button and keep the script in the folder for the technician who needs it twice a year. Take the line off the catalogue. Nobody is embarrassed by this. The candidate list said the fix was rare; the button was built in a burst of enthusiasm; the ledger corrected the enthusiasm. That is the ledger doing its job.

A button whose run count has fallen to near zero for a different reason is the best news on the page. Suppose the drive-remap button goes from sixty runs a quarter to six. The problem record beside it also says the file server was replaced in the meantime. In that case, the problem has been fixed permanently and the button has worked its way out of a job. Retire it with a note, keep the ledger page as the evidence that the permanent fix landed, and move on. A desk that keeps every button it ever built, because retiring one feels like admitting it was unnecessary, ends up with a catalogue of monuments.

And a button whose successes are high but whose tickets reopen is a button that is lying a little. If the drive remap "fixes" the drive and the user is back the next morning with the same symptom, the fix is a workaround for something upstream. In that case, the success count flatters it. Check the reopen rate for tickets the button touched against the desk's usual reopen rate once a quarter. If it is noticeably higher, the minutes saved are real but the button is a splint. In that case, the problem record needs attention more than the ledger does. The ledger has a line for this, and it is the line most often left blank.

The One Page for Management

Ingrid gets one page, once a quarter, and it says four things in this order. First comes what the buttons are, in a sentence each. Then come the hours they gave back, as a range, after their own costs. Next come the ones that were retired and why. Last comes what the desk did with the hours. The last point is the one managers actually want and the one desks forget to make. "The nine to twenty hours went into the laptop refresh and into the backlog, which fell from eighty-one open tickets to fifty-four" is the sentence that turns a ledger into a reason to approve the next batch. Without it, the hours are abstract, and abstract hours do not survive a budget meeting.

Write the page the way Explaining Technical Things to Management says to write anything for a reader with eleven other pages to read. Put the conclusion first and the numbers as ranges with the counting rule in a footnote. Include no exit codes, no script names, and no money unless the reader has asked for money. In that case they supply the hourly rate and you supply the hours. The one-page form that works for a knowledge base's return, with the same two columns and the same honesty about the judgement call, is in Proving It to Management in One Page. And the buttons' page can be a section of that one rather than a document of its own.

Say the failures out loud on the page. "The spooler button failed four times in the quarter and we have fixed the cause" is a sentence that builds more trust than any success figure. It tells the reader the numbers are being counted by someone who counts the bad ones too. Managers have read many reports in which everything worked. They remember the ones where something did not and the author said so.

A Short War Story: The Four Hundred Hours at Pennant Travel

Zara ran the desk at Pennant Travel, a chain of eleven travel agencies. After a year of building buttons she reported to the finance director that they had saved the business four hundred hours. She reached that figure by multiplying every run of every button by forty-five minutes, her estimate of a fully interrupted manual fix. She counted the runs that found nothing to do as savings. The director did what directors do with a round number: divide it by the hours in a working year. The director asked which quarter of a technician she had been able to do without. The desk had not shrunk, had in fact hired, and Zara had no second column to retreat to. The number was not believed. Because of that, the request attached to it for a management console with stored actions was deferred for a year. The following spring she re-did the ledger with the counting rules in this article. She reported a range of sixty to a hundred and forty hours after costs for the same buttons. She named the two she had retired and said where the hours had gone. The console was approved in the same meeting. The lesson she wrote at the bottom of the ledger was that a range she could defend was worth more than a total she could not.

The Remediation Ledger

Keep one ledger per button, filled in each quarter from the log and the tickets. Keep it with the button's specification in the agents-only knowledge base, with the summary line copied to the one page for management. The counting rules are printed on the ledger so that whoever fills it in next quarter counts the same way. That also lets a reader who disagrees disagree with a rule rather than a number. Fill in both columns, always, and leave a line blank rather than guess it. A blank is a question for next quarter, and a guess is a number somebody will repeat.

REMEDIATION LEDGER — ONE BUTTON, ONE QUARTER
Button: ______________________   Fix: ______________________   Owner: ______________
Quarter: ________   Filled in by: ______________   Date: ________
Sources: script log at ______________ ; ticket notes mentioning "______________"

COUNTS (from the log; check against the ticket notes)
  1. Runs (all exit codes) ..................................... ____
  2. Successes: fixed, check passed (exit 0) ................... ____
  3. Nothing to do (exit 1) .................................... ____
  4. Failed after acting (exit 2) .............................. ____
  5. Refused before acting (exit 3) ............................ ____
  Log count and ticket-note count agree?  Y / N   If N, note why: ______________

MINUTES                                       LOW (hands-on)   HIGH (loaded)
  6. Manual fix, minutes (from candidate list) ..  ____             ____
  7. Button, minutes (press, read, reply) ........  ____             ____
  8. Saved per success (6 minus 7) ...............  ____             ____
  9. Gross saved (line 2 times line 8), minutes ..  ____             ____
     in hours ....................................  ____             ____

TICKETS AVOIDED (self-healing rules and sweeps only; buttons write 0)
  10. Acts before anyone reported it ............................ ____
  11. Fall in tickets for this symptom, before vs after ......... ____
  12. Tickets avoided: low = lower of 10 and 11; high = line 10   ____   ____
  13. Minutes per whole ticket avoided ..........................  ____   ____
  14. Saved by avoided tickets (12 times 13), hours .............  ____   ____

COSTS (hours)
  15. Build, once (procedure, script, tests, spec, review) ...... ____
  16. Upkeep this quarter (re-tests, refusals read, review) ..... ____
  17. Failure cost: (lines 4 + 5) times ____ minutes each, hours . ____

RESULT (hours)                                LOW              HIGH
  18. Net this quarter (9 + 14 minus 16 minus 17) ...  ____             ____
  19. Build paid back? (cumulative net vs line 15) ...  by: ________     by: ________

HEALTH
  Failures as a share of runs (4 + 5 over 1): ____   Above one in ten?  Y / N
  Reopen rate for tickets this button touched vs desk rate: ____ vs ____
  Runs trend vs last quarter: rising / flat / falling   Reason if falling: ______
  Candidate for self-healing (many successes, no user cost)?  Y / N
  Retire? (few runs, or problem fixed permanently)  Y / N   Note: ______________

SUMMARY LINE FOR THE ONE PAGE
  "____________ gave the desk back about ____ to ____ hours this quarter after
   its costs; ____ failures, cause ____________; ____ retired / kept / promoted."

Joel filled in five ledgers in an afternoon, which was the first afternoon the buttons had cost him since the spring. He took one page to Ingrid the following Monday. The page said the five buttons had given the desk back between twenty and fifty hours over the quarter after their costs. It said the clock button had been retired. The page also said the spooler button's two failures had the same cause and it was fixed. It said the hours had gone into the backlog, which had fallen by a third. Ingrid read it in about a minute and asked whether there were more fixes on the candidate list. There were. That is what the ledger is for.

Frequently Asked Questions