Getting the Permanent Fix Done, Not Just Approved

Six weeks after the Friday sort found it, problem 3 at Tollgate Foods is in the state most problems reach and stay in. The cause is known: the label printer on packing line 2 sits in the coldest corner of the hall. It will not feed below a certain temperature, so it jams on every cold start. The workaround is published and takes two minutes. The fix is written on the record: "move the printer to the warm side of the line, which needs a socket". Under it, in Lena's handwriting, is "IT manager says put it on the list". The list has eleven things on it. Problem 3 has had thirty-one tickets now, and the packing line has stood still for another three hours since the cause was found. That is the part that bothers her.

A permanent fix is the change that removes the cause, so that the incidents stop and the workaround is no longer needed. It is the point of the whole practice and the step most often left undone. By the time the cause is known the pain has been reduced to two minutes a ticket. Two minutes a ticket does not feel like an emergency to anyone. This article is part of our Problem Management series. And it is about the distance between "fix proposed" and "fixed", and how to cross it.

It covers why fixes stall after the cause is found, then the cost of the repeat ticket as the business case, with the arithmetic. Writing the proposal on one page and asking for a decision follows, then scheduling the fix through change control. Next come getting it done by a date and what to do when the date slips, then verifying it worked by watching the tickets stop. Finally, it covers what to do when the fix is refused, a short war story, and the fix proposal to copy, in that order.

Why Fixes Stall After the Cause Is Found

A permanent fix is the only work on the desk with nobody waiting for it, and so it waits. Every incident has a requester who will chase. The fix has a record with a next-review date and an owner who is also on the queue. Every incident has a clock; the fix has "when things are quiet", which is a time of day that does not occur. The fix usually needs someone outside the desk too, an electrician, a supplier, a budget holder, a change window. And each of them has their own queue in which the desk's fix is also not an emergency. Nothing about this is anyone's fault. It is what happens to work that has no deadline and no complainant.

The workaround makes it worse by making it bearable. Before the known error was published, each label-printer ticket cost the desk ten minutes of investigation and the packing line twenty-five minutes of standing still. That hurt enough to be noticed. After publication, the desk's part fell to two minutes and the ticket was closed before anyone thought about it. The packing line's twenty-five minutes did not change at all, because the line still stops until a technician arrives. But the desk no longer felt it, and the desk is where the problem record lives. Comfort is the main cause of unfixed problems. It is a very reasonable cause.

The third reason is that the fix was never turned into a request anyone could say yes to. "Move the printer" is an idea. "Half a day of my time, two hours of an electrician, and a socket, to stop a packing line standing still for four hours a month" is a request. It has a cost on one side and a saving on the other, and it can be approved, refused or scheduled in a sentence. Most fixes stall as ideas. The rest of this article is about turning an idea into a request and a request into a date. It is about turning that date into a count that falls to zero.

The Cost of the Repeat Ticket Is the Business Case

The business case for a fix is the cost of the repeats. And the cost of the repeats is arithmetic the problem record has been collecting since the fifth ticket. Tickets per month, multiplied by the minutes each one costs the desk, plus the minutes each one costs the requester and anyone standing behind them, gives hours per month. Hours per month multiplied by the months the fix will wait gives the cost of waiting. Put that beside the cost of the fix, in hours of whoever's time it needs plus whatever the part or the contractor costs. Finance can price those. Hours are the right unit for the case, because everyone understands hours and nobody argues about them. A rate can be applied by whoever needs money.

Count the requester's time honestly, and the time of anyone the requester was holding up. That is usually the larger number and it is the one the desk never sees. The desk's two minutes per label-printer ticket is trivial. The packing line's twenty-five minutes is a line of six people standing still, a dispatch that goes out late, and on two occasions a lorry that waited. Lena's record had the desk minutes on it and nothing else, because the desk fills in what the desk knows. The case was in the other column.

Notice what the table does not contain: a percentage, an industry figure, or an estimate of "productivity". It contains counts from the record, minutes from the tickets and from one conversation with a supervisor, and a quote from an electrician. Every number has a source in the right-hand column, which means every number can be checked by the person being asked. That is what makes it persuasive. A case built on numbers the reader can check is read as information. A case built on adjectives is read as a request for sympathy, and sympathy does not buy sockets.

Writing the Proposal on One Page

The proposal is one page, sent to the one person who can say yes, and it asks for a decision by a date. It has eight parts, in order. It starts with the problem in one sentence, then the count and the cost per month. Next comes what has been done so far, meaning the workaround and what it does and does not save. The proposed fix in plain words follows, then what the fix costs, in hours and in money if any. Next come when it could be done and by whom, and how the desk will know it worked. Finally comes what happens if it is not done, which is the cost of waiting from the table. Then a line that says what is being asked for. The template at the end of this article is that page.

Ask for a yes, a no, or a date. "Put it on the list" is none of those. The proposal should make it hard to say, by offering the date itself: "I can do this on the afternoon of the 14th if the electrician is booked by the 7th; can you approve the socket by then?" A manager who is asked a question with a date in it usually answers it. A manager who is asked to "consider" something adds it to a list, because that is what consider means. The way to talk to management about process work without the vocabulary getting in the way is covered in Talking ITIL with Management. The short version is that the word "problem" does not appear in the proposal at all. The packing line does.

Keep the proposal to the fix being asked for. The temptation, once writing, is to add the other ten things on the list, the resilience the hall really needs, and the printer fleet's age. The result is a document that asks for everything and gets a meeting. One problem, one fix, one page, one decision. If three fixes are ready at once, send three pages, and let the reader approve the cheap one today and the expensive one at budget time. Each page gets its own yes or no, which is better than all three getting a "let's discuss".

Scheduling the Fix Through Change Control

A permanent fix is a change. And it goes through whatever change control the desk has, because a fix made quietly is the next problem's unexplained change. Change control is the habit of writing down what will be altered, when, by whom, what could go wrong and how to put it back. Someone other than the person making the change reads that before it happens. On a small desk that is a one-page record and a fifteen-minute read, not a board. The whole practice at that size is the subject of Lightweight Change Management, and the record itself is in The One-Page Change Record. A problem fix is the best possible first customer for that record, because it already has a reason, a cost and a way of knowing it worked.

Not every fix needs the same review, and the test is what the change could break. Moving a printer and adding a socket touches one packing line for an afternoon; it needs the line supervisor told, a time agreed, and a note. Changing a scheduled job on a server that forty people depend on, or a setting on the directory, touches everyone. That change needs the read, the back-out plan and the window. The line between the two, drawn so that a small desk does not review everything or nothing, is in What Needs Review and What Does Not. Whichever side the fix falls on, it gets a record, and the record cites the problem number so that in two years the change log explains itself.

The change record also forces three questions that fix proposals tend to skip. What is the back-out, if the printer jams worse on the warm side? Who is told before, so the packing supervisor is not surprised by an electrician at 14:00? And what is the check immediately after, so the desk knows on the day that the printer feeds, before waiting a month to see whether tickets stop? Each of those is a sentence. Each has prevented a fix from becoming an incident on somebody's desk. And the record is where they get written down instead of thought about on the way to the hall.

Getting It Done by a Date

A fix without a date is a wish. And the owner's job after approval is to turn the wish into a date and then defend the date. The date goes on the problem record and in the change record. The date is the first thing the owner reports at the weekly review: "problem 3, fix approved, electrician booked for the 14th, Lena doing the move that afternoon." A date said out loud weekly is hard to let slip quietly. A date on a record nobody reads is a date that is still there in October.

Dates slip for three usual reasons, and each has a usual answer. The outside party is late: the electrician, the supplier, the person with the budget. The answer is to chase on a fixed interval, written on the record, and to have asked at the start what their lead time was. The quiet week never comes: the desk was going to do it when the queue was short, and the queue is never short. The answer is to block the afternoon on the calendar, as a meeting with the fix, which is the only kind of meeting that reduces tickets. And the owner is pulled onto an incident on the day, which happens, and the answer is to rebook the same week rather than "soon".

Many small desks find that a fixed monthly slot works better than individual dates. It is one afternoon, same week each month, protected on the calendar. The desk does permanent fixes and nothing else in that slot, with one person covering the queue. The problem list supplies the agenda, the proposals supply the approvals, and the change records are written the week before. A desk that does this closes one or two problems a month, every month. Over a year that is the difference between a problem list that shrinks and one that is a monument. I have run both kinds of list. The monument was tidier.

Verifying It Worked by Watching the Tickets Stop

A fix is done when the tickets stop, not when the change closes. The difference between the two is the watching period. That is the time the owner keeps the problem open after the fix, counting linked tickets, before calling it fixed. The period is at least twice the usual gap between occurrences and never less than a month. A problem that happened weekly is watched for a month. One that happened monthly is watched for a quarter. During the watching period, the known error stays published with a "fixed on" line at the top, as described in Known Errors and Workarounds. That way, any new occurrence is recognised as news and reported loudly rather than quietly worked around.

The count is read weekly and it should be zero. Before the fix, problem 3 ran at about two tickets a week. After the 14th, the owner looks every Friday for tickets linked to KE-3 or containing "label printer". The number is the only verdict that matters. Zero for the whole period means fixed. One or two, with the recognition test confirming they are the same symptom, means the fix did not remove the cause. In that case, the record goes back to investigating, with the fix noted as tried. One or two that fail the recognition test, a different symptom on the same printer, mean a second problem that had been hiding behind the first. That happens more often than anyone expects, and the second problem gets its own record.

The trap during the watching period is the technician who keeps applying the workaround from habit. That means a returning symptom is closed in two minutes and never flagged. The "fixed on" line on the known error is the guard against it. So is saying at the weekly review, every week of the period, "problem 3 is in watching; if you see a label-printer jam, tell me, do not just clear it". When the period ends at zero, the problem closes as fixed and the known error is archived with the fix attached. The owner then reports upward in one line: "problem 3 fixed on the 14th; thirty-one tickets and about fifteen hours of packing-line time over four months; none since." That line is the whole return on the practice, and it is worth sending to the person who approved the socket. They will approve the next one faster.

When the Fix Is Refused

Some fixes are refused, and a refusal with a reason is a legitimate outcome that the record should capture. The fix costs more than the repeats, the equipment is being replaced next year anyway, the supplier's update will make it moot, the budget holder said no. The problem's status becomes "accepted". The reason and the date and the name go on the record. And the known error stays published so that each repeat still costs two minutes rather than fifty. What the desk does not do is quietly stop counting. The count continues, because a refusal was a decision made on a number, and the number changes.

Set a threshold for re-raising, and write it on the record: "re-raise if incidents exceed twelve a month or if the line stops for more than an hour in a day". When the threshold is crossed, the proposal goes back with the new count and the old decision quoted. That is not an argument but an update, and updates get read. Accepted problems are reviewed at the weekly hour like any other, in one sentence. That keeps "we have chosen to live with this" a choice someone is making. It does not become a thing that happened once and was never looked at again. A refused fix that is still counted is a case being built. A refused fix that stops being counted is a workaround with a pension.

A Short War Story: The Fix Nobody Watched at Greystone Legal

Greystone Legal's desk of two found the cause of a repeat that had generated forty tickets in a quarter. Every fee-earner's document template opened slowly because a group policy pointed the template folder at a server that had been retired. The policy timed out before falling back. The fix was a one-line change to the policy, made on a Thursday, recorded as a change. The problem was closed as fixed the same afternoon because the change had gone in cleanly. The other reason was that the technician, Felix, had tested it on his own machine, where it worked. Nobody watched the tickets. Over the next two months eleven more slow-template tickets arrived and were closed by the other technician using the old workaround. They were not linked to anything, because the problem was closed and the known-error article had been removed. The fix had been applied to the wrong policy object, one that covered the IT department and nobody else, which was why Felix's machine was fine. The error was found when a partner asked why the thing they had been told was fixed in spring was still happening in summer. The desk could not answer, because it had stopped counting on the day it declared victory. A fix is done when the tickets stop, and the only way to know the tickets have stopped is to keep counting after you think they have.

The Fix Proposal

The proposal below is the one page that goes to the person who can say yes. It is written for them, not for the desk: no problem numbers in the first line, no vocabulary that needs explaining, every number with a source. The "what we are asking for" line is the last thing on the page and the first thing they will read. So make it one sentence with a date in it. Attach nothing. If they want the record, they will ask, and most never do.

FIX PROPOSAL — ONE PAGE
To: ______________   From: ______________   Date: __________
Decision needed by: __________

THE PROBLEM (one sentence, in plain words)
  ___________________________________________________________________

WHAT IT COSTS NOW
  Occurrences per month: ____   (____ in the last ____ months)
  Time per occurrence: desk ____ min   people affected ____ min
  Hours per month: ____    Other costs seen: ______________________
  Source of the numbers: ___________________________________________

WHAT WE HAVE DONE SO FAR
  Workaround in place since ______: cuts desk time to ____ min;
  does not change the time lost by ______________________________

THE FIX
  What we would change: ______________________________________________
  Why that removes the cause: ________________________________________
  What it costs: ____ hours of ________  +  ____ hours of ________
                 Parts or contractor: ______________________________
  Risk and back-out: _________________________________________________
  Who is affected while it is done, and for how long: _______________

WHEN AND WHO
  Proposed date: __________   Done by: __________   Needs: __________
  Change record: __________

HOW WE WILL KNOW IT WORKED
  Watching period: ______ (until __________)
  Expected: occurrences fall from ____ per month to 0
  Reported back to you on: __________

IF WE DO NOT DO IT
  Cost of waiting six months: ____ hours and ____ more occurrences
  Re-raise if: ______________________________________________________

WHAT WE ARE ASKING FOR
  Approve __________________ by __________ so the fix can be done on
  __________.   [ ] Yes   [ ] No, because: ______________   [ ] Date: ______

Lena sent the proposal for problem 3 on a Monday with the table above folded into it. The last line was "approve the socket by Friday so the move can happen on the 14th". The IT manager approved it on Tuesday, having read, she later said, the hours per month and the last line and nothing in between. The electrician came on the 14th, and the printer moved ten metres. The change record took fifteen minutes to write. And the watching period ran for six weeks at zero. Thirty-one tickets. The thirty-second never arrived, and the list of eleven things became a list of ten, which is how lists are supposed to work.

Frequently Asked Questions