Pattern-Triggered Maintenance: From Ticket Pattern to Schedule

Monday, 08:50, at Bluewater Bank. Lena, who runs the three-person desk, is reading the queue before the branches open and already knows what is in it. The card printer at the Castle Street branch is "offline"; the shared PC at Mill Lane "wants a restart before it will let anyone in"; the tablet the Harbour branch uses for new-account photos "won't join the wireless". Three tickets, three branches, three Mondays in a row with the same three tickets. Each is fixed in ten minutes by a technician on the phone while a branch manager stands at a counter with the shutters half up. Lena has been treating them as nine small tickets. On this Monday it occurs to her that they are one pattern, and that the pattern has a day of the week.

Everything in the first three articles of this series read a signal from a machine. This one reads the signal from the queue. Pattern-triggered maintenance is a piece of scheduled work, chosen because a run of similar tickets showed the fault would keep returning. It resets, cleans or replaces something before the next ticket is due. Or it changes the fleet so that the ticket type stops arriving at all. This article is part of our Proactive Remediation series. It sits at the boundary with problem management, which finds the repeat and hunts its cause. This article picks up where the desk decides what to do every week while the cause is being hunted, or instead of hunting it.

It starts with the queue as a signal and the three axes a pattern lives on. Then it explains where this picks up from problem management. Next are the three kinds of maintenance and the arithmetic for choosing one. After that comes scheduling the job, with its window, its notice and its record. Then comes the maintenance that removes a whole ticket type. Next is keeping the schedule honest and knowing when to retire it. It ends with a short war story and the pattern-to-maintenance table to copy.

The Queue Is a Signal Too

A pattern is a run of tickets that share a cause, and it hides in a queue because the queue shows tickets one at a time. Three "printer offline" tickets a week apart, each handled by a different technician and each closed in ten minutes, look like three small successes. Sorted by the words in them and by the day they arrived, they are one fault with a calendar. The desk that notices is the desk that reads its tickets in bulk, monthly, by type and by time, rather than one at a time as they arrive. Finding the Repeat Tickets owns the mechanics of that search; what matters here is what you are looking for once you have the list.

A pattern has three axes: when, where and what. Most desks look only at what, which is why the pattern hides. The when axis is periodic: Monday mornings, the first working day of the month, the hour after a backup, the week after a payroll run. The where axis is population: the same branch, the same model of laptop, the same floor, machines bought in the same order. The what axis is cumulative: after so many days without a restart, after so many print jobs, after so many sign-ins. A ticket type that clusters on any one axis has a cause behind it, and the axis tells you roughly what kind of cause. Monday mornings mean something happened on Sunday night to machines that were alone. The same model means the model. Every twelve days means something fills or leaks.

The simplest tool for seeing the axes is a monthly spreadsheet of closed tickets with four columns: the ticket type as recorded at close, the weekday, the hour, and the machine or location. Sort by each column in turn and the clusters stand out in a way they never do in the live queue. That depends on the type being recorded honestly when the ticket is closed, which is a discipline of its own. That is why the categorisation series in this library insists on categorising at close, not just at open. A ticket filed as "printer" at open and never corrected after it turned out to be the wireless will join the wrong cluster. The wrong cluster gets the wrong maintenance.

Where This Picks Up From Problem Management

Problem management, in plain words, is the practice of treating a run of similar incidents as one underlying problem, finding its root cause, and fixing that. It is the right thing to do, and the problem-management series in this library covers it in full. Pattern-triggered maintenance is the pragmatic branch off that path, and it is taken in two situations. The first is that the cause is known and the permanent fix is far away. It needs a supplier, a budget, a replacement model, or a project. Meanwhile the tickets keep arriving every Monday. The second is that the cause is known and the permanent fix costs more than a scheduled reset ever will, so the reset is the fix. Getting the Permanent Fix Done is about the first situation's long road. This article is about what the desk does on the Mondays in between. It also covers the second situation, where the schedule is the destination.

The industry is faintly embarrassed by the scheduled restart, in the way a surgeon might be embarrassed by a plaster. It works, it is cheap, and nobody puts it on a conference slide. Take a service that leaks memory and falls over every nine days, and whose supplier has not fixed the leak in three releases. It is correctly treated with a restart every Sunday night. The desk that does so has removed a ticket type for the price of a scheduled job. The honest condition is that the schedule is written down with its reason, so that when the supplier does fix the leak, somebody knows the job can go.

Where the two practices meet is the record. Problem management keeps a problem record for the underlying cause, with its known error and its workaround. Pattern-triggered maintenance is very often that workaround, promoted from "a technician does this when the ticket arrives" to "a job does this before the ticket arrives". The maintenance entry should point at the problem record and the problem record at the maintenance. Then whoever closes the problem one day also finds and retires the job. A workaround with a schedule is still a workaround, and it should be filed as one.

Choosing the Maintenance

There are three kinds of maintenance, and the pattern's axis usually tells you which. A reset restarts a service, a device or a machine on a schedule. The schedule is chosen to fall before the point at which the pattern says it fails. It fits the cumulative axis, the thing that leaks or fills. A clean clears a cache, a spool folder, a log or a temporary area on a schedule. It fits the same axis where the thing that fills is a folder rather than memory. It borrows its safe list from the disk-space article earlier in this series. A replace swaps a consumable or a wearing part on a schedule rather than at failure. Examples include printer rollers, batteries, and the keyboard on the kiosk that gets a thousand sign-ins a week. It fits the where axis, the population that wears at a known rate.

The trigger threshold is an arithmetic question, and the arithmetic is simple enough to do on the ticket itself. Here is a worked example from a fictional desk, in ranges. A ticket type arrives three to five times a month. Each one takes the desk twenty to forty minutes, counting the phone call, the fix and the note, and costs the user fifteen to thirty minutes of not working. That is one to three and a half hours of desk time a month and a similar amount of user time. A scheduled reset takes ten minutes to set up once and a minute a week to check. If the reset removes the ticket type, the desk gets back one to three hours a month for a one-off hour of work. The payback is then inside the first month at the low end. Any pattern with two or more tickets a month and a known reset passes that test. Below two a month, the hour of setup still pays, but slowly, and the pattern may be noise.

Cadence comes from the pattern's interval. If the fault returns every nine to twelve days, a weekly job falls comfortably inside the shortest interval; if it returns monthly, a fortnightly job does. The rule of thumb is to schedule at about half the shortest interval observed, so that a slightly early failure is still caught. Tighten it if a ticket slips through. A job scheduled at exactly the interval is a job that races the fault, and the fault has been doing this longer.

Scheduling It: Window, Notice, Record

The job runs in a maintenance window, a time chosen because nobody is using the thing. The window is agreed with the people who use it rather than assumed. Sunday night is empty for a branch; it is the busiest shift of the week for a warehouse. Ask the owner of the service or the location when a two-minute interruption would be invisible, write that time down, and do not move it without asking again. The planned-maintenance notice tells users the window exists, what will happen in it and what they will notice. It is written once and reused. Planned Maintenance Notices is the article that writes it. For a job that restarts a single service nobody will notice, the notice is a line in the knowledge base and nothing more.

Where the thing being maintained is shared, the job is a change. It gets whatever record the desk uses for changes: what will be done, when, by what, how to undo it, and who agreed. The One-Page Change Record is the form for that, and a recurring job needs one record with the schedule on it rather than a record per run. The record is also where the retire-when condition lives. That is the sentence that says under what circumstances the job should stop, and it is written on the day the job is created. Not later. Later is never.

The job itself is usually a scheduled task, run by the scheduler built into the operating system, calling a short script in the shell or scripting language built into it. The script does the reset, checks that it worked, writes a line to a log, and opens a ticket if the check failed. That last part is what separates maintenance from hope: a job that fails silently every Sunday for a month has produced four Mondays of tickets and a log nobody read. Here is the shape in plain words; the real one is about this long.

SCHEDULED RESET: BRANCH PRINT SERVICE (runs Sunday 22:00)
if the print service is running
    stop the print service; wait up to 60 seconds for it to report stopped
start the print service; wait up to 60 seconds for it to report running
status = the print service's current state
write "date, time, machine, status" to maintenance-log.txt on the share
if status is not running
    send a message to the desk's ticket address:
        "Scheduled reset failed on machine; service state is status"

The log line is the second half of the job. Once a week, somebody on the rota opens the log and confirms the job ran and reported running on every machine it covers. This takes a minute and is written on the week's sweep ticket beside the disk-space and hardware results. The job that opens its own ticket on failure covers the loud case. The log check covers the quiet one, where the scheduler never ran the job at all because the machine was off or the account's password changed. Both happen. The second is more common.

The Maintenance That Removes a Ticket Type

The reset keeps a fault from reaching users; the better maintenance changes the fleet so the fault cannot happen, and the where axis is how you find it. A ticket type that clusters on one model of laptop is a model problem: a driver, a firmware setting, a battery design, a dock. A ticket type that clusters on one office is a site problem: a wireless access point, a switch, a shared printer, a power circuit that dips at eight. A ticket type that clusters on machines built from the same image is a build problem. The fix is one change to the standard build applied to every machine. After that, the type stops arriving from new machines and fades from old ones as they are rebuilt.

Removing a type is a project rather than a job, and it is usually small. Change the default save location in the build so the scanning software writes to the share. Update the driver across the model that loses its dock. Replace the one access point that drops every Monday. Turn off the feature nobody uses that asks every user to restart every fortnight. Each of these takes an afternoon, and each removes a column from next quarter's ticket spreadsheet. A desk that does one a month will, within a year, have a queue made mostly of things that genuinely could not have been predicted. That is what the queue is for.

Measure it, because the removal is the single clearest prevention result a desk can show. Count the ticket type for three months before the change and three months after, from the same population of machines. Report the drop as a range with the counting rule stated. The last article in this series, Measuring the Tickets That Never Arrived, sets out how to do that without flattering yourself. The removed ticket type is its easiest case, because the before and after are the same words in the same column.

Keeping It Honest, and Knowing When to Stop

Scheduled maintenance outlives its reason unless somebody checks, and the check is quarterly. For each job on the list, ask three questions. Is the ticket type it was built for still absent, which is the evidence it is working? Has the underlying cause been fixed permanently, by a supplier, a replacement or a build change, which is the evidence it is no longer needed? And did the job run every time, according to the log? A job whose ticket type is gone and whose cause is fixed is retired, with a note on the change record saying why. A job whose ticket type is gone and whose cause is not fixed stays. A job whose ticket type has come back is not working and needs looking at, not tightening.

I once scheduled a weekly reset for a service that leaked, and left it running for two years after the supplier fixed the leak. It restarted a perfectly healthy service a hundred times. The log was immaculate. Nobody was harmed, which is the usual outcome, and the usual outcome is also why nobody checks. The harm from a stale job is mostly the quiet one. The desk has a growing list of things it does on Sunday nights for reasons nobody can remember. One of them will one day restart something that has since become important. The quarterly review is twenty minutes. Put it on the rota beside the sweeps.

The other honesty is about masking. A reset that works perfectly removes the pressure to find the cause, and some causes should still be found. The leak that makes a service fall over every nine days may also be corrupting something every nine days that nobody has noticed yet. The maintenance list should say, for each job, whether the cause is known, and whether anyone is still looking. If you run QuerySolv, the volume reports and a search on the pattern's words give you the monthly count behind each of those answers without exporting anything. The ticket history behind the job's own failure tickets also shows how often the check has fired. In any tool, the test is that someone can look at the maintenance list and say, for every line, why it is there and what would let it go.

A Short War Story: The Monday Handhelds at Bellweather Logistics

Bellweather Logistics' warehouse ran on handheld scanners. Every Monday morning for four months a dozen or so refused to join the wireless network until a technician walked over and reconnected each one by hand. That took about fifteen minutes per handheld and left a picker standing idle for twenty. The tickets were raised by different team leaders, handled by whichever of the desk's two technicians was in first, and closed as "wireless, reconnected". Jonas, who handled most of them, treated each one as new. The pattern surfaced when the desk lead sorted a quarter's tickets by weekday for an unrelated report. The handhelds had failed on sixteen consecutive Mondays, which the review later described as "a tendency". The cause was a network controller that restarted itself every Sunday night for a check. It came back without the warehouse's wireless profile until someone re-applied it. The fix was a scheduled job at 05:30 on Mondays that re-applied the profile and confirmed it. A month later, a permanent setting change from the supplier let the job be retired. The four months had cost roughly fifty tickets and something over thirty hours of desk and picker time. The pattern had been visible in the ticket dates from the third week, to anyone who sorted by the day.

The Pattern-to-Maintenance Table

The table below is the desk's list of patterns and what is scheduled against each, and it is the template this article leaves you with. The rows are a worked example from Bluewater Bank's desk after Lena's Monday; the last row is blank for your first entry. Use one row per pattern, with the three axes in the first column and the arithmetic in the second. Include the maintenance chosen and its schedule, the check that proves it ran, and the condition under which the row is deleted. Keep it on the same page as the signal list from the first article, review it quarterly, and never add a row without filling in the last column.

Lena's three Monday tickets became three rows on that table within a fortnight, and two of them became scheduled jobs the following Sunday night. The third stayed a hand fix with a problem record attached until the access point was replaced six weeks later. At that point, the row went into the "removed" list at the bottom of the page with the date. Bluewater's Monday queue now opens with whatever Sunday night did to the branches that nobody could have predicted. On most Mondays that is nothing, and Lena reads it at 08:50 with a cup of tea rather than a phone.

Frequently Asked Questions