Self-Healing Basics: The Fix That Runs Itself

Wednesday, 03:12, at Ashgrove Housing. Nobody is at the desk and nobody is awake. The service behind the tenants' rent portal stops, as it has done four times this month. It is the one that lets people check their balance and pay from their phones. At 03:15 a scheduled check on the portal server notices that the service is not running. It writes a line to its log, starts the service, waits, confirms the portal's sign-in page loads, and writes a second line. At 03:16 a ticket appears in the housing desk's queue with an internal note that says "rent-portal service was stopped; started; sign-in page loads; second run this week". At 08:05 Femi reads the note with her first coffee. The tenants who paid their rent at half past three never knew there was anything to know.

A self-healing fix is a scripted fix that runs without a person pressing anything. A trigger starts it. The fix checks and acts the way any good script does, and it tells someone it ran. It is the top of the ladder that began with a repeat fix done by hand, became a script, and became a button. And the reason it sits at the top is that every rung below it had a person in the loop and this one does not. That is what makes it valuable at 03:12 and what makes it dangerous at any other time. This article is part of our Scripted Fixes & One-Click Remediations series. And it is about letting a fix run itself without letting it run away.

It covers when a fix may run itself and the two kinds of trigger. Then come the limit on how often it may run, and what happens when the limit is reached. Next is the notification, meaning who is told and how. The off switch, the fix that masks a problem, and a short war story follow. The self-healing rule card to copy and the monthly review of the rules come last, in that order.

When a Fix May Run Itself

A fix may run itself when it has already run many times with a person watching and the result was the same every time. That is the whole qualification, and it rules out most fixes for most of their lives. A button may have been pressed sixty times, with a record that says "fixed" on fifty-eight of them and "nothing to do" on two. That button has earned the right to be considered. The script that was written on Thursday has not, however good it looks. Nothing has yet shown how it behaves on the machine that always does something odd. Self-healing is a promotion, and promotions are earned with a record.

The second condition is that the fix is harmless when it was not needed, and here the bar is higher than for a button. A button has a person reading the confirmation; a self-healing rule has nobody. If the check is wrong, the fix runs on a healthy machine with no one to notice. If the fix has any cost to the user, the user then pays it at random. A service restart whose check is "is the service running?" is harmless when unnecessary. The check cannot be wrong in a way that restarts a running service. A fix that ends an application the user might be working in is never harmless when unnecessary, and does not run itself, whatever its record.

The third condition is that the cost of a wrong run is small and bounded. Ask what the worst case is if the rule fires on the wrong machine, at the wrong time, or ten times in a row. If the answer is "a service restarts that did not need to" the rule is a candidate. If the answer involves anyone losing work or anyone being locked out, it is not. The same question, asked of scripts generally and called the blast radius, is in Guardrails: What Scripts Must Never Do. It applies twice over here because nobody is watching. The decision table below is the short form.

Most desks end up with two or three self-healing rules and a dozen buttons, and that ratio is healthy. The ladder is not a race to the top. It is a set of places a fix can sit, and most fixes belong one rung down from where their author would like them. The thinking behind the whole ladder, from acknowledgements to routing to timers to remediation, is in What to Automate First, and What Never. Self-healing is the last rung, and the one with the longest drop.

Triggers: On a Schedule or On a Signal

A trigger is whatever starts the rule running, and there are two kinds. A scheduled trigger runs the check at fixed intervals: every five minutes, every hour, every night at two. A signal trigger runs the check when something else reports a symptom. An event appears in a log, a monitoring system raises an alert, or a user's machine reports a state. Both start the same script, and in both cases what they start is the check, not the fix. The trigger says "look now". The script decides whether there is anything to do, exactly as it would if a person had pressed the button.

The scheduled trigger is simpler and should be the first one you use. Ashgrove's rule runs every five minutes on the portal server and checks whether the rent-portal service is running. It nearly always finds that it is and exits with "nothing to do" without writing to the ticket. A check that runs every five minutes and does nothing two hundred and eighty-eight times a day is not waste. It is the cost of catching the one time it matters within five minutes rather than at 08:05. Choose the interval from how long the symptom can be tolerated, not from how often the fault occurs. A portal that tenants use at all hours gets five minutes. A report server used at nine gets an hourly check overnight and a check at 08:30.

The signal trigger is more precise and more fragile. Your monitoring may already notice that the service has stopped and already open a ticket, as described in Opening Tickets From Alerts. In that case, the alert itself can start the script, and the script's result can go onto the same ticket. That is neat, and it depends on the alert being right, which it is most of the time. The failure case is an alert that fires on a symptom the script does not fix. The alert says "portal unreachable". The script finds the service running and exits "nothing to do". The real problem, a full disk or a network fault, is a few lines further down the log. A signal-triggered rule therefore always has its "nothing to do" exit go to a person. "The alert fired and the fix found nothing" is itself a finding.

Here is Ashgrove's rule in the same shorthand as the rest of this series. The parts that distinguish a self-healing rule from a button are in the middle: the counter, the limit and the notification. The script it calls is the one that was a button for four months first.

RULE heal-rent-portal-service          (illustrative shorthand, not a real language)
  TRIGGER  every 5 minutes, on portal01.example.com only
  OFF SWITCH  if file heal-rent-portal.OFF exists in the rules folder: EXIT quietly

  CHECK    is service "rent-portal" running?
  IF yes   EXIT 1 quietly                      (nothing to do; no ticket, counter untouched)

  COUNT    runs that acted in the last 60 minutes, and in the last 24 hours
  IF       acted >= 2 in the last hour  OR  acted >= 4 today
           NOTIFY on-call: "rent-portal limit reached; not restarting; needs a person"
           TICKET  update today's ticket, set the escalation marker
           EXIT 3                               (refused: limit reached)

  ACT      start service "rent-portal"
  WAIT     up to 60 seconds for the sign-in page to load
  CHECK    does the sign-in page load?
  RECORD   time, found, action, result to the counter and the log
  TICKET   open today's ticket if none; append the record as an internal note
  IF yes   EXIT 0   ELSE  NOTIFY on-call "restart did not restore portal"  EXIT 2

Read the order. The off switch is checked before anything else. The check for the problem comes before the counter is consulted, so a healthy service costs nothing and counts for nothing. The limit is checked before the action, so the rule can refuse. And the ticket is written whatever happens once the rule acts, so nothing the rule does is invisible. The script that actually starts the service is the one that was already tested; this rule is the frame around it.

The Limit: How Often, and Then Stop

Every self-healing rule has a limit on how many times it may act in a period. When the limit is reached it stops acting and calls a person. The limit is what separates a self-healing rule from a machine that hides a failure indefinitely. A service that stops once a week and is restarted once a week is being healed. A service that stops every six minutes and is restarted every six minutes is broken. A rule without a limit will restart it every six minutes until the heat death of the server. Meanwhile, the ticket queue shows nothing and the users see a portal that works for five minutes in every six. Engineers call that flapping. It is a service going up and down fast enough that each state is a lie about the next one. The limit is how the rule notices it is happening.

Set two limits, a short one and a long one, and make them small. Twice in an hour and four times in a day are reasonable starting points for a service restart. Your numbers will depend on the fault. The point is that they exist and are written on the rule card. When either is reached the rule exits "refused: limit reached" and does not act. It writes the refusal on the ticket and notifies a person in a way that person will notice. The person then has what the rule could never have: the ability to look at why. A limit that is never reached costs nothing. A limit that is reached once has probably paid for the whole rule.

The counter the limit relies on lives outside the script, in a small file or a log the rule reads. The script does not remember the last run and must be told. Keep it simple: one line per run that acted, with a time, and the rule counts the lines in the window. Reset nothing by hand; the window does the forgetting. And test the limit on purpose, before the rule goes live. Stop the service three times in an hour on a test machine. Watch the rule refuse the third time and send its notification. A limit that has never been seen to work is a hope, and hopes do not wake the on-call technician.

The Notification: Tell Someone, Not Everyone

Every run that acts is written to a ticket, and the rule of one ticket per machine per day keeps the queue readable. The first restart of the night opens the ticket; the second appends to it; the limit refusal appends to it and raises its visibility. Nobody wants six tickets from one server before breakfast, and nobody wants zero either. The patterns that keep alert-driven tickets from flooding the queue are in Avoiding Alert-Ticket Floods. So are the reasons a flood is worse than silence because people stop reading. A self-healing rule is an alert with hands, and it floods in exactly the same way if you let it.

Successes go to the ticket and nowhere else, and are read in the morning. Limit refusals and failures go to a person, now, by whatever means wakes them: the on-call phone, the paging channel, the thing that makes a noise. The difference is the difference between "the portal stopped and is back" and "the portal stopped and I could not bring it back". Only the second needs anyone out of bed. If you run QuerySolv, the rule can open and update the ticket through the REST API and set the escalation flag on it. That flag is separate from priority. So the on-call technician sees a thing that needs a person rather than merely a thing that is urgent. In any tool, keep the two channels distinct: quiet for the fixed, loud for the unfixed, and never the same noise for both.

A weekly digest is worth adding once the rule has run for a month. It is one message, every Monday, listing each rule, how many times it acted, how many times it refused, and whether anyone looked. The digest exists because tickets that say "fixed itself" get closed without being read. The rule that acted forty times last week has told you something that none of the forty tickets did on its own. The digest is the self-healing version of the ledger described at the end of this series. And the count of acts per rule per week is the number that eventually gets the root cause fixed.

The Off Switch

Every self-healing rule has one off switch. It is documented on the rule card and works in under a minute without editing anything. The simplest form is a file. If a file with a particular name exists in the rules folder, the rule exits before checking anything. Anyone on the desk may create that file. Another form is disabling the scheduled task that runs the rule. Either is fine. What is not fine is a rule that can only be stopped by someone who knows where the script is and how to comment out a line. The person who needs to stop a rule at 02:00 is the person on call. And they need to be able to do it from a phone call's worth of instructions.

Test the off switch before the on switch, and test it again after every change to the rule. Create the file, stop the service by hand on a test machine, and watch the rule do nothing. Remove the file and watch it act. It is a two-minute test and it is the one that tells you the rule can be controlled. That is the thing you most want to know about something that runs while you sleep. Some desks keep a line in the on-call runbook that says "to stop any self-healing rule, create its OFF file; the file names are in the rules folder". And that single line has saved more than one night.

The off switch also answers the question nobody asks until a planned change: what happens to the rule while the service is being upgraded? A rule that restarts a service whenever it finds it stopped will fight an engineer who is deliberately stopping it. The engineer will lose, repeatedly, and then say things. The change record for any work on a self-healed service has a line that says "switch off rule X first, switch on after". And the rule card lists the changes that must do so. I learned this the usual way. I watched a service I had just stopped start itself again, three times, while I looked for who else was logged on.

The Fix That Masks a Problem

A self-healing rule that runs more than you expected is a problem report you have not read. The rent-portal service does not stop four times a month for no reason; something is wrong with it, or with the server, or with something it depends on, and the rule has made that something painless enough that nobody has to find out. That is the rule's purpose and its danger in one sentence. The rule bought time; it did not buy the right to stop asking why. And the number of times it acted is now the best evidence anyone has about how often the real fault occurs.

Treat every self-healing rule as a workaround for a known error, which means it has a problem record beside it and a review date on it. A known error is a fault you understand well enough to work around but have not yet fixed. The discipline of recording it, with its workaround and its owner, is in Known Errors and Workarounds. The rule card has a field for the problem record's number and a field for the review date. And a rule whose fields are blank is a rule that has quietly become the permanent fix. Nobody decided that. It just happened, which is how most permanent fixes that are actually workarounds come about.

The counter is the argument for the permanent fix. "The portal service stopped forty-one times last quarter and was restarted automatically each time. The restart takes a minute and tenants may see an error during it. The supplier has a fix that needs a maintenance window" is a paragraph that gets a maintenance window approved. "The portal sometimes needs a restart" is not. When the permanent fix lands, the rule's count falling to zero is how you prove it worked. Then the rule is retired. That is the best thing that can happen to a self-healing rule and should be celebrated rather than mourned. The route from a counted workaround to a fix that sticks belongs to problem management. And the count is what you take into that conversation.

A Short War Story: The Rule That Ran Two Hundred Times at Bluewater Bank

Rashid at Bluewater Bank set up a self-healing rule for the branch document service. The service stopped now and then and was fine after a restart. The rule had a ticket per run and no limit and no digest. Over one quarter it acted two hundred and twelve times, each restart opening a ticket that said "fixed itself". Each ticket was closed by whoever was on the queue without being read, because they all said the same thing. The cause was a log folder on the server filling slowly, and the service stopped each time the disk ran short. The restart cleared enough temporary space to carry on for a day or so, which is why it always worked. On a Monday at 09:05 the disk filled completely. The restart could not clear anything. The rule kept trying every five minutes, and twelve branches had no documents until half past eleven. Meanwhile, two people worked out why a service that "fixed itself" had stopped doing so. The rebuilt rule had a limit of three acts a day and a weekly digest. It also had a problem record whose first line read "212 restarts in a quarter". The disk got a bigger partition the following week. Rashid's lesson, written in the rule card's notes field, was that a rule nobody reads is a report nobody reads, with a motor attached.

The Self-Healing Rule Card

Use one card per rule, filled in before the rule is switched on. Keep it in the agents-only knowledge base beside the button specification and the script checklist for the fix underneath. The card is deliberately short, because the script and the button already have their own pages. This one is about the frame: trigger, limit, notification, off switch, review. The reviewer checks the frame, not the fix. A rule card with any blank in the "limit" or "off switch" sections is a rule that is not ready, whatever its author says about the script.

SELF-HEALING RULE CARD — ONE PER RULE
Rule name: ______________________   Owner (role): ______________
Script it runs (name, button spec date, runs as a button so far: ____): __________
Machine or service it applies to (one): __________________________________________

QUALIFICATION
  Runs as a button with a person watching: ____   Of which "fixed": ____
  Cost to the user when it runs: ___________________________________________
  Can the check be wrong in a way that triggers the fix?  N
  Worst case if it fires ten times in a row: ________________________________

TRIGGER
  Type: schedule / signal     Interval or signal: __________________________
  Chosen because the symptom can be tolerated for at most: ____ minutes
  If signal: "nothing to do" result goes to a person?  Y

LIMIT
  At most ____ acts in ____ minutes   At most ____ acts in 24 hours
  On limit: refuse, write to ticket, notify ______________ by ______________
  Counter kept at: ______________   Limit tested on ________ by ______________

NOTIFICATION
  Each act: ticket (one per machine per day) as an internal note
  Failure or limit reached: notify ______________ by ______________ (loud)
  Weekly digest to: ______________ on ______________

OFF SWITCH
  How: create file ______________ in ______________  /  disable task ______________
  Who may use it: anyone on the desk     Tested on ________
  Changes that must switch it off first: ___________________________________

MASKING CHECK
  Problem record for the underlying fault: ____________   Owner: ______________
  Review date (retire, extend or escalate): ________
  Acts last month: ____   Trend: rising / flat / falling

Reviewed by: ______________   Date: ________   Switched on: ________

Femi's card for the rent-portal rule has "button for four months, sixty-three presses, sixty-one fixed" in the qualification box. It has "two in an hour, four in a day" in the limit box. A problem record number in the masking section points at a supplier ticket about a memory fault in the portal's service. The rule has acted eleven times since it went live and refused once. At that refusal, Femi found the server had been left with a stale update pending and finished it. The review date is the week after the supplier's fix is due. If the count drops to zero after that, the card gets a line that says "retired". Then the rent portal will have one fewer thing watching it in the dark.

Reviewing the Rules Monthly

Once a month, read every rule card against its counter and ask three questions. Did it act at all? A rule that has not acted in three months is watching for a fault that has gone away. It should be switched off and archived before it surprises someone by acting on a change. Did it act a lot? A rising count is a problem getting worse under a blanket, and the problem record needs a nudge or a new owner. Did anyone look? A rule whose tickets were all closed unread is Bluewater's rule, and the digest needs a reader with their name on it.

The monthly review is also when the off switch is tested and the limit is sanity-checked against the month's counts. The review date on the masking section is either honoured or moved with a reason then, too. The monthly review takes about half an hour for a desk with three rules. It is the half hour that keeps self-healing from becoming the thing it was meant to prevent. That is a fault nobody understands, fixed by a script nobody reads, on a schedule nobody remembers setting. The desk may already hold a monthly audit of everything it has automated, including the humbler rules for routing and timers. In that case, the self-healing rules go on the same agenda, at the top. They are the only automation on it with hands.

Frequently Asked Questions