Turning a Fix Into a Script Safely: Idempotent, Logged, Tested

Thursday, 15:40, at Redbrook Engineering. Tomasz has the candidate list on the desk beside him and the top line says "remap the drawings drive, twenty-six a month". He has the four manual steps in a text file and he has just typed them, one after another, into a script. It is nine lines long. It works on his own laptop, where the drive was already mapped. It works on the spare laptop in the cupboard, where it was not. He has a list of thirty machines in the drawing office and a finger over the key that would run it on all of them before he goes home. He has not thought about what the script does on a machine where the drive is mapped to the wrong server. Nor has he considered a user in the middle of saving a file, or a network that is down. He has not written down anywhere that it ran. Tomasz is about to learn a great deal, and the drawing office is about to help him.

A scripted fix is a repeat fix written as a small program that a machine can run instead of a technician's hands. Turning one into the other is not a typing exercise, because the technician's hands came with a technician's eyes, and the script has none. Everything the technician checked without noticing, the script must be told to check. Everything the technician would have stopped for, the script must be told to stop for. This article is part of our Scripted Fixes & One-Click Remediations series. It is about giving the script the eyes before giving it the hands.

It covers writing the fix down as a procedure before writing any script. It covers the check-first shape that makes a script safe to run twice; and logging every run so the desk can see what happened. It covers the dry run and the test ladder of one machine, then five, then the rest. It covers leaving a way back; timeouts and failing loudly; and where the script lives and who may change it. It ends with a short war story and the script checklist to copy, in that order. The examples are written in a made-up shorthand rather than any real language, so they read the same whatever shell or scripting language is built into your operating system. Every line has an equivalent there.

Write the Fix Down Before You Write the Script

The first draft of a script is a procedure in plain words, and it is written for a colleague, not a computer. One action per line, each followed by the check that shows it worked and what the check should show. "Remove the stale mapping for the D drive. Check: the D drive no longer appears in the list of drives. Map the D drive to the drawings share on the file server. Check: the D drive appears and opens within twenty seconds." That is the fix Tomasz does by hand, written so a new starter could do it at a user's desk without him. It is also the specification for the script, line for line. The shape of it is the runbook described in Writing the Runbook.

Writing it down finds the step everyone does without noticing. Tomasz's four steps turned out to be five when he wrote them for a colleague. That was because before removing the mapping he always glanced at where it pointed. If it pointed at the old file server he knew the user's machine had missed a policy update and needed a different fix. He had never counted that glance as a step. It was a decision, taken in half a second, and his script had no idea it existed. The procedure caught it on paper, which is where you want to catch it. Scripts are very good at doing exactly what they were told, and terrible at noticing what they were not.

The procedure also tells you what the script should refuse to do. Every "if" in the written-down fix is either a branch the script handles or a condition under which the script stops and hands back to a person. "If the mapping points at the old server, stop; this is the policy problem, see the other procedure." A script that stops and says why is a safe script. A script that carries on regardless is a fast way to do the wrong fix on a lot of machines, which is the one kind of productivity nobody asks for. The habit of writing steps that someone else can reproduce exactly is the same habit as in Writing Reproducible Steps. The only difference is that here the stranger following them is a machine.

Check First, Then Act: The Idempotent Script

A script is idempotent when running it twice leaves the machine in the same state as running it once. The word is from mathematics and the idea is from plumbing: turning a tap that is already off does nothing. An idempotent fix checks whether the problem exists before doing anything, acts only if it does, and then checks again to see whether the action worked. Run it on a machine that was fine and it reports "nothing to do" and leaves. Run it twice by accident and the second run finds nothing to do. That one property is what makes the script safe to hand to other people later, because it removes the question "but what if they press it when it isn't broken?"

The shape is always the same four movements: check, act if needed, check again, report. Here is Tomasz's drive remap in the shorthand, with the policy-server branch he found on paper. Read it for the shape; the real version in your scripting language will be longer and uglier. It should still have these four movements in this order.

SCRIPT remap-drawings-drive         (illustrative shorthand, not a real language)
  INPUT   machine (one name), ticket (number), --dry-run (optional)
  LOG     "start", time, operator, machine, ticket

  CHECK   where does drive D: point on machine?
  IF      D: points at files.example.com\drawings AND opens
          LOG "nothing to do: D: already correct"
          EXIT 1                                (1 = nothing needed)
  IF      D: points at the retired server oldfiles.example.com
          LOG "refused: mapping points at old server; policy problem"
          EXIT 3                                (3 = refused, needs a person)

  IF      --dry-run
          LOG "would remove D: and map it to files.example.com\drawings"
          EXIT 0
  ACT     remove the D: mapping
  ACT     map D: to files.example.com\drawings, persistent for this user
  WAIT    up to 20 seconds for D: to open

  CHECK   does D: point at files.example.com\drawings AND open?
  IF yes  LOG "fixed"                           EXIT 0
  ELSE    LOG "failed: D: not usable after remap"  EXIT 2

Notice that the script does one machine, named on the way in, and nothing else. Notice that it has three ways to leave without acting, and one way to act. Notice that "fixed" is a claim it only makes after checking. A script that performs the action and then prints "done" has not fixed anything; it has finished. The difference matters at four in the afternoon when the user phones back.

Idempotence also decides what the script is allowed to do when it finds the problem half-present. If the mapping exists but points somewhere strange, the right answer is nearly always to stop and say so, not to guess. Every branch Tomasz could not describe in the written procedure became an EXIT 3. The script's log told him how often each one happened. In a month he had learned two new fixes from his own refusals, which is a better outcome than having the script quietly perform one of them wrong.

Everything It Does Is Written Down

Every run of the script writes a line, whatever happens, and that line is the only evidence anyone will have later. The line says when, who ran it, on which machine, for which ticket, what the state was before, what the script did, what the state was after, and how it exited. Eight facts, one line, every time. A script that only logs when something goes wrong leaves no trace of the forty runs that went right. When someone asks "has this ever been run on that machine?" the honest answer is a shrug.

The log lives in two places, and the second is the one people read. The first is a file on the machine where the script ran, or in a shared folder if your scripts run from one, with one line appended per run. The second is the ticket, because the ticket is where the desk already looks. The script's closing line, "remap-drawings-drive on WS-DRAW-14 for ticket 6604: fixed, exit 0", goes onto the ticket as an internal note, either pasted by the technician or sent by the script. If you run QuerySolv, the REST API will take that line straight into the ticket as an internal note. In that case, the script's own account of itself sits in the event history next to the technician's words. In any tool, the convention is the same. The ticket is the audit trail. A run that is not on the ticket did not happen as far as the desk can later tell.

Give the exit codes fixed meanings and keep them for every script you write. The numbers themselves do not matter. Using the same ones everywhere does, because the person reading the log at a glance should not need the script's source to know what happened. Four is enough.

The "what was found" and "what the check saw" parts are not optional. "Failed" on its own is a mood, not a log line. "Failed: D: not usable after remap, error access denied" is a fact that points at a permissions problem and saves the next person ten minutes. Write the log line you would want to read at 16:55 on a Friday, because that is when you will be reading it.

The Dry Run

A dry run is a way of running the script so that it does all its checks and none of its actions, and reports what it would have done. It is a switch on the command line, built into the script from the first draft. It is the single cheapest piece of safety you can add. Run the script with the switch against a real machine. It tells you "would remove D: and map it to files.example.com\drawings", or "nothing to do", or "refused: mapping points at old server". It does this without touching anything. You learn whether the checks see what you think they see, on real machines, before any action exists.

The dry run is also how you test the checks on machines you would never dare act on. Tomasz ran his script in dry-run mode against all thirty drawing-office machines on Friday morning while the users were in them. That would have been reckless with the actions switched on and was free with them switched off. The output told him that four machines pointed at the old server. It said two had no D: mapping at all because the users mapped a different letter years ago. The other twenty-four were as expected. He now knew what the real run would do on every machine before it did it. Nobody in the drawing office knew anything had happened, which is the correct amount.

Make the dry run the default behaviour if your desk is nervous, so that acting requires an explicit switch rather than not acting. Some teams prefer the reverse. Either way the switch must exist, and the log must say which mode ran. The dry run must be a real run of the real checks. It must not be a separate script that prints what the author hopes will happen. A dry run that lies is worse than no dry run, because people trust it.

One Machine, Then Five, Then the Rest

Test on one machine you can afford to break, and that machine is your own. Run the fix by hand first to make sure the fault is really there, then run the script, then confirm by hand that the fix took. Then break it again and run the script again. Then run the script a third time with nothing broken and confirm it reports "nothing to do" and changed nothing. That last run is the idempotence test, and it is the one people skip because the script has already "worked". It is also the one that catches the script that remaps a healthy drive and drops the user's open files on the floor.

Then five machines, chosen to be different from each other and from yours. One old laptop with the slow disk. One desktop that has never been off the network. One machine belonging to a remote user on a home connection. One at the other site, if you have one. And the machine that always does something odd, which every estate has and every technician can name without pausing. The five are chosen to be awkward because awkward is where scripts fail. A script that passes on five awkward machines will mostly pass on fifty ordinary ones. Ask the five users first, and tell them what you are doing and when. Be in a position to put the machine right by hand if the script does not. The dry run goes first on each of the five; the real run follows only when the dry run said what you expected.

Then the rest, but not all at once, and not from a loop inside the script. The script does one machine; the person running it does the loop, machine by machine or in small batches of ten, reading the log between batches. This feels slow and is slow and is the correct speed for the first release of anything that changes other people's machines. A failure in the first batch of ten is a thing you fix before the second batch. A failure in a loop of three hundred is a thing you discover from the phones. The reasons for keeping the loop out of the script, and in a human's hands with a count, are the subject of Guardrails: What Scripts Must Never Do. Those reasons are not negotiable even when the first ten went perfectly.

Leave a Way Back

A script that changes something records what it found before changing it, in the log, in enough detail to put it back. Tomasz's script logs where D: pointed before removing the mapping. That way, if the "fix" turns out to be wrong for one user, the old value is one log line away rather than one guess away. For a service restart there is little to record; the service was running or it was not. For a setting, the old value goes in the log before the new one goes in. For a mapping, a printer connection, a cached entry, the same: write down what was there. It costs one line of script and it is the whole of your rollback.

Some fixes cannot be reversed, and the script must say so in its header, in capital letters, before anyone runs it. The honest question is whether such a fix belongs in a script at all. Clearing a cache the application will rebuild is reversible in effect, because the cache comes back. Removing a user's stored sign-in for a service is reversible in effect, because the user signs in again. However, it costs them a minute and the script should say so. Anything that removes data a person made is not reversible by the script and is not a scripted fix. It is a change, with a record and a reviewer. The small-team version of that process is in What Needs Review and What Does Not. The rule of thumb is simple to say and mildly annoying to follow: if you cannot write the rollback line, do not write the action line.

Reversibility also includes the user's open work. A fix that restarts an application, ends a process or signs the user out of something is reversible for the machine and not for the half-written document. The script cannot see the document. The person running it can, by asking, and the script's header should tell them to. "Warn the user: closes the application; unsaved work is lost" is a line in the header that costs nothing and is read by the right person at the right moment. When the fix later becomes a button, that line becomes the confirmation, which is a subject for From Script to Button.

Timeouts, Failures and Stopping

Every wait in the script has a limit, because a script that is waiting forever looks exactly like a script that is working. "Wait up to twenty seconds for D: to open" is a line that either succeeds or fails. "Wait for D: to open" is a line that may still be running next week. Put a number on every wait, make the number generous, and when the number runs out, log it and exit with a failure. The person running the script would rather see "failed: timed out after twenty seconds" than see nothing and wonder.

Failures are loud and single. The script does not retry in a loop, because a loop is a way of doing the wrong thing with great persistence. It tries once, checks once, and reports. If a second attempt is genuinely sensible, the person reading the log can make that call and run it again, which is exactly what idempotence is for. Nor does the script hide failure behind success. The last line of the log says what happened, the exit code agrees with it, and nothing prints "done" unless it is. I have debugged more than one script whose final line said "complete" because the author put the print statement after the action rather than after the check. It was complete. It was also wrong.

And the script stops when something is not as the procedure described. Unknown state is not an invitation to improvise; it is EXIT 3 with a reason. The great temptation when scripting a fix is to handle every case, so that the script never has to hand back to a person. Resist it, at least for the first release. A script that handles the two common cases and refuses the rest is finished in an afternoon and safe on Monday. A script that handles nine cases is finished in a fortnight, and the ninth case is the one that was never quite understood. Let the refusals teach you which cases are worth adding.

Where the Script Lives and Who May Change It

The script lives in one shared place, with a header that says what it is, and that place is the only copy anyone runs. The header carries the name, the purpose in one sentence, the fix it replaces, the author, and the reviewer. It carries the date the script was last tested and on how many machines. It also says what the script changes, what it refuses, how to roll back, and the warning for the user if there is one. A script without a header is a script that will be run by someone who does not know what it does, usually at a moment when finding out the hard way is inconvenient. Scripts also breed. Leave one in a shared folder with no owner and by spring there are three versions of it, each slightly different, each believed by someone to be the current one.

Changes to a released script go through the same two-person habit as the first release. One person edits, another reads, and the header records both. The edit is tested on one machine then five, like the original, because a one-line change can move the fix from "harmless" to "restarts the service on every run". The person who made the change is the one person guaranteed not to see it. Keep the previous version alongside, named by date, so that a bad edit can be backed out in a minute. This is change management at its smallest and most useful: not a committee, just a second pair of eyes and a dated copy.

Finally, the script is linked from the procedure it replaced, and the procedure stays. The written-down fix is what a technician follows when the script refuses, when the machine is in a state the script does not handle, or when the script itself is broken. The script is the fast path. The procedure is the path that still exists when the fast one is closed. A desk that deletes the procedure because "we have the script now" will rediscover the steps at the worst possible time, from memory, under pressure.

A Short War Story: The Script Without a Check at Stonebridge Insurance

Dana at Stonebridge Insurance scripted the fix for the claims team's remote-access client, which stuck "connecting" after the laptops slept and was cured by restarting its service. She wrote the restart, tested it on her own laptop where the client was indeed stuck, and it worked. The script had no check, because the fix had always been run on machines that were stuck. So there had never been anything to check for. On the last working day of the month, a colleague ran it from a list against every laptop in the claims team to "get ahead of it". The script restarted the service on forty-one machines whose connections were working perfectly. That dropped every claims handler mid-session for about ten minutes during the month-end run. The cost was a morning of re-keying for eleven people and a conversation with the claims manager that Dana still describes as "educational". The rewritten script checked the client's state first and exited with "nothing to do" on a healthy machine. That took her four lines, and would have taken four lines the first time.

The Script Checklist

Run the checklist below against every script before its first real run and again after every edit. It is a one-page gate, not a style guide. It does not care how the script is written, only whether it has the properties that make it safe to run on someone else's machine. Tick each line honestly; a line you cannot tick is a thing to fix before Monday, not a thing to remember. Keep a copy of the filled-in checklist with the script, dated, so the next person knows which version was checked and by whom.

SCRIPT CHECKLIST — BEFORE FIRST RUN AND AFTER EVERY EDIT
Script name: ______________   Fix it replaces (candidate list line): ____
Author: ______________   Reviewer: ______________   Date: ________

WRITTEN DOWN FIRST
  [ ] The manual fix is written as a procedure: action, check, expected result
  [ ] Every "if" in the procedure is either handled or a refusal in the script
  [ ] The procedure is linked from the script header and will stay after release

CHECK FIRST, THEN ACT
  [ ] The script checks for the problem before acting
  [ ] On a healthy machine it changes nothing and exits "nothing to do"
  [ ] Running it twice in a row leaves the machine as running it once would
  [ ] Unknown or unexpected state exits "refused" with a reason, never a guess
  [ ] It acts on exactly one machine, named as input; no lists, no wildcards

LOGGED
  [ ] Every run writes one line: time, operator, machine, ticket, before,
      action, after, exit code
  [ ] Exit codes follow the desk standard (0 fixed, 1 nothing, 2 failed, 3 refused)
  [ ] "Failed" and "refused" lines say what was seen, not only that it happened
  [ ] The closing line goes onto the ticket as an internal note

DRY RUN
  [ ] A dry-run switch exists and runs the real checks with no actions
  [ ] The log line says which mode ran
  [ ] Dry run was run against the five test machines before any real run

TESTED
  [ ] Own machine: broken, fixed by script, broken, fixed again, healthy, "nothing to do"
  [ ] Five awkward machines, users told first, dry run then real run on each
  [ ] Rollout in batches of ten or fewer, log read between batches

REVERSIBLE
  [ ] State found before acting is written to the log in enough detail to restore
  [ ] Anything not reversible is named in the header in capitals, or removed
  [ ] The user warning (closes application, signs out, loses queue) is in the header

STOPS
  [ ] Every wait has a limit and logs a failure when it runs out
  [ ] No retry loops; one attempt, one check, one report
  [ ] Nothing prints "done" or "fixed" before the check after the action

LIVES SOMEWHERE
  [ ] One shared location; the only copy anyone runs
  [ ] Header: name, purpose, replaces, author, reviewer, last tested, changes,
      refuses, rollback, user warning
  [ ] Previous version kept, named by date
  [ ] Edits reviewed by a second person and re-tested on one then five

Tomasz's remap script went through the checklist twice: once on the Friday, when it failed four lines, and once the following Wednesday, when it passed. The four lines were the idempotence test, the dry run, the "before" state in the log, and the refusal for the old server. Every one of them would have bitten someone in the drawing office if he had pressed the key on Thursday. The script has run about two hundred times since. The procedure it replaced is still in the knowledge base, visible to the desk, with a line at the top that says "there is a script for this; use the procedure when the script refuses". That happens roughly once a month, and every refusal has been right.

Frequently Asked Questions