Writing the Runbook: Precise, Complete, Followable at 02:00

Tuesday, 02:10, at Ashgrove Housing. Ravi is on call for the first time and the out-of-hours repairs line has just phoned. The contractors' portal, the one the emergency plumbers use to pick up jobs, is not loading. Ravi opens the laptop, finds the agents-only article called "Repairs portal down", and reads it in full, which takes four seconds. It says: "Restart the portal service. If that does not work, check the database. Escalate if needed." There are two portal servers. He does not know which one the service is on, what the service is called, or what "does not work" looks like. He does not know which database, or who "escalate" means at ten past two. He restarts a service on the server whose name sounds most likely. The portal stays down. There is nothing in the article about what to do now, and the plumber is on hold.

Leila, who wrote the article, could have fixed the portal in six minutes. Everything Ravi needed was in her head. The article was the ten-minute summary she had typed from memory on a quiet Friday, which is a generous act and not a runbook. A runbook is a procedure written so that a competent stranger can run it alone, under pressure, and know whether it worked. It states what it assumes and gives each action with the check that follows it. It says what the output should look like, how to undo the action, and when to stop and who to call. It carries the date it was last run. This article is part of our Technician Docs vs User Docs series. The series is about keeping documents for technicians and documents for users apart. This article is about the technician's document in full.

It covers the competent stranger the runbook is written for, stating the assumptions first, and writing every action with its check. Next come saying what good looks like, the rollback, and when to stop and who to call. Then it covers the date it was last run, keeping it to one page and deciding where it lives. A worked example on a generic file share, a short war story, and the runbook skeleton to copy finish the article.

The Competent Stranger

A runbook has one reader, and it is the competent stranger: someone who knows the trade and does not know this building. They can read a command, understand a service, and find a log. They do not know which of your two servers holds the portal, what the service is called here, or that the database is on a third machine the diagram forgot. Every new hire is the competent stranger for their first six months. Every on-call technician is one at 02:10, when the part of their brain that knows the building is asleep. And the whole team becomes one eighteen months from now, when the person who set the thing up has gone to a job with a shorter commute.

Writing for the stranger changes what you leave in. The expert's summary leaves out everything the expert finds obvious, which is precisely the part the stranger needs. That is because the expert's obvious is the stranger's unknown. "Restart the portal service" is obvious to Leila. The stranger needs the server, the service's name, the command, and what the screen says when it has worked. Those four facts are not padding. They are the runbook. Everything else is a reminder for people who did not need one.

Writing for the stranger also changes the tone. The runbook does not persuade, explain the history, or apologise for the design. It states. "The portal runs on the second application server. The service is called the portal service. Restart it with the service manager. The check is that the status reads running and the portal's sign-in page loads within one minute." Short declarative sentences, one action each, numbered. If a sentence in a runbook made you smile while writing it, take it out and put it in the team chat. The stranger at 02:10 has no sense of humour, and should not be asked to find one.

State the Assumptions First

The first block of a runbook says what must be true before step one, because the stranger will otherwise find out at step four. It names the systems involved, with the names the desk actually uses: which server, which service, which database, which account. It says what rights the reader needs and where to get them if they do not have them. It says where the reader must be: on the office network, over the remote connection, at the console. And it says what the symptom looks like, in enough detail that the reader can tell this runbook is the right one. It also says what looks similar but is not.

The last of those, the recognition test, is the one most often skipped and the one that prevents the worst outcomes. A runbook for "portal not loading" that is run against a portal that is loading slowly, or loading for everyone except one contractor, is a correct procedure applied to the wrong problem. The stranger cannot tell the difference without being told. Two lines do it: "This runbook applies when the portal's sign-in page does not load at all from any device. If the page loads but sign-in fails, this is a different problem; see the account runbook." The same discipline, applied to a known error's workaround, is in Known Errors and Workarounds. It is the same discipline because the runbook for a recurring problem is very often the workaround written up properly.

Assumptions also include what the runbook does not cover. "This does not restore data; if the database is corrupt, stop at step six and escalate." A runbook that is honest about its limits is one the stranger can trust, because they know where its edge is. A runbook that implies it can fix anything has an edge too. It is just a surprise.

Every Action Has a Check

The body of the runbook is a numbered list of actions. Every action is followed by the check that tells the reader it worked. Action: restart the share service. Check: the service status reads running, and the share is listed when you list the shares. Action: clear the stale session. Check: the session no longer appears in the session list. Never two actions in a row without a check between them, because two unchecked actions are a procedure the reader cannot tell apart from a guess. A runbook without checks is a list of things Leila once did, in the order she remembers doing them.

Use generic names for actions where the desk has more than one way to do them, and exact ones where it does not. "Restart the share service" is enough when every technician uses the same service manager and the service has one name. Add the command when there is a command, in a line of its own, with the exact text, so it can be copied rather than retyped. Vendor syntax changes between releases, so the runbook says what the command does in plain words beside it. The "last run" line at the bottom tells the reader how recently the exact text was seen to work.

Write each action as one thing. "Restart the service and check the log" is two actions and one of them is a check; split them. "Confirm the backup completed, then stop the service, then restart it" is three. The reason is not pedantry. At 02:10 the stranger will do the first half of a compound step, be interrupted by the phone, and come back not knowing whether the second half happened. One action per numbered line means every line is either done or not done, and the reader always knows which. The same rule for the steps in a ticket note, where it is about reproducing a fault rather than fixing one, is in Writing Reproducible Steps.

Say What Good Looks Like

Expected output is the single most valuable thing a runbook contains and the thing most often missing. After every check, the runbook says what the reader should see, in the words or numbers that will appear on their screen. Examples are "status: running", "three shares listed, including the one named in the ticket", and "the sign-in page loads within one minute and shows the organisation's logo". Without that line, the stranger has restarted a service and is staring at a screen, deciding whether what they see is good. With it, the decision is made before they looked.

Expected output also includes expected time. "The service takes up to two minutes to start; do not restart it again before then" prevents the most common 02:10 mistake. That mistake is to do the right thing twice because it did not look like it had worked yet. "The portal is slow for about five minutes after a restart while its cache rebuilds" prevents the second most common. That is to decide the fix failed and start on step seven. Time is output. Write it down.

Say what bad looks like too, where you know. "If the status reads stopped after two minutes, the service failed to start. In that case, read the last twenty lines of its log and look for the word 'denied'. If you see it, the service account's password has expired. In that case, go to the account runbook." Each known failure, with its sign and its next step, is a branch the stranger would otherwise have to discover at the worst possible time. You do not need every branch. You need the three that have actually happened, which is why the runbook's "last run" notes matter. It is also why the technician who hits a new branch adds it before going back to bed.

Rollback: How to Put It Back

Every action that changes something has a line saying how to undo it. If an action cannot be undone, the runbook says so in capital letters before the action, not after. Restarting a service is its own rollback, mostly. Changing a setting needs the old value written down first. That means the runbook's step is "record the current value here: ______, then change it to the new one". Deleting a stale session, clearing a cache, rebuilding an index: each has a cost if it turns out to be the wrong move. The stranger is entitled to know the cost before paying it.

The rollback section also answers the question the stranger will ask at step six when nothing has worked: "how do I leave this no worse than I found it?" At 02:10, with the plumber on hold and the fix not working, the right outcome is often not a fix but a safe state and an escalation. A runbook might end with "if steps one to six have not restored the portal, return the setting in step four to its recorded value, confirm the service is in the state you found it, and escalate". That runbook has given the stranger permission to stop, which is the thing they least feel they have.

Where the procedure is itself a change to a shared system, the rollback in the runbook and the rollback in the change record are the same text, and should be. The one-page record that every planned change deserves, with its risk and rollback fields, is in The One-Page Change Record. A runbook that is run often is that record's reusable half. Write the rollback once, in the runbook, and let the change record point at it.

When to Stop, and Who to Call

The runbook ends with an escalation section, and it names a role, a time, and what to send. "If the portal is not restored thirty minutes after starting this runbook, or at any point the database server is unreachable, escalate to the infrastructure lead on the on-call rota. Send: which steps you ran, what each check showed, the current state of the service, and the ticket number." That paragraph converts a stranger's panic into a handover. Without it, the stranger either carries on past the point of usefulness, because stopping feels like failing, or phones the person whose name is on the article, who is on leave, at 02:40.

Name roles, not people, and point at the rota for who holds the role tonight. Leila will not always be the infrastructure lead, and a runbook with her mobile number in it will outlive her by years. The rota is the one document that knows who is on tonight; the runbook knows which role to wake. Where the desk has an escalation marker on the ticket that is separate from its priority, the runbook says to set it. Setting that marker makes the handover visible on the queue and not only in a phone call. How the handover itself should be written, so the next person does not start from nothing, is the business of the escalation series in this library. The runbook only needs to say "escalate, to this role, with these four things".

Set the stopping time deliberately and make it earlier than feels comfortable. Thirty minutes is long enough to run most runbooks twice; it is not long enough to improvise. A stranger who has been told to stop at thirty minutes stops at thirty minutes. A stranger who has not been told stops at dawn. The handover they give then is "I tried everything", which is true and useless.

The Date It Was Last Run

The last line of a runbook says when it was last run, by whom, and on which ticket. That line is the runbook's claim to be true. A procedure that has not been run in a year is a hypothesis about a system that has had a year of changes. A procedure run last Tuesday, by Ravi, on ticket 7318, with the note "step five took three minutes rather than two; updated", is evidence. The date tells the stranger how much to trust each line, which is a thing the stranger badly needs to know and has no other way to find out.

The ticket number is the better half of that line, because it points at the record of what actually happened. The runbook says what should happen; the ticket says what did, with times. The difference between the two is where the next edit comes from. If you run QuerySolv, the ticket named in the "last run" line carries its complete event history. In that case, the evidence that the procedure worked, and how long each step really took, is one search away rather than a memory. In any tool, the convention is the same: finish the runbook, write the ticket number and the date at the bottom, and note anything that differed.

A runbook that has not been run in a long time should be run on purpose, in a quiet hour. It should be run against a test system or during a planned window, by someone who did not write it. That is a rehearsal, and it is the only way to find the step that was always wrong but had never been reached. It is also the only honest way to update the date. The person who writes "last run: today" without running it has made the runbook less trustworthy, not more. The stranger will discover that at 02:10 on their behalf. The review that keeps the runbook and its derived user guide in step with each other, including when to rehearse, is in Keeping the Pair in Sync.

One Page, and Where It Lives

A runbook is one procedure on one page, and if it will not fit, it is two procedures. The stranger at 02:10 cannot hold a twelve-page document in their head and will not try. They will search it for the word "restart" and run the first thing they find. One page means the whole procedure is visible at once, the step numbers are small, and the escalation section is reachable without scrolling. Long procedures get a short runbook that says "run runbook A, then if its check fails run runbook B". That is a page the stranger can follow and a structure that lets A and B be maintained separately.

The runbook lives where the desk already looks when stuck, and it is visible to the desk only. Most knowledge bases can mark an article as visible to agents only, and that is the right place. There, the runbook is found by the same search as everything else, from the words in the ticket. It is never seen by a user. Where the tool has no such setting, a shared internal folder with one file per runbook, named by the symptom, does the job. That works as long as it is the one folder and not one of four. The runbook is the technician's half of a pair whose other half, the user guide, is published. Why the two must be kept apart, and how the split works in practice, is in Two Audiences, Two Documents.

Title it by the symptom, in the words that will be in the ticket. "Contractors' portal sign-in page does not load" will be found by a stranger searching for "portal" at 02:10. "Application tier restart procedure, server two" will be found by its author, once, when checking that it is there. A runbook nobody can find is indistinguishable from one nobody wrote, except that it took longer.

A Runbook Worked Through

Here is the whole thing on a generic file share, written for a stranger. The systems and commands are deliberately generic; your runbook names your server and uses your service manager's words. Read it for the shape: assumptions, then action and check in pairs with the expected output, then rollback, then escalation, then the last-run line. It is about a page, which is the right length, and it is dull, which is the right tone.

FILE SHARE NOT REACHABLE FROM WORKSTATIONS
Runbook RB-14   Owner: file services technician   Visible to: agents only

APPLIES WHEN
  Users on more than one workstation report the shared drive is missing
  and the share does not open by its network path from any machine.
NOT THIS RUNBOOK IF
  One user only: see RB-09 (drive mapping on a single workstation).
  The server itself is unreachable: see RB-02 (server not responding).
YOU NEED
  Admin rights on the file server; connected to the office network or
  over the remote connection. Server name and share name from the
  server list on the desk's systems page.

STEPS
  1. Check the share service is running on the file server.
     Expect: status "running". If "stopped", go to step 3.
  2. List the shares on the server.
     Expect: the share named in the ticket appears in the list.
     If it appears and users still cannot open it, go to step 5.
  3. Restart the share service.
     Expect: status returns to "running" within two minutes.
     Do not restart again before two minutes have passed.
  4. Repeat step 2.
     Expect: the share is listed. If not, go to ESCALATE.
  5. From a workstation, open the share by its network path.
     Expect: the folder opens within thirty seconds.
     If "access denied": permissions, not the service; see RB-11.
  6. Ask the requester to sign out, sign in and check the drive.
     Expect: the drive letter is back.

ROLLBACK
  Steps 1 to 6 change no settings. If any setting was changed during
  troubleshooting, restore it to the value recorded at the time and
  confirm the service is in the state found at step 1.

ESCALATE
  After thirty minutes, or if the service will not start twice, escalate
  to the infrastructure lead on tonight's rota. Send: steps run, what each
  check showed, current service status, the ticket number.

LAST RUN
  Date: ________  By: ________  Ticket: ________
  Differences noted: ______________________________________________

Notice what the example does not do. It does not explain what a share service is, because the stranger knows the trade. It does not give the history of the server, the reason the share exists, or an apology for the design. It gives six numbered actions, each with a check and an expected result. It gives a rollback that honestly says there is almost nothing to roll back, an escalation with a time and a list, and a line at the bottom waiting for a date. That is the whole form. Everything a runbook needs fits in it, and a surprising amount of what runbooks usually contain does not.

A Short War Story: The Three-Line Runbook at Harbourview Clinic

Harbourview Clinic's two-person desk had a runbook for the appointments system called "Appointments down". The runbook read, in full, "Restart the appointments service. Check it comes back. Call the supplier if not." On a Monday at 07:40, with the first patients due at eight, the service stopped. The technician on duty, Sam, who had joined six weeks earlier, restarted it as instructed. It started, showed running, and the appointments screens stayed blank. That was because the service depended on a reporting service on a second server that had to be started first. That fact lived only in the head of Sam's colleague, who was on a ferry with no signal. Sam restarted the appointments service four more times, then phoned the supplier. The supplier took fifty minutes to call back and asked first whether the reporting service was running. The clinic ran its morning on paper, two consultants saw patients without their notes, and the practice manager spent the afternoon re-keying ninety appointments. The rewritten runbook had eleven numbered steps, with an expected output after each. It had a line reading "the appointments screen stays blank until the reporting service on the second server shows running; start that one first", and a last-run date. The fix had always been six minutes long; the runbook had simply never said which six.

The Runbook Skeleton

The skeleton below is the blank form of the worked example. Copy it into an agents-only article or a file in the desk's internal folder, one copy per procedure. Fill it in from the ticket on which the procedure was last done, while the steps are fresh and the person who did them is still in the building. The expected-output lines are the ones to fight for. A skeleton filled in without them is the three-line runbook with more headings. Fill in the last-run line honestly or leave it blank, because a blank is a warning and a false date is a trap.

RUNBOOK — ONE PAGE
Title (symptom, in the words a ticket would use):
  ___________________________________________________________________
Runbook ID: RB-____   Owner (role): ______________   Visible to: agents only
Paired user guide (if any): ______________   Last updated: ________

APPLIES WHEN
  ___________________________________________________________________
NOT THIS RUNBOOK IF (looks similar but is not; which runbook instead)
  ___________________________________________________________________
YOU NEED
  Rights: ______________   From where: ______________
  Systems (server, service, database, account): ________________________
  What this runbook does NOT do: ______________________________________

STEPS (one action per line, then its check and expected output)
  1. Action: ___________________________________________________________
     Check: ______________________   Expect: ______________________
     If not: _____________________________________________________
  2. Action: ___________________________________________________________
     Check: ______________________   Expect: ______________________
     If not: _____________________________________________________
  3. Action: ___________________________________________________________
     Check: ______________________   Expect: ______________________
     If not: _____________________________________________________
  (continue; if more than about ten, split into two runbooks)

TIMING
  Step ____ takes up to ____ minutes; do not repeat before then.
  After the fix, expect ______________ for about ____ minutes.

ROLLBACK
  Settings changed and their recorded original values: ________________
  How to return to the state found at step 1: __________________________
  Steps that CANNOT be undone (say so before the step): _______________

ESCALATE
  Stop after ____ minutes, or immediately if: _________________________
  Escalate to (role): ______________   Found on: the on-call rota
  Send: steps run, what each check showed, current state, ticket number

LAST RUN
  Date: ________  By: ________  Ticket: ________
  Differences noted and fixed in this runbook: ________________________

Leila rewrote "Repairs portal down" from the skeleton the morning after Ravi's night, from the ticket, with Ravi reading each step back to her as a stranger would. The runbook came to fourteen numbered lines and named both servers and the service. It said that the sign-in page should load within one minute and show the housing association's logo. It ended with "after thirty minutes, escalate to the infrastructure lead on the rota". Ravi ran it for real five weeks later at 23:50. It took eight minutes. The plumber never knew. The last-run line at the bottom now has his name on it, which he mentions more often than is strictly necessary.

Frequently Asked Questions