Preparing Before the Big One: The One-Page Playbook

Friday, 16:10, at Copperline Printing. The server that drives the two big presses stops answering. The production manager appears at the desk inside ninety seconds. Dana, who runs the three-person desk, spends the next four minutes discovering things. She does not know whether she may restart the server without asking production, because nobody has ever said. She does not know the press supplier's support number without searching her mailbox for the last invoice. She does not know whether the other two technicians, both out at a branch, can log in to it from there, because it has never come up. At 16:14 the server comes back on its own, the presses resume, the production manager says "well, that's fine then", and it is not fine. The incident fixed itself. The four minutes did not.

A playbook, for a major incident, is one page that tells whoever is on the desk what to do in the first fifteen minutes. It says who to call, what they may decide without asking, and where the notes for each important system live. Preparing is writing that page and filling in the contact list by actually ringing the numbers. It includes agreeing the decision rights with the people who hold them, and running the page once, on a quiet morning, before the day it is needed. This article is part of our Major Incident Management for Small Teams series and is the last in it. That is because everything the other five describe is easier if this page exists first.

It covers what the playbook is for and what it is not. Next come the one page and what goes on it, in order; the contact list and how to test it. Then come decision rights, which are the part nobody writes down; the per-system pages for the handful of systems that matter. Next come the practice run, in thirty minutes, twice a year; keeping the page alive after it is written. Finally come a short war story and the playbook template, in that order.

What the Playbook Is For

The playbook turns the first fifteen minutes of an incident from a debate into a checklist. It is written for the most junior person on the desk on the worst possible day. Picture a Friday afternoon, the senior technician on leave, the manager in a car, and the production manager standing at the desk. If that person can pick up the page and get to a declaration, a first notice and a named fixer in a quarter of an hour, the playbook works. If the page assumes they know the supplier's name, where the diagram is, or that restarting the press server is allowed, it does not. Write it for them and the senior technician will be fine with it too.

It is one page, printed, in three places: at the desk, in the playbook folder on the shared drive, and in the lead's bag. It is printed because the day the playbook is most needed is a day when something is down. That something is occasionally the file share, the wiki, or the login that reaches both. Eleven pages, laminated, in a binder, is also a thing some organisations have, and it is read exactly as often as it is updated.

It is not a disaster recovery plan, which is the longer document about rebuilding systems from backups after the building floods. The playbook points at that plan and does not contain it. It is not a policy, and it does not need approval beyond the people whose names are on it. And it is not the per-system notes, which are separate pages, one each, for the handful of systems whose loss stops the organisation. The playbook is the page you read first. Everything else is a page it points to.

The One Page, in Order

The page has nine parts, in the order they are needed on the day, and the whole thing fits because most of the parts are pointers. First, the criteria: the three questions and the never-pause list from the criteria card in What Makes an Incident Major, copied in or pinned beside. Second, the declaration sentence, with blanks. Third, the roles and the default split for this desk: who is lead and who is fixer when both regular people are in, and what happens when one is out. Fourth, the bridge: the name of the chat channel or the call to open, or the ticket notes if there is neither. Fifth, the update interval for this desk, as a number.

Sixth comes where the three messages are, so the first notice can be sent from a template in two minutes. The templates themselves are in Communicating During the Incident and belong in the drafts folder, not on this page. Seventh, the contact list, which is the longest part and has its own section below. Eighth, decision rights: what the lead may do without asking and whom to ask for the rest. Ninth, the close: the pointer to the close checklist and the rule that the review is booked before the all-clear is sent. The page ends with the list of per-system pages and where they are.

The order matters because the page is read top to bottom while the phone rings. Nobody at minute three needs decision rights; they need the declaration sentence. Nobody at minute twelve needs the criteria; they need the deputy's mobile number. A page in the right order can be used without being read.

The Contact List

The contact list is every person and company the lead might need in the first hour. It includes the number that will be answered, the thing the lead will need from them, and the thing they will need from the lead. It is the part of the playbook most often half-done, because half of it feels obvious (the desk's own mobile numbers) and the other half feels like somebody else's job (the supplier's contract number). Both halves are needed at 16:11, and the second half is the one that costs twenty minutes when it is missing.

Six groups belong on it. The desk itself, with mobiles, and who is on today, which the coverage calendar in The Coverage Calendar already answers if the desk keeps one. The manager and the manager's deputy, because the manager will be in a car. Include the owner of each never-pause process, who is the person the lead calls to agree the workaround and who has to be told first. At Copperline, that is the production manager. Include each supplier whose system is on the never-pause list, with the support line, the out-of-hours line if different, and the account or contract number. Also include the name of the system as the supplier calls it, which is rarely what the desk calls it. The building: power, the landlord, the internet provider, the alarm company. And the alternate-channel list: the people who get a text message when e-mail is the thing that is down.

Test it by ringing. Once a quarter, somebody on the desk calls every supplier line on the list. They say "this is a test of our incident contact list, can you confirm this is the right number for support on account such-and-such", and write the date next to the entry. It takes an hour, and it is tedious. Every changed number or lapsed account it finds is one that the first fifteen minutes of a real incident will not have to. The entries for people are tested the same way, with a message rather than a call. A deputy who does not know they are the deputy is found in the same hour.

The table shows the shape of the list for the groups that are hardest to fill. The "what they need from you" column is the one to prepare, because a supplier's first three questions are always the same. A lead who has the answers on the page shortens the call by the ten minutes it would take the fixer to find them.

Decision Rights

Decision rights are the written list of what the lead may do during an incident without asking anyone, and whom to ask for everything else. They are the part of the playbook nobody writes, because on most small desks the answer has never been needed. The senior technician does what they think best, and if it goes wrong there is a conversation. That works until the senior technician is on leave and the junior one spends four minutes at 16:10 not restarting the press server because nobody has ever said they may.

Write two lists. The "may do without asking" list is for the lead. It includes restarting a service or a server on the never-pause list, and failing over to a standby. It includes restoring a system from backup when the fixer says it is the fastest route. It includes calling any supplier on the contact list, including out of hours, including when the call costs money up to an agreed amount. It includes putting a process on its paper workaround, having told the process owner; pausing the queue; and declaring and standing down. The "must ask" list says who to ask. It includes sending staff home or closing a site (the manager), and telling a customer or a regulator (the manager, and whoever the manager says). It includes spending above the agreed amount (the manager or the deputy), and taking a working system down to fix a broken one (the owner of the working one). It also includes anything to do with payroll, banking or patient data, which usually has its own rules already.

Agree the lists with the people on them, once, in a twenty-minute conversation that most managers are glad to have. That is because it is also the conversation in which they learn they will not be rung at 16:11 to approve a restart. Write the agreed amount as a number. Write the deputy's name. And add one line that the manager signs, in effect if not in ink: "If the lead cannot reach me or my deputy within ten minutes, the lead decides and tells me afterwards." That sentence is worth the rest of the page. It is the difference between a desk that acts and a desk that waits, and waiting is the most expensive thing a desk can do with an incident.

The solo administrator's version is shorter and more important. There is nobody to ask except the manager, who is the person most likely to be in a meeting. The "may do" list is the administrator's licence to act, and the ten-minute line keeps the incident moving when the meeting runs long. The pages that stand in for a colleague on a solo desk are the subject of Documentation as Your Second Person. The playbook is the one with the most urgent reader.

The Per-System Pages

Behind the playbook sits one page for each system on the never-pause list. Most small organisations have between three and six such systems. These are the thing that takes orders, the thing that pays people, the thing that runs the presses, the link between the sites, e-mail, and the login that reaches all of them. The page is written by the person who knows the system best, for the person who will be fixing it when they are away. It is written in the same order the fixer will need it.

Each page has eight short parts. What the system is and does, in plain words, including what the business calls it. How the desk finds out it is down: the alert, if there is one, or the person who will phone. The first three checks, in order, each one sentence, each with what a bad result looks like. The restore-first option: the standby, the failover, the paper process, and how long it takes to switch to. The supplier, with the details from the contact list repeated so that nobody flips pages. The known workaround for the people affected, in their words, ready to paste into the first notice. What not to do, which is usually a restart that clears evidence or a change that looks harmless and is not. And the last incident on this system, with the date and the one-line cause, so that the fixer knows whether this is a repeat.

The "what not to do" line is the one the senior technician most resists writing and the one that saves the most time, because it is the knowledge that lives only in their head. "Do not restart the application service before copying the log folder; the restart clears the in-memory state that shows which outside call was timing out" takes thirty seconds to write and once saved a desk two hours. Write it even if it feels like admitting that the system is fragile. The system is fragile. The page is where that gets to be useful.

The Practice Run

A practice run, sometimes called a tabletop exercise, is thirty minutes in a meeting room with the playbook, a scenario on a card, and a clock. The desk walks through the first fifteen minutes of a pretend incident out loud, doing everything except touching a system. It is the only way to find out whether the page works before the day. It finds, every time, something that the people who wrote the page could not see because they knew too much. Run it twice a year, and run it with the most junior person as lead at least once, because they are the reader the page was written for.

The scenario is one paragraph, written by someone who will not be lead: "It is 10:15 on a Tuesday. The order system has stopped loading for the warehouse. Three tickets have arrived. The senior technician is at a branch with no laptop. Go." The lead reads the playbook, says the four-question test aloud, and writes the declaration sentence on the whiteboard. They name a fixer, open a pretend bridge, draft the first notice from the template, and say who they would ring and what they would say. The facilitator plays the production manager, the supplier's switchboard, and the manager in a car, and answers only what is asked. Someone watches the clock and writes down when the declaration sentence was finished, when the first notice was ready, and where the lead stopped and said "I don't know where that is".

Measure three things and nothing else: minutes to the declaration sentence, minutes to a sendable first notice, and the number of times the lead had to ask for something the page should have given them. Good numbers on a small desk are under three, under eight, and zero. The third number is the point of the exercise; every "where is the supplier number?" and "am I allowed to restart it?" is a line to add to the page. The practice run is the cheapest way to find those lines ever invented. The debrief is ten minutes, blameless, with the lead's gaps written as actions against the playbook, not against the lead.

Include the manager once a year, as themselves, so that they experience being told "you are on the update list, reply here for decisions" and discover that it works. Include the owner of one never-pause process, so that they agree the paper workaround in one sentence and see what a first notice looks like before they receive a real one. And run the surprise variant occasionally: a declaration at 10:00 on a quiet day, announced as a drill in the sentence itself, with the first notice sent to the desk only. Nobody enjoys it. Everyone is faster the next time, which is the only thing a drill is for.

A practice run is not a technical drill. Restoring a server from backup to prove the backup works is a different, longer, and equally necessary exercise, and it belongs with the disaster recovery plan. The thirty-minute run tests whether the desk can get to a declaration and a notice; the restore drill tests whether the fixer can get the system back. Do both, and do the thirty-minute one first, because it is cheaper and because most desks have never done it at all.

Keeping the Page Alive

The playbook is edited after every incident and every practice run, by the lead, as one of the review's actions. It is read in full once a quarter by someone who did not write it, on the same day the contact list is rung. It has an owner, which is a person's role and not the desk in general. It has a "last reviewed" line at the top that the owner is slightly embarrassed to see get old. Those four habits keep it true. Without them it is a souvenir of the week it was written, accurate about a supplier that has since been replaced and a deputy who has since left.

The test of a live playbook is a new starter. In their first week, hand them the page and the scenario card, give them thirty minutes, and see how far they get. A page that stops them at "who is the deputy?" or "what is the bridge?" has a gap that the people who know the answers could not see. The new starter has just found it for free. Most new starters, asked to do this, assume it is a test of them. Tell them it is a test of the page. It is.

Resist the drift toward length. Every incident teaches something, and the instinct is to add a paragraph. The discipline is to add a line, and to move anything longer to a per-system page or the close checklist in Closing the Incident and the Review. A playbook that has grown to four pages is on its way to being the eleven-page binder. The binder is read by nobody at 16:10. One page, in order, printed in three places, rung quarterly, run twice a year. That is the entire discipline.

A Short War Story: The Playbook in the Leaver's Mailbox at Larkspur Nurseries

Larkspur Nurseries, a chain of six garden centres with a two-person desk, had a good incident playbook. It was written two years earlier by a technician called Ben, who had since left. It was found on the Saturday morning in spring when the till system at all six centres stopped taking card payments at 09:20. That was the busiest trading hour of the year. The remaining technician, Zoe, read it standing up, got to the declaration in four minutes, and then reached the contact list. The till supplier's support entry gave a number that had changed, an account reference that was Ben's login to the supplier's portal, and the note "contract details in my mailbox". Ben's mailbox had been deleted eleven months earlier. Zoe found the right number by ringing the supplier's sales line, which does not open on Saturdays, and then the regional manager, who had the account manager's mobile from a lunch. The supplier's engineer was on the phone at 10:35. The centres took card payments again at 10:52. They had turned away or taken cash from customers for ninety-two minutes on a morning that the finance director later estimated at a fifth of the day's takings across the chain. The playbook had been right about everything except the one thing that had changed. Nobody had rung the list since Ben wrote it. Larkspur's playbook now has an owner by role, a "last rung" date next to every supplier line, and a rule that the quarterly ring is a ticket in the queue. That ticket ages like anything else that is not done.

The One-Page Playbook

The template below is the page. Fill it in with the desk, print it, put it in the three places, and run it once with the most junior person as lead before anything breaks. The pointers at the bottom are to pages described elsewhere in this series: the criteria card, the first-fifteen-minutes checklist, the three messages, the incident log, and the one-page review. The playbook does not repeat them. It tells the lead where they are, in the order they will be needed.

MAJOR INCIDENT PLAYBOOK — ONE PAGE
Desk: __________________   Owner (role): ____________   Last reviewed: ________
Printed copies: [ ] at the desk   [ ] shared folder   [ ] lead's bag

1. IS IT MAJOR?  (criteria card pinned beside this page)
   Who cannot work? ______   Process stopped? ______   Workaround holds? ______
   Never-pause processes: ____________  ____________  ____________  ____________
   Doubt rule: after ____ minutes, declare.

2. DECLARE (write this in the bridge and in the incident ticket)
   "Major incident declared __:__ : ____________ unavailable, ____________
    affected, workaround ____________, lead ________, fixer ________,
    next update __:__"

3. ROLES ON THIS DESK
   Both in: lead ____________  fixer ____________
   Only one in: that person is lead and fixer, with a timer every ____ min
   Third person arriving takes: the updates, the phones, the linked tickets
   Who may declare: ________________ (role).  Manager may stand down, not hold.

4. THE BRIDGE: ______________________ (channel / call / ticket notes)
   Incident ticket subject prefix: "MAJOR:"   Priority: ______

5. UPDATE INTERVAL: every ____ minutes, on the clock, whether or not there is news.

6. THE THREE MESSAGES: saved as ______________________ (drafts / saved replies)
   First notice goes out by minute ____, before the cause is known.
   Alternate channel when e-mail is down: ______________________

7. CONTACT LIST                                   number        last rung
   Desk: ____________ ______   ____________ ______   ____________ ______
   Manager: ____________ ______   Deputy: ____________ ______
   Process owners: ____________ ______   ____________ ______   ____________ ______
   Suppliers (system / account ref / support / out of hours):
     ____________ / ________ / ____________ / ____________   ________
     ____________ / ________ / ____________ / ____________   ________
     ____________ / ________ / ____________ / ____________   ________
   Internet: ________ ref ________   Power: ________   Landlord: ________
   Text-message list when e-mail is down: ______________________________

8. DECISION RIGHTS
   Lead MAY, without asking: restart or fail over a never-pause system; restore
   from backup when the fixer says it is fastest; call any supplier on the list,
   up to ________ in cost; put a process on its workaround after telling its
   owner; pause the queue; declare and stand down.
   Lead MUST ASK: send staff home or close a site -> ________;
   tell a customer or regulator -> ________; spend above ________ -> ________;
   take a working system down -> its owner; payroll / banking / patient data ->
   ________ (existing rules apply).
   If the manager and deputy cannot be reached in ____ minutes, the lead
   decides and tells them afterwards.   Agreed by: ____________  on ________

9. AT THE ALL-CLEAR
   Two confirmations (fixer + one affected person) before sending.
   Book the review BEFORE sending the all-clear.  Close checklist: __________

PER-SYSTEM PAGES (one each; location: ______________________)
   [ ] ____________   [ ] ____________   [ ] ____________   [ ] ____________

PRACTICE RUN: last held ________  lead was ________  minutes to declaration ____
              minutes to first notice ____  gaps found ____  next run ________

Back at Copperline, Dana wrote the page the following Monday, in about two hours, and spent the third hour ringing the press supplier. The supplier confirmed a different support number from the one on the invoice and an out-of-hours line nobody at Copperline had known existed. The production manager agreed the "may restart" line in a four-minute conversation. He added, unprompted, that he would rather lose ten minutes of printing to a restart than an hour to a phone call about whether to do one. The practice run, three weeks later with the newest technician as lead, got to the declaration in two minutes and the first notice in six. It stopped once, at "where is the per-system page?" That is now the last line of the playbook. The press server has not failed since. It will, and the four minutes are ready for it.

Frequently Asked Questions