Coordinating a Major Incident With Two People

15:10 at Bellweather Logistics, and the handheld scanners in the warehouse have stopped talking to the stock system. Jo is the fixer, head down in the service logs. Ravi is the lead. Behind Jo's chair, in order of arrival, stand the warehouse operations manager, two shift supervisors, and the finance director. The finance director has come down "just to understand the situation". The operations manager has asked Jo three questions in four minutes. The finance director has suggested a restart. One of the supervisors is on the phone to the courier, describing what he can see on Jo's screen. Jo has read the same log line five times and has not yet understood it, which is not Jo's fault. There are four people breathing on it.

Coordination, during a major incident, is the work of keeping one accurate picture of what is happening in one place. That way, the fixer can fix, the lead can decide, and everyone else can see the picture without asking the fixer for it. On a large desk this is the incident manager's whole job. On a two-person desk it is half of one person's job, done while that person also sends the updates and answers the phone. The only way it fits is with three tools: a bridge, a log, and a door. This article is part of our Major Incident Management for Small Teams series, and it is about those three.

It covers what coordination means when there are two of you and the bridge, which is the one place the incident is discussed. Next come the lead's three jobs and the timeline log, including how to keep it while also talking to people. Then come keeping the crowd out of the fixer's way, including the crowd that outranks you, and deciding when to change course, escalate, or roll back. It finishes with a table of who posts what, a short war story, and the incident log template to copy.

What Coordination Means When There Are Two of You

The roles are settled before this article starts: one person is the lead and also the communicator. The other is the fixer. And the split is made in the first two minutes as described in Declaring an Incident and the First Fifteen Minutes. Coordination is what the lead does with the remaining seventy minutes. It is not fixing, not even a little. And the hardest thing about leading a two-person incident is that the lead is usually also a competent technician who could be helping. They are helping. They are helping by not being a third pair of hands on a system that needs one.

The lead has three jobs, and they are the same three on a desk of two and a desk of twenty. Keep the picture: know, at every moment, what is broken, what has been tried, what is being tried now, and what happens next. Have it written down. Make the decisions: what to try next, when to stop trying and escalate, when to roll back, when to tell the manager, when to send the all-clear. Keep the crowd out: be the person every question goes to, so that none of them reaches the fixer. Everything else the lead does, including every update sent, is a product of those three.

Coordination fails on a small desk for a reason that does not apply on a large one: everybody in the building knows where the desk is. On a large desk the incident happens on a call that managers can join and leave. On a two-person desk it happens at two chairs in an open office, and the crowd arrives on foot, in person, with opinions. A method that assumes the fixer is in another building does not survive contact with a finance director. The method below assumes the finance director is already there.

The Bridge

The bridge is the one place where the incident is discussed, and it is the first thing the lead sets up after the declaration. It is usually a chat channel named for the incident, or a standing voice call that stays open. And the rule is that there is exactly one and that everything said about the incident is said in it. A question asked in a corridor is answered with "ask it in the bridge". A finding the fixer mentions aloud is typed into the bridge by whoever heard it. The bridge is where the picture lives, and the picture only stays accurate if there is nowhere else for pieces of it to go.

For two people, a written channel beats a call, and the reason is that the fixer can ignore it. A call demands attention in real time. A channel can be read when the fixer reaches a natural pause. It keeps its own history, which is most of the timeline for free. Name the channel for the incident ticket, "inc-5120-scanners" or similar, so that it is findable afterwards and so that nobody posts about Tuesday's incident in Monday's channel. A call is better in two cases. One is when a supplier's engineer is working with the fixer and needs to be talked through screens. The other is when several sites are affected and each has a person who needs to report in. Then the call is the bridge, and the lead writes the log from it.

The bridge includes the lead, the fixer, the manager, and anyone whose job depends on the picture. That includes the operations manager who needs to decide whether to hold the courier. As for who posts in the bridge, the fixer posts one-line findings and one-line requests for a decision. The lead posts decisions and the update as it goes out. Everyone else reads, and asks questions of the lead by name. The phrase "any update?" is banned from the bridge on the grounds that the update has a time on it and the time is in the channel header. (It will be typed anyway. The lead answers with the time and nothing else.)

Many small desks have no chat tool, or have one that nobody opens. The bridge is then the notes of the incident ticket, or a document with a link in the ticket. It is less immediate and it works; the property that matters is that there is one place, not that the place is fast. A desk whose bridge is a shared document has a better incident than a desk whose bridge is four conversations, three of them with the fixer.

The Lead's Three Jobs, Minute by Minute

Keeping the picture is a loop the lead runs every few minutes. Read the fixer's last line, write the log entry, check the next update time, and look at the door. The fixer's line arrives in the bridge, "15:24 tried restarting the scanner gateway, no change, looking at the certificate next". And the lead turns it into a log line and into the sentence of the next update. The lead does not ask the fixer for status. The fixer posts it when there is a change. The lead asks only when a decision is needed or when the fixer has gone silent for longer than the update interval. That silence is itself a signal worth a one-line nudge.

Making the decisions is the part the lead cannot delegate, and the decisions come in a known set. What does the fixer try next, when there are two candidates and time for one? When does the desk stop trying to fix and start trying to restore, by failing over, restoring from backup, or putting the warehouse on paper while the fix continues? When is the supplier called, and who calls them? When does the manager need to decide something only a manager can, such as holding the courier or sending the evening shift home? Each decision is one line in the log with the time and the reason. And a decision made by the lead is better than a better decision made by nobody.

Keeping the crowd out is the door, and it has its own section below. On a small desk it is the job that decides whether the other two get done. A lead who spends the incident at the next keyboard, also fixing, has left the door open and the log blank. The review will have to be written from what the finance director remembers.

The Timeline Log

The timeline log is a list of one-line entries, each with a time. The entries say what was seen, what was tried, what was decided, what was changed, and who was told. The log is kept by the lead, written as the incident happens. And it lives in the incident ticket's notes or in a document linked from the ticket. It is the only record of the incident that will exist on Thursday. That is when the review tries to work out why the scanners were down for ninety minutes and not thirty. The log is the raw material for every update sent while the incident runs. A lead who keeps the log has already written the next update; it is the last three lines, in plainer words.

Entries have five kinds, and tagging them makes the log readable in a hurry. SEEN is a symptom or a finding: "15:12 SEEN scanners show 'host unreachable'; stock system web page loads normally." TRIED is an action and its result: "15:24 TRIED restart of scanner gateway service; no change." DECIDED is a choice and its reason: "15:31 DECIDED warehouse picks on paper until 16:00 at the latest; ops manager agreed; fix continues." CHANGED is any alteration to a live system, which matters later when something else breaks: "15:40 CHANGED replaced expired gateway certificate with the one from the spares folder." TOLD is a message sent: "15:45 TOLD update 3 to all warehouse staff and ops; next 16:05." Five tags, one line each, the time first. That is the entire format.

Write it as it happens, not after, and accept that it will be ugly. The log written at 16:30 from memory is neat, confident, and wrong about the order of three things. And the order is the part the review needs. The log written at 15:24 says "no change" with a typo and is right. If the lead is on the phone when the fixer posts a finding, the finding is in the bridge with a time on it. The lead then copies it into the log at the next pause. On a solo desk the log is kept by the fixer, in the two-minute stop every fifteen minutes, and it is shorter and still right.

Keep ideas separately from actions. The crowd produces suggestions, some of them good, and the fixer cannot try all of them. The lead writes each one at the bottom of the log under a heading of its own, with who suggested it. The lead picks from the list when the current attempt finishes. "Have you tried restarting it?" becomes "IDEA (finance director, 15:18): restart" and the finance director, having been written down, usually stops saying it. A parked idea is a dealt-with idea. A dismissed one comes back every four minutes.

Keeping the Crowd Out of the Fixer's Way

The crowd is everyone who is near the fixer and not fixing. That includes managers, supervisors, the colleague who had this exact problem once, the supplier's account manager on the phone, and the person who has come to watch. Every one of them is there for a good reason, which is that something important is broken and they want it fixed. Every one of them costs the fixer the same thing: the five minutes it takes to get back to the log line they were reading. The lead's job is to take the good reason seriously and the five minutes away.

The first tool is physical. Put the fixer somewhere the crowd is not, or put the crowd somewhere the fixer is not. Do it in the first ten minutes before the crowd has settled. A meeting room with a door, a desk at the far end, headphones with nothing playing in them: anything that makes the fixer harder to reach than the lead. If the fixer cannot move, the lead stands up, which sounds trivial and changes where people walk to. The lead's chair, facing the room, is the door. The fixer's chair, facing the wall, is the workshop.

The second tool is the standing sentence, said in the same words to everyone who arrives: "Jo is on it, the next update is at 15:30, and you will get it in the bridge; is there anything you need from me before then?" The first clause answers "what's happening". The second answers "when will we know", and the third answers "where do I look". The question at the end turns a spectator into someone with a decision to make or a reason to leave. Said five times in ten minutes it starts to sound like a recording. That is fine. Recordings are reliable.

The third tool is the bridge itself, offered as the place to watch. A manager who wants to stand behind the fixer wants to know what is going on. And the bridge is where that is, in writing, with times, and with the next update promised. "Join the channel, everything Jo finds goes there within a minute" is an invitation, not a dismissal. Most managers accept it with relief, because they did not want to stand in a warehouse office either. The ones who stay get the standing sentence and a chair that faces the lead.

The crowd that outranks you is the hard case, and the rule does not change. The finance director who suggests a restart gets the suggestion written in the ideas list, thanked, and told when it will be considered. The owner of the company who wants to talk to the fixer directly gets the lead. The lead says "I can answer anything Jo can, and Jo is thirty seconds from the next finding; what do you need to decide?" This is not insubordination. It is the lead doing the one job the organisation most needs done at 15:20. And most senior people, told plainly that the fixer is being protected on their behalf, are on the lead's side. The one who is not will be at the review, where the log will show what the interruption cost.

Deciding When to Change Course

Every attempt gets a time limit before it starts, and the lead sets it. "Try the certificate; if it has not worked by 15:50 we fail over to the old gateway." A timebox is just a decision made in advance about when to stop. It is the lead's main defence against the fixer's natural and admirable refusal to give up on a theory. Fixers chase causes. Leads watch clocks. The incident needs both, and the lead is the only one of the two who can see the time.

Restore before repair when the two are different. Suppose the scanners can be put back on the old gateway in ten minutes while the certificate problem takes an hour to understand. In that case, the warehouse should be scanning in ten minutes and the understanding can happen afterwards, in a problem ticket. The decision to restore service by a different route, and to keep the fix going in the background, is the single most valuable decision a lead makes. It is the one most often missed, because the fixer is close and the lead is listening to the fixer. Ask it out loud every timebox: "Is there a faster way to get them working that is not the fix?"

Escalate to the supplier on the second failed attempt or at the half-hour, whichever is first. Have the lead make the call so the fixer does not stop. The supplier will want the symptom, what has been tried, the system's version and the contract number. All of these are in the log and the playbook if the playbook was written. The call is ten minutes shorter for every one of them that does not have to be found. The mechanics of escalating to an outside party, including what to say and what to write down, are in Escalating to Vendors and Other Teams, and an incident escalation is that method at speed.

Roll back when a change made during the incident has made it worse, and treat the roll-back as a change too. The CHANGED lines in the log are the list of what can be rolled back; if there are none, nothing has been changed, which is itself a finding. A lead who cannot answer "what have we changed since 15:10?" from the log in ten seconds has a gap in the log. That gap is exactly where the next hour will be spent.

The table below is the bridge etiquette by role, for a two-person desk with the manager and the crowd in the channel. Most of a well-run bridge is the fixer's one-liners and the lead's decisions, with everyone else reading.

The Solo Desk and the Desk of Three

With one person, the bridge is still worth opening, because it gives the manager and the department heads somewhere to look that is not the administrator's shoulder. The solo administrator posts their own one-liners in it during the two-minute stop every fifteen minutes. And the log is those lines copied into the ticket at the all-clear. The door is harder to hold alone. The standing sentence does most of the work: "I am fixing it; the next update is at 15:30 in the channel; if you need a decision, write it there." Then the administrator turns back to the screen, and keeps turning back.

With three people, the communicator role leaves the lead. The communicator posts the updates and relays the department heads' facts. The lead's posts shrink to decisions and timeboxes. This is the point at which the lead can keep a genuinely good log, because they are no longer also on the phone. Plan that split in the playbook, which Preparing Before the Big One describes. Do not discover it at 15:35 when the third person walks in and asks what they can do. The answer, pre-written, is "take the updates and the phones; here is the interval and the last message".

A Short War Story: Four People Behind One Chair at Stonebridge Insurance

Stonebridge Insurance's claims system failed at 10:50 on a Monday, the busiest claims morning of the week. The three-person desk's most experienced technician, Mei, sat down to fix it. Behind her chair stood the claims manager, the operations director, a team leader and a colleague from the desk, each with a question or a theory. Nobody had been named lead, so nobody held the door. At 11:05 the operations director's suggestion, delivered from behind her shoulder for the third time, was followed: Mei restarted the application server. That cleared the error from the screen and also cleared the in-memory state. That state would have shown, in about four minutes, that a licence check to an outside service had been timing out since 10:48. The restart brought the system up for six minutes and down again. With the evidence gone the desk spent the next two hours testing theories. Meanwhile, claims were taken by hand on a form that the team leader found in a cupboard. The incident ran for three hours and ten minutes. Roughly two hundred claims were keyed in twice over the following days. And the review's one finding was that nobody had been standing between the fixer and the room. Stonebridge's playbook now names the lead before the fixer, and the lead's first instruction is to turn their chair to face the door.

The Incident Log Template

The template below is the timeline log on one page. It has a header, the entries with their five tags, the running list of decisions, and the parked ideas. It also has the roll-back list drawn from the CHANGED lines. Keep it in the incident ticket's notes or in a document linked from the ticket. Write it as the incident happens, and copy its last three lines into each update. At the all-clear it goes, unedited, into the review, which Closing the Incident and the Review builds from it. Do not tidy it. The untidy version is the true one.

INCIDENT LOG — ONE PAGE
Incident ticket: ________   Declared: ____:____   Lead: ____________   Fixer: ____________
Bridge: ______________________   Update interval: ____ min   Next update: ____:____
What is broken (plain words): _______________________________________________
Who cannot work: ____________________   Workaround in use: ____________________

TIMELINE (time first, one line each; tags SEEN / TRIED / DECIDED / CHANGED / TOLD)
  __:__  SEEN     ________________________________________________________
  __:__  TOLD     first notice to ____________________; next update __:__
  __:__  TRIED    ______________________________________ result: __________
  __:__  DECIDED  ______________________________ because ___________________
  __:__  CHANGED  ______________________________ on __________ by __________
  __:__  TOLD     update __ to ____________________; next update __:__
  __:__  TRIED    ______________________________________ result: __________
  __:__  DECIDED  ______________________________ because ___________________
  __:__  SEEN     ________________________________________________________
  __:__  CHANGED  ______________________________ on __________ by __________
  __:__  TOLD     all-clear to ____________________

DECISIONS (copied from the DECIDED lines, in order)
  1. __:__  ____________________________________  decided by __________
  2. __:__  ____________________________________  decided by __________
  3. __:__  ____________________________________  decided by __________

TIMEBOXES
  Current attempt: ______________________  stop and change course at __:__
  Supplier called at __:__ by __________  reference: __________

IDEAS PARKED (who, when, what; picked from when an attempt ends)
  [ ] __________  __:__  ____________________________________________
  [ ] __________  __:__  ____________________________________________
  [ ] __________  __:__  ____________________________________________

ROLL-BACK LIST (every CHANGED line, newest first)
  [ ] __:__  __________________________  how to undo: ______________________
  [ ] __:__  __________________________  how to undo: ______________________

ROLE CHANGES
  __:__  ____________ takes ____________ from ____________; briefed for __ min

Two parts of the sheet earn their place more than the rest. The roll-back list, because at 16:10 when the second thing breaks the question will be "what did we change?", and the answer should take ten seconds. And the parked ideas, because the list is where the crowd's energy goes, and a crowd whose ideas are written down is a crowd that has been heard.

Back at Bellweather, Ravi stood up at 15:14 and said the standing sentence four times. Ravi opened a channel called inc-5120-scanners that the finance director joined from his own office. Jo found an expired certificate on the scanner gateway at 15:38, with the log line she had been trying to read at 15:10. She replaced the certificate by 15:52. The warehouse picked on paper for twenty-six minutes, by Ravi's decision at 15:31, which the operations manager agreed to in the channel in one word. The log has nineteen lines, three decisions, two parked ideas, one of which was "restart it", and one CHANGED entry with a roll-back note. Nobody stood behind Jo's chair after 15:16, and the courier left on time.

Frequently Asked Questions