Communicating During the Incident: The Lead's Routine

09:05 at Northgate Retail, and the stock lookup that every till in twelve stores depends on has stopped answering. Lena declared at 09:02 and is the lead; Sam is the fixer. The desk phone has rung six times in three minutes, each time a store manager. Each asks the same two things: what is going on, and what should they tell the queue at the till. Lena does not know what is going on. Sam has been looking for three minutes and has a hunch about a database connection and nothing else. The regional manager has sent a message that reads, in full, "???". Lena has to write something now, to twelve stores, with no cause, no fix time, and a hunch she is not allowed to repeat.

Incident communication is the routine of telling the people affected what is broken, what to do in the meantime, and when they will hear next. It comes at fixed times, from one person, in the same shape every time. It is a routine rather than a skill: the three messages have fixed parts, and the interval is on a clock. The hard sentence, "nothing has changed", has a form that can be written in forty seconds. This article is the lead's view of that routine. It is part of our Major Incident Management for Small Teams series. A later series in this library covers outage communication in its own depth. This one stays with what the lead does, message by message, while the incident runs.

It covers who the lead is talking to and why each audience needs something different. Then come the first notice, sent before anyone knows the cause, and the update interval and why it runs on a clock. Next is what to say when there is nothing new. Then it covers which channels carry the messages, including the one that still works when e-mail is the thing that broke. The manager's version of each message follows, then the all-clear and the two confirmations it waits for. A table of the three messages, a short war story, and the three message templates to copy finish the article.

Who You Are Talking To

The lead sends messages to three audiences, and each needs something different. The people affected need to know what to do in the next twenty minutes and when they will hear more; they do not need the cause and will not read a paragraph about it. The manager needs the same message plus any decision only they can make, framed as a choice. The desk itself needs the bridge, not messages, and is covered in Coordinating a Major Incident With Two People. Everything in this article is about the first two audiences, and most of it is about the first.

The people affected are not one group, and the lead should name them. At Northgate they are twelve store managers, who each have a till queue and staff to instruct. There is the regional manager, who has twelve store managers. There is also head office, where stock lookups fail silently in a back-office screen. The first notice goes to all of them, in words that make sense at a till. "The stock lookup system" is their phrase. "The inventory service's database connection pool" is Sam's phrase, and Sam is not on the distribution list.

Write to the person at the till, every time. The test for any sentence in an incident message is whether someone who has never seen the desk could act on it in the next minute. "Sell what you can see on the shelf, note the barcode on a slip, and we will reconcile stock afterwards" passes. "We are investigating a connectivity issue" fails, because it leaves them exactly where they were, holding a slip, wondering what to write on it.

The First Notice

The first notice goes out between five and ten minutes after the declaration and before anyone has a cause. It has four fixed parts: what is broken, in the affected people's words; who is affected; what to do in the meantime; and when the next update will come. "The stock lookup on the tills is not working in all stores since about 09:00. Tills can still take payment. Please sell from the shelf and write the barcode and quantity on a slip for anything you cannot look up; we will reconcile stock with you afterwards. Next update at 09:30, whether or not there is news." That is the whole first notice, and Lena sent it at 09:08.

The first notice never contains the cause, because at minute eight there is not one. A guessed cause is a sentence the lead will spend the rest of the morning taking back. It never contains an estimated fix time, for the same reason. It does not apologise at length, because an apology at minute eight reads as a message with no content. One clause, "sorry for the disruption", is plenty, and the workaround is the real apology. And it never names a person as responsible, including the supplier, because the message will be forwarded and the supplier will read it. The notice is what, who, what to do, and when next. Anything else is decoration, and the store managers are not reading for decoration.

The hardest part of the first notice is sending it before the cause is known. The only cure is to decide in advance that the notice is sent at minute eight whatever is known. The store managers do not need the cause to instruct their tills. They need the workaround and the time. A desk that waits until it understands the problem to say anything has, by the time it understands, taken forty phone calls. A message from the regional manager has overtaken it, written from what one store manager overheard. That is now the official version.

The Update Interval

Updates go out on a clock, not on a condition, and the clock is set in the first notice. "Next update at 09:30" is a promise that can be kept with no news. "We will update you as soon as we know more" is a promise that the people affected cannot plan around. The lead will forget to keep it, because there is never a moment when "more" is clearly known. The interval is a playbook decision, fifteen to thirty minutes for most desks. It is shorter when people are standing at tills, and longer when the workaround is comfortable and the incident is in its third hour. Northgate's is twenty minutes for anything that touches a till.

Each update has the same four parts as the first notice, in the same order, with one addition at the front: what has changed since the last one. "Update 2, 09:30. No change: the stock lookup is still unavailable in all stores. We have ruled out the network and are now working on the database behind it. Keep selling from the shelf and keeping slips. Next update at 09:50." The shape is fixed so that the reader can find the one line they need, which is nearly always the last one. It also lets the lead write it in a minute from the last three lines of the timeline log.

Send an update early when something the reader must act on changes, and never skip one because nothing has. A workaround that stops working, a store that can now look up stock, a decision to send part-time staff home: each is an update now, and the clock restarts from it. A scheduled update with no news goes out anyway, on time, because its job is to say that someone is still there. The messages people remember from an incident are the ones that did not arrive. The 09:30 update that never came is talked about for a year.

What to Say When There Is Nothing New

The no-news update is the one leads dread, and it has a form that makes it the easiest to write. The form is: no change, what we are doing now, what to keep doing, when next. "Update 3, 09:50. No change; the stock lookup is still unavailable. We are working with the database supplier, who is now on the call with us. Keep selling from the shelf and keeping slips. Next update at 10:10." Four sentences, forty seconds, and every store manager knows three things: it is still broken, someone is still on it, and 10:10 is when to look again. The absence of news is itself information. It says the desk has not gone to lunch.

Resist the three usual substitutes for content. The first is the vague reassurance: "we are working hard to resolve this as quickly as possible", which the reader translates as "nothing to say". The second is the soft estimate: "we expect this to be resolved shortly". Here, "shortly" is a word that has never once meant anything and will be quoted back at the lead at 11:15. The third is the cause-in-progress: "it appears to be a database issue". It will be forwarded and discussed at the tills. When the cause turns out to be a certificate, it will be remembered as the time the desk got it wrong. Say what you are doing. Doing is a fact. Expecting is a hope.

Honesty about not knowing is not weakness in an update; it is the thing that makes the next update believable. "We do not yet know the cause" is a plain sentence that the people at the tills accept without difficulty. They are not expecting the lead to be an oracle, only to be present. What they will not accept is being told at 09:30 that it will be fixed by 10:00, and at 10:00 that it will be fixed by 10:30. A desk that is right about the next update time every time can be wrong about the cause for an hour and keep every ounce of trust it had. The time is the promise. Keep the promise and the rest is forgiven.

Give an estimated fix time only when the lead would bet their own afternoon on it. That means the fix is known, the fixer is applying it, and the remaining work is a restart with a known duration. Then say it with a margin: "we expect the lookup back by 10:25; next update at 10:25 either way." The "either way" is the part that keeps the promise if the estimate fails, and it should be on every estimate the lead ever sends. The rules for what a desk may promise and what it must never promise are in What to Promise and What Never to Promise, and an incident estimate is the sharpest case of them.

The Channels

The messages go wherever the affected people already look, and the playbook names which that is. For most small organisations it is an e-mail to a distribution list, plus one reply to every ticket linked to the incident. That way, each reporter hears in their own thread. Add whatever the organisation uses for announcements: a chat channel that everyone is in, or a banner on the support portal if the tool can show one. It could be a line on the phone greeting if the phone system allows it. Some tools have a broadcast or mass-notification feature; most small-team tools do not. A reply sent to many linked tickets, or an e-mail from the mail system to a list, does the same job. Pick the two channels that reach the most people fastest and use the same two for every message, so that nobody has to guess where to look.

The first notice also goes to one human at each affected site, by phone, when the site has a public counter. A store manager who receives an e-mail at 09:08 while standing at a till with a queue does not read it until 09:20. A store manager who gets a thirty-second call reads the e-mail afterwards for the details. On a two-person desk twelve calls is too many for the lead, and this is where the playbook's phone tree earns its place. The lead calls the regional manager, who calls the twelve, with the e-mail as the script. The message is identical in every mouth because it was written down once.

Plan for the day the channel is the thing that is broken. When e-mail is down, the e-mail about e-mail being down reaches nobody. The lead needs the alternate channel to have been agreed before the day. It could be the chat tool, a text message to a list of department heads kept in the playbook, or a phone tree. It could be a notice on the door of the building. The alternate channel is tested once a quarter by sending a dull test message through it. The person who sighs at the test is the one who will be most relieved on the day. How the desk's channels fit together in ordinary times is the subject of One Queue, Many Doors; during an incident the doors are used in reverse, for messages going out.

Talking to the Manager

The manager gets every message the affected people get, at the same time, through the same channel, plus one private line when there is a decision only they can make. The private line is a choice, framed by the lead, with a recommendation and a deadline: "We can put the stores on the old lookup server in about fifteen minutes with stock figures from last night, or keep fixing the live one for up to another forty. I recommend the old server now. I need a yes by 09:45." That is a message a manager can answer with one word from a train, and most will.

Do not send the manager problems without options, and do not send them the fixer's theories. "The database is doing something strange" is not a message to a manager; it is a worry, and worries sent upward come back down as visits. The manager also does not need to be told the interval has been kept; they can see it, because they are on the list. What they need from the private line is the decision, the recommendation, and the time by which silence will be taken as agreement. The lead should state that time and then act on it. A manager who does not reply by 09:45 has delegated, whether they meant to or not.

The All-Clear

The all-clear is the message that says the incident is over, and it waits for two confirmations. The fixer confirms that the fix is applied and the system looks healthy from the desk. An affected person confirms that it works from where they sit. The second is the one desks skip, and it is the one that matters, because a service that is up on the server and down at the till is down. Lena's rule at Northgate is that the all-clear waits for Sam to say "healthy". It also waits for one store manager, picked from the list, to say "I just looked up a barcode and it worked". Two confirmations. Then the message.

The all-clear says five things. It says it is fixed, since when. It says what the reader should do now, such as restart, log in again, or reconcile the slips. It says what was lost or needs redoing, if anything. It says what happens next, which is the review and any follow-up tickets. And it says thanks, in one line, for the workaround they ran. "All clear, 10:32. The stock lookup is working again in all stores since 10:28. Please log out and back in on each till. Slips from this morning: enter them into the stock adjustment screen by end of day, or send them to the desk and we will do it. We will review what happened and send a short summary by Thursday. Thank you for keeping the tills moving." It does not explain the cause, beyond a phrase, because the cause is the review's job and the review will be more accurate than anything written at 10:32.

Send a partial all-clear when service is restored but the fix is temporary, and say so in plain words: "The lookup is working again on a standby server. Stock figures may be a few hours behind until this evening. We will move back to the main server overnight and tell you before we do." A partial all-clear that pretends to be a full one costs the desk the next incident's credibility, because the people at the tills will discover the difference. Closing the incident ticket, the linked tickets, and the conversation that follows the all-clear are in Closing the Incident and the Review. The all-clear message is the last thing the lead's routine sends, and the first thing the closing routine reads.

The table below puts the three messages side by side. The "never contains" column is the one to read twice. Every item in it is something a well-meaning lead has typed at minute eight, with the best of intentions, and regretted by minute forty.

The Routine on a Solo Desk

With one person, the routine is the same three messages on a longer interval, written in the two-minute stop every fifteen or twenty minutes. The manager is told at the declaration that this is how it will work: "You will get an update every twenty minutes in the chat channel; if you need a decision, write it there and I will see it at the next stop." The first notice is still sent before the cause, still at about minute eight. The solo administrator has even less time for phone calls than a lead with a fixer does. The first notice is what stops the phone. A solo desk that sends the first notice on time has bought itself twenty quiet minutes to work. A solo desk that does not has bought itself twenty phone calls and the same problem.

A Short War Story: The Skipped Update at Ashgrove Housing

Ashgrove Housing's tenancy system went down at 09:40 on a Monday, during the hour when its five neighbourhood offices open to tenants. The lead, Priya, sent a good first notice at 09:48 with paper forms as the workaround and the next update promised for 10:10. At 10:10 nothing had changed. Priya was on the phone to the supplier and felt she had nothing to say. She let the update slide "until there was something worth sending". Between 10:10 and 10:40, the desk's phone took forty-one calls from office staff asking whether the system was back. The housing director, having heard nothing, sent her own message to all offices. It said the system would be "back within the hour" on the strength of a corridor conversation. Two offices stopped using the paper forms because the director's message implied they would not be needed. When the system returned at 11:25, those offices had an hour of visits with no record at all. The review counted the cost at roughly sixty tenant visits to be reconstructed from memory. The cost also included a director who had, in good faith, published a fix time the desk had never given. Ashgrove's playbook now has the sentence "an update with no news is sent on time and says no news" in bold on the first page. Priya, who wrote it, says it is the only line in the playbook she has never been tempted to skip twice.

The Three Messages

The templates below are the first notice, the update, and the all-clear, with blanks where the incident's facts go and nothing else to write. Keep them in the playbook, in the lead's drafts folder, and as saved replies in the ticketing tool if it has them. The first notice should be sendable in under two minutes by whoever is lead, and the no-news update in under one. The words outside the blanks are deliberately plain. They are read at a till.

THE THREE INCIDENT MESSAGES

1. FIRST NOTICE (send 5-10 minutes after declaring; cause not needed)
Subject: ____________________ not working since about __:__

____________________ (the thing, in your words) is not working
since about __:__. This affects ____________________ (who).
In the meantime: ____________________________________________
(the workaround, or "please wait; there is no workaround yet").
Next update at __:__, whether or not there is news.
Sorry for the disruption.
-- ____________ (lead), on behalf of the desk

2. UPDATE (send on the clock; early if what to do changes; never skip)
Subject: Update __ : ____________________ at __:__

Since the last update: ____________________________________
(what changed, or "No change: ____________ is still unavailable").
We are now ____________________________________ (what the desk
is doing, not what it expects).
Please keep ____________________________________ (the workaround).
Next update at __:__, either way.
-- ____________ (lead)

3. ALL-CLEAR (send after the fixer confirms AND an affected person confirms)
Subject: All clear: ____________________ working again since __:__

____________________ is working again since __:__.
What to do now: ______________________________________________
(restart, log in again, nothing).
Work done during the outage: _________________________________
(what to re-enter or send to the desk, or "nothing was lost").
What happens next: we will review what happened and send a short
summary by ____________. Any follow-up will come as a ticket.
Thank you for ______________________________ (the workaround they ran).
-- ____________ (lead)

PARTIAL ALL-CLEAR: use message 3 and replace the first line with
"____________ is working again on ____________ (standby / temporary
fix). ____________ may be ____________ until ____________. We will
____________ and tell you before we do."

Three things about using them. Fill the first notice's "in the meantime" line from the never-pause list in the criteria card. That list should already say what each process does when its system is gone. If it does not, the first incident is when the line gets written, and the second is when it is ready. Number the updates, because "Update 3" tells the reader instantly whether they missed one. And send the all-clear to every linked ticket as a reply, not only to the list. That way, each person who reported the problem hears in the thread they opened. A ticket closed by a status change alone tells the requester nothing. The article Avoiding the Silent Close explains why that silence costs more than the reply.

Back at Northgate, Lena sent six messages between 09:08 and 10:32. There was a first notice and four updates, one of which said "no change" in its first line. There was an all-clear after a store manager in the Harbor Street store looked up a barcode and said "yes". The phone rang once after 09:10, from a store that had not been on the list, which is now on the list. The regional manager's second message of the day was "thanks for the updates". The cause, a connection pool that had run out after a batch job, was in the review on Thursday, where it belonged.

Frequently Asked Questions