Choosing the Channels for Outage Messages

Monday, 07:50, at Harbourview Clinic. E-mail has stopped. Priya on the desk knows inside two minutes, because the practice manager phoned. She does the right thing: she writes a first notice, three lines, next update at 08:15, and sends it to the all-staff list. It sits in her outbox. She watches it sit there for a full minute before the obvious occurs to her. Then she laughs, briefly, in the way you laugh at 07:52 on a Monday. Reception opens at 08:00 with thirty patients booked. The nurses live in the chat tool and have not noticed anything. The receptionists live in e-mail and have noticed everything. Joan on the front desk has already been asked nine times whether the referrals have gone. She has no idea, because the only message anyone has written about it is in an outbox upstairs.

A channel is any route a message takes to reach a person. It could be an e-mail to a list, a post in a chat tool, a page people visit, a phone call, a sheet of paper on a door, or a sentence said by the receptionist. Choosing the channels for outage messages means deciding, before anything breaks, which routes carry which messages for which kinds of outage. That way, on the day it is a lookup rather than a decision. This article is part of our Outage Communication series. The content of the messages is covered in The First Notice and the articles that follow it. This one is about the doors the messages go through.

It covers the principle of one message through several doors and e-mail and its blind spot. Next comes what to do when e-mail is the thing that is down. Then come chat and the status page as a door people open themselves. After that come the phone tree, the phone greeting and the receptionist. Then come paper and the physical world; reach against noise, with a comparison table; and a short war story. Last comes the channel matrix by outage type to copy.

One Message, Many Doors

Write the message once and post the same text through two or three doors. The content does not change between channels, only the length. The full notice goes by e-mail and into the chat channel. The subject line goes to the phone greeting and the receptionist. The whole thing goes on the status page where people can read it when they choose. Writing a different message for each channel takes longer and produces versions that disagree. It also guarantees that the person who read the chat version will forward it to someone who read the e-mail version. At that point the desk is answering questions about its own messages. One text. Several doors. The same clock time for the next update on all of them.

The doors are chosen by who is affected, not by what the desk prefers. The desk prefers e-mail because it is written, it is kept, and it goes to everyone with a mailbox. The warehouse does not read e-mail until the shift break. The nurses do not read it at all between patients. The drivers do not have it. For each kind of outage, the question is "where are the affected people looking right now?" The answer is often a door the desk never uses for anything else. The same thinking in the opposite direction, how requests come in through many doors to one queue, is in One Queue, Many Doors. During an outage the doors are used in reverse, and the same list applies.

Replies come back through the doors too, and they should all land in one place. In QuerySolv the way to do this is to link every report of the outage to one incident ticket. Then send each message as a public reply on the linked tickets as well as through the announcement doors. Each person who reported it then hears in the thread they opened, in the channel they used. The "me too" replies arrive as tickets in the same queue instead of as forty chat mentions and an inbox. Whatever your tool, the rule is the same: announce outward through the doors, and collect inward through the queue.

E-mail: The Default and Its Blind Spot

E-mail is the default channel for outage messages because it reaches everyone with a mailbox. It is written down, and it is still there at lunchtime for the person who was in a meeting. For an outage that affects people at desks, a message to the right distribution list is the single most effective thing the desk can send. A distribution list is one address that delivers to many people; "all-staff" and "sales-office" are the usual kind. The lists have to exist before the day. The desk has to be allowed to send to them. On many systems, someone has to grant that permission. Ask for it on a quiet afternoon. Asking during the outage is a second ticket.

E-mail's blind spot is the person who is not at a desk. A receptionist with a queue of patients reads e-mail every twenty minutes at best. A warehouse supervisor reads it at the break. A nurse reads it after clinic. A driver does not have it. For all of these people, an e-mail at 08:05 is a message that arrives at 08:25. That is after they have phoned, after they have asked three colleagues, and after they have formed a view. E-mail is a written record that reaches people slowly. It is excellent at being the record and poor at being the alarm, and most desks ask it to be both.

Make the subject line do the work, because many people will read nothing else. "Order system not working since 08:50; next update 09:20" is a complete message in a notification on a phone screen. "Service disruption notification" is a message that will be opened later, or never. Keep the sender the desk's shared address so that replies become tickets. Use the same subject prefix for every message of the outage. That way, the five messages sort together and the all-clear is visibly the last of a series rather than a stranger in the inbox.

When E-mail Is the Outage

The e-mail about e-mail being down reaches nobody, and the only cure is a second channel agreed before the day. Choose it now. It could be the chat tool, if most people are in it. Or it could be a text message to a short list of department heads whose numbers are kept somewhere other than the e-mail system. Other choices are the receptionist, by phone, with the one-line version; or a sheet of paper on the door of each office. Write the choice into the playbook under the heading "if e-mail is down", with the list of numbers and the name of the person who holds the list. Then the day e-mail stops, the first notice goes through the second door at minute five instead of sitting in an outbox while someone works out what to do.

The same problem exists for every channel, and the matrix at the end of this article is built on it. When the network is down, the chat tool is down, the status page is unreachable from inside, and e-mail is down. What remains is the phone, paper and people. When the phone system is down, e-mail and chat still work. But the people who were going to phone the desk cannot. So the notice has to tell them how to reach the desk instead. When the website is down, the public status page may be down with it, if it lives on the same server. Every channel fails in a specific way, and for each kind of outage at least one door should be one that the outage cannot close. Paper has never once been affected by a certificate expiring.

Test the second channel once a quarter by sending something dull through it. A text message to the department heads reading "test of the outage notification list; no action needed" costs thirty seconds. It finds the number that changed when someone got a new phone. The quarterly test is on the readiness list in Preparing Before the Big One. It is the item most often skipped, because nothing is wrong on the day it is due. The person who sighs at the test message is the person who will be most relieved on the day it is not a test.

Chat: Fast, Noisy, and Only for People Who Are In It

The chat tool is the fastest written channel, and the right one for people who already work in it all day. A message in a channel that everyone is in appears on their screen within seconds, which is exactly what the first notice needs. Its weakness is that it only reaches people who are in it. The clinic's nurses were, but the receptionists were not. An organisation that has not made a deliberate decision about who uses chat usually has both groups without knowing it. Check who is actually in the channel before relying on it. "Everyone" in a chat tool often means "everyone who set it up when it was new".

Use a dedicated announcements channel and keep the replies out of it. An outage notice posted in a general channel is followed within a minute by eighty replies. Each reply pushes the notice further up the screen. Eventually, the person who arrives at 08:20 sees a wall of "same here" and has to scroll to find the three lines that matter. An announcements channel that only the desk and a few others can post in keeps the notice visible. A line at the end of it, "please reply to the desk, not here", moves the conversation to the queue. Pin the current message if the tool allows pinning, and replace the pin with each update, so the latest state is always the first thing seen.

For a small organisation the pinned message in the announcements channel is often the status page, and it does the job well: one place, one message, replaced with every update. The difference between that and a page of its own, and when a desk needs the page, is the subject of A Status Page for a Small Organisation. The only thing the pinned message cannot do is reach a person outside the chat tool, which is why it is one door and not the only one.

The Status Page: The Door People Open Themselves

Every channel so far pushes a message at people. A status page is a place people go to look. The difference matters more than it sounds. A pushed message reaches the people on the list at the moment it is sent. A page reaches the person who arrives at 09:40 and the manager who deleted the e-mail. It reaches the person working from home who heard from a colleague and the customer who was never going to be on an internal list at all. It also answers "is it still down?" without anyone having to ask. During a long outage, that is the question that otherwise generates most of the phone calls.

The page can be anything people can reach and the desk can edit in a minute. It could be a pinned chat message, a shared document everyone has the link to, a single web page, a line on the login screen, or a whiteboard in reception. What it needs is a known address that people are told about before the outage. It needs a timestamp on every entry so that staleness is visible. It also needs the discipline to update it on the same clock as the pushed messages. A status page that says "investigating" at 11:00 about something fixed at 09:30 is not a channel. It is a rumour with a web address.

Phone: The Tree, the Greeting and the Receptionist

The phone reaches the people no written channel does, and it reaches them in thirty seconds. That is why the first notice to a site with a public counter goes by phone as well as in writing. A store manager with a queue at the till does not read the e-mail until the queue is gone. A thirty-second call tells them what to do now. They read the e-mail afterwards for the detail. The cost is the desk's time, and on a two-person desk twelve calls is twelve minutes nobody has. This is what the phone tree is for. A phone tree is a short list of who calls whom. The desk calls three people, each of whom calls four. Every one of them reads the same written notice aloud. The message is identical in every mouth because it was written down once, and the desk made three calls instead of fifteen.

Change the phone greeting if the phone system allows it, because the greeting reaches exactly the people who are about to phone the desk. "If you are calling about the order system, we know it is down since 08:50 and the next update is at 09:20" turns a call into a hang-up, politely. The caller has what they needed. Most small phone systems can change the greeting in a few minutes. Find out how before the day and write the steps on the playbook page. If the phone system is the outage, the mobile numbers in the e-mail-is-down list serve the same purpose, and the notice tells people to use them. How a phone fits into a desk that otherwise works by ticket is in Phone Support in a Ticket World. During an outage the phone is briefly the most important door in the building.

The receptionist is a channel, and a good one, if the desk treats them as one. Joan at Harbourview was asked nine times before anyone told her anything. That is nine answers of "I don't know" that could have been nine answers of "yes, the referrals are delayed, IT is on it, next update at 08:15". Give the person at the front desk the one-line version of every message, by phone or by a direct message they will see. Do this before the people they deal with have a chance to ask. Update it with every message. A receptionist with the current sentence is the most reliable status page a small organisation has, and the only one that can read the room.

Paper and the Physical World

A sheet of paper reaches people who have no screen, and it works when nothing else does. A notice on the door of the office, on the printer that is not printing, on the till that is not scanning, or on the whiteboard by the loading bay carries the three lines. It reaches shift workers, visitors and anyone whose device is the thing that broke. It cannot be filtered, and it does not need a password. It is in exactly the place where the problem is being experienced, which no electronic channel can claim. Print a few copies of the blank template in advance and keep them in a drawer with a marker pen. It is a low-tech plan, and the technology has been known to let it down less often than the other way round.

Paper's weakness is updating it. A sheet that says "down since 08:50, next update 09:20" is correct for thirty minutes and then wrong. A wrong notice on a door is worse than none, because it tells the reader the desk has forgotten. Write the update time large and put a name on it. Make the walk to replace it part of the update routine, or give the walk to someone on site who will do it when the message arrives. For a one-site organisation this is two minutes. For a site forty minutes away it is a phone call to a supervisor with a marker pen, which is exactly what the phone tree is for.

Reach Against Noise

Every channel added to a message increases the chance it is seen and the chance it is resented. Three channels for one notice is normal. Six is a desk that has stopped choosing. An organisation that receives the same message six ways learns, quickly, to ignore five of them. It does not always keep the right one. The aim is the smallest set of doors that reaches everyone affected quickly. The set is different for each kind of outage, which is why the matrix at the end exists. Pick two channels that push and one that people can pull from. Use the same three for every message of that outage, so nobody has to guess where the next one will appear.

Noise also comes from the wrong audience. A notice about the sales office's order system sent to the whole organisation is noise to the two-thirds who never use it. After three of those the whole organisation treats outage notices as something that happens to other people. Send to the people affected and their managers where a list exists, and to everyone only when everyone is affected or when the scope is unknown. The lists are the preparation work: a distribution list, a chat channel or a phone-tree branch for each group that shares a set of systems. Most organisations have the groups already. They are called departments.

The table compares the channels on what matters during an outage: who they reach, how fast, when they fail, and what they are best at. No row is best at everything, which is the argument for using more than one, and no row is good for nothing, including paper.

A Short War Story: The Drivers Who Never Heard at Bellweather Logistics

Bellweather Logistics ran a depot where thirty drivers scanned parcels onto handheld devices before loading. On a Wednesday at 13:10 the scanning service stopped. Omar on the desk sent a good first notice at 13:15 by e-mail and in the chat tool. The notice gave a paper manifest as the workaround and the next update at 13:35. Both channels reached every office worker in the building and not one driver, because the drivers had neither. For forty minutes the drivers kept trying to scan, assumed the fault was in their own handhelds, and queued at the dispatch window to swap them. They phoned the dispatch supervisor, whose line was engaged for the whole period because the other drivers were phoning it. Two afternoon loads missed their departure slots and the depot paid the contractual penalty on both. The review found that the workaround had existed from 13:15 in a channel none of the affected people could see. Bellweather added two lines to its matrix: for anything touching the handhelds, Omar phones the dispatch supervisor first, and the supervisor writes the three lines on the whiteboard at the loading bay. A message that does not reach the people affected has not been sent, however good it was.

The Channel Matrix by Outage Type

The matrix below is the decision made in advance. There is one block per kind of outage the organisation could plausibly have. Each block lists the two channels that push the message, the one channel people can pull from, and the physical fallback. It also lists the one thing to do first. The example blocks are a starting point and will be wrong for your organisation in details that matter. The blank block is for the outages that are yours. Fill it in on a quiet afternoon with the people who run the front desk and the warehouse in the room. They know where their people look and the desk does not.

OUTAGE CHANNEL MATRIX — ONE PAGE
Organisation: ______________________   Owner: ______________   Reviewed: ________

Alternate contact list, kept OUTSIDE the e-mail system: ______________________
  Held by: ______________   Last test message: ________   Next test: ________
Phone greeting, how to change it: _____________________________________________

For each outage type: two channels that push, one people pull from,
the physical fallback, and the first thing to do.

E-MAIL DOWN
  Push 1:    Chat announcements channel
  Push 2:    Text message to the department-heads list
  Pull:      Pinned chat message or the status page
  Physical:  Sheet on each office door
  First:     Phone the receptionist with the one-line version

NETWORK OR INTERNET DOWN
  Push 1:    Phone tree
  Push 2:    Text message to the department-heads list
  Pull:      Nothing reachable from inside
  Physical:  Sheets on the doors; the whiteboard
  First:     Phone the site supervisors; change the phone greeting

PHONE SYSTEM DOWN
  Push 1:    E-mail to all staff
  Push 2:    Chat announcements channel
  Pull:      Status page
  Physical:  Sheet at reception
  First:     Put the mobile numbers for reaching the desk in the notice

ONE BUSINESS APPLICATION DOWN (orders, HR, finance, booking)
  Push 1:    E-mail to the affected group's list
  Push 2:    Chat channel for that team
  Pull:      Status page
  Physical:  Sheet on the device, if there is one
  First:     Phone the team lead of the group affected

PRINTING OR SCANNING DOWN
  Push 1:    E-mail to the affected group's list
  Push 2:    Phone to the floor supervisor
  Pull:      Status page
  Physical:  Sheet on the printer or scanner
  First:     Phone the supervisor; write the whiteboard

REMOTE ACCESS DOWN
  Push 1:    Text message to the department-heads list
  Push 2:    Phone to the people known to be remote today
  Pull:      Status page at a public address
  Physical:  None
  First:     E-mail will not reach them; text and phone only

PUBLIC WEBSITE OR CUSTOMER SERVICE DOWN
  Push 1:    Public status page or a line on the website
  Push 2:    E-mail to the people who talk to customers
  Pull:      Public status page
  Physical:  None
  First:     Brief whoever takes external calls with the one-liner

SITE POWER LOSS
  Push 1:    Phone tree
  Push 2:    Text message to the department-heads list
  Pull:      None
  Physical:  Sheet on the main door
  First:     Phone tree first; then the other sites by e-mail

______________________________ (your outage type)
  Push 1:    ______________________   Push 2:    ______________________
  Pull:      ______________________   Physical:  ______________________
  First:     ______________________________________________________

RULES THAT APPLY TO EVERY ROW
  [ ] The same text goes through every door; only the length changes
  [ ] The same next-update time appears on every door
  [ ] Replies are directed to the desk address, never to the channel
  [ ] The receptionist or front desk gets the one-line version first
  [ ] Paper notices carry the next-update time and a name, and are
      replaced on the clock

Priya rewrote Harbourview's matrix the week after the Monday it was needed. The block for e-mail now reads: chat channel, then a phone call to Joan on reception with the one-line version, then a sheet on the door of each clinic room, in that order. The first two are done before the desk touches the mail server. The clinic has had one e-mail outage since. Joan was told at minute four. The nurses saw the pinned message at minute five. The referrals question was asked once, by a patient, and answered by someone who knew.

Frequently Asked Questions