The All-Clear and the Follow-Up: Declare It Fixed Only Once

11:52 at Acme, after a two-hour e-mail outage. Sam restarts the last service on the mail server, watches the queue of stuck messages drain to zero, and feels the particular relief of a Tuesday morning coming back. At 11:53 he sends "All fixed, e-mail is working again" to the whole company. At 12:10 the sales floor still cannot send, because the mail program on every desk needs closing and reopening before it notices. Nobody has told them that. At 12:15 the phones start again, and this second wave is angrier than the first. The first wave was a problem and this one is a problem the desk said was over. The server was fixed at 11:52. For the sales floor, the outage lasted until a colleague worked out the restart at 12:40 and shouted it across the room.

The all-clear is the message that says the outage is over. It is sent only when the service works from the seat of the people who use it, not from the server. It tells them what to do now, what may have been lost, why it happened in a sentence or two, and it apologises once. The after-note is the shorter message that follows a day later, to management, saying what happened, what it cost, and what will change. This article is part of our Outage Communication series. And it is about the two messages that end an outage well, which are the two most often sent too early or not at all.

It covers what "fixed" has to mean before the word is used, then the partial fix and the watching state. Next comes what to do when the all-clear turns out to be wrong. We cover what people should do now, with a table by outage type, then the short explanation and the apology that works. Next come who gets the all-clear and where, then the after-note for management a day later. We finish with a short war story and both messages to copy, in that order.

Fixed Means Fixed Where They Sit

A service is fixed when a person who relies on it can do the thing they could not do an hour ago. They must do it from their own seat, with their own login, and the all-clear waits for that. The server being healthy is necessary and not sufficient. Between the server and the counter there are caches, sessions, programs that need restarting, and phones that need signing in again. There is also a browser that remembers the error page. The fixer confirms the first half: the service is up and looks right from the desk. An affected person confirms the second: "I have just sent an e-mail and it arrived". The lead's routine of waiting for both confirmations is described in Communicating During the Incident. This article is about what happens to the message when the second confirmation is skipped, and about what the message says once it is not.

Pick the confirming person from the group that complained first, and ask them to do the real thing, not to look at a screen. "Can you open a customer record and save a change?" is a test. "Does it look all right?" is a conversation. If the outage touched several groups or several sites, get one confirmation from each that differs in some way that matters. That could mean a branch and head office, a desk user and a phone user, or someone in the building and someone at home. Two minutes of asking saves a second wave of calls. The second wave always costs more than the first, because it arrives with a sentence attached: "but you said it was fixed."

Where the fix requires the user to do something, the service is not fixed until they have been told what. Sam's mail server was fine at 11:52. The all-clear that should have gone at 11:55, after one person on the sales floor confirmed, would have said "close and reopen your mail program, then check your outbox". That line was the fix for everyone who was not Sam. A desk that sends "all fixed" without it has announced the end of its own outage and the start of everyone else's.

The Partial Fix and the Watching State

When the service is back but not as it was, say so in plain words and do not call it an all-clear. "The order system is working again on the standby server; it will be slower than usual and stock figures may be a few hours behind until this evening" is a true message that people can work with. "All clear, order system restored" followed by an afternoon of complaints about slowness is a message that cost the desk the next outage's credibility. The people on the floor discovered the difference themselves. A partial fix gets a partial all-clear, with the limits listed, and a time at which the desk will say more.

Use the watching state when the fix is in but the desk is not yet sure it will hold. A watching state is a status, on the page and in the message, that says "working since 10:28; we are keeping an eye on it; tell the desk if it misbehaves". It is honest about uncertainty and tells people what to do if the symptom returns. It gives the desk a defined period after which it either declares the full all-clear or admits the fix did not take. Most outages that recur do so within the first hour. A watching period of an hour or two, said out loud, turns that recurrence from a surprise into something the message already allowed for. The states a status page uses, including this one, are described in the status page article of this series.

The second all-clear, when the standby is swapped back or the watching period ends, is a short message that closes the loop. For example: "The order system is now back on its main server and running normally since 18:40; stock figures are current." It is two lines and it matters. Without it the people who read the partial all-clear are still, in their minds, on the standby server, and will be for weeks. A partial fix announced and never completed in writing is an outage that, as far as its readers know, is still going on quietly.

When the All-Clear Was Wrong

If the service fails again after the all-clear, send a new first notice within five minutes and say what happened in the first line. "The order system has failed again since 11:15, after being restored at 10:28. We are back on it. Keep using the paper pad. Next update at 11:35." The message does not pretend to be update seven of the earlier outage, because for the reader it is a new outage with a worse beginning. It does not hide the earlier all-clear, because everyone remembers it. The honesty costs one clause and buys back most of what the premature all-clear spent.

Do not argue with the people for whom it never worked. Some will reply to the all-clear saying it is still broken for them. And a proportion of those will be local problems, a cached page, a program not restarted, a different fault with the same symptom. Treat every one as a report and answer it as a ticket. And watch the count: three "still broken" replies from the same site is a site the fix has not reached. And that is an update, not three tickets. The desk that sent the all-clear is the desk best placed to notice that it was wrong, provided it is reading the replies rather than defending the message.

What People Should Do Now

The all-clear tells every reader what to do next, in a numbered list if there is more than one thing. The list is the most useful part of the message. Restart the program; sign in again on the phone; refresh the page; reconnect to the drive. Then the harder part: what may have been lost, and how to check. "Anything you saved in the order system between 09:02 and 10:28 may not have saved; open your last three orders and confirm they are there" is a sentence that prevents a week of quiet discoveries. If the desk does not know whether work was lost, it says so, and says how to find out. A message that says "all fixed" and leaves the lost-work question unasked has left it for each reader to discover alone, usually at month end.

Say what happens to the work done on the workaround. The paper orders, the handwritten slips, the applications filled in on the form: somebody has to key them in. The all-clear says who and by when. "Send this morning's paper orders to the desk by 15:00 and we will enter them" is a plan. "Please enter any orders taken on paper" is a plan that will be executed differently by every branch and not at all by two of them. Where the users are expected to do it themselves, give them the steps and a time, and say what to do with a slip that does not match anything.

The table below lists the usual "do this now" actions for common outages. Each row is a starting list to be adjusted for your own systems. The point of having it in advance is that the all-clear can be written in two minutes. It need not be composed from memory while the sales floor waits. The lost-work column is the one most often left blank, and the one that should never be.

The Short Explanation

The all-clear explains what happened in two or three sentences of plain words, and no more. The review will know more tomorrow than the desk knows at 10:33. Anything written now in detail will be corrected later in public. The shape is the one in The Three-Sentence Explanation. Say what broke, in the reader's words. Say why, in one clause, only if the desk is sure. Say what stops it happening again, or "we are reviewing why and will say more". "The order system lost its connection to its database at 09:02. The connection was restored at 10:28. We are reviewing why it dropped and will send a short note tomorrow." That is enough for a person at a counter, and it is true.

Leave out the things that make explanations go wrong. Leave out blame, including the supplier's name and "a change made last night". The message will be read by the person blamed and by people who will quote it before the review has established anything. Jargon, because "the connection pool exhausted its limit" is a sentence that tells the reader only that the desk speaks a language they do not. Minimising, because "a minor configuration issue" describes two hours of a sales floor on paper as minor, and the sales floor will notice. And the word "unprecedented", which is never true and reads as the desk being surprised that a computer stopped.

If the cause is not known, say so in those words. "We do not yet know why the connection dropped; we are looking, and the review will say" is a sentence people accept, because they were not expecting the desk to be certain at 10:33. What they will not accept is a confident cause at 10:33 that becomes a different cause on Thursday. The full account, when it exists, goes into the review described in Closing the Incident and the Review. The after-note the next day carries as much of it as is settled by then. The all-clear carries the sentence the desk is sure of and no more.

The Apology That Works

Apologise once, specifically, and in your own words. "Sorry for the disruption this morning; we know it cost the sales office two hours on paper" is an apology that names what happened to the reader. That is the only thing that makes it land. "We apologise for any inconvenience this may have caused" is a sentence that has been printed on so many signs that it now means "a sign was required here". Readers skip it without noticing they have. The word "may" in it is doing something unkind. It suggests the inconvenience is hypothetical, to people who have spent the morning living in it.

Own it without grovelling and without explaining it away. One sentence of apology, one of thanks for the workaround they ran, and then the message moves on to what to do. A paragraph of apology reads as the desk asking for sympathy during the reader's bad morning. And a sentence that begins "unfortunately" and ends with the supplier's name is not an apology but a forwarding address. Do not promise it will never happen again, because it may, and the promise will be quoted. Promise what the desk can do: "we are reviewing why, and will tell you what we change".

Thank the people who did the workaround by name of group, because they did the organisation's work during the outage and the desk did not. "Thank you to the branch teams for keeping customers moving on paper" costs nothing and is remembered longer than the explanation. It also, quietly, makes the next workaround easier to ask for. I have never had a workaround refused by a group that was thanked for the last one. I have had several refused by groups that were not.

Who Gets the All-Clear, and Where

The all-clear goes to every audience the first notice went to, through every channel the first notice used. Then it goes to the status page, the phone greeting and the receptionist, in that order. It is the one message people most want to receive and the one most often sent to fewer channels than the first. The person sending it is tired and the relief is making them generous with assumptions. Every door the first notice opened has a person standing at it waiting for this message. The channels and the matrix of which to use are in Choosing the Channels for Outage Messages. For the all-clear the rule is the same doors, all of them, plus the clear-down of the page.

Each person who raised a ticket about the outage gets the all-clear as a reply on their own ticket, with the "do this now" list. The ticket is resolved with that reply. This is the step that finds the people for whom it is not fixed, because a reply to a resolved ticket is a signal. If you run QuerySolv, a requester's reply to that all-clear reopens the resolved ticket. In that case, the people for whom the service is still down come back to the desk as reopens it can count. They do not come back as a second outage discovered by phone an hour later. What a cluster of reopens after an all-clear is telling you, and how to read it honestly, is in What a Reopen Is Telling You.

The public version of the all-clear, for customers and people outside, is shorter and later. It goes after the internal one has been out for a few minutes without a wave of "still broken" replies. It says only that the service is available again since a time and what to do if something they did during the outage did not go through. It adds one line of apology. No cause beyond a phrase, no internal names, and no promise. The receptionist gets the one-line version first, because the next caller is already dialling.

The After-Note for Management, a Day Later

The after-note goes to management the morning after the outage, and it is six to eight lines. It waits a day because the facts settle overnight. The cost is countable, the cause is known or honestly not, the actions have owners, and the desk has slept. It goes before the review, not instead of it. Management should hear from the desk what the outage was before they hear it from a department head in a corridor. The review will take a week. The after-note is the desk saying, in its own words and first, what happened and what it is doing about it.

It contains what happened, in one sentence a person who was not there can read. Next comes who was affected and for how long, in business terms. Then comes what it cost, as honestly as it can be counted the next morning. Next comes the cause as far as it is known, with "not yet known" where that is true. Then comes what will change, with a name and a date against each item. Last come when the review will be held and who will see it. It does not contain the timeline, the technical detail or the fixer's account, all of which belong in the review and the incident ticket. It is the summary that the one-page review will later confirm or correct, and it says so.

Send it to the managers who received the updates. Consider a two-line version to everyone affected: "Yesterday's outage was caused by a database connection failing; we have changed the setting that allowed it and will say more after the review on Thursday." Most desks never send the all-staff version and most organisations would be better for it. It tells the people who ran the workaround that the morning they lost was noticed and is being acted on. The after-note is also where the apology to management lives, in one line, with the cost beside it. A manager reading "two hours, sixty orders on paper, no data lost" has been given a fact to repeat. A fact is what they were going to ask for.

Tone matters more here than anywhere else in the outage, because the after-note is the only message written when nothing is on fire. It will be read as the desk's considered view. Plain, specific, owning the parts the desk owns and naming no one. The desk that writes "the outage was caused by a change the supplier made without telling us" has started a dispute by e-mail. The desk that writes "a change on the supplier's side disconnected the database; we are agreeing a notification process with them, owner Sam, by the end of the month" has reported the same fact and the fix. One of those sentences will be forwarded to the supplier. Both, in fact, will be. Only one of them should be.

A Short War Story: The Early All-Clear at Oakmoor College

Oakmoor College lost its timetable system at 09:10 on the first day of term. Dana on the desk ran a good outage for eighty minutes, with paper timetables at reception and updates every twenty minutes. At 10:30 the system came back on the server. Dana confirmed it from her own screen, and she sent the all-clear at 10:31: "Timetables are working again; thank you for your patience." The forty corridor screens that show room allocations were still displaying the page they had cached at 09:10. Nobody had told anyone they needed restarting, and the all-clear had told every lecturer that the system was fine. So at 11:00 roughly four hundred students followed the corridor screens to the rooms the system had shown before the outage. Those were the wrong rooms. The review counted an hour of teaching lost across twelve rooms. It counted a reception desk that handled more questions between 11:00 and 11:30 than during the outage itself. It also counted a line in the student newsletter about the day the timetables lied, which Dana still has pinned above her desk. The all-clear now waits for a lecturer to confirm from a corridor screen, and the first line of it is the restart. Fixed on the server is a fact about the server. The all-clear is a message about everybody else.

The All-Clear and the After-Note

The two templates below are the all-clear, with its "do this now" list and the lost-work line, and the after-note for management the next morning. Fill the blanks, delete nothing else. The all-clear takes two minutes if the "do this now" rows were prepared in advance. The after-note takes fifteen the following morning, before the first meeting, and should be sent before anyone asks for it. The plain words outside the blanks are the ones that have been found to be read by people at counters and by directors with four minutes. Those are the two readers these messages have.

THE ALL-CLEAR
Send only after: [ ] the fixer confirms the service is up
                 [ ] one affected person has done the real thing from their seat
Subject: All clear: ____________________ working again since __:__

____________________ (the service) is working again since __:__.

Do this now:
  1. ______________________________________________ (restart, sign in, refresh)
  2. ______________________________________________
  3. ______________________________________________

Work from the outage:
  Anything you did in ____________________ between __:__ and __:__ may not
  have saved. Check: ________________________________________________
  Paper or workaround records: send to ____________ by __:__, or
  enter them yourself by following ____________________.

What happened, briefly: ______________________________________________
  ____________________________________________ (two or three sentences;
  cause only if certain; otherwise say the cause is under review)

Sorry for the disruption; we know it cost ____________________ (what,
to whom). Thank you to ____________________ for ____________________
(the workaround they ran).

Partial fix, if it applies: ____________________ is working but
  ____________________ (the limit) until ____________. We will confirm
  when it is fully back.
Watching: we are keeping an eye on it until __:__. If ____________ (the
  symptom) comes back, tell the desk straight away.
-- ____________, help desk

THE AFTER-NOTE TO MANAGEMENT (the next morning; six to eight lines)
Subject: Yesterday's ____________________ outage: what happened and what changes

What happened: ________________________________________________ (one sentence)
Who was affected, and for how long: ____________________________________
What it cost: _________________________________________________________
  (hours, orders, customers, rework; "no data lost" if true)
Cause, as far as it is known: ___________________________________________
  ("not yet known; the review will say" where that is true)
What changes:
  - ____________________________  owner ________  by ________
  - ____________________________  owner ________  by ________
Review: ________ (date), attended by ____________; summary to follow.
Sorry for the disruption; ____________________ (one line).
-- ____________, help desk

Sam's all-clear at Acme, the next time e-mail went down, went out at 14:21, seven minutes after the server was healthy. It was two minutes after someone on the sales floor had sent a message to the desk and watched it arrive. The all-clear's first line was "close and reopen your mail program, then check your outbox". The desk took one call after it, from a person who had not read past the subject line. The after-note went to the management team at 08:40 the next morning. It gave the cause, a two-hour cost, and a change with Sam's name and a date against it. The sales manager replied with one word, which was "thanks". Sam says it is the shortest reply he has ever been pleased to receive.

Frequently Asked Questions