Closing the Incident and the Review That Follows

Thursday, 14:50, at Bluewater Bank. The link between head office and the fourteen branches came back at 14:28, and a branch manager confirmed it at 14:30. Tomas, who has been lead since 13:20, sent the all-clear at 14:31. He has now had nineteen minutes of quiet, which he has spent looking at what is left. There is an incident ticket with thirty-one log lines, and twenty-three linked tickets from branch staff, all still open. There is also a temporary fix on a backup router that will have to be undone, and two messages. One is from his manager and says "can you write this up for the exec meeting on Monday". The other is from the branch manager at the Harbor Street branch and says "will this happen again?". Tomas does not know, and the only honest way to find out is the part of the incident nobody budgets time for.

Closing an incident is the sequence that runs from the all-clear to the moment the record is complete and every follow-up has an owner. It includes the linked tickets closed with a reply, the incident ticket closed with a summary, and the review held and written up. The review is a short, blameless meeting, within a few days. It turns the timeline into a handful of findings and a shorter list of actions with names and dates on them. This article is part of our Major Incident Management for Small Teams series. It is about the ninety minutes after the all-clear and the forty-five minutes on Monday.

It covers why the all-clear is not the close. Next come closing the linked tickets so every reporter hears; the summary note that closes the incident ticket. Then come what a blameless review is and is not; building the timeline and the four gaps to measure in it. Next come findings and action items, with a table of where each kind of action goes. Then come writing the review for people who were not there; tracking the actions so they happen. Finally come a short war story and the one-page review template, in that order.

The All-Clear Is Not the Close

The all-clear ends the outage. It does not end the incident. The gap between the two is where small desks lose the value of everything they did well during it. After the all-clear, branch staff are still holding paper forms that have to be keyed in. A backup router is carrying traffic it was not sized for. Twenty-three people are waiting to hear that their ticket, specifically, is done. The only account of the afternoon exists in one person's log and three people's memories. Each of those will decay by Monday. The paper forms will be keyed in late, the router will be forgotten until it fails, the tickets will be bulk-closed at the end of the month, and the memories will disagree.

The close is a fixed sequence, done while the incident is still fresh, in this order. Reply to and close the linked tickets, then write the summary note and close the incident ticket. Book the review, and record any temporary fix and emergency change as a ticket with an owner. On a two-person desk it takes about an hour, and the hour is best spent immediately, before the queue that was paused comes back. The queue will wait another hour. It has been waiting since 13:20.

Closing the Linked Tickets

Every ticket linked to the incident gets a reply. The reply is the all-clear message, pasted or sent as a saved reply. One sentence at the top addresses the ticket itself: "This was part of the branch link outage this afternoon, which is now resolved." Then the ticket is resolved or closed according to the desk's normal workflow. The reply is the whole point. A status change alone, in many tools, tells the requester nothing. The person at the Harbor Street branch who raised ticket 7311 at 13:05 should hear in the thread they opened, not infer it from the fact that the till works. Whether a reply on a closed ticket reopens it automatically depends on the tool, and it does not matter here. A requester who replies "thanks" has confirmed, and a requester who replies "still not working for me" has told you something the all-clear missed.

Read each ticket before closing it, quickly, because some of them are not about the incident. Of Bluewater's twenty-three, two were raised during the outage about other things, a password reset and a printer. They had been swept into the link because they arrived in the same half hour. Those are unlinked and worked as ordinary tickets, with an apology for the delay. A ticket closed with the wrong reply is a silent close wearing a costume, and the requester will know. The shape of a proper close, with a note somebody can read in a year, is in What a Proper Close Looks Like. The incident's tickets deserve the same shape with less typing, because the note is shared.

Do not bulk-close in silence, however many there are. If the tool can send one reply to many tickets, use it, and the twenty-three take five minutes. If it cannot, the saved reply pasted twenty-three times takes fifteen. Either way the reply goes before the status. Fifteen minutes is cheap for twenty-three people who ran paper forms for an hour and a half on the desk's instruction. It is the moment the desk says thank you in the one place they will each read it.

Closing the Incident Ticket

The incident ticket closes with a summary note on top of the log, and the summary has eight lines. Start with when it started, when it was declared, when service was restored, and when the all-clear went out. Then name who and what was affected in plain words, and what the cause was in one line or "not yet known". Finish with what fixed it and whether the fix is temporary or permanent, and what follow-up tickets exist. Tomas's read: "Started about 12:58 (first branch call 13:02). Declared 13:20. Restored 14:28 on the backup router. All-clear 14:31. All fourteen branches lost the head-office link; tills ran offline, counter staff used paper. Cause: primary router rebooted into a failed state after a firmware update applied at 12:55; exact reason under investigation. Fix: traffic moved to the backup router; temporary. Follow-ups: problem ticket 7340, change ticket 7341, review booked Monday 10:00."

The note is written by the lead, from the log, before the log is tidied. The log stays under it exactly as it was written. The temptation is to make the record neat; the value of the record is that it is true. The truth at 13:40 had a typo and a wrong guess in it that the review needs to see. The incident ticket is then closed, and it stays the record of the incident forever. It is the thing the next lead searches for when the same symptom appears. That is why the subject prefix matters and why the summary names the symptom in the words a technician would search for.

Two kinds of follow-up ticket are opened at the same time, when they apply. If the cause is not yet known or not yet fixed for good, a problem ticket is opened and linked. That is because an incident ticket records an outage and a problem ticket records the pursuit of its cause. The distinction, and why keeping them apart stops the cause being forgotten the moment service is back, is in Problems vs Incidents. And every CHANGED line in the log that is still in effect is an emergency change that was made without the usual look and now needs one. Examples are the backup router carrying traffic, a certificate swapped in a hurry, and a service account's password reset. Emergency Changes describes the after-the-fact record, and the close is when it is written. A temporary fix without a ticket is a permanent fix with a surprise in it.

The Review: What It Is and Is Not

A blameless review is a meeting that asks what the system, the process and the tools let happen, and never asks who is at fault. It is blameless as a matter of engineering rather than kindness. The moment a review assigns fault to a person, the next incident's log gets shorter. That is because the fixer who wrote "14:05 TRIED restart, made it worse" has learned what that line costs. A review that wants true logs has to make true logs safe to write. The firmware update that rebooted Bluewater's router was applied by Dana at 12:55. The review's question is not why Dana did that. It asks why a router firmware update could be applied at 12:55 on a Thursday with no second person and no maintenance window. It also asks why there was no way to know it had failed except a branch phoning.

The review is short, soon, and small. It takes forty-five minutes, inside three working days while memories are intact, with the lead, the fixer, the manager, and one person from the affected side. At Bluewater, that person was the Harbor Street branch manager with the question. More people make it a presentation; fewer make it a conversation between the two people who already agree. The affected person is the one most desks leave out. They are also the one who most often supplies the finding the desk could not see. For example, the branches had no idea the backup router existed and had been told by someone to switch the tills off.

It is not a report for the executive meeting, although a page of it will be sent there. It is not a search for a root cause, which is the problem ticket's job and may take weeks. It is not a performance conversation about anyone, and if a manager tries to make it one, the lead's job is to say so and move on. And it is not optional because the incident went well. The incidents that go well are where the desk learns what to keep. A review of a good incident is forty-five minutes of writing down what worked before it stops being obvious.

Building the Timeline and Measuring the Gaps

The review starts with the timeline, read aloud from the log, uncorrected, by the lead. Where the log is silent, the room fills in what it remembers, and those additions are marked as memories, not records. The timeline is then reduced to five timestamps and four gaps, and the gaps are where every finding lives. The timestamps are: when the problem started, when the desk first knew, when the incident was declared, when service was restored, and when the all-clear went out. The gaps between them are: time to notice, time to declare, time to restore, and time to all-clear.

Bluewater's gaps: started 12:55, noticed 13:02, seven minutes, because a branch phoned; monitoring did not. Declared 13:20, eighteen minutes after noticing, because the first ten were spent assuming it was one branch's line and the next eight working out who was lead. Restored 14:28, sixty-eight minutes after declaring, of which about twenty were spent trying to recover the primary router before the decision to move traffic to the backup was made. All-clear 14:31, three minutes after restoring, which is the only gap in the four that nobody needed to discuss.

Each gap has its own question. Time to notice asks whether the desk should have known before a person told it, and the answer is usually a monitoring check or an alert. Time to declare asks whether the criteria card gave a fast answer and whether the roles were settled quickly. The answer is usually an edit to the card or the playbook. Time to restore asks whether the restore-before-repair decision was made early enough, and whether the supplier was called at the right moment. It also asks whether anything needed on the day was missing, such as a password, a diagram, a contact number. Time to all-clear asks only whether the two confirmations were quick. Four gaps, four questions, and the review has its structure without anyone having to design one.

Findings and Action Items

A finding is one sentence about what the gaps showed. It comes in three kinds: what went well and should be kept, what was slow and should change, and what was lucky and should not be relied on again. Bluewater's findings were these. The paper-form workaround worked at all fourteen branches within ten minutes because the criteria card named it (keep). It took eighteen minutes to declare because the card did not say that one branch's loss is a ticket and all branches' loss is major (change). And the backup router happened to have been powered on and patched by chance in the spring (lucky). Three to six findings is the right number. Twelve findings is a list nobody will act on, with the real one hidden in the middle.

An action item is a change that would move one of the four gaps, with an owner, a date, and a ticket. The owner is a person. The date is within a month for most and within a quarter for anything that needs money. The ticket is in the same queue as everything else so that it ages visibly like any other work. Five actions is the ceiling, and "no action" is a legitimate verdict on a finding that was lucky in a way that will not recur. The test for each action is the sentence "if this had been in place on Thursday, the gap would have been shorter by about ___". An action that cannot complete the sentence is not an action; it is a wish.

Certain actions are worth considering at every review, because they are where small desks reliably find time. Edit the criteria card and the playbook with whatever the day taught; this is always worth a ticket and usually takes an hour. Add the monitoring check that would have shortened the time to notice. Confirm the problem ticket has an owner, so that the cause gets chased. Schedule the undoing of the temporary fix, with a date, before it becomes a permanent one by neglect. And write down the thing that was lucky, so that next time it is planned. The table pairs each kind of finding with where its action goes, because an action filed in the wrong place is an action that will not be found again.

Writing It Up for People Who Were Not There

The review produces one page, and the top five lines of the page are written for someone who was not there and has four minutes. What happened, in plain words. Who was affected and for how long. What the cause was, as far as it is known. What is being done so that it is shorter or rarer next time, as a list of actions with owners. Include one line on what went well, because the executive meeting will otherwise hear only the first four and conclude the desk had a bad day. The desk had a good incident on a bad day. The rest of the page is the timeline, the gaps, the findings, and the action list, for anyone who reads past the fold.

Leave out the things that feel like content and are not. Names attached to mistakes: the write-up says "a firmware update was applied without a maintenance window", not who applied it. The full log: it is in the ticket for anyone who wants it, and nobody at the executive meeting does. Promises that it will never happen again: the write-up can say what will make it shorter or rarer, and nothing else, because the next incident will be read against this page. The branch manager at Harbor Street gets the same page as the executive meeting, with a two-line note on top. The answer to "will this happen again?" is the action list, which is the only honest answer there is.

Send it by the date promised in the all-clear, which Communicating During the Incident recommends naming in the message itself. A review that is promised for Thursday and sent for Thursday is a desk that keeps its times. The people who ran paper forms notice. A review that is promised and never sent teaches everyone that the desk's interest in the incident ended with the all-clear, which is, after all, when theirs would have.

Tracking the Actions

Each action is a ticket, and the tickets are reviewed twice. One is at the next incident's review, where the first question is "what did we say we would do last time, and did we?" The other is at a monthly glance by the manager at the open ones. An action that is still open at the next review is itself a finding, and a more important one than anything new. That is because it says the desk's reviews produce lists rather than changes. The lead reads the old list aloud before the new timeline. It takes two minutes, and it is the two minutes that make the rest of the process real.

Close the loop with the people who were affected when an action lands. "The monitoring check from the branch outage review is now live; the desk will know about a link failure inside a minute" is a one-line message to the branch managers. It costs nothing and is the only evidence they will ever see that the review happened. Most desks never send it. The ones that do are the ones whose next first notice is believed.

A Short War Story: The Review Nobody Held at Acme

Acme's two-person desk had its e-mail system fail for most of a Wednesday morning when a disk filled on the mail server. It ran a competent incident with a declaration, a first notice, five updates and an all-clear. Then, because both technicians had been on it since 08:00 and the queue was eighty tickets deep, they agreed to "do the review next week". Next week had its own queue, and the review was never held. The log sat in the incident ticket, and the disk that had filled was cleared but not enlarged. The alert that would have caught it at eighty percent was discussed in the corridor and not ticketed. Six weeks later, on a Monday, the same disk filled again. E-mail was down for three hours and forty minutes, and the sales team missed a tender submission deadline that had been due by e-mail at noon. The sales director costed that at a contract the company had spent four months pursuing. The review that was finally held found that the second incident was the first one with a worse day attached. It also found that the two actions, a bigger disk and an alert, would have taken a morning between them. Acme now books the review before sending the all-clear, as a calendar entry the lead cannot leave the incident without creating. The two actions from the first review were the first two tickets on the list.

The One-Page Review

The template below is the review on one page. The summary for people who were not there is at the top, followed by the timeline reduced to five timestamps and four gaps. Next come the findings in three kinds, the actions with owners and dates, and the check on last time's actions. Fill it in during the meeting, from the log, with the lead typing. Do not write it afterwards from notes, because the forty-five minutes are the only time the four people will be in one room with the same facts. Send the top section by the date the all-clear promised.

INCIDENT REVIEW — ONE PAGE
Incident ticket: ________   Date of incident: ____________   Review held: ____________
Attending: lead ________  fixer ________  manager ________  affected side ________

SUMMARY (for people who were not there; five lines, plain words)
  What happened: ______________________________________________________
  Who was affected and for how long: __________________________________
  Cause, as far as it is known: _______________________________________
  What we are doing about it: see actions below
  What went well: _____________________________________________________

TIMELINE (from the log, uncorrected)
  Started      __:__   how we know: ______________________________
  Noticed      __:__   by: [ ] alert  [ ] a person phoned  [ ] desk saw it
  Declared     __:__   lead: ________  fixer: ________
  Restored     __:__   by: ____________________  [ ] permanent  [ ] temporary
  All-clear    __:__   confirmed by fixer [ ] and by ____________ [ ]

THE FOUR GAPS (minutes, and the question each one asks)
  Time to notice    ____  Should we have known before a person told us?
  Time to declare   ____  Did the criteria card and the roles settle quickly?
  Time to restore   ____  Was restore-before-repair decided early enough?
                          Was the supplier called at the right moment?
                          Was anything missing on the day (password, diagram, contact)?
  Time to all-clear ____  Were the two confirmations quick?

FINDINGS (one sentence each; three to six in total)
  Keep:    ___________________________________________________________
  Change:  ___________________________________________________________
  Change:  ___________________________________________________________
  Lucky:   ___________________________________________________________

ACTIONS (five at most; each one shortens a gap; each one is a ticket)
  1. ______________________________  owner ________  by ________  ticket ____
  2. ______________________________  owner ________  by ________  ticket ____
  3. ______________________________  owner ________  by ________  ticket ____
  Criteria card or playbook edited: [ ] yes, ticket ____   [ ] nothing to change
  Problem ticket for the cause: ____   owner ________
  Temporary fix to be undone: ______________  change record ____  by ________

LAST TIME'S ACTIONS (read aloud before the timeline)
  [ ] done ______________________   [ ] still open ______________________
  [ ] done ______________________   [ ] still open ______________________

SENT TO (by the date promised in the all-clear: ________)
  [ ] manager   [ ] affected departments   [ ] desk   [ ] filed in the incident ticket

The last section is the one that earns its place. A review that is not sent is a meeting, and a meeting is a thing that happened to four people. A page in the incident ticket and in three inboxes is a thing the organisation knows. Tomas's page for the Bluewater review went to the executive meeting on Monday, to the fourteen branch managers with two lines on top, and into ticket 7338. The next lead will find it by searching for "branch link". The page had three actions on it. The monitoring check was live by Wednesday, and the Harbor Street branch manager got a one-line message to say so.

Frequently Asked Questions