Emergency Changes: Moving Fast and Still Writing It Down

Tuesday, 14:10, at Bluewater Bank. The mail gateway has started bouncing every message from outside the bank with a "policy violation" notice. And the branches have noticed, because a bank branch notices its mail stopping faster than any monitoring system does. Imani finds the cause in six minutes. A rule added by the supplier's 14:00 signature update is classifying an entire class of attachments as hostile. She has the fix, which is to disable that one rule. In the next four minutes she does four things. She types to Callum in the team chat: "Disabling rule ATT-EXEC-2 on the mail gateway; it has been bouncing all external mail since 14:00; shout if you object." She disables the rule. She sends herself a test message from a personal address and watches it arrive. She types two lines into the incident ticket with the time. Mail flows from 14:14. The full record is written at 16:30, when it has been flowing for two hours.

The outage lasted twenty-one minutes and the writing took four. Both numbers are the point. An emergency change is a change made before the weekly review can read it. That is because waiting would cause harm now or very soon. It is reviewed after the fact instead. It is not an unrecorded change. It is a change whose record is written afterwards, within one working day, with three extra lines. This article is part of our Lightweight Change Management series, and it is about moving fast without losing the record.

It covers what counts as an emergency and what is just impatience, with a table of both. Next come the four emergency rules, telling one person, and the smallest change that stops the harm. Then we cover the after-the-fact record and why it is written within a day, and the emergency inside a major incident. We move on to reviewing emergencies afterwards and counting them. We finish with a short war story and the emergency change record to copy, in that order.

What Counts as an Emergency

The test is one question: what gets worse between now and the next review? If the answer is a named harm that is happening or about to happen, the change is an emergency. If the answer is nothing, or "someone will be annoyed", it is a normal change, however strongly the person asking feels about it. The harm comes in three shapes on a small desk. A service is down and the fix is a change. An attack or a leak is under way and a change stops it. Data is being damaged and a change halts the damage. A fourth shape is the rollback of a change that just broke something. That is an emergency in its timing although its record already exists.

"About to" counts, provided it is specific. The certificate on the public site expires at midnight and the renewal has not been done. The disk holding the finance database will be full by morning at the current rate. The firewall log shows a known hole being probed from outside and the patch for it is available. Each of those has a time and a consequence. And a change that prevents a certain harm at a known hour is an emergency on the afternoon before. "Something might happen" has neither a time nor a consequence, and it is not.

The person who decides is the technician on duty. And the way they decide is by naming the harm in one sentence to one other person. "External mail has been bouncing since 14:00" is a harm. "The patch for the hole in the log is out" is a harm. If the sentence cannot be finished with something that is or will be broken, the change is not an emergency. It is a normal change with a feeling attached, and feelings go on Thursday's list like everything else.

What Is Just Impatience

The reasons people want to make a change now are mostly good reasons to want it and not one of them is harm from waiting. "I'm here now and it's quiet." "The user is waiting." "The director asked." "It's Friday and I don't want it hanging over the weekend." "It's a small change." "The review isn't until Thursday." Each is a reason to prefer now. The emergency test does not ask what you prefer. It asks what breaks if you wait, and the honest answer, most afternoons, is nothing.

The table puts the two kinds side by side, because the difference is easier to see in examples than in definitions. The last column is what to do instead when the answer is no, and it is never "nothing". A normal change still gets made, at the review's window, with its record. That window is usually a few days away and occasionally the next morning.

The two Friday rows are the same event with different harms. And they are in the table together because the question is never about the job. It is about what the job protects. A desk that can tell those two rows apart at 17:00 on a Friday has understood the whole subject. And a desk that treats them the same is either working every Friday evening or losing a backup now and then. It depends on which way it got them wrong.

The Four Emergency Rules

Four rules, in the order they happen. Tell one other person before you act. Make the smallest change that stops the harm. Write the record within one working day. Bring it to the next weekly review. The four fit on an index card, and each gets a section below. But the card is the whole process. A technician who can recite it at 03:00 with a service down does not need the sections.

Rule One: Tell One Person

Telling one person is not asking for approval. It is a witness and a second brain, acquired in thirty seconds. The message has three parts: what you are about to change, why, and an invitation to object. "Disabling rule ATT-EXEC-2 on the mail gateway; it has been bouncing all external mail since 14:00; shout if you object." Chat, phone, or the next desk, whichever is fastest. The person does not need to be senior. They need to be awake and to have heard the sentence before the change rather than after.

What the one person does is read it, ask one question if they have one, and say go. The question is where the value is. "Is that the rule that catches the payroll attachments?" "Did you check it's not just our domain?" "Are you on the right gateway?" Nine times in ten there is no question and the reply is "go". The tenth time the question stops a change that would have been a second outage. And the thirty seconds spent on the other nine were the price of the tenth.

If nobody is available, write the message anyway. You might be the solo administrator at 03:00, the technician on a site visit with the office closed, or the one person on the desk during lunch. Type the sentence into the incident ticket or the team chat, with the time, before acting. It becomes the first line of the record. Phone the on-call manager if there is one, and tell the supplier if it is their system. But do not wait for either. A message to nobody, timestamped, is still a message that existed before the change, and that is the thing the rule is for.

Rule Two: The Smallest Change That Stops the Harm

Fix the bleeding, not the patient. Disable the one rule; do not rewrite the ruleset. Block the one address; do not redesign the firewall. Restore the one service; do not patch the server while you are in there. The emergency change is a tourniquet. The proper fix comes through the normal route with a record and a review, next week, when there is time to do it well. Every extra thing done during an emergency is an extra thing that might have caused the next one. It is also an extra thing nobody can separate from the first when the review asks what happened.

The temporary fix gets a follow-up ticket before the day ends. "CHG 7311: replace the disabled rule with a corrected one; normal; Thursday review." Without that ticket the temporary becomes permanent, and the permanent is undocumented. In a year there is a rule on the gateway called ATT-EXEC-2 that is switched off and nobody knows why. Every network I have inherited has had at least one "temporary" setting older than the person explaining it. The workaround that holds a problem open while the fix is found is described in Known Errors and Workarounds. The emergency change is very often that workaround, which is why it needs the same follow-up.

Rule Three: The Record, Within One Working Day

The emergency record is the same seven fields as a normal change record, written after the change rather than before. It adds three lines: why it could not wait, who was told and when, and the follow-up ticket for the proper fix. The seven fields are what, why, when, risk, rollback, test and who. What each one is for is laid out in The One-Page Change Record. The three extra lines are what make an emergency record reviewable after the fact. They let Thursday's review judge the decision, not only the change.

It is written within one working day because memory is a poor place to keep a configuration change. What was typed at 14:14 is exact at 16:30, approximate on Wednesday, and a story by the following week. The actual times come from the chat timestamps, and the what from the console you still have open. The rollback comes from what you would do right now if the fix turned out to be wrong. That is a question worth answering while the fix is fresh.

Two lines at the time are the minimum, and they are written before anything else. "14:14 disabled rule ATT-EXEC-2 on the mail gateway, Imani. Told Callum 14:13." Those lines go in the incident ticket the moment the harm stops. If the afternoon then goes wrong in some new way and the full record never gets written, the desk still has the essentials. It has a time, a system, a change and a name. The full record follows the same day, in whatever place the desk keeps change records. It is marked with the kind EMERGENCY and linked to the incident in both directions. The conventions for the place and the links are in Tracking Changes Without a Change Module.

The Emergency Inside a Major Incident

During a major incident, changes are made fast and often: restart this, fail over that, block this address, roll back that patch. Each one is an emergency change, and the incident gives it a ready-made process. The incident channel is the "tell one person", because everyone working the incident is in it. The incident timeline, kept by whoever is coordinating, is the record's first draft, because it already has the times. Whoever is coordinating says "go" before each change, and nobody makes one that the coordinator has not heard. The first minutes of that structure are set out in Declaring and the First Fifteen Minutes.

One change at a time. During an incident the temptation is to try three things at once. The result is a service that recovers without anyone knowing which of the three did it. Or it breaks further without anyone knowing which of the three did that. Make one change, test it, say what happened in the channel, then make the next. It feels slower. It is faster, because it is the only way the fourth change is chosen on evidence rather than on hope.

When the incident closes, the changes come out of the timeline. Each becomes a one-line entry in the change log with the incident number and the kind EMERGENCY. And each temporary one gets a follow-up ticket for the proper fix. That takes ten minutes at the close and is part of closing, in the same way that the review of the incident is. Closing the Incident and the Review puts it in the sequence. An incident whose changes stay inside the timeline is an incident that will be repeated. That is because the next person to touch the firewall will not know what the last one did at 02:40.

Reviewing Emergencies Afterwards

The last two minutes of the weekly review belong to the emergencies since the previous one, and each is asked four things. Was it an emergency, which means: name the harm again, in daylight. Did the fix hold, or has it been undone or redone since. Is the follow-up ticket for the proper fix open and dated. And what would have prevented it? That question turns an emergency into a problem record when the answer is "nothing we could have seen". Or it turns it into a calendar entry when the answer is "the certificate should have been on it". The habit of asking that last question without a committee is in Root Cause Analysis Without a Committee.

Once a month, count them. On a desk of five, one or two emergency changes a month is ordinary. One a week is a signal, and it points one of two ways. The environment may be fragile, with certificates, disks and jobs failing on their own schedule, which is problem management's territory. Or the word "emergency" has come to mean "I'm here now". That is the classification's territory and is fixed by reading the table above aloud. Count also the emergencies that had to be rolled back. An emergency change that needed undoing is two outages, and two in a month is worth a conversation about rule two.

Never punish the emergency. The technician who acted at 14:14 and wrote it down at 16:30 did the right thing even if Thursday's review decides, in daylight, that it was not quite an emergency. The review's job is to adjust the sort, not the person. The lesson goes on the classification table, and the technician is thanked for the record. A desk that is hard on emergencies does not get fewer of them. It gets changes made at 14:14 that nobody is told about and nobody writes down, which is the thing the whole subject exists to prevent.

A Short War Story: The Friday Port at Acme

At Acme, a manufacturer with a desk of three, Soren opened a firewall port at 17:05 on a Friday. The sales team needed a new conferencing unit working for a customer call at 08:00 on Monday, so he called it an emergency. He told nobody, and opened the range of ports listed in the supplier's document rather than the one the unit used. Nothing was written down, because it was Friday and the change was small. A vulnerability scan on Monday morning found the range exposing a management interface on the conferencing unit to the internet. The security contractor asked who had opened it and when. The answer took most of a day to establish because the firewall's own log recorded the rule and not the reason. The cost was a weekend of exposure, a day of two people's time, and an audit finding that took a quarter to close. The call on Monday would have worked just as well with one port opened at 08:30 after a two-minute review. Or it would have worked with one port opened on Friday, with one person told and the exact port named. The lesson Acme wrote on its card was that the harm from waiting had been an awkward conversation. And an awkward conversation is not an outage.

The Emergency Change Record

The record below is the normal change record with its three extra lines and the two-line minimum at the top. The top two lines are written at the time, in the incident ticket, before anything else. The rest is written the same working day. Keep the headings as they are so that an emergency record reads like every other record on the desk. The extra lines stay in the same place every time, where Thursday's review will look for them first.

EMERGENCY CHANGE RECORD — ONE PAGE
AT THE TIME (two lines, in the incident ticket, before anything else)
  ____:____  changed ______________________ on ______________________ (name)
  Told ______________ at ____:____ by ____________ (chat / phone / in person)

WITHIN ONE WORKING DAY
Change number: ________   Kind: EMERGENCY   Incident: ________

WHY IT COULD NOT WAIT
  The harm, in one sentence: ______________________________________
  Happening now [ ]   Certain by ____:____ on ________ [ ]

WHAT      System: ______________________  Setting or component: __________
          Before: ______________________  After: ______________________

WHEN      Actual start: ____:____   Harm stopped at: ____:____   Date: ________

WHO       Doer: ____________   Told first: ____________   Coordinator (if incident): ________
          Informed after: ___________________________________________

RISK      Who would have noticed if the change itself went wrong: __________

ROLLBACK  To undo this change now: ______________________________________
          Takes about: ____ min   Anything saved first: ____________________

TEST      Checked at the time: _____________________________________________
          Checked the next morning by ____:____: _________________________

FOLLOW-UP Temporary fix [ ]  Permanent fix [ ]
          Proper fix ticket: CHG ________   Kind: normal   Review on: ________
          Anything still disabled, bypassed or widened: ______________________

REVIEW    Read at the review on: ________
          Was it an emergency?  [ ] yes  [ ] no: lesson for the classification table:
          ____________________________________________________________
          Did the fix hold? [ ]   What would have prevented it: ________________

Imani's record went to Thursday's review with every line filled. The review agreed it had been an emergency and noted that the fix had held. It read CHG 7311, the corrected rule, as a normal change for the following Tuesday at 11:00 with the supplier's reference on it. The two lines she typed at 14:14 are still the first two lines of the record. They are the ones Callum read, three weeks later, when a branch manager asked why an attachment had bounced on a Tuesday afternoon. He had the answer in under a minute. That is what the four minutes were for.

Frequently Asked Questions