Root Cause Analysis Without a Committee, on One Page

Monday, 06:10, at Bellweather Logistics. Sam on the night dispatch desk phones the on-call number: the dispatch board is blank. That is the big screen that shows every driver which bay to load from. Dana restarts the board service from home, the screen comes back at 06:19, and twelve drivers who had been standing by their cabs go to their bays. It is the seventh Monday this has happened. Dana has a problem record open with seven linked tickets, a hypothesis line that says "something overnight?", and a next-review date of Friday. On Friday she has an hour to find out what the something is. There is no committee. There is Dana, a notebook and the ticket history.

Root cause analysis is the work of finding the thing that, if changed, would stop a repeat from repeating. The phrase has a grand sound and a committee-shaped reputation, and neither is deserved. Done well on a small desk, it is one person, one or two hours, and three plain tools. Those are a timeline of what happened, the question "why?" asked a few times honestly, and a sketch of the candidate causes in plain words. This article is part of our Problem Management series. It is about doing the analysis with those three tools and knowing when to stop.

It covers what "root cause" means on a small desk and what it does not. Starting with a timeline before asking anything follows, then the five whys, done honestly. Next comes the cause-and-effect sketch in plain words, for when the chain branches, then stopping at the actionable cause, with a picture. It then covers why a committee makes it slower and proving the cause before fixing it. A short war story and the one-page RCA sheet to copy finish the article, in that order.

What "Root Cause" Means on a Small Desk

The root cause, for a help desk, is the first cause in the chain that the desk can change and that would stop the repeats if changed. It is not the deepest cause. Every chain of causes, followed far enough, ends somewhere like "nobody has time" or "there is no budget" or "the company grew faster than its IT". And those are true and useless. The dispatch board at Bellweather went blank because a service stopped. The service stopped because a disk filled, because logs were never rotated. Logs were never rotated because the server was built by hand, because there was no build standard. There was no build standard because the person who built it was alone that year. Somewhere in that chain is a cause Dana can fix on Friday afternoon. The analysis is finished when she finds it.

Root cause is never a person. "Marcus restarted the wrong service" is a step in a timeline, not a cause. The cause is whatever made the wrong service the easy one to restart. An analysis that ends at a name has stopped one question early. And it has also made sure nobody will tell the truth in the next one. The question "why?" is asked of the system, every time. The person is a witness, and witnesses who are blamed stop appearing.

The aim is modest and should stay modest: stop the repeats. Not to understand the system completely, not to produce a document that satisfies an auditor, not to assign responsibility. A one-page sheet that says "the disk fills because logs are not rotated; set rotation and add an alert; expected to stop the Monday outages" is a complete root cause analysis. Anything longer is either a different problem or a report, and this desk has an hour.

Start With the Timeline

Before asking why anything happened, write down what happened, in order, with times. A timeline is a list of events with timestamps. It records when each incident was raised, when it was restored, what the restore was, and what else was going on around it. The ticket history supplies the first three. The rest comes from wherever the desk records change. Sources are the change log, the calendar, the patch schedule, the list of scheduled jobs, and the memory of whoever was on shift. Most "mysteries" are not mysterious once the events are in a column in time order. They were mysterious because the events were in seven tickets and three heads.

Dana's timeline for the dispatch board took twenty minutes. Seven incidents, all Mondays, all between 06:05 and 06:20, all restored by restarting the board service. Against them she set the overnight jobs: a backup at 01:00 daily, a data export at 03:00 on Mondays only. She also set a log file against them. When she looked at the server, that file was larger than everything else on the disk combined. The disk was at a hundred percent on Monday morning and sixty percent by Tuesday, because the export's temporary files were cleaned up on Tuesday. Seven tickets had said "service stopped". The timeline said "the disk fills on Monday nights".

Two things about the timeline matter more than the format. First, it includes what changed, not only what broke, because the commonest cause of a new repeat is a change nobody connected to it. A desk that tracks its changes somewhere, even without a change module, can lay them beside the incidents in five minutes. That is the case made in Tracking Changes Without a Change Module. Second, it is written down, not held in one head. A timeline on paper can be shown to the one person who knows about the export job. And they will spot the gap in it in a minute. A timeline in Dana's head cannot be shown to anyone.

The Five Whys, Done Honestly

The five whys is the simplest analysis method there is. State the symptom, ask why it happened, write the answer, and ask why of the answer. Repeat until you reach a cause you can act on. "Five" is a rough number, not a rule; some chains reach an actionable cause in three and some need six. The method's strength is that it forces each step to be stated. So a jump from "the board went blank" to "we need a new server" has to pass through the steps in between, and usually cannot.

Its weakness is that it is only as honest as the answers, and three habits keep it honest. Each answer must be a fact that was checked, not a guess that sounds right. "The service stopped because the disk was full" is only allowed on the chain once somebody has seen the disk full. Each why is asked of the thing, not the person; "because Marcus did not notice" becomes "because nothing alerted anyone", which is a cause with a fix. And when a why has two true answers, the chain branches, and both branches get followed. That is when the sketch in the next section earns its keep. A five-whys chain that has no branches and no checked facts is a story, and stories are very persuasive.

Dana's chain ran like this: the board was blank because the service had stopped. The service stopped because the disk was full. The disk was full because the Monday export's temporary files landed on top of a log that had grown for a year. The log had grown because rotation was never configured. Rotation was never configured because the server was built by hand from memory with no checklist. Five whys, each one checked against the timeline. The actionable cause is the fourth: configure log rotation and add a disk-space alert. The fifth is true, and worth a line in the record, and is not Friday's job.

The Cause-and-Effect Sketch in Plain Words

When the chain branches, or when the first why has four possible answers and none has been checked, a sketch does what a single chain cannot. The cause-and-effect sketch is the symptom written on the right-hand side of a page, and every candidate cause written on the left. The causes are grouped under four plain headings. They are the thing itself (the device, server or application), its settings and changes, and the steps people take with it. The last heading is its surroundings (timing, load, temperature, network, power). An arrow from each candidate to the symptom. That is all. The formal version has a name and a fish-shaped diagram; the plain version is a list in four columns, and it works the same.

The sketch's job is to stop the analysis from marrying the first plausible cause. Written out, Dana's left-hand side held nine candidates. It listed the service itself crashing, a memory leak, the disk, the backup locking a file, and the export locking a file. It also listed the night dispatcher's screen, the network to the screen, a Monday reboot of a switch, and the building's power. Most desks would have fixed the first two and been surprised in a week. With nine on a page, each one is tested against the timeline in turn. The ones that cannot explain "Mondays only, 06:00, every week" are crossed out in thirty seconds each. Three survived the timeline. One survived a look at the server.

Draw it on paper or a whiteboard, in the words the desk uses, and photograph it for the record. The point of plain words is that the night dispatcher can read it. Sam, shown the sketch, said "the screen is never the problem, it says 'no data' when the service is down and goes black when the screen is off". That removed two candidates and took him ten seconds. A diagram with a technical name would have been shown to no one outside the desk.

Stopping at the Actionable Cause

The analysis stops at the first cause that passes three tests. The desk can change it. Changing it would stop the repeats, and it can be done inside a month. A cause that passes all three is actionable, and it is the root cause for the desk's purposes whatever lies beneath it. A cause that passes the second test but not the first or third is contributing. It is true, worth one line in the record, worth raising with whoever can change it, and not the desk's fix. Beyond that is the territory of budgets and headcount, and the analysis does not go there, because nothing it finds there can be fixed on a Friday.

The picture below shows the dispatch board chain with the stopping point marked. The symptom is on the left, and each box to the right is one honest answer to "why?". The third why, log rotation never set up, is marked green. The desk can change it. Changing it stops the Monday outages and takes an afternoon. The fourth, the server built by hand with no checklist, is marked amber: true, contributing, written down, and raised as a request for a build checklist. The fifth, no build standard, is dashed, because it is where the desk stops asking. Every chain has a dashed box eventually. The skill is in noticing the green one before you reach it.

Both directions have a trap. Stopping too shallow produces a fix that is really a restore with a schedule. "Restart the service every Sunday night" stops the Monday outages and leaves the disk filling. The outages move to Wednesday by autumn. Going too deep produces a fix nobody can make. "Adopt a server build standard" is right, and will take a year. The board will go blank fourteen more Mondays while it is adopted. The three tests catch both. A fix that is a restore fails the second test; a fix that needs a budget fails the third.

Why a Committee Makes It Slower

A committee is the natural response to a repeat that has embarrassed someone senior. And it is almost always the wrong one for a desk of this size. A committee has a calendar rather than a timeline. It meets when nine people are free, which is in eleven days, and by then there has been another Monday. It seeks agreement rather than evidence, so the cause it settles on is the one nobody in the room objects to. That is usually "insufficient resilience" or "a lack of monitoring", both of which are true of everything. And the committee generates a document, because a committee that has met four times must produce something, and the something is rarely a configured log rotation.

The analysis needs one person with the timeline and the authority to go and look. It needs other people for exactly one thing: facts. The person who set up the export job knows what it does with temporary files. The dispatcher knows what the screen looks like when the service is down. The person who built the server knows there was no checklist. Each of them is asked one question, in a corridor or a two-line message, and each answers in a minute. That is the whole of the consultation. A desk that turns it into a meeting has spent nine hours of other people's time to learn three facts. It will be asked to present the findings as a slide deck.

There is one case for a group, and it is small. When the chain crosses teams, the facts and the fix sit with different people. In that case, bring exactly those people together for thirty minutes, with the timeline printed and on the table. Not to discuss the problem; to check the timeline for gaps and agree who does the fix. Thirty minutes, a printed timeline, and a named owner for the fix is the most committee a small desk's problem ever needs. The review after a major incident is a related but different exercise, with its own shape. And it is covered in Closing the Incident and the Review.

Proving the Cause Before Fixing It

A cause is proved by a prediction that comes true, and the prediction is made before the fix, not after. If the disk filling on Monday night is the cause, then the disk will be nearly full at 03:30 next Monday. In that case, the service will stop within the hour of it reaching full. Dana can check that with a disk-space reading at 03:30, set to run automatically, without changing anything. If the reading shows forty percent free and the board still goes blank, the cause is wrong and nothing has been wasted but a line in a log. A cheap check that could prove you wrong is worth more than an expensive fix that could only prove you right.

Resist "the fix will prove it". It will, eventually. The problem with eventually is that a fix applied to a wrong cause produces a quiet month by coincidence and a loud one by spring. By then nobody remembers that the cause was never checked. Where the check is cheap and safe, do it first. Where reproducing the failure is possible without hurting anyone, do that: fill a test disk and watch the service. Where neither is possible, apply the fix and say in the record that the cause is "probable, unproven". That way, the owner watching the tickets afterwards knows what they are watching for.

Once the cause is proved or probable, the analysis hands over two things and is finished. The workaround, the fastest way to restore the symptom while the fix waits, is written up for the technicians. That is the subject of Known Errors and Workarounds. And the fix becomes a proposal with a cost, a change, and a way of knowing it worked. That is the subject of Getting the Permanent Fix Done. The one-page sheet at the end of this article is the handover document for both. Dana configured the rotation on Friday, added the alert, and set the export to write its temporary files elsewhere, which was a bonus from the sketch. The board came up on Monday. It also came up the Monday after, which was the one that counted.

A Short War Story: The Licence Server Committee at Redbrook Engineering

Redbrook Engineering's design office lost its drawing software for twenty minutes at a time, several times a week, when the licence server stopped answering. After a client deadline was missed the operations director convened a working group of nine. It had two from the desk, three from engineering, two project managers, a finance representative and a facilities manager. The group met four times over eleven weeks. It produced a fourteen-page report recommending a resilient licence server pair, a monitoring platform and a quarterly review. In those eleven weeks the design office lost its software fourteen more times. Priya, the junior technician on the desk, built a timeline in the second week from the fourteen tickets and the switch logs. She found that every drop matched a four-minute window in which a network switch in the plant room rebooted itself. The switch had been doing that since a firmware update had left its power-saving setting on. She was not on the agenda until meeting four. The setting took two minutes to change. The thirty-six hours of the working group's time, the eleven weeks, and the fourteen outages had bought a report that nobody read. One person with a timeline and an afternoon had already found the cause and nobody had asked her.

The One-Page RCA Sheet

The sheet below is the whole analysis on one page: timeline, chain, sketch, stopping point, proof, and handover. Fill it in the order it is printed, because the order is the method. The chain is not started until the timeline exists, and the fix is not named until the cause has a check against it. The sketch section is four short lists, not a diagram. Keep the completed sheet with the problem record. It is the thing the next technician reads when the symptom comes back in two years and the desk has changed hands.

ROOT CAUSE ANALYSIS — ONE PAGE
Problem ID: ________   Analyst: ______________   Date: __________
Symptom (requester's words): ____________________________________
Incidents linked: ____   First: __________   Last: __________

1. TIMELINE (events in time order, with the source of each)
   When            What happened                          Source
   ____________    ___________________________________    ____________
   ____________    ___________________________________    ____________
   ____________    ___________________________________    ____________
   ____________    ___________________________________    ____________
   Changes made in the period (from the change list): _______________
   Pattern in the timeline (day, hour, trigger): ______________________

2. CANDIDATE CAUSES (the sketch, in plain words)
   The thing itself:        ____________________  ____________________
   Its settings / changes:  ____________________  ____________________
   Steps people take:       ____________________  ____________________
   Surroundings / timing:   ____________________  ____________________
   Crossed out by the timeline: _______________________________________
   Still standing: ______________________________________________________

3. THE CHAIN (each answer a checked fact; ask why of the thing)
   Symptom: ________________________________________________________
   Why 1: __________________________________  checked by: ___________
   Why 2: __________________________________  checked by: ___________
   Why 3: __________________________________  checked by: ___________
   Why 4: __________________________________  checked by: ___________
   Why 5: __________________________________  checked by: ___________

4. STOPPING POINT
   Actionable cause (desk can change it, stops repeats, inside a month):
   ____________________________________________________________________
   Contributing causes (one line each; who is told): ___________________
   Out of reach (noted, not pursued): ___________________________________

5. PROOF
   Prediction if the cause is right: _____________________________________
   Cheap check, before the fix: __________________  Result: ______________
   Cause is:  [ ] proved   [ ] probable, unproven   [ ] wrong, back to 2

6. HANDOVER
   Workaround for technicians: ___________________________  published [ ]
   Proposed fix: ____________________________  Change record: __________
   How we will know it worked: ______________  Watch until: __________
   Fix owner: ______________

Most sheets are finished in under two hours, and most of that is the timeline. If section 3 cannot be filled in with checked facts, the sheet is not finished. In that case, the right next step is one more look at the thing, not one more meeting about it. If section 4 has no actionable cause, the problem goes back on the list with the contributing causes written up and a note of who has been told. That is a real outcome, and it is written down. Dana's sheet for the dispatch board took an hour and forty minutes, including the walk to the plant room. The sheet is still in the record. When the export job was changed two years later and the Monday outages briefly returned, the technician who found the sheet read it. The technician checked the disk and had the board up before the drivers arrived.

Frequently Asked Questions