The Bus-Factor Audit: Finding the Knowledge Only One Person Holds

Thursday, 08:50, at Redbrook Engineering. The CAD licence server has stopped handing out licences, and thirty engineers are sitting in front of drawings they cannot open. Dana, who has run the desk for six years, is looking at a sign-in page she has never signed into. Lena set the licence server up. Lena renews its licence file every spring. Lena is the person the engineers e-mail when it sulks. And Lena is off sick with her phone on the bedside table, switched off, as a sick person's phone should be. The desk has five people in it. For this problem, this morning, it has none.

Nobody at Redbrook had thought of the licence server as a risk, because it had never gone wrong in a way that reached the desk. It went wrong, and Lena fixed it, and the two events were so close together that from outside they looked like nothing happening. That is how single points of failure stay hidden. The person covering them is so good at it that there is never a ticket. And a desk that measures itself by tickets never sees the gap until the morning the person is not there.

The bus factor of a system is the number of people who would have to be unavailable at once before nobody left on the desk could run it. They might be under a bus, on a beach, in bed with a fever or at a new job. A bus factor of one means one person; lose them for a day and the system has no owner. The bus-factor audit is an hour, once a quarter. In that hour the team draws a grid of its systems against its people, counts the ones, and picks three to fix. This article is part of our Team Knowledge Sharing series. The series covers how a small desk gets what is in one head into several. This is the article to start with, because the audit tells you where the other five articles should be pointed.

It covers what the bus factor measures and the standard behind the word "can". Next come building the grid in an hour and reading it for the two shapes of trouble. Then comes a worked example from Redbrook's five-person desk. It covers choosing the first three to fix and why those three, then what "fixed" has to mean. It ends with the quarterly rhythm, a short war story, and the grid itself to copy, in that order.

What the Bus Factor Measures

The bus factor counts people who can run a system alone, and "alone" is the whole standard. Picture the expert unreachable, on a bad Tuesday, with the requester waiting. "Alone" means this person could sign in, find the problem, fix it or safely get round it, and know whether it worked. It does not mean "has seen it done". It does not mean "could probably work it out". It does not mean "knows where the password is". Those are all useful, and none of them is a second person at 08:50 on a Thursday.

Between "can run it alone" and "cannot" there is a third, honest state: can follow the runbook. A runbook is a procedure written so that a competent stranger can run it alone and know whether it worked. And a person who could get through one with the document open and a phone call allowed is halfway to being a second owner. The grid records that state separately, because it is the cheapest next person to train. There is another reason. Counting it as a full owner is how a desk convinces itself it has a bus factor of two. A week later, it discovers it has one.

The bus factor is a property of a system, not of a person, and that matters for how the conversation goes. The question is never "is Lena a risk?" Lena is an asset; she is the only reason the licence server has ever worked. The question is "how many people can run the licence server?", and the answer is a number the whole team owns. Keep the grid about systems and the audit stays a planning exercise. Let it become about people and it becomes a performance review nobody asked for, with the wrong person being reviewed.

Building the Grid in an Hour

The grid is systems down the side, people across the top, one mark per cell. And it takes an hour because the hard part is the list of systems, not the marks. Build the list from three places. First is the server and service inventory, if the desk has one. Second is the ticket categories from the last quarter. Anything that generated tickets is a system whether or not anyone calls it one. Third is a question to each technician, "what do people come to you for that nobody else does?" The third source is where the surprising rows come from. The label printers in the warehouse. The door-entry system. The spreadsheet that finance treats as an application. The thing that is "not really IT" and stops the building when it breaks.

Aim for ten to twenty rows and resist going finer. "The directory" is a row; "resetting passwords in the directory" is not, because a grid of eighty rows takes a day to fill and nobody reads it afterwards. If a system is really three systems with different owners, split it; if two rows would always have the same marks, merge them. The test for a row is whether a reasonable technician could run all of it or none of it. When the answer is "some of it", you have found two rows wearing one name.

Fill the grid in the room, with everyone present, and let each person mark their own column first. Then open it to challenge. The challenge is not "I don't believe you". It is "when did you last do it, and was anyone helping?" A mark that cannot answer that question drops from "alone" to "runbook". And the room should expect a few of those; the only grid that gets no downgrades is one nobody looked at. If you run QuerySolv, the per-technician reports show who resolved what in the last quarter. Those reports are a less flattering and more accurate first draft than memory. They are worth having open while people mark themselves.

The honesty rule runs one way. Over-claiming is the risk. A cell that says "alone" when it means "watched once" is a hole the grid has painted over. And the hole is found at 08:50 on a Thursday. Under-claiming costs a half-hour of cross-training that turns out to be unnecessary, which is the cheapest mistake in this article. When a technician hesitates, mark the lower state. The grid is not a place to be modest about your colleagues or generous about yourself. (It is also, I have found, not a place to be generous about yourself.)

Reading the Grid

Two counts fall out of the finished grid, and they show two different shapes of trouble. Count the "alone" marks across each row and you have the bus factor of each system. Every row that totals one is a single point of failure, and those rows are the audit's first output. Count the "alone" marks that are the only mark in their row down each column. That gives you the concentration: how many single points of failure sit with one person. A desk with five singles spread across five people has five small problems. A desk with five singles that all sit with Dana has one large problem with Dana's name on it. And it should be read that way.

The row shape tells you what to cross-train; the column shape tells you how urgently. A person who holds three singles is three systems away from being irreplaceable and one resignation away from a very bad quarter. And the cost of them being away is not three separate outages but three outages on the same day, which is worse than the sum. The column count is also the one that shows the hero, the one person the desk has quietly been built around. That is what The Seven Ways Support Quietly Fails describes as a failure that never pages anyone. The grid is the first time most desks see the hero in numbers.

The picture below is the finished grid for Redbrook's desk, which the next section walks through. Each row is a system, each column a technician. And each cell says whether that person can run the system alone, can follow its runbook, or cannot yet. The bus-factor column counts the "alone" marks. The rows that score one are outlined in red, and the three the team chose to fix this quarter carry an orange tag. Read it across for a system and down for a person, and notice how different the two readings are.

A Worked Example: Redbrook Engineering's Five-Person Desk

Redbrook's desk is Dana, Marcus, Priya, Owen and Lena, supporting about two hundred staff. Thirty of those are engineers who cannot work without the CAD licence server. The grid came out with ten rows. Five of them scored a bus factor of one: the mail service and the firewall and VPN with Dana, backup and restore with Marcus, the CAD licence server with Lena, and the phone system with Priya. Owen, the newest, holds no singles, which is normal for the newest person. It is also the reason he is the obvious next owner of several rows. The table adds the two things the grid does not show and the choice needs: how often each single breaks, and how bad a day without it would be.

The first thing the table does is separate the five singles, which the grid alone could not. All five are red. Two of them are on fire and one is a bomb; the other two are red in the way a parked car with a flat tyre is red, worth fixing, not worth fixing today. The grid finds the singles. The two questions, how often and how bad, order them.

The second thing it does is make the column count visible as a cost. Dana holds two of the five. The mail service and the firewall are not the sort of systems that fail on the same day by chance. They are the sort that fail on the same day when Dana is on leave and nobody has touched either for a fortnight. Fixing the firewall single does two jobs: it adds a second VPN owner, and it takes Dana from two singles to one. That is why it beat the phone system, which breaks about as often and matters less.

Choosing the First Three to Fix

Three, because three is what a small desk can cross-train in a quarter without the queue noticing. Each fix is a few half-hours, a runbook and at least one rehearsal. That is a real cost on a desk that also has tickets. And a list of eight fixes in a quarter is a list of eight fixes that will all be half done in June. Pick three, finish three, and put the rest at the top of next quarter's list. A quarter in which three single points of failure become double is a good quarter, and the grid will show it.

Order them with the two questions, and let the second question beat the first. How often does the system need its owner's hands, and how bad is a day without it? A thing that breaks monthly and costs an hour is cheaper to leave single than a thing that breaks yearly and costs the business a day. That is why backup and restore is first at Redbrook though it is the quietest row on the grid. The backup is the system whose bus factor you only discover on the worst day you will have that year. Do not wait for the day to tell you.

Add the concentration as a tie-breaker, and sometimes as more than a tie-breaker. When two singles score the same on the two questions, fix the one held by the person who holds the most, because it buys two things for one price. When one person holds three or more singles, fix one of theirs even if it would not otherwise make the list. In that case, the risk is no longer the system; it is the week they are away. And that week is already in the calendar whether or not anyone has looked.

Beware the interesting one. There is always a system on the grid that the team would enjoy cross-training on, usually the one with the nicest console. And it is rarely the one that should be first. The phone system is boring. The label printers are boring. The payroll export that runs on the last working day of the month is so boring that nobody wants to own it even once. That is exactly how it ended up with one owner. Fix the boring single that stops the business, then reward yourselves with the interesting one. The grid does not care which is more fun, and neither does the business on the day it breaks.

What "Fixed" Has to Mean

A single point of failure is fixed when two things have happened. A second person has run the system alone, at least once, with the expert reachable but not touching. And the grid's cell has changed from "runbook" or blank to "alone". Not when there is a runbook. Not when the second person has "been shown it". Not when the expert says "Owen could do it". A runbook that nobody but its author has run is a hypothesis, and a demonstration is a memory that fades in about a month. The only evidence the grid accepts is a cell that changed because someone did the thing. That is a high bar on purpose. The bar is 08:50 on a Thursday.

The three ways to get a cell to "alone" are the subjects of the next two articles in this series, so here they are only named. The runbook route starts with the expert writing the procedure. The second person runs it in a quiet hour with the expert in the room and their hands in their pockets. What the second person could not follow is what the runbook was missing. The shadow route: the second person takes the next real ticket for that system with the expert watching. The swap route: the system's tickets are routed to the second person for a month, with the expert as the escalation. The rhythms that make these happen without the queue collapsing are in Cross-Training Rhythms That Survive a Busy Month. What to write down first is in Documentation as the Sharing Medium.

Write the cell change down with a date and the ticket or rehearsal it happened on, because the grid is also the record. Next quarter, the team asks whether the backup single is really fixed. The answer is "Priya restored a folder from backup on the fourteenth, on ticket 5127, with Marcus watching". That is an answer. "We did the backup thing in the spring" is not. The grid with its dates becomes the first page of the knowledge-spread measurement this series ends with, which is why it is worth keeping tidy from the first quarter.

The Quarterly Hour

The audit is an hour a quarter, same grid, same room. And it takes less each time because the list of systems changes slowly and the marks are a correction rather than a survey. Three things move between audits. People move: a technician leaves, a new one arrives with no marks and inherits the newest-person job of being everyone's second. Systems move: something new arrives. A new system has a bus factor of one by definition, because the person who set it up is the only person who has touched it. And marks decay. A cell that said "alone" a year ago about a system nobody has run since should be challenged with the same question as before. Ask "when did you last do it?" Downgrade the cell if the answer is "not since the audit".

Add a new system to the grid the week it arrives, with the one name on it. And treat the row as a single from the start. This is the cheapest moment to fix a bus factor there will ever be. That is because the person who built it still remembers why every setting is what it is. And a second person shadowing the setup learns in an afternoon what will take a week to reverse-engineer in two years. Most desks do not do this. They add the system to the grid at the next audit. By that time it has been quietly working for three months and has acquired the reputation of a thing that never goes wrong. Those are the rows the audit exists to find.

The grid also pays for itself outside the audit. The next time someone books two weeks of leave, read their column before approving the dates. And the singles they hold are the list of things to hand over before they go. That is the sensible half of Vacation Coverage Without the Pile. The next time someone resigns, their column is the first page of the handover plan. And the shorter that column is, the calmer the notice period. A grid that only comes out once a quarter is a grid that is not being used. Keep it where the rota lives and it earns its hour several times over.

Report the audit upward in one line per quarter: how many systems, how many with a bus factor of one, which three were fixed, which three are next. Managers understand "we had five single points of failure and now have two" without any help from the desk. It is one of the few numbers a support desk can report that gets better without anyone working harder. How to track that number honestly over several quarters, and the other measures that show whether knowledge has really spread, is in Measuring How Far Knowledge Has Spread.

A Short War Story: The Payroll Export at Tollgate Foods

Tollgate Foods ran a food-packing plant with six hundred staff on shifts. Once a month Hugo on the IT desk exported the hours from the time-clock system into the file the payroll bureau needed. The job took him twenty minutes and had never once failed in five years. It was not on any list, because it was not a system; it was a thing Hugo did. Hugo took a fortnight's leave in a month when the export's signing certificate expired on the last working day. The two technicians covering for him spent that day and the next working out which of four servers the export ran on. They were also working out what it was called, and why it produced an empty file. Meanwhile the bureau's deadline passed. Six hundred people were paid three days late. The finance team worked a weekend re-keying hours by hand from the clock printouts. The plant manager asked, reasonably, how a twenty-minute job had become a three-day failure. The grid Tollgate drew the following week had the payroll export on its own row, with one name in it. And that row was fixed first. A thing one person does every month is a system, whether or not anyone has called it one.

The Bus-Factor Grid

The grid below is the audit on one page, as plain text. So it can be pasted into a document, a spreadsheet or a knowledge base article visible to the desk. There is one row per system, one column per person, one letter per cell. A means can run it alone. R means can follow the runbook with a phone call allowed. A dash means cannot yet. The bus factor is the count of A marks in the row. The concentration is the count of rows where a person's A is the only A. Fill in the two questions for every row that scores one, choose three, and write the date and evidence when a cell changes.

BUS-FACTOR GRID — ONE PAGE
Desk: ______________   Quarter: ________   Filled in by: all present

Marks: A = can run it alone   R = can follow the runbook   - = cannot yet
Bus factor = count of A in the row.  Rows scoring 1 are single points.

System                     ______  ______  ______  ______  ______   Bus
                           (name)  (name)  (name)  (name)  (name)   factor
1.  _____________________    __      __      __      __      __      __
2.  _____________________    __      __      __      __      __      __
3.  _____________________    __      __      __      __      __      __
4.  _____________________    __      __      __      __      __      __
5.  _____________________    __      __      __      __      __      __
6.  _____________________    __      __      __      __      __      __
7.  _____________________    __      __      __      __      __      __
8.  _____________________    __      __      __      __      __      __
9.  _____________________    __      __      __      __      __      __
10. _____________________    __      __      __      __      __      __

Concentration (rows where this person's A is the only A):
                             __      __      __      __      __

FOR EVERY ROW SCORING 1
  System: _______________   Only owner: ________
  How often it needs the owner's hands: [ ] weekly [ ] monthly [ ] quarterly [ ] rarely
  A day without it: [ ] stops the business [ ] stops a department [ ] annoying
  Next owner (the R, or the newest person): ________

THE THREE TO FIX THIS QUARTER
  1. _______________  second owner: ________  route: [ ] runbook [ ] shadow [ ] swap
  2. _______________  second owner: ________  route: [ ] runbook [ ] shadow [ ] swap
  3. _______________  second owner: ________  route: [ ] runbook [ ] shadow [ ] swap

CELL CHANGES (evidence)
  System: ___________  Person: ________  now A on: ________  ticket/rehearsal: ________
  System: ___________  Person: ________  now A on: ________  ticket/rehearsal: ________
  System: ___________  Person: ________  now A on: ________  ticket/rehearsal: ________

NEXT AUDIT: ________   Singles last quarter: ___   Singles now: ___

Frequently Asked Questions