Measuring How Far Knowledge Has Spread Across the Desk

Thursday, 15:30, at Oakridge Academy, in the bursar's office for the quarterly review. Nikhil, who runs the five-person IT desk, has spent a year on the things in this series. Those were the grid in the spring, half hours every Tuesday, a huddle every Wednesday, and a handover when a technician left in the autumn. The bursar asks a fair question. "Are we less dependent on Dev than we were?" Nikhil says yes, he thinks so, definitely. The bursar asks how he knows. There is a pause in which Nikhil discovers that he has a year of effort and a strong feeling and no number at all. The feeling, however right, is not going to appear in the minutes.

The feeling is probably correct. That is what makes the pause so annoying. A desk that has done the work can usually tell that fewer tickets wait for one person. It can usually tell that the newest technician takes things she could not have taken in the spring, and that the huddle has stopped producing surprises. None of that is a count, and a manager who is paying for the half hours is entitled to one. The question is which counts, because the obvious ones measure the wrong thing. Measuring the wrong thing for a year is how a desk ends up praised for its documentation the week its only expert leaves.

Measuring knowledge spread is a small set of counts, taken once a quarter from data the desk already has. They show whether what the desk knows is in more heads than it was. Four counts do it. They are how many people resolve each kind of ticket and the bus factor by system over time. They also count how many tickets had to be reassigned because only one person could do them. The last count is how many documents each person wrote, read carefully as coaching. This article is part of our Team Knowledge Sharing series. This article is the last one, because it is the one that tells you whether the other five worked.

It covers why to measure and what not to, then each of the four measures with its counting rule and a worked example from Oakridge's desk. Next come reading the per-person count as coaching rather than ranking and putting the four together. The honest line in the quarterly report, a short war story, and the knowledge spread scorecard to copy come last, in that order.

Why Measure, and What Not To

Measure because the alternative is a feeling, and feelings do not survive the bursar, the budget round, or the week the expert resigns. A desk might say "a year ago five systems had one person who could run them; now two do". That sentence justifies the half hours, points at the next quarter's work, and will still be true when the person who said it has moved on. A desk that can say "we've done a lot of cross-training" has a sentence that is believed in proportion to how much the listener already liked the desk. The counts take about an hour a quarter, from the grid, the tickets and the knowledge base, which the desk already has. They are the only hour in this series that produces something a manager can read.

Do not measure inputs, and most of what desks measure under this heading is an input. Training hours attended measure attendance. Documents written measure typing. Forty articles a quarter is forty articles, and if thirty-eight are user guides to the printing portal, the desk's single points of failure are exactly where they were. Quiz scores measure quizzes. The thing the bursar is actually asking about is whether, on a Thursday when Dev is away, the desk can do what Dev does. The only measures that answer it count what would happen, or what did happen, when a ticket met a person who was not the expert.

Keep every count cheap enough to repeat, because a measure taken once is an anecdote with a decimal point. If a count takes a day, it will be taken once, in the quarter when someone is enthusiastic. In that case, the trend, which is the only interesting part, will never exist. Each of the four below is a filter on the tickets, a row count on the grid, or a column in the knowledge base. None should take more than fifteen minutes once the counting rules are written down. The rules are written down once, at the end of this article, so that next quarter's count is a reproduction rather than an argument. (Arguments about counting rules are how an hour becomes an afternoon.)

Measure One: Resolvers per Category

The first count is, for each kind of ticket, how many different people resolved at least two of them this quarter. A category here is whatever the desk already uses to sort tickets by what they are about. On most small desks, that is close to the list of systems on the grid. The counting rule is to take tickets closed in the quarter and group by category. For each category, count the distinct technicians who were the resolver on two or more. Two, not one, because one is a lucky Thursday; two is a person who can do it. The resolver is whoever did the fixing. On a ticket that was reassigned, that is the person it was reassigned to, not the person who triaged it or the person who clicked close. Those rulings matter enough to write down. And the general rules for which tickets belong to which quarter are in Counting Tickets Properly, which this measure leans on rather than repeats.

Oakridge's count, from a fictional quarter of about four hundred closed tickets, looked like this. Printing: four of five technicians. Accounts and passwords: five of five. Classroom displays: three of five. Wi-Fi: two of five. The timetable system: one of five, and it was Dev. The finance system: one of five, also Dev. Four categories with a single resolver in the spring; two in the autumn. The number to report is the count of categories with only one resolver, and the trend in it. That number is the bus factor measured by what actually happened rather than by what people say they can do.

Two traps. A category with very few tickets cannot score well no matter how spread the knowledge is. If the finance system produced three tickets all quarter, it cannot have four resolvers. In that case, the honest entry is "fewer than ten tickets; not scored" rather than a one that looks like a single point of failure. And a category that scores five of five may still hide a single point of failure inside it. "Accounts and passwords" at Oakridge was five of five because everyone resets passwords. Meanwhile, the one person who could fix the directory synchronisation was still one person. That is the limit of the ticket count, and it is why the second measure exists.

Measure Two: Bus Factor by System Over Time

The second count is the number of systems on the bus-factor grid with one, two, and three or more people who can run them alone. It is taken at each quarterly audit and kept as a series. The grid itself is the subject of The Bus-Factor Audit. This measure is just its row totals, written down each quarter beside the previous quarter's, so that the movement is visible. Oakridge's series ran five singles, then four, then four, then two, over four audits. The flat quarter in the middle was the one in which a technician left and took two "alone" marks with her. The series shows that, and the feeling would have forgotten it.

Count "alone" marks only, with their evidence dates, and the measure stays honest. The grid's weakness is that it is filled in by people describing themselves, and people are optimistic in rooms. So the audit's rule applies with force here. A cell is "alone" only when the person has done the thing without help, on a dated ticket or rehearsal. A series built on "could probably do it" marks goes down beautifully and tells you nothing. A series built on dated evidence goes down more slowly and is the one you want in the report.

Measures one and two should agree, and where they do not, one of them is wrong, and it is usually the grid. Suppose the grid says three people can run the timetable system and the tickets say only Dev resolved any this quarter. One possibility is that the other two never got a timetable ticket, which the triager can confirm. Or they got one and passed it to Dev, which is the third measure's subject. Or the grid is over-claiming. Put the two numbers side by side each quarter. The grid is what the desk believes about itself; the ticket count is what the desk did. The gap between them is the most useful thing on the scorecard, and it should be closing.

Measure Three: Reassignments Because Only One Person Could

The third count is the number of tickets this quarter that were reassigned to a specific person because nobody else could do them. A forced reassignment is the moment the bus factor costs something. A ticket arrives, and the person who picked it up cannot do it. It goes to the one person who can, who was doing something else. Each one is a wait for the requester, an interruption for the expert, and a small confirmation that the knowledge has not spread. Counted per quarter, and by system, it is the cost side of the grid. It is the number that falls fastest when cross-training works. That makes it the most encouraging thing on the scorecard in the first year.

The counting rule needs one habit: whoever reassigns a ticket for this reason writes a fixed phrase in the reassignment note. The phrase is "reassigned: single owner" or whatever three words the desk agrees, used the same way every time, so that a search finds them all. Some tools have a reason field or a tag that does this; most small desks use the note, and the note works everywhere. Without the phrase, the count is every reassignment. That includes the ordinary ones, the misrouted tickets, and the handovers at the end of a shift, and the count means nothing. With it, the count is exactly the tickets that waited for one person. The general counting of tickets that moved between people, and what the other kinds of move tell you, is in Measuring the Cracks. This measure borrows the method and keeps only one reason.

Oakridge counted thirty-one forced reassignments in the spring quarter, twenty-two of them to Dev, nineteen of those for the timetable system. In the autumn the count was twelve, five to Dev, and none for the timetable system, because by then two other people had reached "alone" on it. The requester-facing cost, in hours waited, fell with it. And the interruption cost to Dev, which nobody had ever counted and Dev had been quietly carrying, became a number that could be shown to the bursar. If you run QuerySolv, the complete event history on each ticket holds every reassignment with its note. In that case, the count is a search for the phrase rather than a memory of who passed what to whom. In any tool, the phrase is the measure; the tool only has to keep it.

Measure Four: Articles Written per Person, Read as Coaching

The fourth count is how many knowledge base articles, runbooks and known-error records each technician wrote or substantially revised this quarter. And it is the only one of the four that names a person, which changes everything about how it is read. The number is for coaching and never for ranking, and the rule from the metrics series applies in full. Do not read a per-person figure until there is a minimum sample behind it. Here that means at least a full quarter and at least a dozen documents across the desk. Three articles in a month by five people is noise with names attached. A quarter in which the desk wrote forty documents, and one of the five wrote none, is a conversation.

The conversation is not "you wrote nothing". It is "you resolved more timetable tickets than anyone and there is no timetable runbook; shall we fix that in the half hours?" That is the hero being found by a number and offered a route rather than a mark. The other pattern the count finds is the person who wrote eleven articles that nobody has used. That is not a problem to fix but a thing to notice. It usually means the articles are about the wrong systems or are not findable from the words in a ticket. Both uses are coaching. Neither works if the number has been on a wall screen. The moment a count of documents is public and compared, the desk produces documents, short ones, about easy things, in quantity. The reasons a per-person number should never be a league table are in Dashboards That Motivate Instead of Surveil. And they apply to this count more than to most.

Add a second column beside "written": "used by a second person", taken from the last-used line that every runbook should carry. A technician who wrote four runbooks that three colleagues have since run has moved knowledge. A technician who wrote nine that only they have opened has written a diary. The second column is the one worth more. And it is also the one that cannot be gamed by writing more, which is exactly why it belongs beside the first.

Putting the Four Together

The four measures answer four different questions, and the scorecard works because of the way they check each other. Measure two says what the desk believes it could do. Measure one says what it did. Measure three says what it cost when it could not. Measure four says who is turning the first three into something that outlasts them. A quarter in which the grid improves and the forced reassignments do not fall is a quarter in which the grid is lying. A quarter in which the ticket count improves and the grid does not is a quarter in which the audit was skipped. A quarter in which everything improves and nobody wrote anything is a quarter that will have to be done again when someone leaves.

Take all four once a quarter, in the same week as the audit, and keep them on one page as a series rather than a snapshot. The first quarter's numbers are a baseline and mean nothing on their own. The fourth quarter's numbers beside the first are the entire case for the series. Oakridge's page, after a year, read: singles five to two; categories with one resolver four to two. It continued: forced reassignments thirty-one to twelve; documents used by a second person nine to twenty-six. Nikhil did not need to say he thought so. The bursar read it in less time than the pause had taken.

The Honest Line in the Report

The quarterly report carries one line of knowledge-spread numbers and one line of caveats, and the second line is the one that earns trust. The numbers line is the four counts, this quarter beside last: "Systems only one person can run: 4 to 2. Ticket types with a single resolver: 3 to 2. Tickets that waited for one person: 19 to 12." The caveat line says what moved the numbers that was not the desk's work. A technician left and took two "alone" marks with her; volume in one category fell so its score is not comparable. The finance system had too few tickets to score. A report that gives the good numbers without the caveats is believed once. The format of the page these lines sit on, and the discipline of a number per question, is in The Monthly Report They Will Read.

Report the bad quarter in the same shape, because the bad quarter is where the measures do their most useful work. "Singles: 4 to 4. Forced reassignments: 12 to 15. Cross-training half hours skipped: nine of thirteen" is a line that says the desk had no slack this quarter, in numbers that a manager can act on. It says it without anyone having to describe how the quarter felt. A desk that only reports when the numbers are good is a desk whose reports are read as advertising. A desk that reports the flat quarter with its reason is a desk whose good quarters are believed.

Never let the line become a target. The moment "singles down to zero by year end" is written in a plan, the grid will show zero singles by year end. It will do so by marking people "alone" who have watched a thing once. The measures work because they describe; they stop working the day they are asked to persuade. Report them, discuss them, and let them choose the next quarter's three systems. Leave the targets to things that cannot be gamed by optimism in a room. The bus factor is not one of those things.

A Short War Story: Forty Articles a Quarter at Pennywell Credit Union

Pennywell Credit Union's four-person desk reported one knowledge-sharing number to its board every quarter: the count of knowledge base articles written. That rose from eleven to forty-three over a year and was praised each time as evidence that the desk was reducing its reliance on individuals. The articles were real and mostly good, and almost all were guides for members and staff to the online banking portal. Those were the tickets that came in most and the articles that were easiest to write. The member statement run produced and dispatched statements on the first working day of each month. It had one person who could run it, Agnes, and no article, because it generated no tickets and nobody had drawn a grid. Agnes left for a rival with three weeks' notice in a quarter the report called the best yet. The first statement run after her last day failed on a step nobody had seen. Statements went out eight days late, and the regulator asked why. The desk spent two of those days finding out what the step was. The chief executive had a question at the next board, with the forty-three articles on the page in front of him. How had a number that had tripled described a desk that could not produce a statement? The desk now reports the bus factor by system beside the article count, and the article count is the smaller line. Count what could happen, not what was typed.

The Knowledge Spread Scorecard

The scorecard below is the four measures on one page, as plain text, with the counting rules written in. That way, next quarter's count is taken the same way as this one. Fill it in the week of the audit. Keep the previous quarter's column beside the current one; after four quarters, keep the first and the latest and archive the rest. The honest line and its caveats go to the manager; the per-person block stays on the desk.

KNOWLEDGE SPREAD SCORECARD — ONE PAGE, ONCE A QUARTER
Desk: ______________   Quarter: ________   Taken in the week of the audit by: ________

COUNTING RULES (write once, do not change to improve a number)
  Ticket in the quarter = closed date in the quarter.  Resolver = who fixed it, not who closed it.
  Resolver counts for a category only with 2 or more tickets in it.  Categories under ____ tickets: "not scored".
  "Alone" on the grid = done without help on a dated ticket or rehearsal.
  Forced reassignment = reassignment note contains exactly: "________________________".
  Per-person reading only after a full quarter and at least 12 documents desk-wide.

                                              Last quarter   This quarter   Movement
1. CATEGORIES WITH ONLY ONE RESOLVER              ____           ____          ____
   Which: _________________________________________________________________
   Not scored (too few tickets): _____________________________________________

2. BUS FACTOR BY SYSTEM (from the grid)
   Systems with 1 "alone"                         ____           ____          ____
   Systems with 2                                 ____           ____          ____
   Systems with 3 or more                         ____           ____          ____
   Cells that changed this quarter, with dates: ____________________________

3. FORCED REASSIGNMENTS (search the agreed phrase)
   Total                                          ____           ____          ____
   To the single most reassigned-to person        ____           ____          ____
   By system (top three): _______ ____  _______ ____  _______ ____

4. DOCUMENTS (desk total)
   Written or substantially revised               ____           ____          ____
   Used by a second person (last-used lines)      ____           ____          ____

   Per person (coaching only; never displayed; minimum sample met? [ ] yes [ ] no)
   Name ________  written ___  used by others ___  heavy resolver with no runbook? [ ]
   Name ________  written ___  used by others ___  heavy resolver with no runbook? [ ]
   Name ________  written ___  used by others ___  heavy resolver with no runbook? [ ]

CROSS-CHECK
   Grid says more people can run a system than resolved its tickets?  Which: ______________
   Reason: [ ] no tickets came in  [ ] tickets were passed on  [ ] grid over-claims

THE HONEST LINE (to the manager)
   "Systems only one person can run: ___ to ___.  Ticket types with one resolver: ___ to ___.
    Tickets that waited for one person: ___ to ___.  Documents used by others: ___ to ___."
   Caveats: ______________________________________________________________________
   Half hours skipped this quarter: ___ of ___

NEXT QUARTER'S THREE SYSTEMS (chosen from the above): 1. ________  2. ________  3. ________

Frequently Asked Questions