Keeping Dashboards Honest: The Stale Tile and the Monthly Check

Monday, 08:20, at Pinecroft Academy. Greg leads the school's three-person desk. The overdue tile on the shared view has read zero for three weeks. He has mentioned this, modestly, in two heads-of-department meetings. At 08:25 Helen, who runs the science department, phones about the data-logging software in the physics lab. It has not worked since a ticket she raised a fortnight ago. "Is anyone looking at it?" Greg finds the ticket. It is open, unowned, and has no due time at all. Three weeks ago the desk replaced its request form, and the new form does not ask for a priority. No priority means no service-level target. No target means the ticket can never be overdue. The tile had been telling the truth about what it counted. It had just stopped counting the thing everyone thought it did.

An honest dashboard is one on which every figure means what the people reading it believe it means, today, and the word "today" is the hard part. A figure that was true when the tile was built becomes untrue when the data stops arriving. The same happens when a form or a schedule or a routing rule changes under it, or when a filter quietly leaves something out. In every one of those cases the tile goes on showing a number with the same confidence as before. This article is part of our Team Dashboards series. It is about the four ways a dashboard lies, how to catch each one, and the monthly check that keeps the figures worth reading.

It starts with the four ways a dashboard lies: the stale tile, the number nobody questions, the definition that drifted, and the quiet exclusion. Then comes cross-checking against the aging report, with a table of symptoms and tests. The monthly check itself, a short war story, and the dashboard review sheet to copy finish the article.

The Four Ways a Dashboard Lies

A dashboard does not lie by showing a wrong number. It lies by showing a right number for a question nobody is asking any more. It does this in four ways that have different causes and the same symptom, which is a figure that looks fine. The stale tile shows yesterday's number because the data stopped updating. The number nobody questions has been the same comfortable value for so long that it has stopped being checked. The drifted definition counts something slightly different from what the label says, because something upstream changed. The quiet exclusion counts a subset, because a filter leaves out a department, a status or a channel that it should include.

The four are listed in the order they are usually found, which is the reverse of the order they do damage. A stale tile is caught within a day by anyone who notices the number has not moved. A quiet exclusion can run for a year, because the figure it produces moves, changes colour and behaves exactly like a true one. The tickets it leaves out wait somewhere the dashboard never looks. Pinecroft's zero was a drifted definition, and it had been drifting for three weeks while being praised in meetings. The praise is a symptom. A tile that is always good news is either a well-run desk or a broken tile, and the only way to know which is to check.

The Stale Tile

A stale tile is one whose data has stopped arriving, so that it shows the state of the desk at the moment the feed broke and goes on showing it. The causes are dull. The browser tab on the wall screen lost its session and is showing a cached page. The account that the display logs in with had its password expire. The tool was restarted and the display was not. A report that fed the tile was scheduled by someone who has left. The symptom is a number that is right, for a while ago, and the danger is that a stale zero looks identical to a live one. I once ran a desk whose wall showed zero overdue for most of a Thursday. The desk found this reassuring until somebody noticed that the clock in the corner also said Wednesday.

The defence is a time on the screen. Every dashboard, on a wall or in a tab, shows the time its figures were last updated, small and in a corner. The lead's fixed reads begin with it. A time more than a few minutes old is the first thing said aloud, before any count. Some tools show this time themselves; some do not. A desk whose tool does not can put a clock beside the screen and compare, or make "reload the page" the first step of every read. Whichever you have, write it on the specification. A display that announces its own staleness is honest and one that does not is a liar with good posture.

Test the feed on purpose, monthly, with a test ticket. Raise a ticket from a personal address and watch the unassigned count move. Claim it and watch the count move back. Resolve it and close it with a note saying it was a test. If any count does not move within the time the tool is supposed to take, the tile is stale and the cause is found that morning. The test ticket costs three minutes and is the only way to know that the feed works. Otherwise you are believing that it does because the numbers look plausible. Plausible is what stale numbers look like. That is what makes them stale rather than plainly wrong.

The Number Nobody Questions

The number nobody questions is the one that has been comfortable for so long that it has become part of the room. Zero overdue for a month. Unassigned always under three. Waiting on us never above one. Each of these marks either a desk that has got its routines right or a tile that is counting something smaller than its label. From the outside the two are indistinguishable. The desk at Pinecroft had earned its zero for the first week, when the old form was still in use. For the next two weeks it had inherited it.

The cure is the hand count, done quarterly for every tile and immediately for any tile that has been good news for longer than feels plausible. Open the list behind the figure and count it by hand, with the definition in front of you. Count every open ticket past its target, from the whole list, not from the filter. If the hand count matches the tile, the tile is honest and the desk is good, and both deserve saying. If it does not, the difference is the lie, and the next two sections are about where it came from. The hand count for a desk of ten or fewer takes about twenty minutes a tile and is the most boring twenty minutes in this series. It is also the only one that catches a tile that looks right.

Ask the question out loud, in the team meeting, once a quarter: which of our figures would we be most embarrassed to find was wrong? The answer is usually the one that has been best for longest, because that is the one that has been quoted upward most often, and the desk knows it. A figure that has been quoted to a director three times is a figure that has acquired a stake in staying the way it is. Count it first.

The Definition That Drifted

A definition drifts when something upstream of the tile changes what the tile is counting without anyone changing the tile. The overdue count depends on a service-level target, which depends on a priority, which depends on a form field. Change the form and the chain breaks at the top while the tile at the bottom goes on reporting zero with perfect sincerity. The unassigned count depends on which departments are included; add a department for a project and its tickets are unassigned somewhere the count does not look. The waiting-on-us count depends on which statuses are "active". Add a status called In Progress alongside Open and the tickets in it may or may not be counted. It depends on who built the filter and what they knew. Every tile has a chain above it, and every link in the chain is something somebody can change on a Tuesday afternoon for a good reason.

The defence is a written definition for every tile, kept with the dashboard's specification, that names each link in the chain. For example: this tile counts tickets where the status is one of these, the department is one of these, and the owner is empty. The target is calculated from this priority on this schedule. The discipline of defining a metric so that two people get the same number from it, with worked examples of where the definitions wobble, is the subject of Defining Each Metric Exactly. The dashboard inherits those definitions and adds a line for the source. A definition in writing is a definition that can be checked against the form, the schedule and the status list. That check is a line on the monthly review sheet.

Make a change to anything upstream trigger a look at the tiles, and write that rule where the people who change things will see it. A new form, a new department, a new status, a new priority, a new service-level schedule or a new routing rule each gets recorded. Each gets a one-line entry in the desk's change notes, and the line ends "dashboard checked: yes". It is a small habit and it is the one that would have caught Pinecroft's zero on the afternoon the form went live. Forms are changed by people thinking about forms. Nobody changing a form is thinking about a tile in another room.

The Quiet Exclusion

A quiet exclusion is a filter that leaves something out, and it is the hardest of the four to catch because the figure it produces is alive, moving and partly true. The unassigned count that covers the IT department and not the facilities department is right about IT. Suppose the overdue count covers tickets from the web form and not the ones that arrived by e-mail. This happens because the e-mail channel assigns a different default priority with no target. That count is right about the form. The waiting-on-us count that covers Open and not In Progress is right about Open. Each is a correct count of a subset wearing the label of the whole. A desk can run on one for a year while the excluded tickets wait in the part the dashboard never looks at.

The defence is the counting test, weekly, which the queue article introduced and which this article makes a duty. The total number of open tickets in the whole list must equal the number reached by adding the dashboard's figures. Allow for tickets that appear in two. If the whole list says thirty-one and the figures account for twenty-two, nine tickets live somewhere the dashboard does not look. Then the morning's job is to find where. The usual places are a department nobody staffs, a status nobody filters, a channel with a different default and an owner who has left. These are the same four places tickets hide from the standing views, described in Keeping Tickets From Hiding. The dashboard inherits the hiding places. It does not cure them.

Check the exclusions in the other direction too, because a filter can include what it should not. A count of overdue tickets that includes resolved ones awaiting closure inflates the figure. So does an unassigned count that includes tickets in a closed department. An inflated figure is a tile the desk learns to discount, which is a different kind of lie with the same ending. The hand count catches both directions at once, because the person counting is holding the definition and the list. Anything on one and not the other is a finding either way.

Cross-Checking Against the Aging Report

The dashboard says what is happening now. The aging report says how long every open ticket has been waiting, in bands. The two should agree in a way that is easy to test. If the dashboard's overdue count is zero and the aging report shows tickets older than the desk's longest target, one of the two is wrong. It is nearly always the dashboard. The aging report is built from the ticket's creation date, which nothing upstream can change. The dashboard's overdue count is built from a target, which several things can change. Greg had not opened Pinecroft's aging report in a month because the dashboard was green. It showed five tickets older than two weeks on a desk whose longest target was five working days. How to read the report, and what each band is telling you, is the subject of Reading an Aging Report. The cross-check here is one line of it.

Do the cross-check monthly, as part of the review, and immediately whenever the dashboard has been quiet for long enough to be suspicious. It takes five minutes: open the aging report, find the oldest band that should be empty if the dashboard were right, and see whether it is. A desk that does this has a second witness to its dashboard that does not share the dashboard's assumptions. A figure with a second witness is one you can quote upward without the small cold feeling that comes with quoting a figure you have not checked. The table below gathers the symptoms this article has described, with the likely cause, the test that confirms it and the fix.

The Monthly Check

The monthly check is one person, about half an hour, the review sheet at the end of this article, and a fixed day. It is the whole of this article as a routine. The person rotates, because a dashboard checked by the person who built it is checked by someone who already believes it. The half hour goes on the test ticket, the hand count of one tile chosen in turn, the counting test, and the aging cross-check. It also covers reading the definitions against the current form, schedule and status list, and looking at whether anything upstream changed in the month. The sheet is filled in and filed with the previous ones. One line goes in the team chat: "Dashboard checked, all tiles honest" or "Dashboard checked, overdue tile fixed, see sheet".

When a tile fails, take it down until it is fixed, rather than leaving it up with a note. A tile that is known to be wrong and still displayed teaches the desk that the screen is sometimes wrong. A desk that has learned that stops reading the screen, including the five tiles that were right. A blank where the overdue count was is uncomfortable for a day and is honest. An overdue count everyone knows to ignore is comfortable for a month and is the end of the dashboard. If you run QuerySolv, the event history on every ticket already records when it was created, claimed, replied to and resolved. In that case, the hand count behind a tile is a filter rather than an archaeology dig. On any tool, the hand count is the step that must not be skipped, however the figures are reached.

Report the corrections upward yourself, before anyone else finds them. A desk lead who says "the overdue figure I quoted last month was wrong, because the new form dropped the priority field; the true figure was five, the fix is in place, and here is how we check it now" has lost a little and gained a great deal. Every figure quoted after that carries the knowledge that the desk checks its own numbers. A lead whose wrong figure is found by a head of science has lost the figure and the next six. Honesty about a tile is cheaper on the day the tile is found than on any day after.

A Short War Story: The Green Tile at Westmere Council

At Westmere Council, a local authority with a six-person desk, the overdue tile on the wall had read zero for six weeks. The desk lead, a man called Ayo, had quoted it in a service review as evidence that the new service-level targets were working. The targets were calculated on a business-hours schedule. An administrator had been asked to add a public holiday to that schedule. By selecting the wrong range, the administrator had marked every day for the rest of the year as non-working. No ticket's clock had advanced since. Twenty-three tickets were past their target by the time a councillor wrote to the chief executive about a planning officer's broken mapping software. That ticket had been open for nineteen working days and showed a target date in the following year. The cost was a morning of the chief executive's attention and a written apology to twenty-three requesters. It also cost about fifteen hours of catching up on tickets that had been quietly waiting. And a service review in which the previous service review was discussed at some length. The desk added the aging cross-check to a monthly review the same week, and put "schedule changed" on the list of upstream events that trigger a look at the tiles. A tile that has been green for six weeks has either earned it or been excused from the clock. Ask which.

The Dashboard Review Sheet

The sheet below is one page, filled in once a month by whoever's turn it is, and filed with the previous months' so that a tile's history can be read. Section A is the stale test. Section B is the hand count of one tile, in rotation, with the next month's tile named at the bottom so that nobody has to choose. Section C is the counting test. Section D is the aging cross-check. Section E reads each definition against the things that could have changed it. Section F records what was fixed, what was taken down, and the line that went to the team. Nothing on the sheet needs a report; every step is a filter, a list and a count.

DASHBOARD REVIEW SHEET — MONTHLY, ONE PAGE
Desk: ______________________   Dashboard: ______________________   Reviewed by: ______________   Date: ____________

A. THE STALE TEST
   Last-updated time shown on the screen: [ ] yes   [ ] no, so we reload before every read
   Test ticket raised from a personal address at ____:____
   Unassigned moved within the expected time [ ]   Claimed; count moved back [ ]   Resolved and closed with a test note [ ]
   Feed verdict: [ ] live   [ ] stale, cause: ______________________   fixed [ ]

B. THE HAND COUNT (one tile per month, in rotation)
   Tile counted this month: ______________________
   Written definition (statuses / departments / owner / target rule): ________________________________________
   Tile showed: ____   Hand count from the whole open list: ____   Difference: ____
   Tickets on the list and not the tile: ____   Tickets on the tile and not the list: ____
   Cause of any difference: ______________________________________   fixed [ ]
   Next month's tile: ______________________

C. THE COUNTING TEST
   Total open tickets in the whole list: ____
   Sum of dashboard figures, less tickets counted twice: ____   Difference: ____
   Where the uncounted tickets live (department / status / channel / owner): ______________________   fixed [ ]

D. THE AGING CROSS-CHECK
   Longest service-level target on the desk: ____ working days
   Open tickets older than that on the aging report: ____   Overdue tile shows: ____
   If the report shows more than the tile: cause ______________________   fixed [ ]

E. UPSTREAM CHANGES THIS MONTH (each one triggers a look at every tile)
   [ ] form changed   [ ] department added or closed   [ ] status added or renamed   [ ] priority or target changed
   [ ] service-level schedule changed   [ ] routing rule changed   [ ] channel added   [ ] someone left with tickets
   For each ticked: tiles checked by ______________ on ____________

F. OUTCOME
   Tiles honest: ____ of ____   Tiles fixed: ______________________   Tiles taken down until fixed: ______________________
   Corrections reported upward on ____________ by ______________
   Line for the team chat: "Dashboard checked: ____________________________________________"
   Next review: ____________   Next reviewer: ______________

Greg ran the sheet at Pinecroft on the Monday afternoon, with Helen's ticket as the first entry in section B. The hand count found five open tickets with no target where the tile had shown zero overdue. He fixed the form on the Tuesday and added "no target set" to the overdue count's definition. That means a ticket with no clock counts as late rather than as never. On the Thursday, he told the heads of department that the zero he had mentioned twice had been wrong. It took thirty seconds and was received with less interest than he had feared. Helen's data loggers were working by Wednesday. The tile has read zero on most days since, and the desk now knows which kind of zero it is.

Frequently Asked Questions