Measuring the Password Problem Month by Month

The first Tuesday of the month, 10:20, at Oakhurst Veterinary Group. Grace runs the desk for a chain of nine practices. She has the monthly numbers half-built when the operations manager, Tom, leans into the doorway. "Is the password thing fixed yet? You said the reminder went out and the expiry change happened. The number was lower last month, wasn't it?" It was. It was lower the month before, too. Grace is looking at a spreadsheet in which password tickets fell from 140 to 96 to 71 across three months. She is also looking at a calendar in which those three months were June, July and August. In those months, half the practices run on a skeleton rota. Nobody changes a password because nobody is at a desk long enough to be asked. She would very much like to say yes. She says, "Ask me in October."

Measuring the password problem means counting the password and lockout tickets every month under the same rules. It means cutting them by cause and by day, and reading the result against the calendar and the changes you made. That way, you can say what moved, why, and whether it will stay moved. It is the difference between a desk that feels better and a desk that is better. This article is part of our Password & Lockout Tickets series. And it is the last one because it is the one that tells you whether the other five worked.

It covers counting under the same rules every month, the count and the share, and the cut by cause. Next come the Monday peak and the calendar, the spike after a change, and per-technician numbers, which are for coaching. Then it covers the month you can honestly declare the problem solved, the report line, and a short war story. It finishes with the monthly report line to copy.

Count Under the Same Rules Every Month

The monthly count is the census from Why Password Tickets Dominate the Help Desk Queue repeated. And the only thing that makes month two comparable with month one is that the rules did not change between them. Write the rules on the report sheet once. A password ticket is any ticket whose cause was a credential, decided from the resolution note and not the subject line. Count one ticket per contact, with duplicates merged and counted once. Log walk-ups and phone calls as tickets. A ticket belongs to the month it was opened. Those are the same rules the whole library uses for counting anything, set out in Counting Tickets Properly. And the password count inherits them rather than inventing its own.

The first census is slow because it is read from the notes. Every month after it is fast because the cause is recorded at close. The mechanism is a convention, not a feature. It is a field on the desk's form if the tool has one, or a fixed phrase at the start of the resolution note. The phrases are "expired", "forgotten", "locked: phone", "locked: mistyped", "second system: finance", "new starter". The month's count is a filter on that field or a search for those phrases. If you run QuerySolv, the standard volume report gives the month's total for the bottom of the fraction. In that case, the password count still comes from your own cause convention. That is because no help desk report knows why a ticket was raised unless somebody wrote it down at the time. The convention takes technicians five seconds a ticket. Its absence takes the desk lead an afternoon a month.

Resist changing the rules to make a month look better. Write down any change you do make with the month it took effect. A desk that decides in March that unlocks are no longer password tickets will see its number fall in March and will have measured nothing but its own definitions. If the rules must change, recount the earlier months under the new rules. Or draw a line on the chart and treat the two sides as different series. The number is only ever as honest as the rules. And the rules are only ever as honest as the sheet they are written on.

The Count and the Share

Report two numbers every month, the password tickets and all tickets, in that order, "71 of 390". Let the share follow as a word rather than a decimal: "a little under a fifth". The count is what changes when a layer works. The share is what tells you whether the count fell because the problem shrank or because the whole desk went quiet. The whole desk does go quiet in August and over the winter break. A desk reporting only the count will take credit for the holidays. The share without the count hides the other direction. A merger doubles the headcount. Password tickets double with it, and the share stays flat while the desk drowns.

Small desks should be careful with the share altogether. A desk that logs sixty tickets a month will see the share move by several points when four tickets change sides. A month-to-month comparison at that size is noise with a line drawn through it. Report the two counts and say that the volume is too small for a rate, or report quarters, where the counts are large enough to mean something. Nobody has ever been helped by a share quoted to one decimal place from a desk that had thirty-one password tickets. The decimal is a costume the number wears to look like a measurement.

The Cut by Cause

The monthly count is the census lines, repeated as columns, and the lines are where the information is. The total can stay flat while the lockout line falls by half and a new line grows to replace it. In that example, the new line is a system bought in the spring with its own password. Only the cut by cause shows that one thing worked and another thing arrived. Keep the lines the same as the census: expired, forgotten, locked with a device involved, locked by mistyping, changed and broken, second system by name, new starter, and not recorded. Add a line when a new cause appears and keep it. Never fold two lines together to tidy the sheet, because the fold is where next year's question hides.

Each line has its own expected shape, and reading the lines against their shapes is most of the skill. The expired line is a wave on the expiry cycle and should fall only when the policy or the staggering changes. The locked-with-device line should fall sharply within a month of the reminder going out and stay down. And a rise in it means the reminder has stopped going out, usually because the person who sent it left. The forgotten line falls when self-service enrolment rises and not before. The second-system line falls per system as each is connected to the directory, and rises per system as each new one is bought. The not-recorded line measures the desk's notes and should be small and shrinking. If it grows, the convention is slipping and the whole sheet is getting softer. There is a method for following one line over time and for telling a real change from a wobble. It is the subject of Trends and Seasonality. The password sheet applies it line by line rather than reinventing it.

The Monday Peak and the Calendar

The Monday peak is the cleanest single measure of the stale-credential problem, and it is worth its own line. Count the password tickets opened on each Monday of the month, average them, and do the same for the other weekdays. The ratio of the two is how much worse Monday is than an ordinary day. At Oakhurst before the reminder, Mondays averaged eleven password tickets against five on other days, a ratio a little over two. After the reminder had been in the reset reply for two months, Mondays averaged six against four. The ratio had fallen to about one and a half. The weekend had not changed. The phones had been updated on the Friday instead of locking the account on the Monday.

The calendar needs annotating on the sheet, not remembering. Mark the Mondays after public holidays, the first week after the long break, and the expiry waves. Also mark the first week of term or the first week of the season on desks that have one. Each of those produces a peak that is nobody's fault. A reader six months later will otherwise take that peak for a bad month. Mark the quiet months too. A low August is the roster, not the plan. A sheet with the calendar on it reads itself. A sheet without it needs the desk lead in the room to explain every bump, and desk leads leave.

The Spike After a Change

Every change the organisation makes to its credentials produces a spike in password tickets. The spike is expected, bounded and temporary. That means it should be written on the sheet before it arrives rather than explained after. A forced password change after an incident produces a spike of changed-and-broken lockouts the following week. A new expiry policy produces a spike on the first wave under the new period. A new system with its own password produces a new line that starts high and settles. A laptop rollout produces a spike of forgotten passwords on the new machines, and a spike of lockouts on the old ones that were left on. An onboarding wave produces new-starter tickets. Each of these is the cause doing what causes do. And the measurement's job is to show that the spike was the size expected and faded within a month or two.

The spike that does not fade is the information. A forced change produced forty lockouts in week one and twenty in week two, as expected. Then it produced fifteen in week six, which is not the fade. It is a set of devices that nobody found. The lockout diagnostic sheets for those fifteen will say which. A new system's line settled at twenty a month instead of five. That is not the novelty wearing off. It is a system that expires its passwords monthly and has to go to the top of the single sign-on list. Annotate the spike, give it an expected size and an expected end, and read the sheet against both. A spike with a date on it is a measurement. A spike with no date on it is a grievance in a chart.

Per-Technician Numbers Are for Coaching

Some password measures can be cut by technician. The useful ones are about the quality of the work rather than its quantity. They include the share of a technician's reset tickets that carry a verification line and the share that record a cause. They also include the share where the device warning was given and the share of their unlocks that recurred within the week. These numbers are read for coaching, never for ranking, and never from fewer than forty or so comparable tickets. A technician with twelve resets in a month and one missed verification line is not a problem. They are a small sample. A technician with a hundred and twenty resets and thirty missed cause lines has been given a convention nobody explained. The fix is a conversation, not a league table.

The number of resets a technician did is not a measure of anything except who answered the phone. Resets arrive by phone and walk-up. They go to whoever is nearest. A technician who did eighty resets this month was the one sitting by the door. Do not report resets per technician as throughput. Do not compare it between people, and do not put it on a dashboard. The person at the bottom of that list will learn to be further from the phone. The queue will learn to wait. The measures worth cutting by person are the ones about doing the reset properly. And the right response to a low one is to read the person's tickets with them and find out which step they were never shown.

The Month You Can Declare It Solved

"Solved" does not mean zero, because people forget passwords and mistype them. A desk that has removed every removable cause still has a residual that is mostly those two. Solved means small and understood. The count sits below a share the desk chose in advance. Every remaining line has a known cause the plan has decided not to chase, and no line is growing. To declare it, the sheet should show three consecutive months inside the target, under the same rules. Those months should include at least one busy month, one Monday after a public holiday, and one expiry wave. The Monday ratio should be flat and the not-recorded line small. Three quiet months in summer do not count. Grace knew this, which is why she said October.

After the declaration the count continues, at a lower cost. The sheet now has a baseline, so a new line will stand out against it the month it appears. Move the password line from its own section of the monthly report to a single row in the standing numbers. Keep the cut by cause in the sheet rather than in the report. What stops is the project, not the counting. I have watched a desk stop counting the month after it declared victory. And I have watched the laptop image lose the self-service link. I have watched the tickets climb back for half a year before anybody noticed. That is the story told at the end of the self-service series and the reason this paragraph exists.

The Report Line

The monthly report to leadership gets one line and one chart for the password problem. And the line carries the two counts, the share in words, the largest cause, and the change since last month with its reason. "Password and lockout tickets: 71 of 390 this month, a little under a fifth, down from 96; the largest cause is now forgotten passwords; the fall is mostly the summer rota and is not yet confirmed." The last clause is the one that earns trust, because it is the one a reader could not have known. A report that says "not yet confirmed" in August is believed when it says "confirmed" in October. The chart is the monthly count by cause, stacked, with the calendar marked. The report the chart sits in has a format: one page, three numbers, the thing that changed and the thing that is being asked for. That format is the subject of The Monthly Report They Will Actually Read. This line is written to fit that format.

When the line is asking for something, the layer it is asking for and the slice it would remove are in The Layered Plan That Makes Password Tickets Rare. In that case, the ask is in hours: "the lockout line was 48 last month; the reminder costs an afternoon and should remove most of it". When the line is reporting a result, the result is the line that fell and the layer that made it fall, and nothing else. A leader who is told that the reminder removed thirty lockouts a month will fund the next layer. A leader who is told that "password tickets are improving" will ask whether the desk is still needed. Be specific about the line, the cause and the layer. The number is a story with the plot removed, and the plot is what gets the next layer paid for.

The table below is Oakhurst's sheet for the six months that included the summer, with the fictional desk's numbers. So the shape of a sheet that reads itself is visible. The June to August fall was the rota. The September rise was the rota ending and the expiry wave landing together. October was the first month the desk believed, and November confirmed it. The lockout line fell in June, when the reminder started. It stayed down through September when everything else rose. That is what a layer that worked looks like on a sheet.

A Short War Story: The Quiet Summer at Stonebridge Insurance

At Stonebridge Insurance, the desk lead, Owen, reported in August that the password problem was solved. His evidence was two months in which password tickets had fallen by nearly half. The operations director, pleased, cancelled the planned conversation with the security owner about staggering the expiry waves on the grounds that it was no longer needed. September brought the end of the holiday rota and the expiry wave in the same week. The count doubled. The Monday after the autumn bank holiday was the worst day the desk had logged. The director, who had told her own board the problem was solved, asked Owen what had gone wrong. Nothing had gone wrong. Nothing had gone right either, and the two low months had been the calendar. The cost was a quarter's delay on the staggering decision. When that decision was finally made, it removed most of the September spike. The cost was also a year in which every number from the desk was received with a raised eyebrow. A month is a weather report. Three months across a holiday and an expiry wave are a climate, and only the climate gets reported as solved.

The Monthly Password Report Line

The sheet below is the one Grace keeps, and it is in three parts. Part one is the rules, written once and never changed without a dated note. That way, every month is comparable with every other. Part two is the month's row, in the same columns as the census. The Monday ratio and the calendar note sit beside the counts. The per-technician quality measures are kept separately as coaching notes. Part three is the line for the report and the declaration test, with its three conditions. That way, "solved" is a box that is either ticked or not and never a feeling in August.

MONTHLY PASSWORD REPORT LINE — RULES, ROW, LINE
Desk: ______________________   Sheet owner: ______________   Rules in force from: ____________

1. THE RULES  (written once; any change dated here: ______________)
   A password ticket is any ticket whose cause was a credential, read from the resolution note.
   Cause recorded at close as one of: expired / forgotten / locked, device / locked, mistyped /
      changed and broken / second system: name / new starter / not recorded
   One ticket per contact; merged duplicates count once; walk-ups and calls are logged.
   A ticket belongs to the month it was opened.  Lockouts are counted; unlocks are lockouts.

2. THIS MONTH'S ROW
   Month: ________   Password tickets: ______ of ______ all tickets
   Expired ____  Forgotten ____  Locked, device ____  Locked, mistyped ____  Changed and broken ____
   Second system: ______________ ____   ______________ ____   New starter ____   Not recorded ____
   Average Monday ____  Average other weekday ____  Ratio ____   Busiest day ______ on ______
   Calendar: [ ] public holiday  [ ] expiry wave  [ ] forced change  [ ] rollout  [ ] onboarding wave
             [ ] quiet month, rota or season     note: ______________________________
   Layer applied this month, if any: ______________   Expected effect on which line: ______________
   Spike expected?  [ ] yes: line ______ size ______ expected to fade by ______   [ ] no
   Coaching notes, kept apart from this row, read only above ____ comparable tickets per person:
      verification line present ____ of ____   cause recorded ____ of ____   device warning given ____ of ____

3. THE LINE FOR THE REPORT
   "Password and lockout tickets: ______ of ______ this month, ______________ (share in words),
    up / down from ______; the largest cause is ______________; the change is because ______________
    and is / is not yet confirmed."
   Asking for: layer ______________, cost ______ hours, expected to remove most of line ______________.

   DECLARATION TEST  (all three, or not solved)
   [ ] Three consecutive months at or below ______ of ______ under these rules
   [ ] Including one busy month, one holiday Monday and one expiry wave
   [ ] Every remaining line understood; no line growing; not-recorded below ____

In October Grace put the line in the report with "believed" where "confirmed" would go, and in November she changed the word. Tom asked whether that meant the password thing was fixed. She said the lockouts were fixed. The expired ones were smaller and staggered. The forgotten ones were now the largest line and were what self-service would be for, if he wanted to see the plan. He did. It was the first time anyone at Oakhurst had asked to see a plan from the desk rather than a number. And it had taken six months of the same sheet, filled in the same way, to get there.

Frequently Asked Questions