Measuring Security Culture From the Help Desk

Month-end, 16:30, at Larkspur Accountancy. Grace, the managing partner, stops at the desk on her way out and asks Farah, who runs it with two others, whether the security training is working. It is not a hostile question; the firm paid for a programme, the insurer asked about it, and Grace would like to know. Farah has the answer open already, because she looked this morning. This month the desk logged twenty-three reports of suspicious messages. Two people clicked and both rang within ten minutes. The technicians gave sixty-one one-sentence lessons inside ordinary tickets. The same month a year earlier: six reports, one click, found on day eight.

Grace asks where those numbers came from. The training platform? No, says Farah, the queue. The platform can say who finished the module. The queue says what people did on a Tuesday when a convincing e-mail arrived, which is the only thing anyone actually wanted to know. Grace writes the four numbers on the back of an envelope and takes them to the partners' meeting, which is more than the module's completion rate ever achieved.

Measuring security culture from the desk means counting the behaviours that show whether people report, how fast they report, and whether the desk is teaching. It uses tickets the desk already has. It never means counting people. This is the last article in our The Helpdesk's Role in Security Culture series, and it closes the loop. The lessons, the signals, the blameless culture and the working agreement from the earlier articles each leave a mark in the queue. This article is about reading the marks honestly and reporting them without a single name.

It covers what you are measuring and what you are not. Next come reports in, time to report, time to acknowledge, lessons delivered, and the trend rather than the month. Then it covers never shaming and never per person, the line in the leadership report, and what the numbers cannot prove. A short war story and the security culture line to copy finish the article.

What You Are Measuring, and What You Are Not

Culture is behaviour, and behaviour leaves tickets. A person who rings about an odd e-mail has made a decision, and the ticket is the record of it. A person who rings two minutes after clicking has made a braver one. The ticket's opening time against the time of the click records exactly how brave. A technician who says the one sentence and writes "lesson given" in the note has taught, and the note is the proof. None of these required a survey. They are side effects of the desk doing its job, which is why the desk is the right place to count them.

Four numbers carry the story, and the fifth is their direction over time. Reports in: how many people told the desk about something odd. Time to report: how long between the thing happening and the desk hearing. Time to acknowledge: how long the desk took to reply to the reporter, which is the desk's own contribution to whether people report again. Lessons delivered: how many tickets carried the one sentence. Each is defined precisely below, with a worked example from Larkspur, and each is reported as counts beside any rate.

What the desk cannot measure is whether people care, whether the module was good in the abstract, or whether the organisation is "secure". The honest report says so. The desk measures what it can see, which is what people did. If the organisation wants to know what people believe, that is a survey and belongs to somebody else. If it wants to know whether a given control works, that is the security function's. The desk's numbers are narrow, which is what makes them trustworthy.

Reports In: The Number That Should Go Up

Reports in is the count of tickets in the month where a person told the desk about something that might be a security matter. That might be a suspicious e-mail, an odd sign-in alert, a found device, a call that felt wrong, or a click. The counting rules, including what to do with three reports of one e-mail and with the technicians' own reports, are set out in Measuring Reports and Clicks Without Shaming Anyone. This article uses them rather than restating them. The tag from the signals sheet makes the count one search.

The number should go up, and that needs saying to leadership before they see it. That is because a rising count of "security tickets" reads, to a partner, like things getting worse. It is the opposite. Larkspur's six reports a year ago were not six suspicious e-mails; they were the six people out of perhaps sixty recipients who bothered to tell anyone. Twenty-three this month is not more phishing; it is more people choosing to speak. The honest caveat is that more phishing would also raise the count. So the number is reported beside the share that turned out to be real: "twenty-three reports, of which nine were genuine attempts". That separates the organisation's attention from the attackers' effort.

Report the count, not a rate, until the volume is large enough that a rate means something. At Larkspur's size a difference of three reports is the difference between two people being away and nobody being away. A rate to one decimal place would be a decoration. Counts, the share that were real, and the direction over a quarter are enough. A desk that reports "up from six to twenty-three over the year" has said something a partner can repeat. A desk that reports a percentage has started an argument about the denominator.

Time to Report

Time to report is the interval between the thing happening and the desk hearing about it. It is the number that most directly measures whether blame has been removed, because a person who expects to be blamed waits. The thing happening is the click, the odd sign-in, or the message's arrival. The desk hearing is the ticket's opening time. For a click, the event time comes from the person, asked once and kindly as part of the first-hour runbook, with the message's received time as the floor if they are unsure. For a suspicious e-mail that was not clicked, the received time is the event and the forward or the call is the report.

Report the median and the longest, never the average, because the long tail is the finding. Larkspur's two clicks this month were reported in four and nine minutes. A year ago, the one click was reported after eight days. That single number was the whole of the firm's security culture at the time. A median of twenty minutes with one report at six days tells a partner exactly where to look. An average of a day and a half tells them nothing. Where volumes are small, bands do the job: reported within the hour, same day, or later. The count in each band is the report.

Watch this number hardest in the month after a bad incident, because that is when it moves. If the organisation handled the incident without blame, time to report holds or falls. If somebody sent the all-staff e-mail from Security Without Blame, it climbs for a quarter. In that case, the climb is the cost of the e-mail, visible in the queue to the day.

Time to Acknowledge

Time to acknowledge is the desk's own number: the interval between a security report arriving and a human reply reaching the reporter. Its arrival is the moment they learn that reporting was welcome. The reply itself, and why it goes out within the hour whatever the triage finds, is the subject of The User-Reported Suspicious Email. The measurement is simply the response time on tagged tickets, which most tools report. If you run QuerySolv, the ticket's event history holds the opening and first-reply times and the standard response-time report can be read for the tagged tickets. In that case, this number is a filter rather than a stopwatch.

This number belongs in the same line as the others because it explains them. A desk that acknowledges reports in twenty minutes will see reports rise. A desk that acknowledges them the next day will see reports fall, however good the module. When reports in goes flat, the first thing to check is not the users but this. It is the one number in the line the desk controls entirely, and the one it is most tempted to leave out.

Lessons Delivered

Lessons delivered is the count of tickets in the month whose resolution note carries the "lesson given" line, or the agreed tag, from Every Ticket a Teachable Moment. Beside it goes the count of eligible tickets, the ones whose type is on the lines sheet: resets, lockouts, found devices, shared-account requests, remote sessions, new-starter setups. Larkspur's month was sixty-one lessons across about a hundred and forty eligible tickets. Farah reports that as "sixty-one of about one hundred and forty", not as a percentage, for the same reason as before.

The share tells the desk whether the habit has formed. Below about a third of eligible tickets, in my experience, the lines are a thing one technician does. Above two thirds, they are a thing the desk does. Between is a desk that is building the habit and should keep going. These are the bands one desk settled on, not a standard. The right band for yours is the one where the number stops moving once the sheet is on the wall. The count is read for the desk as a whole. It is never ranked by technician. Where a desk lead looks at one person's share for coaching, it is read on the coaching terms set out in Satisfaction by Category, and Carefully by Agent. In that case, use the same minimum sample, and never show it on a wall.

Pair lessons with reports from the same people over the following months, if the tool makes it easy. That is the closest the desk gets to seeing the lesson land. A reset ticket with "lesson: ring us first" in March and a suspicious-e-mail report from the same requester in June is the mechanism working once. It proves nothing on its own and is the kind of thing a partner remembers. (Dev at Harbourview Clinic, from the first article in this series, is that pairing with a name on it.)

The Trend, Not the Month

A month is weather; the year is climate. The four numbers are only worth reading as a line across at least twelve months. Reports rise in a month when a convincing campaign hits the whole firm and fall in August when half of it is away. Neither movement says anything about culture. The way to read a monthly count against its own seasons, and to tell a step from a wobble, is in Trends and Seasonality. The security line simply applies it to four small counts instead of one large one.

Mark the chart at the things you changed, because the steps are the evidence. The month the lines sheet went on the wall. The month the no-blame statement was signed and said aloud. The month of the bad incident, and the month of the all-staff e-mail if there was one. The month a simulated phishing run went out, which produces a spike in reports that is real behaviour and should be labelled as a test so that nobody reads it as an attack. A chart with four marks on it and a line that steps at each one is the most persuasive thing the desk will ever show a partner. It costs a pencil.

Small desks report quarters. Three months of counts is enough to see whether time to report is falling. A quarterly paragraph is read where a monthly line with tiny numbers is doubted. The monthly line still gets written, for the desk and the security contact, so that the quarter is a sum rather than a memory.

Never Shaming, Never Per Person

The numbers are counted for the organisation, and at most by department where the department is large enough that no count points at a person. The line never, in any report to anyone, names an individual. A per-person count of clicks is the single most damaging document a desk can produce, for the reasons the third article in this series sets out. It leaks, it ends up in a review, and from that day the reporting stops. The desk does not keep one, even privately, even for curiosity. The counts it keeps are of behaviours, and a behaviour has no name.

The department cut needs care even when it is allowed. A team of three with "two reports, one click" is a list of names with the names removed, and everyone in the building can put them back. Cut by department only above a size where that is impossible. Report the direction rather than the count where the count is small. Never show the cut in a meeting where the managers of those departments will compare them. The purpose of the cut is to find the team nobody has briefed, so the desk can go and brief them. It is not a league table. The moment somebody tries to use it as one, the desk withdraws the cut and reports the total.

The desk will know names, because it did the resets, and it will be asked. The answer is the one from the working agreement: what happened to the account is in the ticket. The security function decides who needs to know. The same principle governs any screen the desk puts on a wall. A security number on a team dashboard is a total with a direction and nothing else, on the terms set out in Dashboards That Motivate Instead of Surveil. If the organisation runs simulated phishing, the desk does not hold the per-person results and does not ask for them. Nothing the desk reports should make anyone in the building sorry they rang.

The Line in the Leadership Report

Security culture gets one line in the monthly report to leadership. The format of that report, what goes above the fold and what leaders actually read, is the subject of The Monthly Report They Will Read. The line is four counts and a direction: "Security: 23 reports (9 genuine), up from 14 last quarter's monthly average; both clicks reported inside ten minutes; reports acknowledged in a median of 18 minutes; 61 lessons given in about 140 eligible tickets." Then one sentence of what the desk did about it and one of what it will do next. Then stop.

Leaders ask three questions about the line, and the answers are prepared. "Why are reports going up?" Because more people are telling us, and here is the share that were real. "Who clicked?" The account is handled and the person is in the ticket. We report behaviours, not names, and that is why people ring in minutes. "Is the training working?" Reports are up and time to report is down since the lines and the statement went in. We cannot prove the module caused it. Here is the chart with the marks on it. The third answer is the honest one, and it is more persuasive than the claim it declines to make.

The quarterly version is a paragraph with the chart, and the yearly version is the chart with its marks and three sentences. A partner who has seen the line every month for a year will read the chart in ten seconds and ask what the desk needs. That is the moment the headcount case, or the case for a security retainer, is made. The line earns that moment by being the same shape every month and never once containing a name.

What the Numbers Cannot Prove

The numbers show that reports rose and time to report fell after the desk changed what it did. They do not prove that the desk's changes caused it, and the report never claims they did. A new campaign, a news story, a colleague's bad week in another firm: all of these move the counts. A desk that claims credit for every rise will be blamed for every fall. The honest sentence is "reports rose and time to report fell in the months after the lines sheet and the statement were introduced", and it is enough.

A month with no clicks proves nothing, and a desk should say so before a partner says "so the problem is solved". It usually means no convincing e-mail reached the firm that month, which the attackers decide. A year of many reports, fast reporting and quick acknowledgement proves something real about readiness. In that case, when the convincing e-mail comes, the firm will hear about it in minutes rather than days. That is the claim the numbers support, and it is the one worth making. Security culture is not the absence of clicks. It is what happens in the ten minutes after one.

A Short War Story: The Spreadsheet at Ashgrove Housing

Owen, who ran the desk at Ashgrove Housing, built a spreadsheet with the best of intentions. It had one row per employee, a count of security reports made, a count of clicks, and a "security score" that subtracted one from the other. That was so the security committee could see who was engaged. He saved it in the shared managers' folder so the committee could find it. A tenant-services manager found it first and used a row in a performance review. Word of that reached the rest of the team inside a week. Reports from tenant services, which had been the best-reporting team in the organisation, fell to zero for five months. The next click in that team was hidden for nine days and took about thirty hours of desk and contractor time to clean up. Owen deleted the spreadsheet, apologised to the team in person, and rebuilt the monthly line as four totals with no names. That is the version Ashgrove still uses. The lesson he wrote at the top of it was that the per-person list had cost more than any click it was meant to prevent. It also said that the desk counts behaviours, never people.

The Security Culture Line

The sheet below is the format Ashgrove uses now. The first section is the line itself, in the shape it appears in the leadership report. The second is the counting rules, written once and dated, so that next month's line is a reproduction rather than an argument. The third is the quarterly version, and the last is what never appears, which is the most important section on the page. Fill the blanks from the queue, keep the shape the same every month, and date the rules.

THE SECURITY CULTURE LINE — MONTHLY, FOR THE LEADERSHIP REPORT
Desk: ______________________   Month: ____________   Rules dated: ____________   Prepared by: ______________

THE LINE (one line, same shape every month)
  Security: ____ reports (____ genuine), [up / down / level] from ____ last quarter's monthly average;
  ____ clicks, all reported inside ____ [minutes / hours / the day]; reports acknowledged in a median
  of ____ minutes; ____ lessons given in about ____ eligible tickets.
  What the desk did this month: ________________________________________________________________
  What the desk will do next month: ____________________________________________________________

THE COUNTING RULES (written once, changed only with a date)
  Reports in ......... tickets opened by a person this month carrying the tag ____________;
                       several reports of one message count once each; technicians' own reports noted separately
  Genuine ............ reports the security contact closed as a real attempt
  Time to report ..... event time to ticket opened; the person's stated time for a click, the message's
                       receipt as the floor; reported as median and longest, or in bands:
                       within the hour ____ / same day ____ / later ____
  Time to acknowledge  ticket opened to first human reply on tagged tickets; median
  Lessons delivered .. tickets whose note ends "Lesson given: ____" or carries the tag ____________;
                       eligible tickets are the types on the lines sheet: ______________________________
  Trend .............. the four counts by month for at least twelve months, marked at:
                       lines sheet introduced ____ / statement signed ____ / incident ____ / campaign ____

THE QUARTERLY VERSION (one paragraph and the chart)
  Three months of the line summed; the chart with its marks; one sentence on direction; one on what
  the desk needs. No claim that any change caused any movement; "after", never "because".

WHAT NEVER APPEARS
  Any individual's name, count or score, in any report, to anyone, including privately.
  A department cut where the count could point at a person.
  Per-person results from any simulated phishing.
  A percentage where the counts are small enough that three tickets would move it.
  The sentence "the training worked".

Farah's four numbers went into the partners' meeting on the back of an envelope and came out as a standing line in the monthly report, which Grace reads first. Nobody at Larkspur has been named in it. The two people who clicked this month will ring again next time, which is the only outcome any of the counting was for.

Frequently Asked Questions