Measuring Reports and Clicks Without Shaming Anyone

Thursday, 17:05, at Greywell Surveyors. Fiona runs a desk of two for sixty-odd staff across three offices. Alistair, the managing partner, has stopped at her desk on his way out. Last week a quantity surveyor typed his password into a fake document page, reported it eleven minutes later, and the desk had it contained within the hour. Alistair has heard the story and has one question: "Are we getting better at this phishing thing, or just lucky?" Then, before she can answer: "And who was it?" Fiona has fourteen months of tickets in a category called "security: suspicious message". She can answer the first question properly by Monday. The second one she answers now, and the answer is no.

Measuring reports and clicks means counting, each month, how many suspicious messages people reported and how quickly the desk acknowledged them. It also means counting how many turned out to be phishing, and how many people clicked before or instead of reporting. These are desk-wide numbers that show whether the organisation is getting safer. The numbers are never a list of names, and the article's title is not a nicety. A measurement that shames anyone stops the reports it depends on, and a month later measures nothing. This article is part of our Phishing Reports & Security Tickets series, and it is about answering Alistair's first question honestly and his second question never.

It starts with what the numbers are for, then the four numbers, defined exactly. Next come counting reports properly, time to acknowledge, and the click rate as a worked example. After that: the trend over a year, the line in the leadership report, and why there is never a per-person list. It ends with a short war story and the monthly security line to copy.

What the Numbers Are For

The numbers exist to show whether the loop described across this series is working, and the loop has a shape that numbers can see. If people trust the desk, reports go up and then settle. If the desk is doing its part, the time to acknowledge stays short. If the organisation is learning, the share of people who click when a phishing message arrives goes down. In that case, the first report also arrives sooner after the message does. Those four movements, together, are the answer to "are we getting better?", and none of them needs a name attached.

The numbers also do a second job, which is to justify the desk's hour. Acknowledging every report by a person within the hour, sending the outcome message, and writing the monthly note all take time. A desk lead will be asked whether it is worth it. The honest answer is a comparison in hours of the kind in The User-Reported Suspicious Email. A report that arrives early costs the desk under two hours, and a compromise found late costs the organisation days. The monthly numbers are what make that comparison concrete instead of rhetorical.

The third job is to find the silent corners. A department that has never reported anything is not a department that never receives phishing. It is one where nobody knows the address, or somebody once sighed at a report, or the manager handles it "in-house". Counting reports by department, never by person, shows where the channel is unknown, and the fix is a visit and a poster rather than a metric. Zero is the most interesting number on the sheet, and it is always worth a walk to find out why.

The Four Numbers, Defined Exactly

Four numbers, each with a written definition, because a number without its counting rule is an opinion that two people will reproduce differently. The general discipline is to define a metric before counting it; the four below are that discipline applied to this one subject. Write them on one sheet, date it, and count the same way every month whether or not the result flatters the desk.

Two numbers are deliberately not on the table. There is no "reporting rate" as a share of all phishing received, because nobody knows how much phishing was received. The filter catches most, the security function knows some of the rest, and the remainder is invisible. There is no per-person anything, for reasons that get their own section. A fifth number is optional and useful once the first four are routine: the time from a campaign's first delivery to its first report. The security function can supply it because it knows when the first copy arrived. It is the purest measure of the loop, and it falls when the loop works.

Counting Reports Properly

Reports are counted from the category, which is why every report has to become a ticket with the category set, including the spam and the real invoices. The count is the number of tickets in "security: suspicious message" with an open date in the month, and it is a filter rather than a reading. Most tools will produce it from a report on volume by category; a spreadsheet export counts rows where the category matches and the open date is in the month. The general method, and the traps in it, are in Counting Tickets Properly. The trap that matters here is the report that was handled by a reply in the mailbox and never became a ticket. It simply does not exist as far as the count is concerned.

Count campaigns beside reports. Three people reporting the "mailbox almost full" message is three reports and one campaign, and both numbers are true. The three is the organisation doing the right thing; the one is the desk's workload. A month with twenty reports and two campaigns is a different month from one with twenty reports and twelve campaigns, and the leadership line should say which. Linked tickets make the campaign count easy, because each cluster of linked reports is one campaign. Where the tool does not link, a column on the sheet with the campaign's subject line does the same job by hand.

Simulated phishing reports are counted on their own line, never mixed into the real count. A month in which the organisation ran a test and thirty people reported it is a good month for the test. It says nothing about the real messages, and a trend chart with the test month in it has a spike that will be misread by everyone who sees it. If you run QuerySolv, the standard reports on volume by department give the report count without a spreadsheet. This requires reports to be a department of their own. The simulation count is a filter on the subject line in any tool, and a line of its own on the sheet.

Time to Acknowledge

Time to acknowledge is measured from the report arriving to the first reply from a person, in working time. It is reported as a median because the median is the number the typical reporter experienced. An average adds every time up and divides by the count, so one report that waited a whole weekend drags it up. A median is the middle value when the times are sorted, and it tells you what the middle reporter saw. Report the median, and beside it the count of reports that waited longer than the target, because the one who waited four hours is the one who will remember.

The automatic reply does not count, however warmly worded, because the point of the number is whether a person looked. Most tools record the time of the first agent reply on the ticket, and that is the timestamp to use. Where the tool keeps a full history of events on each ticket, the first public reply after the opening is the acknowledgement. The arithmetic is a column subtraction. Reports that arrive outside working hours start their clock at the next working minute. That is the same rule most desks use for any response target. A reports queue with a one-hour acknowledgement target in working time applies that rule to the most time-sensitive ticket the desk receives.

This is the one number in this article that the desk can be measured on. It is the only one that can reasonably be cut by technician, because it is about the desk's behaviour and not the organisation's. Cut it by technician for coaching only, never for ranking, and never on fewer than about forty acknowledgements for one person. Below that, two slow Fridays are the whole story. A desk of two will rarely reach that sample in a month and should read the desk-wide figure instead.

The Click Rate as a Worked Example

The click rate is the number everyone wants and the one most easily misread. Here it is as a worked example from Greywell's fourteen months, which is fictional and offered as a shape rather than a benchmark. A click rate, for one campaign, is the number of people who clicked, typed, opened or replied, divided by the number of people who received the message. The desk knows the clicks from the first-hour tickets and the did-you-click answers. The recipients come from the security function, which can see who the message reached. Where the security function cannot say, the desk reports the count of clicks alone and says so.

Fiona's sheet for one month in the spring reads like this. One campaign, a fake shared-document message, reached thirty-eight people according to the security function. Nine of them reported it, the first within fifteen minutes of delivery. Two clicked, one of whom typed a password. Both were reported, one by the clicker and one found from the sign-in history. Two of thirty-eight is a little over one in twenty. That is how she writes it: "two clicks of thirty-eight recipients, about one in twenty, both contained within the hour". She does not write a percentage with a decimal point. A decimal point invites comparison with a figure from a different organisation counted under different rules, and nothing good follows.

Across the fourteen months, Greywell's per-campaign click share ran between none and about one in eight in the first half of the period. In the second, it ran between none and about one in twenty-five. In a couple of campaigns in each half, no recipient count was available and only the click count was reported. The ranges are the honest form. A single month's figure at Greywell's size rests on one or two campaigns and a handful of clicks. Two clicks either way moves it enormously. The trend across quarters is readable; the month-on-month figure is noise with a story drawn on it. Report ranges, report quarters, and let the direction speak. The direction is what Alistair asked about.

The Trend Over a Year

The year's shape, when the loop is working, looks like this. Reports rise sharply in the first few months after the channel is announced, as people learn the address and discover that a report gets a reply. They then settle at a level that moves with how much phishing arrives. Time to acknowledge drops once the queue has an owner and a target, and then stays flat; a flat line here is success. The click share per campaign drifts down and gets noisier as it gets smaller, because one click of thirty is a bigger share than one of sixty. And the time from a campaign's arrival to its first report falls from hours to minutes, which is the line that most directly shows the race being won.

Fiona's fourteen months showed exactly that, with the usual untidiness. Reports went from two or three a month to between twelve and twenty. One month was at thirty-four because of a simulation that was then moved to its own line. Median acknowledgement went from about three working hours, when reports shared the general queue, to under an hour once they had their own, and stayed there. The click share fell across the year with two awkward months in the middle where a single well-made campaign caught three people. The first-report time went from "the next morning" to "within the half hour". It is the figure she is proudest of, because it is the organisation's number rather than the desk's.

Draw the trend as two simple charts: reports and campaigns by month on one and the click share by quarter on the other. Mark the simulation months and reference the rules sheet underneath. A chart that hides the simulation, smooths the noise or extends a line into next year is a chart that will be believed and then found out. The difference between a chart that helps and one that hides is set out in Charts That Help and Charts That Hide. Fiona's charts have four points on the quarterly one. That is enough to see a direction, and she says so on the chart rather than letting the eye invent precision.

The Line in the Leadership Report

Leadership gets one line a month, and it carries the four numbers, the direction, and one sentence of what changed. "Phishing: 17 reports this month across 3 campaigns (12 and 2 last month); 6 confirmed as phishing; median acknowledgement 40 minutes; 1 click of about 40 recipients, contained within the hour; first report arrived 12 minutes after delivery. The warehouse team reported first for the third month running." That is the whole line, and it fits in the monthly report described in The Monthly Report They Will Read. It sits beside the desk's other handful of numbers rather than in a security report of its own. A quarterly version adds the two charts and the ranges.

The line is written for a reader who will ask two questions, and the answers are prepared. The first is "is that good?", and the answer is the direction over the quarters. Add the honest note that the organisation's own history is the only benchmark available. Comparisons with other organisations are not meaningful because nobody counts the same way. The second is "who clicked?", and the answer is the one Fiona gave Alistair at 17:06. The person has been helped, the security function has the record, the number is one, and the desk does not report names. Prepare that answer in advance, because it is asked every time and it is asked by people who mean well.

Leave off the line anything that could be read as a league table: no department ranking, no "worst month", no comparison of office against office except as praise for a first report. The line is there to show the loop working and to justify the hour. It is not there to be interesting at anyone's expense.

Never a Per-Person List

There is no list of who clicked. Not for leadership, not for managers, not for the security function's monthly summary, not as "just for context", and not as a column on the sheet that is never printed. The reason is the one this whole series rests on. The report channel runs on people believing that telling the desk never makes anything worse for them. A list that exists will eventually be seen. The first time it is seen the reports from that part of the building stop. A measurement that destroys the thing it measures is not a measurement, and the per-person click list is the clearest example of one that I know.

What exists instead is a per-person thank-you, which is the opposite thing. The desk knows who reported first and can say so with their permission in the monthly note. The desk knows which department reports most and can thank it by name. The desk knows how many clicks there were and can say the number. None of that needs a name attached to a mistake, and all of it rewards the behaviour the desk wants more of. The thinking behind it, and the careful handling of the leaderboard idea when someone proposes one, is in Rewarding the Reporters. The measuring article only needs to say that the list does not exist and that the desk's own numbers prove nothing is lost by its absence.

The manager who asks for the names will ask more than once. The answer is the same each time, delivered as a redirect rather than a refusal. The person has been helped and told what to do. The security function holds the record and decides whether anything further is needed. The desk can offer a team-level count, the knowledge base article, and an introduction to the security function. It is a no said upward, and it is one of the easier ones. The reason is short, and nobody has yet produced an argument for the list that survives being written down. The same principle governs the desk's own dashboards. A number on a wall changes behaviour, and the desk had better want the behaviour it gets. This is laid out in Dashboards That Motivate Instead of Surveil. A click list is the surveillance end of that scale, with a name on it.

A Short War Story: The Column at Linden Court Schools

The desk at Linden Court Schools, a trust of four schools with a desk of three, kept a careful monthly sheet for a year. It included, for completeness, a column with the names of staff who had clicked. The column was never printed and was mentioned to nobody. In the autumn a head of department asked the desk lead, Priyanka, for "the numbers for my school" before a staff meeting. The sheet went as an attachment. The head read out the names of two teachers in the meeting as an illustration of why everyone should be careful. Reports from that school fell from a steady five or six a month to none for the rest of the term. A message imitating the trust's payroll provider reached the whole school in November. Four people clicked it, and nobody reported it for four days, until the payroll provider rang. The clean-up ran to four resets, a contracted firm's review of what the sign-ins had reached, and a conversation with the trust's data-protection officer. It took something over fifty hours of staff time across the desk and the school office. Priyanka deleted the column the same week and wrote the rule on the sheet's first line instead, where it has stayed. There is no per-person figure of any kind, and a sheet with one on it has already been shared.

The Monthly Security Line

The sheet below is Fiona's, filled in on the first working day after month end from the category count, the acknowledgement timestamps and the security function's recipient figures. Section A is the counting rules, written once and dated. B is the month's four numbers, and C is the line that goes to leadership. D is the quarterly addition, and E is the rule that never changes. Keep it with each month's count so that a question in a year has an answer, and when a number moves, check the rules sheet before checking the organisation.

MONTHLY SECURITY LINE — REPORTS AND CLICKS
Desk: ______________________   Month: __________   Counted by: ______________   On: ________

A. COUNTING RULES (written once, dated: ________)
   Reports = tickets in category "______________" opened in the month; every verdict counts
   Campaign = one distinct phishing message; linked reports = one campaign
   Acknowledged = first reply from a person; automatic replies do not count; working time
   Clicks = clicked, typed, opened or replied to confirmed phishing; reported or found
   Recipients = the security function's count where known; otherwise clicks reported alone
   Simulations = counted on their own line, never in the real figures
   Per-person figures of any kind: NONE. Per-technician acknowledgement: coaching only,
     not read below ____ acknowledgements for one person

B. THIS MONTH
   Reports: ____  (last month ____)      Campaigns: ____  (last month ____)
   Confirmed phishing: ____   Spam: ____   Legitimate: ____   Undecided, escalated: ____
   Median time to acknowledge: ____ (working minutes/hours)   Reports over target: ____ of ____
   Clicks: ____ across ____ campaigns   Recipients known: ____   Share: about one in ____
   Contained within the hour: ____ of ____ clicks
   First report after delivery (where known): ____ minutes
   Simulation this month: no / yes: ____ reports, ____ clicks (own line, not above)
   Departments with zero reports this quarter: ______________ (visit, poster, ask)

C. THE LINE FOR THE LEADERSHIP REPORT (one sentence each)
   "Phishing: ____ reports across ____ campaigns (____ and ____ last month); ____ confirmed;
    median acknowledgement ____; ____ click(s) of about ____ recipients, contained within
    the hour; first report ____ minutes after delivery."
   What changed, one sentence: ____________________________________________
   Thanks, by team only: ____________________________________________

D. QUARTERLY ADDITION
   Reports by month (3 points): ____ / ____ / ____      Campaigns: ____ / ____ / ____
   Click share range this quarter: none to about one in ____  (last quarter: one in ____)
   First-report time range: ____ to ____ minutes
   Charts: reports and campaigns by month; click share by quarter; simulation months marked
   Comparison with other organisations: none offered; counting rules differ

E. THE RULE THAT NEVER CHANGES
   No name is attached to a click, on this sheet or anywhere it is copied.
   Names appear only beside a thank-you, and only with permission.

Fiona sent Alistair the sheet on Monday, with the four quarterly points. She added a sentence saying that the first-report time had fallen from the next morning to within the half hour over the fourteen months. That was the organisation's doing rather than the desk's. He asked whether the click share was good. She said it was lower than a year ago and that there was no outside figure worth comparing it with. Two clicks either way would move it, so she would rather he watched the direction. He asked, once more, who it had been. She said the number was one, the person had been thanked, and the desk did not keep that list. He said that seemed fair, which is the most a measuring article can hope for.

Frequently Asked Questions