Reviewing the Monitoring Integration Monthly: The Tuning Loop

The first Tuesday of the month, 10:00, at Acme. Hugo, who runs the desk, filters last month's tickets by the monitoring address and gets 214. Sixty-one of them were closed inside ten minutes with a note reading "cleared, no action". Forty-eight of those sixty-one came from one alert type, a database connection warning that nobody has ever done anything about. The backup failure on the fourteenth, by contrast, took thirty-one hours to be claimed. It opened at ordinary priority at six on a Saturday evening and nothing about it said hurry. The warehouse's power supply, which went on battery on the twenty-second, opened no ticket at all, because it was never connected. The desk learned about it from the warehouse. The integration is six months old. Nobody has looked at it since the week it went live. It has been drifting, quietly, in three directions at once.

The monthly review is one hour, on a fixed day. One person pulls the month's alert tickets and reads five numbers. They find the alert that should not have fired and the alert that should have. Then they make one change per alert type for the month ahead. The review is the maintenance the integration needs to keep earning its place in the queue. Without it the integration becomes, within a year, a thing the team has learned to close unread. This article is the hour, in order. It is part of our Monitoring-to-Ticket Integration series. It closes the series, because the review is where every earlier decision gets checked against what happened.

It covers why the integration drifts, then pulling the month's alert tickets and the five numbers. Next come false positives, time to acknowledge, and the alert that never earned a ticket. We cover the tuning loop and the record it keeps, then the one line to send upward. We finish with a short war story and the review sheet to copy, in that order.

Why the Integration Drifts

The integration drifts because everything around it moves and nothing tells it. Whoever administers the monitoring system adds checks, and some of them notify the desk's address because that is where the last one went. Devices are added, retired and renamed, and the device sheet is not. Thresholds are loosened during an incident and never tightened again. The desk itself changes: a new technician treats alert tickets as noise because the old one did. A habit of closing them unread spreads in a fortnight. Each change is small and reasonable. The sum of them, after six months, is Hugo's 214.

Drift has a direction, and it is always towards more noise and less attention. Alerts get added more often than they get removed, because adding one is a reasonable response to an incident. Removing one requires somebody to decide that it was never useful, which nobody is asked to do. Attention falls as noise rises, because people are sensible. The review is the only force on the desk that pushes the other way. And it works because it asks the removal question on a schedule rather than waiting for someone to volunteer it. An integration is not set up. It is tended, or it rots.

One hour a month is enough for a desk of one to ten people with a few dozen alert types connected. And the hour should be on the calendar with a name against it, the same way the problem review in problem management in an hour a week is. The two reviews feed each other: the alert that fires every month is a problem, and the problem that recurs is often an alert that should exist. Hold them in the same week and the hand-off is a conversation in the corridor.

Pulling the Month's Alert Tickets

The review starts with a filter, not a report. Take every ticket whose requester is the monitoring address, opened in the last calendar month, and export it to a spreadsheet. Most tools can export a filtered list of tickets to a spreadsheet file. The columns you need are the ticket number, the subject, the alert key and the priority. You need when it was opened, when it was first claimed, and when it was resolved. You also need whether the integration or a person resolved it, how many times it was reopened, and the closing note. If your tool cannot export the first-claimed time, the first assignment event in the ticket history is the same thing. The history is where every number in this article ultimately comes from. If you run QuerySolv, the reports on volume, response and resolution times and reopen rate cover alert tickets like any other ticket. In that case, the numbers below are a filter on the ticket list rather than a separate system to build.

Add two columns by hand, and they are the ones that make the review worth doing. The first is the alert type, derived from the key or the subject. That way, the spreadsheet can be grouped by type rather than by individual ticket. The second is the outcome, read from the closing note and reduced to one of four words. Fixed means a person did something; cleared means it resolved with no action. Noise means it should not have fired; unknown means the note does not say. The outcome column is tedious to fill for 214 tickets and takes about twenty minutes. It is the only column that will tell you which alerts are earning their place, and twenty minutes a month is what that knowledge costs.

Keep every month's spreadsheet, in one folder, named by month. The review's best questions are comparisons: "is this type getting noisier?", "did the change we made in March work?" Comparisons need last month's sheet beside this one. Twelve sheets in a folder is the integration's whole history, and it is a history that nobody will reconstruct from memory in a year.

The Five Numbers

The first number is tickets opened by alerts, in total and by alert type. And its job is to show where the volume comes from. Sort the types by count. In most desks the top three types produce more than half the tickets, and the question for each is whether that many was right. The second number is the false-positive rate per type: of the tickets of that type, how many had the outcome "cleared" or "noise". A false positive is an alert that fired when nothing needed doing. A type whose false-positive rate is above a third is a type that is teaching the desk to ignore it. That type gets tuned or disconnected this month, not discussed.

The third number is time to acknowledge: the time from the ticket opening to the moment a person claimed it. Take the median and the worst case, per priority. Report the median because it tells you what normally happens, and the worst because the worst is the one that cost something. Never report the average alone. One thirty-one-hour Saturday will drag the average of fifty ten-minute pickups into a number that describes neither. The fourth number is time to resolve, split by who resolved it, the integration or a person. That way, a fast auto-close does not flatter the desk's own speed. The fifth is reopens and flaps: tickets reopened by a returning alert, by key. That tells you which fixes did not hold and which thresholds are set on the edge.

Read the five together before acting on any one of them. A type with high volume and a low false-positive rate is doing its job loudly. A type with low volume and a high rate is a nuisance that costs little, and may be left for a quieter month. A type with high volume and a high rate is the one that is burning the desk's attention, and it is nearly always the first thing the review fixes. The numbers point. The reviewer decides.

False Positives

False positives have three causes, and each has a different fix. The threshold is too tight: the disk alerts at eighty percent on a server that has lived at eighty-two for a year. And every month it crosses and comes back. The check is watching the wrong thing: the connection warning fires when the pool is busy, which is Tuesday, not when it is broken. The condition heals itself faster than the clear delay: the service restarts on its own in forty seconds. The ticket opens anyway because the alert fired at the first failed check. The first is fixed at the monitoring system by moving the line. The second is fixed by changing or removing the check. The third is fixed by requiring the condition to persist for several checks before alerting, which is the same control that stops flapping.

Fix false positives at the source, not at the desk. It is tempting to add a rule in the integration that drops the connection warning on the floor, and it works. It leaves the monitoring system still raising an alert that nobody acts on and a dashboard tile that is red for no reason. That is how dashboards stop being believed. The honest fix is to go to whoever administers the monitoring system with the spreadsheet. Say that this alert fired forty-eight times last month and was acted on zero times. That is a sentence that ends arguments. (Most monitoring systems ship with a set of default alerts written for somebody else's building. Nobody is obliged to keep them.)

The rule the review enforces is that no alert type lives another month with a false-positive rate above a third. It is tuned, by one change, or it is disconnected, with a note on the decision table saying why and when. The dashboard keeps it if the dashboard wants it. The desk is not the monitoring system's audience of last resort. An alert that cannot earn a ticket does not get one.

Time to Acknowledge

A long time to acknowledge is almost never a technician's fault. And the review should look for the route before it looks for the person. The backup failure that waited thirty-one hours opened at ordinary priority because the priority table said backups were ordinary. They are on a Tuesday afternoon. They are not at six on a Saturday with the weekend's data at stake. It opened outside business hours, when nobody was reading the queue and no on-call route existed for anything below high priority. Its subject said "[ALERT] bk-01: job status" because the job name was in the body. Each of those is a setting, and each is why the ticket waited. The technician who claimed it on Monday did so within ten minutes of sitting down.

The fix is one of four, and the spreadsheet usually says which. If the slow pickups cluster on one priority, the priority mapping for that alert type is wrong. If they cluster outside business hours, the desk needs a route for the alert types that cannot wait until morning. That applies even if the route is a phone that rings one person. If they cluster on one type with an unreadable subject, the subject template is wrong. If they are spread across everything, the alert tickets are landing somewhere nobody reads, and the fix is a queue decision rather than a tuning one. The number is only meaningful against a target. And the target per priority is the response-time half of the pair described in response vs resolution targets. Alert tickets should carry the same targets as human ones, and be measured against them in the same report.

Look at the worst case every month, by name, and write one line about why. "Backup failure, fourteenth, thirty-one hours: opened ordinary on a Saturday evening; mapping changed to high for backups outside business hours." Twelve of those lines, one per month, are the most useful document the integration will ever produce, because each one is a route that was fixed. A median that improves is pleasant. A worst case that was understood is progress.

The Alert That Never Earned a Ticket

The review's hardest question is about the alerts that did not open tickets, and there are two kinds. The first is the alert type that was deliberately left on the dashboard by the decision table. And the check is a quick one: does the decision still hold? A disk alert that was "dashboard only" when the server had a spare terabyte may deserve a ticket now that it does not. Re-read the dashboard rows of the table in why alerts should become tickets, sometimes once a quarter and move any that have changed their minds.

The second kind is the gap, and finding it means reading the month's human tickets rather than the alert ones. Filter the month by everything that is not the monitoring address, and look for infrastructure in the subjects. Examples are "file server slow", "cannot reach the warehouse", "printing stopped for the whole floor", and "the portal was down this morning". For each, ask two questions. Did the monitoring system see it? If the dashboard was red and no ticket opened, the alert exists and is not connected, and connecting it is this month's change. If the dashboard was green, the monitoring system has no check for it, and adding one is a request to whoever administers it. The warehouse power supply at Acme was the first kind: monitored, red for four hours, connected to nothing.

The gap list is the review's most valuable output. Every entry on it is an outage the desk learned about from a person when it could have learned from a machine. It is also the list management understands without translation. So keep it in plain words: "Three times this month users reported something the monitoring system saw first and the desk did not." Quiet failures of this kind, the ones nobody pages anyone about, are the subject of how support quietly fails. An unconnected alert is one of the quietest.

The Tuning Loop

The tuning loop is the discipline of making one change per alert type per month and writing it down. It includes reading the result at the next review before making another. Suppose the threshold and the clear delay and the priority mapping all move in the same month. Then next month's number cannot say which one worked, and the month after that you will be guessing in three dimensions. The changes available are few and should be named plainly. You can move the threshold at source, change the alert's policy, or disconnect it. Changing the policy means ticket or dashboard, auto-close or human confirmation, the priority it opens at, or the clear delay. Pick one, note it, wait.

The record is one page, kept beside the decision table and the auto-close rules. It lists every connected alert type with its current policy and a line per month of what was changed and why. It is the integration's memory, and it is what makes the second year of the integration cheaper than the first. When the connection warning starts firing again after a server move, the record says it was tuned in March from eighty to ninety percent and why. The new administrator does not start from zero. A record that is not kept is a tuning loop that runs once.

Four housekeeping checks ride along with the loop, and each takes five minutes. Spot-check six rows of the device sheet against reality, picking the devices that had tickets this month. Read the last-run date on the runbooks for the three busiest alert types, and chase any that are older than the last change to that device. Trip the flood breaker on purpose with a burst of test alerts and confirm it holds and reconciles, which is the monthly test the floods article asked for. And check that the relay's log file is being written, has not filled its disk, and still has an owner who is in the building. None of these produce a number. All of them produce the silence in which the numbers can be trusted.

The One Line Upward

Management needs one line from this review, and it is not the false-positive rate. It is the count of things the machines caught before people did, against the count of things people caught first. The line includes the single story that explains the month. "Machines opened sixty useful tickets, people reported three things the machines should have caught, and here is the backup we now page for on weekends." That is a sentence a director can repeat in a meeting, which is the test of a sentence for a director. The shape of a brief that survives that meeting is in explaining technical things to management. The review supplies the numbers and the story, and the brief supplies the shape.

Resist the urge to send the spreadsheet. The spreadsheet is the review's working material, and its value upward is roughly that of a mechanic's receipt for parts. Send the line, keep the sheet, and be ready to show it when asked, which will be the month the line says something uncomfortable. A review that produces an uncomfortable line once a year is a review that is looking in the right places.

A Short War Story: The Unread Queue at Greyfield Motors

At Greyfield Motors, Rosa inherited a monitoring integration that had been connected to every alert the monitoring system could raise. A predecessor had connected it two years earlier and then left. Rosa inherited with it a team that closed alert tickets in a Friday sweep without opening them. That was because about seven in ten of them had always said "cleared, no action". On a Thursday in her second month a disk-full alert for the dealer management server opened a ticket in the usual way. That server produces every invoice in the showroom. On the Friday a technician swept the ticket closed with the other forty-one, unread, as the team had done every Friday for a year. The disk filled on the Saturday, and the server stopped. The showroom could not invoice, register or order for the whole of its busiest day. The dealer principal costed that at about thirty lost or delayed sales and a Monday of apologies. Rosa's first monthly review found that two alert types, a connection-pool warning and a processor-load notice, had produced four fifths of the year's alert tickets. They had been acted on by nobody, ever. She disconnected both, moved three more to the dashboard, and set the backup and disk alerts to open high outside business hours. The next month the desk received fifty-eight alert tickets instead of three hundred. Every one of them was opened and read, and the Friday sweep was retired, unmourned. An integration nobody reviews becomes an integration nobody reads. And the one ticket that mattered is closed with the same click as the forty-one that did not.

The Review Sheet

The sheet below is the hour, in order, on one page. Fill it from the month's spreadsheet and keep the filled sheets in the same folder as the spreadsheets. Bring the previous month's sheet to the review. That way, the "changed last month" line can be checked against this month's numbers before anything new is decided. The blanks are deliberate. A review that cannot fill them has not pulled the data. A review that fills them in under an hour has a well-tuned integration or a small one, which are both fine.

MONITORING INTEGRATION — MONTHLY REVIEW SHEET
Month: ____________   Reviewer: ____________   Spreadsheet saved as: ____________

1. VOLUME
   Alert tickets opened: ____   Human tickets opened: ____   Alert share: ____ %
   Top three alert types by count:
     ______________ ____    ______________ ____    ______________ ____

2. FALSE POSITIVES (outcome = cleared or noise)
   Type                      Tickets   No action   Rate     Action this month
   ______________________    ______    ______      ____ %   [ ] tune  [ ] disconnect  [ ] keep
   ______________________    ______    ______      ____ %   [ ] tune  [ ] disconnect  [ ] keep
   ______________________    ______    ______      ____ %   [ ] tune  [ ] disconnect  [ ] keep
   Any type above one in three left untouched?  [ ] no  (if yes, say why: ______________)

3. TIME TO ACKNOWLEDGE (opened to first claimed)
   Priority     Median      Worst       Target
   high         ______      ______      ______
   ordinary     ______      ______      ______
   low          ______      ______      ______
   Worst case this month: ticket ______, type ______________, ______ hours
   Why: ______________________________________________   Route fixed: [ ] yes

4. TIME TO RESOLVE
   Resolved by integration: median ______    Resolved by a person: median ______
   Person-resolved slower than last month?  [ ] yes  [ ] no   Type to look at: ______

5. REOPENS AND FLAPS
   Keys reopened 3 or more times: ______________  ______________  ______________
   Problem ticket opened or linked: ______   Clear delay or threshold adjusted: ______

6. THE ALERT THAT NEVER EARNED A TICKET
   Dashboard-only decisions re-read this month:  [ ] yes   Moved to ticket: ______
   Human tickets about infrastructure this month: ____
     Monitoring saw it, no ticket opened (connect):   ______________________
     Monitoring did not see it (request a check):     ______________________

7. CHANGES
   Changed last month: ______________________   Did it work? [ ] yes [ ] no [ ] unclear
   One change per type this month:
     ______________  ->  ______________________________________
     ______________  ->  ______________________________________
   Recorded on the integration page: [ ]

8. HOUSEKEEPING
   [ ] Six device-sheet rows spot-checked   [ ] Runbook dates read for top three types
   [ ] Breaker tripped on purpose and reconciled   [ ] Relay log written; owner confirmed

9. THE LINE UPWARD
   "Machines caught ____ useful things; people caught ____ first; this month we ______."

Hugo's first sheet took ninety minutes, because the outcome column had never been filled and the decision table had never been written down. The second took fifty. By the fourth, the connection warning was gone and backups paged on weekends. By then, the warehouse power supply had opened a ticket the first time it went on battery. By then, the alert share of the queue had fallen from a third to about a tenth, every ticket of which somebody read. The integration had not become cleverer. It had been looked at, once a month, by someone with a spreadsheet and the authority to disconnect things. That is the whole method, and it fits in an hour.

Frequently Asked Questions