Enriching the Alert Ticket So the Technician Starts Informed

Saturday, 03:20, at Ashgrove Housing. Ines is on call for the second time and the phone wakes her with a ticket: "[ALERT] app-03: service stopped: tenant-portal". That is the whole of it, apart from the seven payload lines, which tell her the service stopped at 03:12 and give her a link to a graph. She does not know what app-03 is, where it lives, or who uses the tenant portal. She does not know whether anyone is using it at twenty past three on a Saturday, or whether anything changed on it yesterday. Nor does she know what the desk normally does when it stops. She spends twenty-five minutes finding out, from a wiki, a shared folder and a search of old tickets, before she touches anything. Afterwards she learns that Owen patched app-03 at 17:30 on Friday, that there is a one-page runbook for exactly this, and that the same alert fired last Saturday at about the same time. All of it existed. None of it was in the ticket.

Enrichment is adding to the alert ticket the things a technician needs that the monitoring system does not know. These are what the device is for, who owns it, where it is, what changed on it recently, how the desk fixes this, and whether it has happened before. The monitoring system supplies the measurement. Enrichment supplies the context, and the ticket is where the two meet. This article is about what to add, where each fact comes from, and how it gets into the ticket without a person typing it at three in the morning. It is part of our Monitoring-to-Ticket Integration series, and it assumes the ticket already exists with the payload described in the mechanics article.

It covers what the monitoring system does not know, then the six things to add and where each fact lives and how it gets in. Next comes the device sheet, which is the simplest source and usually enough. Then it covers automatic against manual enrichment, the runbook link, recent tickets for the same thing, and the last change. Business-hours hints that help without deciding, a short war story and the enrichment checklist to copy finish the article.

What the Monitoring System Does Not Know

The monitoring system knows the device and the number, and that is the full extent of its knowledge. It knows that app-03 exists, that a service called tenant-portal was running and is not, and when that changed. It does not know that app-03 is the tenant portal server. It does not know that the portal is used by housing officers during office hours and by tenants paying rent at any hour. It does not know that Owen owns it, that it sits in the second rack in the Harbour Street office, or that it was patched last night. It does not know that this has happened before. It knows the number because the number is what it was built to watch.

Everything else lives somewhere on the desk already, scattered: in the change records, in the wiki, in the heads of two people, in the closed tickets, in a spreadsheet of servers that somebody started. The technician at 03:20 has to assemble it, and assembly is the twenty-five minutes. Enrichment is doing the assembly once, in advance, per device and per alert type, so that the ticket arrives with the context attached and the technician's first twenty-five minutes go into the fix.

The test of good enrichment is the competent stranger, the same reader the runbook is written for. Could a technician who knows the trade and has never seen this building open the ticket and know what the thing is within a minute? Could they also know, within that minute, whether it matters right now, who to ask, and where the procedure is? If yes, the ticket is enriched. If they have to open a second window to find out what app-03 is, it is not. The ticket should be the one window.

The Six Things to Add

The device's identity comes first: its friendly name, its role in one phrase, and what runs on it. "app-03: tenant portal server; also hosts the repairs booking site" tells the stranger three things the hostname never will. Role matters more than specification; nobody at 03:20 needs the processor count, and everybody needs to know that this is the server tenants use to pay rent. Where the desk has more than one of a kind, say which one: "second of two portal servers; the other is app-04 and takes over automatically", or "the only one".

The owner comes second, and there are two. The technical owner is the person on the desk who knows the thing best and would be woken for it if waking were allowed. The business owner is the person outside IT whose service it is, who can say whether it matters tonight and who must be told if it is down tomorrow. Both are names with a way to reach them, and both are roles that change, which is why they live in a lookup rather than in the runbook. The location comes third: the physical place, for the technician who may need to press a button, and the logical one, which network and which site. The logical location is for the technician who needs to know whether a site link is the real problem.

The last change comes fourth and is the single most useful fact a technician can be handed. Most things that stop working at 03:12 on a Saturday were touched at 17:30 on the Friday. The runbook link comes fifth: the one-page procedure for this alert type on this kind of device, written for the stranger. And the recent history comes sixth: how many tickets have carried this alert key before, and when the last one closed. Together the six turn "app-03: service stopped" into a ticket that says what it is, whose it is, and where it is. It says what happened to it yesterday, how to fix it, and that this is the second Saturday running. Ines could have been back asleep by half past three.

Where Each Fact Lives and How It Gets In

Each of the six facts has a home, and the homes are different, which is why enrichment is a set of lookups rather than one. The identity, the owners, the location and the runbook link are stable facts about a device. They belong in a device sheet, described in the next section, keyed by the device name. The last change lives in whatever the desk uses to record changes, and gets into the ticket by a search of that record for the device name over the last few days. The recent history lives in the help desk itself, and gets in by a search of closed tickets for the same alert key.

Two of the six need no upkeep at all, and that is worth noticing. The recent-ticket history is the help desk's own record and costs nothing to maintain, because every alert ticket closed adds to it. If you run QuerySolv, every ticket carries its full event history, so the recent-tickets line is a search by the alert key and a glance at the dates. The last change is nearly free as well, provided changes are recorded somewhere searchable by device name, which is the whole of the argument in tracking changes without a change module. The other four need a sheet, and the sheet needs an owner.

Most small-team help desks have no asset module; QuerySolv is one of them. For enrichment that does not matter, because a custom field on the ticket carrying the device name is all the ticket needs. The device sheet does the rest. Where a desk does hold device data in the ticket, how to do that without an asset system is the business of linking devices without an asset system. Enrichment only needs the name to be in a predictable place so that the lookup has something to look up.

The Device Sheet

The device sheet is a plain table with one row per monitored device. It is the simplest piece of infrastructure the integration has and the one that does the most work. It can be a spreadsheet, a text file with one line per device, or a page in the agents-only knowledge base. What matters is that it has a fixed set of columns and that the device name in it matches the device name the monitoring system uses exactly. A program or a person must be able to find a row by that name in seconds. Here is the shape, with two rows filled in the way a real sheet would be.

DEVICE SHEET — ONE ROW PER MONITORED DEVICE

name:        app-03.example.com
role:        tenant portal server; also hosts repairs booking site
pair:        app-04 takes over automatically if app-03 is down
owner:       Owen (technical)   /   Grace, housing services (business)
location:    Harbour Street office, rack 2, shelf 3   /   office LAN, site A
hours:       housing officers 08:00-18:00 weekdays; tenants pay rent any hour
runbook:     agents-only article "Portal service stopped"
notes:       patched monthly, first Friday evening

name:        fs-02.example.com
role:        main file server; all department shares
pair:        none; the only one
owner:       Dana (technical)   /   Tomas, operations (business)
location:    Harbour Street office, rack 1   /   office LAN, site A
hours:       08:00-18:00 weekdays; backup runs 23:00
runbook:     agents-only article "File server disk nearly full"
notes:       drive D fills at month end; see problem ticket 5571

Keep the sheet where the integration and the on-call technician can both reach it at three in the morning, which rules out a document on somebody's laptop. The agents-only part of the knowledge base is the natural home for the human-readable copy. A plain text or spreadsheet file next to the relay is the natural home for the machine-readable one. Many desks keep both and copy from one to the other at the monthly review. A sheet that is slightly stale is still worth ten times a sheet that does not exist. A sheet nobody can find is worth nothing, however current.

The sheet goes stale by exactly one mechanism, which is a device being added, moved or re-purposed without the row changing. The fix is to make the row part of the act. Adding a device to monitoring adds its row. Moving a device updates its location. Handing a service to a new owner updates two names. Write that into the change checklist and the sheet stays current enough; leave it to goodwill and by spring the sheet describes a building that no longer exists. Device sheets do not decay. They are decayed, one unrecorded change at a time, by people with better things to do.

Automatic or Manual Enrichment

Enrichment can happen when the ticket opens, by the relay reading the sheet and writing the six lines into the ticket. Or it can happen at first touch, by the technician reading the sheet and adding the lines themselves. Start manual. The checklist at the end of this article is the manual version, and a desk that runs it for a month learns which lines are mechanical and which need a person before anyone writes code. The mechanical lines, identity, owners, location and runbook link, are a lookup by device name and are safe to automate on the webhook or API route. The judgement lines, whether the last change is relevant and whether the history means anything, stay with the technician.

Automating the lookups is worth doing once the sheet has a shape, because it turns twenty-five minutes into none and does it at every hour of the night. A relay can read the device sheet and write "role: tenant portal server; owner: Owen; runbook: Portal service stopped" into the ticket body as it opens. That is a few more lines of script than the relay already is, and the lines are the dull kind. What not to automate is anything that decides: the priority the alert should really have, whether to wake someone, whether this is the same problem as last Saturday. The order in which a desk should automate things, and the things it should never automate, is in what to automate first and what never. Enrichment sits comfortably in the first group as long as it stays a lookup.

On the e-mail route, where the integration cannot write to a ticket after the mail lands, enrichment is manual by necessity, and the checklist is the whole method. That is fine. The twenty-five minutes shrink to five when the technician knows where the sheet is and what six things to pull from it. Five minutes at first touch is a cost most desks never notice. The point of enrichment is that the technician starts informed, not that a program did the informing.

The Runbook Link

The runbook link is the most valuable line enrichment adds, because it is the one that turns a diagnosis into a procedure. A runbook is a one-page set of steps a competent stranger can follow alone, with a check after every action and a note on when to stop and who to call. Writing one is the subject of writing the runbook. Every alert type the integration opens tickets for should have one, and the ticket should link to it by name. Ines, with "Portal service stopped" in front of her at 03:21, restarts the right service on the right server with the right check in four minutes. That is because somebody who knew wrote it down on a Tuesday.

Runbooks belong to alert types, not to devices. "Service stopped" on a portal server, a print server and a database server are three runbooks, because the service, the check and the stop-and-call line differ. "Service stopped" on two identical portal servers is one runbook with the device name as a parameter. The device sheet's runbook column points at the right one for that device and that alert, which is how a lookup by device name produces a procedure. Where no runbook exists yet, the ticket says so plainly, "runbook: none; write one from this ticket's notes". The technician's working notes on the ticket become the draft. That is the capture habit this library describes elsewhere and which alert tickets feed better than most.

A runbook link is only as good as the runbook's last-run date. A link to a procedure nobody has run since the server was rebuilt is a trap with a helpful name. The technician who follows it at 03:20 will do the wrong thing confidently. Enrichment therefore adds the date the runbook was last run beside the link, pulled from the runbook's own footer. A date more than a few months old is a signal to read carefully. The monthly review of the integration, which closes this series, checks those dates in bulk.

Recent Tickets for the Same Thing

The history line says how many tickets have carried this alert key before and when the last one closed, and it changes the technician's first question. No previous tickets: this is new, read the runbook and look at the last change. One or two, months apart: familiar and probably benign, fix it and note it. Three or more in a quarter: this is a problem in the formal sense, a recurring fault with a cause nobody has found. In that case, the right move after the fix is to link this ticket to the others and raise it with whoever runs the weekly problem review. The method for spotting those patterns across the whole queue is in finding the repeat tickets; the history line does it for one alert, at the moment it matters.

Getting the line is a search of closed tickets for the alert key, which is why the key must live in a searchable field and read identically every time. The integration can count and date the results automatically on the API route. On any route the technician can run the search in ten seconds if the key is where the template put it. Write the result in the ticket in one line: "previous tickets with this key: 2; last closed nine days ago, ticket 4418, 'restarted service, cause not found'". The last note from the previous ticket is the most useful thing in the line, because it says what the last person tried.

Keep the line honest about what it counts. It counts tickets with this key, which is this check on this device. It does not count the same service failing on the other portal server, or a different check on this device, and both of those might be the real pattern. The technician who sees "0 previous" on app-03 and remembers app-04 doing the same thing last month has found something the lookup cannot. The history line should be read as a start, not a verdict.

The Last Change

Most things that break at 03:12 were touched the day before, and the last-change line is how the ticket says so. It is a search of the change records for the device name over the last few days. Its result is one line: "last recorded change: patched by Owen, Friday 17:30, change ticket 5590, 'monthly updates, rebooted'". A technician who reads that line before anything else has a hypothesis in ten seconds and a rollback target in the change record. One without it has a server that stopped for no reason, which is a server that will take an hour.

The line is only as good as the recording habit behind it, and this is the honest edge. A desk that records changes as tickets, with the device name in the subject, gets the line for free from a search. A desk that records changes in a spreadsheet gets it from a filter. A desk that records changes in nobody's memory but Owen's gets it from phoning Owen, which at 03:20 is the thing the whole integration was built to avoid. If the last-change line is always blank, the finding is not that nothing changes. It is that changes are not being written down where the ticket can see them. That is a finding to take to the next change review.

Include scheduled changes that have not happened yet. A device inside a maintenance window tonight, or due for patching in an hour, is context the technician needs just as much as yesterday's patch. It may mean the alert is expected, or that the fix should wait, or that the alert is the maintenance going wrong. The device sheet's notes column carries the regular schedule; the change record carries the one-off. Both belong in the line.

Business Hours and the Hint That Does Not Decide

The device sheet's hours column lets the ticket say whether anyone is likely to be affected right now. That is the line that lets an on-call technician go back to sleep with a clear conscience, or not. "Housing officers 08:00-18:00 weekdays; tenants pay rent at any hour" tells Ines that the portal matters at 03:20 after all, because a tenant on a night shift may be trying to pay. "Finance reporting server; used month-end and 09:00-17:00" tells the next technician that a disk alert at midnight on the twelfth can wait until morning. Enrichment adds the fact. The person decides what it means.

Resist the temptation to let the sheet set the priority directly. It is tempting, because the hours column looks like a rule, and the first month it will work. Then the month-end falls on a weekend, or the tenants' rent day moves, or the finance server picks up a second job nobody wrote in the sheet. The rule confidently downgrades the thing that matters. The hint belongs in the ticket as a line a technician reads; the priority belongs to the alert-type table and to the human who adjusts it. The sheet informs. It is not allowed to conclude.

A Short War Story: The Unknown Server at Bluewater Bank

At Bluewater Bank, Tomas was woken at 02:50 by a ticket reading "[ALERT] db-07: free space on drive E: CRITICAL", with the payload and nothing else. He did not know what db-07 was, and the bank had thirty servers with names like it. So he judged it a reporting database that could wait, added a note saying so, and went back to sleep. At 08:05 twenty-three branches opened and discovered that the appointment booking system would not take bookings. That was because db-07 was its database and its transaction log had filled the drive at about 03:30. Branch staff took appointments on paper for two hours. During that time, Lena, who owned db-07 and had a runbook for exactly this, cleared the log in twelve minutes once she was in and had been asked. The review costed the morning at roughly forty-six staff-hours across the branches, plus a week of re-keying paper appointments. That was against a fix that the runbook said took a quarter of an hour. Tomas had done nothing unreasonable with what he had; what he had was a hostname. Bluewater built the device sheet that week. The next db-07 alert arrived reading "branch appointments database; owner Lena; runbook: clear transaction log; last change: none; previous tickets: 1". It was fixed before the branches opened. The technician was not uninformed because the information did not exist. He was uninformed because the ticket did not carry it.

The Enrichment Checklist

The checklist below is the manual version of everything above, run at first touch on every alert ticket until the lookups are automated, and run on the judgement lines forever. Keep it pinned beside the device sheet. The first block is filled from the sheet and takes two minutes. The second is filled from searches and takes three. The third is the technician's reading of the first two and takes as long as it takes. A ticket that has all three blocks filled is enriched, and a ticket that is enriched is one a stranger can work at 03:20 without opening a second window.

ALERT TICKET ENRICHMENT — AT FIRST TOUCH
Ticket: ________   Alert key: ____________/____________   Technician: ____________

FROM THE DEVICE SHEET (lookup by device name)
  [ ] role, in one phrase:          ______________________________________
  [ ] pair or the only one:         ______________________________________
  [ ] technical owner:              ______________   business owner: ______________
  [ ] location, physical:           ______________   logical: ______________
  [ ] hours it matters:             ______________________________________
  [ ] runbook, by name:             ______________   last run: ____________
  [ ] notes from the sheet:         ______________________________________

FROM SEARCHES
  [ ] last recorded change on this device, past 3 days: ____________________
      change ticket: ______   by: ______   when: ______
  [ ] scheduled change or maintenance window pending:    ____________________
  [ ] previous tickets with this key: ____   last closed: __________  ticket ______
      last note on that ticket: ______________________________________

TECHNICIAN'S READING
  [ ] anyone affected right now?    [ ] yes  [ ] unlikely   because: ______________
  [ ] change in the last 3 days?    [ ] yes  [ ] no    relevant? [ ] yes [ ] no
  [ ] third or more occurrence?     [ ] yes  [ ] no    linked to problem ticket ______
  [ ] priority adjusted?            from ______ to ______   because: ______________
  [ ] runbook missing or stale?     [ ] write one from these notes  [ ] flag for update

Ines ran the checklist the following Saturday, when app-03 stopped at 03:14, because by then the desk had a sheet and she knew where it was. The ticket said tenant portal, said Owen, said patched the previous evening, said Portal service stopped, said second occurrence. She followed the runbook, restarted the service, linked the two tickets, left a note for Owen about the patch schedule, and was asleep again by half past three. The information had not changed since the week before. The ticket had.

Frequently Asked Questions