Auto-Categorization and the Limits of Keyword Rules

Thursday, 14:30, at Stonebridge Insurance. Priya is building the quarterly report and the category chart has come out looking confident. Of 1,480 tickets closed this quarter, 611 are filed as Password and 402 as Other. The remaining 467 are spread across eleven categories in slices too thin to read. Priya has worked the desk for three years and knows the desk did not reset six hundred passwords. She opens ten of the Password tickets. Three are password resets. Two are about the claims system refusing a login because the user's account expired. One is a VPN certificate. One is a new starter who has no account yet. Two are about the HR portal, which belongs to a supplier. One says "I don't think it's my password but I'm not sure what else it could be", and the rule, which was looking for the word "password", had agreed.

Auto-categorization is a rule that sets a ticket's category from the words in it, without a person reading it. It is the most tempting rule on the desk, because categorizing by hand is tedious and the report at the end of the quarter depends on it. It is also the rule with the lowest ceiling. A category describes what was wrong and a requester describes what they saw, and those are different things. This article is part of our Ticket Workflow Automation series, and it is about what a keyword rule can honestly do for your categories, which is less than the vendor's demo suggests and more than nothing.

It covers what a category is for, briefly, because the Ticket Categorization series owns the subject. Next come how a keyword rule categorizes and why it reads the symptom, then the misfile cost and who pays it. The rule that quietly files everything as Other follows, then why categorizing at close is still needed whatever the rule does at open. Next come the narrow places where keyword rules are safe and measuring the misfile rate each month. A short war story and the keyword rule table to copy finish the article, in that order.

What a Category Is For

A category is a label on a ticket that says what kind of problem it was, and its only purpose is to let you count tickets of the same kind. Counting is what makes every other use possible. It makes possible the report that says what the desk spends its time on and the trend that shows one kind of ticket climbing. It makes possible the problem investigation that pulls every ticket about the same failing thing. It makes possible the knowledge article written because forty tickets this quarter asked the same question. A category that nobody counts is decoration. A category that is counted wrongly is worse than decoration, because somebody will act on the count.

Categories are also, for a small desk, the field most often left on its default value, and that is the pressure that makes auto-categorization attractive. A technician closing their ninth ticket of the afternoon sees a drop-down with fifteen options and picks the top one, or the one they picked last time, or Other. A rule that sets the field at creation seems to solve that, and the report afterwards looks beautifully complete. The problem is not that the rule is lazy. The problem is that it is reading the wrong moment of the ticket's life, which is the subject of the next two sections.

How a Keyword Rule Categorizes

A keyword rule has the same three parts as every other automation: a trigger, a condition and an action. The trigger is that a ticket is created. The condition is that the subject or the body contains a word or phrase from a list. The action is to set the category field. "When a ticket is created, if the subject contains 'password', set the category to Password." Some tools offer this directly in a rules editor. Some can only set a category from a choice on the web form. Some have no category field at all and desks use a tag or a custom field instead. The mechanics vary. The limit does not, because the limit is in the words rather than the tool.

The limit is that the rule reads what the requester wrote at the moment they wrote it. At that moment the requester knows the symptom and not the cause. "I can't log in" is a symptom. Its cause is a forgotten password, an expired account, a locked account, a certificate, a supplier's portal being down, a new starter with no account yet, or a keyboard with the wrong layout selected. The requester has no way of knowing which. A technician finds out by working the ticket. A rule finds out never, because it runs once, at the start, and then the ticket moves on without it. Every keyword rule is a confident guess about a cause, made from a description of a symptom, before anyone has looked.

The second limit is that words have more than one meaning and requesters do not consult your list before choosing them. "Password" appears in tickets about passwords, and in tickets where the requester has guessed that it might be their password. It appears in tickets where someone is asking for the password to the shared printer. It appears in tickets about a supplier's system whose password the desk cannot see. The rule matches all of them. It does not pause at the one that says "I don't think it's my password". It is not reading. It is matching, and matching is a different activity with the same vocabulary.

The Misfile Cost

A misfile is a ticket in a category it does not belong to, and its cost is paid by whoever uses the count. The report is wrong first. Priya's chart says the desk spends most of its effort on passwords, which would argue for a self-service reset project. The argument would be built on four hundred tickets that were not about passwords. The article on Categories That Make Reports Worth Reading explains what a trustworthy category report looks like. A keyword rule that misfiles a third of its matches cannot produce one, however neat the chart.

Problem management pays next. A problem investigation starts by pulling every ticket in a category and looking for the common cause. If the category holds three different causes under one label, the investigation finds a blur. The claims-system login failures at Stonebridge, which were one problem with one fix, were spread across Password, Other and Claims System. No single pull would have shown that forty of them had happened in a fortnight. The misfile did not just miscount them; it hid a pattern that a correct count would have shown in a week.

The technicians pay last, and quietly. A category set by a rule at creation is a category someone will have to correct. Correcting it means noticing it is wrong. That means reading the category field on a ticket you are trying to close, which most people do not. So the misfile stays. Worse, a field that is already filled in looks finished. A blank category at close prompts a technician to choose; a wrong category at close prompts nothing. The rule has not saved the technician's attention. It has removed the cue that would have asked for it.

The Rule That Quietly Files Everything as Other

Every category list has a last resort, usually called Other or General, and every auto-categorization scheme has a rule that sends unmatched tickets there, whether or not anyone wrote it. If the keyword rules are narrow, most tickets match none of them and land in Other. If the keyword rules are broad, most tickets match the wrong one. There is no setting between the two that is right, because the gap between them is the gap between a symptom and a cause. The rule is on the wrong side of it.

Other grows for a second reason, which is that nobody owns it. A ticket filed as Password has a wrong label somebody might notice. A ticket filed as Other has a label that is never wrong, because Other means "none of the above", and it is impossible to misfile into "none of the above". So Other fills, month by month, with tickets that were never categorized and tickets a rule could not place. The report shows it as a bar that gets taller every quarter. Priya's 402 was not a category. It was the queue's unsorted tray with a chart drawn around it.

The honest number to watch is Other's share of closed tickets, read monthly. If it is small, the categories and the rules are describing the work. If it is the largest category, either the list is wrong, which is covered in Avoiding the 400-Category Monster, or the categorizing is being left to a rule that cannot do it. (Usually both, and the rule gets blamed for the list.) An Other pile that is shrinking is the sign that people are reading tickets at close. An Other pile that is growing is the sign that a rule is reading them at open and nobody is reading them after.

Why Categorize at Close Is Still Needed

The category can only be right when the cause is known, and the cause is known at the end of the ticket, not the beginning. That is the whole argument of Categorize at Close, Not Just at Open, and auto-categorization does not change it; it sharpens it. Whatever a rule sets at creation is a label for the symptom. The technician who closes the ticket knows the cause and is the only party who ever will. If they do not set the category then, the ticket carries the rule's guess into the report forever.

The workable arrangement is to treat the rule's category, if you keep one, as a provisional label and to make the close the moment the real one is set. Some desks run two fields, one for what the requester reported and one for what was actually wrong. That gives the report both the symptom pattern and the cause pattern. It lets a problem investigation ask "how many login symptoms had an expired-account cause?" Most small desks run one field and simply require it to be confirmed at close. The technician cannot resolve the ticket without looking at the category and either keeping it or changing it. The requirement costs four seconds a ticket. Those four seconds are the entire difference between a chart and a report.

Make the confirmation a habit rather than a hope. Put the category check in the resolution routine beside the resolution note. Then closing a ticket means writing what was wrong and filing it under what was wrong in the same motion. A desk that does this can afford a keyword rule at open, because the rule's errors are corrected at close by the person who knows better. A desk that does not do this should not run the rule at all, because it will produce a report that looks complete and is not. A complete-looking wrong report is harder to argue with than a blank one.

Where Keyword Rules Are Safe

A keyword rule is safe when the keyword is a label, not a description. A monitoring tool that sends alerts with a fixed subject line is labelling each message with the thing that failed. A rule that files "ALERT: backup job failed" under Backup is reading a fact. An HR system that e-mails "New starter: Ravi Mehta, starts Monday" is labelling a request type. A supplier's portal that sends "Ticket closed: case 48812" is labelling an update. In each case a machine wrote the words from a fixed template, the words have one meaning, and the rule can be right every time. The test is whether the word was chosen by a system or by a person. Systems are consistent. People are describing.

The form is the other safe source, because a choice on a form is a fact the requester supplied rather than a word they happened to use. If you run QuerySolv, each department has its own web form. What the requester picks on that form is a fact rather than a guess. That is a far more honest basis for any label than anything typed into the description box. A form that asks "which system?" with five plain choices will be right more often than any keyword list, because the requester is answering a question rather than composing one. Keep the choices few and in the requester's words, and map each one to exactly one label.

Where a keyword rule fails the single-meaning test but the pattern is still useful, turn the rule into a hint. Instead of setting the category, have it add an internal note: "subject mentions 'password'; check whether this is a reset, an expired account, or a supplier login." The technician sees the hint at first touch, the category stays blank until a person fills it, and the rule's knowledge is used without its confidence. This is the same move that turns a bad routing rule into a useful one in Auto-Routing Rules That Do Not Misfire, and it works for the same reason. A guess offered to a person is help. A guess written into a field is a decision nobody made.

Measuring the Misfile Rate

Once a month, pull thirty tickets that a keyword rule categorized and have two people, separately, write down the category each ticket should have had, judged from the resolution note. Count the tickets where the rule's category differs from both people's answer; that count out of thirty is the misfile rate. Count also the tickets where the two people disagree with each other. Those are the tickets where the category list itself is unclear, and no rule will fix that. The exercise takes about forty minutes and tells you more about your categories than the quarterly report ever will.

Decide in advance what rate switches the rule off. One misfile in thirty is the kind of error a close-time confirmation will catch. Five in thirty means the rule is wrong one time in six and the report cannot be trusted without a manual pass. Ten in thirty means the rule is a random label generator with a plausible name, and it should become a hint or be retired that afternoon. Write the threshold in the register before the first sample, because a threshold chosen after the count is always set just above the count.

• Pull thirty closed tickets from the last month that the rule categorized.
• Two people, separately, write the correct category from the resolution note.
• Count tickets where the rule's category differs from both people.
• Count tickets where the two people disagree; those are list problems, not rule problems.
• Compare the misfile count to the threshold written in the register.
• Record the count, the date, and the decision: keep, make it a hint, or retire.
• Read Other's share of the month's closed tickets and note whether it rose or fell.

A Short War Story: The Scanner Count at Bellweather Logistics

At Bellweather Logistics, a rule filed any ticket containing the word "scanner" under Hardware. Over one autumn the Hardware category climbed from a handful of tickets a month to more than sixty, nearly all from the two distribution warehouses. The operations director, reading the quarterly chart, concluded that the handheld scanners were wearing out and approved the replacement of fourteen units before the busy season. A technician called Dev spent two weekends swapping them. The new scanners failed in exactly the same way, because the tickets had never been about the scanners. The warehouse application's barcode module rejected any label printed by the newer label printers. Every requester had reasonably described the problem as "the scanner won't read it". Dev found the cause in an afternoon once he read twenty of the tickets end to end, and a configuration change on the application fixed all of them. The cost was fourteen scanners nobody needed, two weekends, and a quarter of a problem hidden inside a category that had been counted with great precision. Bellweather now confirms the category at close, and the rule that filed "scanner" under Hardware adds a note instead, which says "scanner mentioned; check the label printer first".

The Keyword Rule Table

The table below is the register for every keyword rule the desk runs, one row per rule. It is meant to be filled in before the rule is switched on and updated after each monthly sample. The first rows are examples from Stonebridge after Priya's clean-up, kept because they show the three outcomes. A rule stays because its keyword is a system label. A rule becomes a hint because the word has too many meanings. Or a rule is retired. Copy the header, delete the examples, and fill in your own. The "other meanings seen" column is the one that stops arguments, because it is a list of real tickets rather than an opinion.

Priya's next quarterly chart had Password at 148 tickets, which was roughly how many resets the desk remembered doing. It had Other at 71, which was the smallest it had been since the rule was written. The claims-system login failures appeared as their own bar for the first time, forty-four of them in one fortnight, and became a problem ticket the following week. The chart was less tidy than before. It was also, for the first time in a year, about the desk.

Frequently Asked Questions