Knowing Whether Your Templates Actually Help

Tuesday, 15:30, at Halden Transit, a bus operator with four depots. Bea has run the IT desk for a year and brought in a template library three months ago. The operations director, Marcus, stops at her desk with a question he has clearly been saving: "Since you brought in those canned replies, are things better, or just faster?" Bea has the first-response time, which fell from a little over three hours to eleven minutes. She says so. Marcus nods and adds that the manager at the north depot told him last week that IT "sends robot mail now". Marcus adds that the manager had rather liked the old slow replies because at least someone had read her ticket. Bea has no number for that. She has one number, and it is the one that makes the desk look good.

Measuring templates means deciding whether the library is helping the requester, the technician and the desk, or only helping the clock. It uses numbers the desk mostly already has plus one it must start collecting. The clock is the easy one and it is where most desks stop. This article is part of our Templates & Canned Responses Done Right series. This article is the last in the series because it is the one that tells you whether the other five worked.

It covers why "help" has three readers who disagree, and reply time, which is necessary and not sufficient. Next come back-and-forth, the number that catches the form, and satisfaction scores and, more usefully, the comments. Then come the "it sounded automated" complaint treated as a defect report, and which templates are used and which are not, and what both tell you. The technician's time, which you find out by asking, and reading the whole scorecard follow. A short war story and the template scorecard to copy come last, in that order.

"Help" Has Three Readers

A template can help the technician, help the requester, and help the desk lead, and it can do any one of those while hurting another. The missing-information template helps the technician, who sends it in ten seconds. If it asks for things the requester already provided, it hurts the requester, who must now reply to say so. In that case, it also hurts the desk, which gets a second exchange and a dented reputation. A reply-time chart shows the first effect and hides the other two. Marcus's question was really whether anyone had looked at the other two, and the honest answer at Halden was no.

So measure from all three positions. The requester's position is measured by back-and-forth, satisfaction and the specific complaint. The technician's position is measured by time saved and by which templates they reach for and which they avoid. The desk's position is measured by reply time, by reopens, and by whether the numbers that got better did so without the others getting worse. None of the measures is hard to collect. The difficulty is in agreeing to look at the ones that might come back unflattering, which is a decision rather than a technique.

Take a baseline before the library goes in, or as soon as possible after, because the question "is it better?" needs a "than what". Use a month of the numbers below from before templates, or from the first month with them if that is all you have. That is the comparison every later quarter is read against. A desk that cannot answer "better than what?" will end up defending the library with the one number it has, which is what Bea was about to do.

Reply Time: Necessary, Not Sufficient

First response time is the gap between a ticket arriving and the first human message going back to the requester. It is the number templates move most and fastest. It should fall when a library goes in. If it does not, the templates are not being used, or they are so hard to find that typing is quicker. The usage count later in this article will say which. A falling first response time is good news for the requester too, up to a point. The acknowledgment that arrives in eleven minutes tells them a person has their ticket, and nothing else about the desk reassures as cheaply.

The point past which it stops being good news is the point where the reply says nothing. A canned acknowledgment sent in forty seconds with no name, no specifics and no time is the black hole with a timestamp. The requester has been answered and is no better informed than before. Reply time measures how fast the desk said something. It has no opinion on what. That is why it is the first number on the scorecard and never the only one. It is also why the next number exists to catch what this one hides.

Read reply time by template as well as by desk. If the acknowledgment's reply time fell and the cannot-reproduce message's did not, the second template is not being used. Or it is being used as a draft and then rewritten at length, which may be right. If every template's reply time is a few seconds, nobody is personalizing. In that case, the ten-second routine from Personalizing a Canned Response in Ten Seconds is not happening. A reply time that is too good is a finding.

Back-and-Forth: Messages per Resolved Ticket

Back-and-forth is the number of messages, in both directions, between the first reply and the resolution, counted per resolved ticket and averaged over a month. It is the number that catches the form. A template might ask for something the requester already gave, or ask two things and get one answered. Or it might give no time and provoke "any update?". Each of those templates adds a message to every ticket it touches. A library that is helping the requester brings this number down, because the one ask is clearer and the time is there. A library that is helping only the technician pushes it up. And the technician does not notice, because each extra message is only one more reply.

Count it by template where you can. Tag or search the first reply on each resolved ticket for which template it came from, then count the messages that followed. The missing-information template should be followed by one requester reply and then work. If it is routinely followed by two or three, it is asking badly. The resolved template should be followed by one confirmation or by nothing. If it is routinely followed by a question, it is not saying what was done in words the requester understands. The numbers will be small and noisy on a small desk. They are still the clearest signal you have of what the requester experienced.

Keep a separate tally of a specific kind of reply. These replies say "did you read my ticket?", "I already sent that", "as I said in my first message", or anything with the same meaning. Each one is a requester telling you that a template was sent unread. And each one can be traced to the template and the sender. On a desk running the ten-second routine the tally should be close to zero. On a desk that is not, it is the single most persuasive number to put in front of the technicians. That is because it is in the requesters' own words.

Satisfaction and the Comments

The satisfaction score, where the desk has one, is the number everyone looks at. It is also the one that tells you least about templates on its own. That is because it moves slowly, is answered by few, and mixes together everything the desk did on the ticket. Watch it, compare it to the baseline, and expect it to drift upward slightly as the first-reply experience improves. Do not expect it to prove anything by itself about the library. Requesters do not fill in surveys to say the acknowledgment was adequate.

The comments are a different matter. Where the survey has a free-text field, read every comment for a month and count the ones that mention the desk's messages, in either direction. The words that mean trouble are "automated", "robot", "generic", "copy and paste", "template", "form letter", "didn't read" and "the same reply as last time". The words that mean the library is working are quieter: "quick", "clear", "knew when I'd hear back", "told me what was happening". Tally both kinds each month. A rising count of the first kind is the earliest warning the desk will get that a template has gone wrong. And it arrives weeks before the score moves.

Where there is no survey, the comments still exist. They are in the tickets. A search of requester replies for the same trouble words finds them. A desk that reads its own tickets for a month with that list in mind will come away knowing which templates are the problem. The reopen rate belongs here too. A resolved template that fails to say what was done or how to tell if it comes back produces reopens. The honest way to count those is in Measuring Reopens Honestly. A library that lowers reply time and raises reopens has moved the work from the first message to the fourth.

The "It Sounded Automated" Complaint as a Defect Report

Treat every complaint that a message sounded automated as a defect report against a specific template. Handle it the way a good desk handles any defect. Find which one, find why, fix it, and thank the person who reported it. The complaint is not an opinion about templates in general, though it will be phrased as one. It is a fact about one message that one requester received. And that message came from one template, sent by one technician, with its slots in some particular state. All of that can be looked up in a minute on the ticket.

The cause is nearly always one of three. The slots were not filled. In that case, the message went out with the directory's name, the template's example time, or no specific thing, and the requester recognized the shape. The added sentence was missing. In that case, everything was filled in correctly and the message still contained nothing that could only have been about this ticket. A requester feels that even when they cannot say why. Or the specifics were wrong. In that case, the message named a different problem, the previous ticket's details, or a time that had already passed. The first and third are habit problems and are fixed with the technician, kindly, by showing them the message. The second is a template problem, and the fix is a slot for the sentence with a note that it is not optional.

Do not defend the template to the requester, and do not explain that it was a template. Reply by hand with the message they should have received, which proves the point better than any explanation. Then, at the monthly tally, record the complaint against the template. That makes a template with three complaints in a quarter the first thing the owner looks at in the review described in Keeping the Template Library Alive and Trusted. The complaint is the most useful thing a requester will ever send the desk for free, and it arrives already prioritized.

Which Templates Are Used, and Which Are Not

Count how many times each shared template was sent in the month. Some tools report this. Where yours does not, search sent replies for a short phrase that appears only in that template, such as "either way, I will update you by", and count the results. The count is approximate and it is enough, because the two findings that matter are at the extremes. One is templates sent never. The other is templates sent far more often than the situation they are for should arise.

A shared template sent never, for two quarters, may be about a situation that has stopped happening. Or it is about one that technicians are handling by hand because they do not like the message. Cannot reproduce and the plan has changed are the usual unused ones, and in both cases the situation has not stopped. The fix is a conversation, not a retirement. Ask what people send instead. Read a few of the hand-written versions. Then either improve the template so people will use it or discover that the hand-written versions are better and adopt one. A template nobody uses is a question, and the question is "what are you sending instead?"

A template sent far more often than it should be is a finding about the desk, not the template. If the missing-information template goes out on a large share of new tickets, the desk is routinely receiving tickets without the information it needs. In that case, the problem is the form or the first-touch discipline, not the message. For that problem, the fix is in Gathering the Right Information Up Front. If the no-reply-closing-soon template is sent on a large share of tickets, the desk is asking for things requesters do not understand or do not think they need to give. If the reopened template is busy, the resolved template is not. Usage counts are the library telling you what the desk is like. Listen to them before changing the words.

Technician Time: Ask Them

The technician's side of the ledger is measured by asking. No report shows how long it took to find the right template or how much the ten seconds of personalizing actually cost on a bad afternoon. Once a quarter, at the same time as the library review, ask each technician three questions. Which template saves you the most time? Which one do you avoid, and what do you send instead? Which one takes longer to find than to type? The answers are short and they are the most direct evidence you will get about the technician's experience of the library.

Expect the third answer to point at naming. A template that takes longer to find than to type is a template that is badly named, or buried in a list of eighty. Or it is stored somewhere other than where the technician is when they need it. It is not a reason to drop the template. It is a reason to rename it, cut the list, or move it. Expect the second answer to point at the awkward templates, and treat it the way the previous section treats an unused one. And expect the first answer to be the acknowledgment or the status update, which is a useful reminder of where the library actually earns its keep. That is on the dull message, sent forty times a day, by a person who would otherwise be typing it.

Technicians will also tell you when the templates are being used to avoid thinking, if you ask in a way that makes it safe to say. A desk may send the resolved template without testing the fix, because the template makes closing quick. That desk has a template problem that no requester-side number will show until the reopens arrive. Ask "where do the templates make it too easy to do the wrong thing?" and write the answers on the scorecard. The question is worth more than the answer to any other on the sheet.

Reading the Scorecard

The table is the scorecard's logic. It shows each signal, where to get it, what healthy looks like, what unhealthy means, and the first thing to try. Read it once a month for the tallies and once a quarter alongside the library review, with the baseline month beside it. The pattern to look for is not any one number but the relationship between the first row and the rest. Reply time down and everything else flat or better is a library that helps. Reply time down and back-and-forth, complaints or reopens up is a library that moved the work to the requester. In that case, the fix is in the specific templates the other rows point at, not in abandoning the library.

If you run QuerySolv, the reports on response times, reopen rates and satisfaction already exist as standing reports. In that case, the scorecard is a monthly tally on top of them plus one search rather than a project. On any tool, the two rows you will have to collect by hand are the "did you read my ticket?" count and the comment tally. Those two are the ones most worth the effort. A scorecard with only the first row filled in is the report that answers an easy question. That is the failure that The Seven Ways Support Quietly Fails calls metric theater. A template library is unusually good at producing it.

Decide in advance what a bad quarter means you will do, because deciding afterwards produces either denial or an overreaction. A bad quarter means the two or three templates the scorecard points at get fixed. The technicians whose messages produced the complaints are shown the messages. And the next quarter is compared to this one. A bad quarter does not mean the library is removed. The alternative to templates is forty hand-written acknowledgments a day, and the scorecard from that era, if you have one, will not be flattering either.

A Short War Story: The Faster Robot at Granite Peak Outfitters

Granite Peak Outfitters, an outdoor retailer with thirty stores, brought in a template library. Its desk lead, Tomasz, reported at the next quarterly review that first response time had fallen from about four hours to under fifteen minutes. That earned a round of thanks from the retail director. Tomasz did not report, because he had not looked, that messages per resolved ticket had risen from three to just under five. Nor did he report that the reopen rate had climbed because the resolved template said "this has been resolved". It said nothing about what had been done or how to tell if it came back. He also did not report that the satisfaction comments for the quarter contained "copy and paste" eight times and "did you read" four. The store manager at the flagship, Luis, wrote in his survey comment: "I would rather wait an hour for a real answer than get a robot in ten minutes." The retail director read the comments before the following review and asked Tomasz why the report had shown one number. The desk spent the next month rebuilding the resolved and missing-information templates with required lines and running the ten-second routine. After that, the back-and-forth fell below the pre-library figure and the comments went quiet. The cost was a quarter of worse service that the desk had reported as a triumph. Tomasz's lesson, which he wrote at the top of the scorecard, was that the clock measures the desk and the comments measure the requester. A report that shows only the first is a report about yourself.

The Template Scorecard

The scorecard is one page, filled in monthly for the tallies and read quarterly with the baseline beside it. The top block is the desk-level numbers. The middle block is one row per shared template, which is where the findings live. The bottom block is the technician answers and the decisions. Fill in the baseline column once and never change it. Fill in the rest honestly, including the rows that are hard to collect, because a scorecard with gaps in the requester-side rows is Tomasz's report with better formatting.

TEMPLATE SCORECARD — ONE PAGE, MONTHLY TALLY, QUARTERLY READ
Desk: ______________________   Owner: ______________   Month: ________   Baseline month: ________

DESK-LEVEL NUMBERS                               Baseline     This month    Direction
  First response time (median)                   ________     ________      up / down / flat
  Messages per resolved ticket                    ________     ________      up / down / flat
  "Did you read my ticket?" replies (count)       ________     ________      up / down / flat
  Satisfaction score                              ________     ________      up / down / flat
  Comments naming the messages: trouble words     ________     ________      up / down / flat
  Comments naming the messages: praise words      ________     ________      up / down / flat
  Reopen rate                                     ________     ________      up / down / flat

PER TEMPLATE (one row for each shared template)
  #   Name                        Sent    First reply   Messages      "Automated"   Reopens      Finding
                                  (count) (median)      after it      complaints    after it
  __  _______________________     ____    ________      ____          ____          ____         ______________
  __  _______________________     ____    ________      ____          ____          ____         ______________
  __  _______________________     ____    ________      ____          ____          ____         ______________
  __  _______________________     ____    ________      ____          ____          ____         ______________
  (continue for every template)

  Unused this quarter: ____________________   What is sent instead: ______________________
  Dominant this quarter: __________________   What it says about intake or closing: ______

COMPLAINTS TRACED (each "it sounded automated" or equivalent)
  Ticket ____  Template ____  Cause: unfilled slot / no added sentence / wrong specifics   Fixed with: ______
  Ticket ____  Template ____  Cause: unfilled slot / no added sentence / wrong specifics   Fixed with: ______

TECHNICIAN ANSWERS (quarterly, each person)
  Saves most time: ______________   Avoided, and what is sent instead: ______________________
  Harder to find than to type: ______________   Where templates make the wrong thing too easy: ______________

READING
  Reply time fell AND nothing else got worse:  [ ] yes  [ ] no
  Templates to fix this quarter: ____  ____  ____     Habits to address (names, kindly): ______________
  Compared with last quarter, the library is:  [ ] helping  [ ] helping the desk only  [ ] hurting
  Decisions and owner: ______________________________________________________________________

Bea filled in the first scorecard at Halden the week after Marcus's visit, with the month before the library as the baseline. Reply time was down, as she had said. Messages per resolved ticket were up by nearly one. The "did you read my ticket?" count for the month was six. Five of the six traced to the missing-information template sent by two technicians with the slot for the specific thing left as the template's example. The north depot manager's "robot mail" was one of the six. Bea fixed the template so the slot could not be missed and showed the two technicians their own messages. She brought Marcus the full scorecard a month later, with the back-and-forth below baseline and the count at one. He asked why she had not shown him that the first time. She said that the first time she had only had the number that made the desk look good. She said she had since stopped trusting any report with one number on it.

Frequently Asked Questions