Designing a Fair Trial: Real Tickets, Two Weeks, Five Scenarios

Wednesday, 11:15, at Harbourview Clinic. Dev has had the trial instance of a ticketing tool for nine days. It contains eleven tickets, all raised by Dev, all assigned to Dev, most with the subject "test". The vendor's sample data is still in it. So the queue also holds forty tickets from a requester called Jane Sample about a printer at a company that does not exist. Nobody else on the three-person desk has signed in, because nobody was asked to. The practice manager puts her head round the door and says "so, is it any good?" Dev has clicked on most of the menus and found nothing alarming. He says "seems fine". That is the entire evaluation. The trial will expire on Friday having answered no question anyone could write down.

A trial is a fixed period in which the desk runs a candidate tool on real work, under conditions written down in advance. Its purpose is to answer the questions that documentation and demos could not. A fair trial gives every candidate the same tickets, the same people, the same staged scenarios and the same pass marks. That way, the result says something about the tools rather than about which fortnight was quieter. This article is part of our Choosing a Ticketing System series. And it is about designing a trial that produces a result instead of a feeling.

It covers why a trial is not a play session and how to feed it real tickets. Next come why two weeks and why the whole team, then the five scenarios every candidate must survive, in turn. We cover the exit criteria you write before day one and how to run two candidates fairly. The scenario table, a short war story, and the trial plan to copy follow, in that order.

Why a Trial Is Not a Play Session

A play session is one person clicking through a tool to see whether it feels all right. And it answers the question "did this person like it", which is a real question and not the one you are paying to answer. The questions a trial exists to answer are the ones left over after the requirements and the scoring sheet. Does the thing documented as working actually work on our mail, our departments, our people and our Friday afternoons? Those are behaviours, and behaviours only show up under load. A tool with eleven test tickets in it is not under load. It is being admired.

The difference is written down in advance. A trial has a start date, an end date, and a list of people who will use it and in which roles. It has a set of scenarios that will be staged on known days, and a statement of what passing looks like for each. All of that is agreed before anyone signs in. A play session has none of those, which is why it ends with "seems fine". That is why "seems fine" is the most expensive sentence in this series. It has bought more unsuitable tools than any brochure.

There is a second reason to write it down first, and it is about fairness to the candidates rather than to you. Consider two tools trialled on different weeks, by different people, on different tickets, with the scenarios improvised. They will produce two sets of impressions that cannot be compared. The comparison will quietly become "which one did the loudest technician prefer". A written plan run twice is the only way to make the result about the tools. The vendors, if they are any good, will thank you for it. The ones who do not are telling you something too.

Real Tickets, Not Sample Data

Delete the sample data on the first morning. Jane Sample's printer has never been your problem. The sample tickets were written to make the tool look tidy. Every hour spent reading them is an hour the trial is not spent on your work. A queue full of invented tickets also hides the thing you most need to see: how the tool behaves on messages from real people. Those messages are written in a hurry, with the wrong subject line, three attachments and a forwarded chain underneath.

Feed the trial your real intake, which is harder than it sounds because your real intake is already going into the old tool. There are three honest ways to do it, in rising order of effort and faithfulness. The cheapest is to copy: set the live mailbox to forward a copy of everything to the trial tool's intake address. Then the trial queue fills with the same tickets as the live one while the old tool keeps doing the work. The technicians work the real ticket in the old tool. They mirror the key actions, claim, reply, resolve, in the trial tool for a chosen subset. This doubles some typing for two weeks. It is also the only way to see how the tool handles the volume and the mess of real mail without risking a single requester.

The middle route is to move one department. Pick the smallest department with its own mailbox, and point that mailbox at the trial tool for the two weeks. Run that department's work there for real, with the old tool handling everything else. This produces genuine behaviour, including requesters replying and reopening. The cost is one department's tickets living in two places afterwards; keep an export. The most faithful route, moving everything, is not a trial. It is a migration with a return ticket, and return tickets on migrations are rarely honoured.

Whichever route you take, the trial must see at least a hundred real tickets or the whole of a typical fortnight, whichever is more. Under that number the tool has not met your edge cases, and your edge cases are the reason you are changing tools. Keep a note of which route you chose and write it in the decision record. After all, "we trialled it on copied mail" and "we trialled it on the fleet department live" are different strengths of evidence. The person reading the record in two years deserves to know which they are looking at.

Two Weeks, and the Whole Team

Two weeks is the shortest trial that contains everything a desk does. It has two Mondays, which is when the queue is deepest, and two Friday afternoons, which is when corners get cut. It has one weekend, which is when the service-level clock either pauses or does not. And it has enough ordinary days for a ticket to be opened, waited on, chased, resolved and reopened. One week has none of the second instances and you cannot tell a pattern from an accident. Four weeks is better in theory and in practice means the second fortnight is spent not quite using either tool. Two, fully used, beats four half-used.

Everyone who will touch the tool touches it during the trial, in the role they will hold. Every technician works tickets in it, not just the one who was keen. The desk lead moves a ticket between departments, changes a target and reads the reports. A manager who was not involved in choosing signs in as a manager and tries to find the number they are usually given. Two or three real requesters, warned and willing, raise tickets through the portal and reply to them by e-mail from their own addresses. A trial run by the one enthusiast on the desk measures the enthusiast. Consider the quiet technician who has used the old tool for six years and will have to use the new one for the next six. That is the person whose opinion the trial most needs, and the person least likely to volunteer it.

Make the trial somebody's job for the fortnight. One person, usually the desk lead, owns the plan. They stage the scenarios on the right days, collect the notes, nag the quiet technician, and write the result. Without an owner, the trial becomes a thing everyone meant to look at, which is what happened at Harbourview. With one, it becomes two weeks of ordinary work with five deliberate experiments inside it, which is all it ever needed to be.

Scenarios One and Two: The E-mail Loop and the Misrouted Ticket

The five scenarios are staged events, each designed to make the tool do something that demos avoid and documentation describes optimistically. They are run on the same day of the trial for every candidate, by the same person. And each has a pass mark written before it is run. The first is the e-mail loop. Two systems each reply automatically to the other's automatic reply, forever, until somebody notices or the mailbox fills. The mechanics are in loops and auto-reply storms; the trial only needs to provoke one safely and see what the tool does.

Stage it with a test mailbox you control. Set an out-of-office or automatic reply on that mailbox. Then raise a ticket from it to the trial tool's intake address, so that the tool's acknowledgement meets the automatic reply. Wait an hour and count the messages in both directions. To pass, there must be one ticket and one acknowledgement. The automatic reply must be either recognised and ignored or attached to the ticket as a reply with no further acknowledgement sent. Fail: a growing chain of acknowledgements and automatic replies, or a new ticket for every automatic reply. Run it on day two, early. A tool that fails this one will fail it on a real requester's holiday within the first month of ownership. That will be at three in the morning, on a mailbox with a quota.

The second scenario is the ticket sent to the wrong place. Requesters pick the wrong form, reply to the wrong address and forward to the wrong team. A tool that cannot recover gracefully from that turns every mistake into a lost ticket. Stage it on day three. Send a request to the facilities address whose body plainly says it is an IT problem, or pick the wrong department on the form. Then watch.

Pass is four things. A technician in the wrong department can see it and tell, from the list, that it does not belong. Someone with the right permission can move it to the right department in a few clicks, without recreating it. The ticket keeps its number, its history and its original timestamps after the move, so the service-level clock does not quietly restart. And the requester is either told, or at least not sent anything that suggests a brand-new ticket was raised. Fail is any version of the ticket being closed and re-raised or losing its history. It also fails if it is movable only by an administrator who is not on the desk. Watch also for the quieter fail, where the ticket moves but nothing tells the receiving department it arrived.

Scenarios Three and Four: The Breached Target and the Reopen

The third scenario is a ticket that misses its service-level target, which every desk will have and most demos will not show. Set a deliberately short response target on one priority for the trial, an hour or so. Raise a ticket at that priority, and leave it alone past the hour. Then look at what the tool did. What a tool does at the moment a target breaches is the clearest view you will get of how it thinks about promises.

To pass, the ticket must be visibly marked as approaching the target before the hour and as breached after it. That must happen in the list the technicians actually look at, not only on a report. The lead must be able to see every breached ticket in one place. The breach must appear in a report afterwards with the ticket number and the time by which it was missed. Run the scenario so that the hour spans a lunch break. Or, better, stage a second one on the Friday so that the target runs across the weekend, and read the due time on Monday. Some tools pause the clock outside business hours and some do not. Some pause when a ticket is waiting on the requester. The pass mark is that the tool does what your service-level document says. The point of staging it is that you will not have to find out from a complaint. What a breach teaches once you are live is the subject of service-level breaches and what they teach. During the trial, the lesson is only whether the tool will let you learn it.

The fourth scenario is a requester replying to a ticket you have already resolved. It is the commonest event a demo never shows and the behaviour most often assumed. Getting it wrong splits every returning problem into two tickets that nobody connects. Stage it on day eight: resolve a real or staged ticket, and wait a day. Then reply to the resolution e-mail from the requester's own mailbox, ideally a personal address rather than the test one, with "this has come back".

To pass, the same ticket must reopen, with its full history and the new reply at the end, visible in the active list again. The reopen must be recorded so that it can be counted later. Tools differ here more than anywhere else in the five. Some reopen the ticket in place whatever channel the reply came through, and QuerySolv is one of those. Some reopen only within a window after resolution and start a new ticket after it. Some always create a new ticket and leave the desk to link the pair by hand. None of those behaviours is dishonest, but only the first matches what a requester believes they are doing when they press reply. Only the first gives you a reopen count worth reading. That is the number what a reopen is telling you depends on. Stage it from a closed ticket too, if your workflow closes tickets after a confirmation window, because the answer is sometimes different.

Scenario Five: The Report to a Manager

The fifth scenario happens on the last day. It is the one that decides whether the desk will be believed for the next five years. Produce, from the trial data alone, one page for a manager. Include tickets in and out over the fortnight, the age of the oldest open ticket, response time against target by priority, and the number of reopens. Hand it to a manager who took no part in the trial. Ask two questions: can you read it, and what would you ask next? Then try to answer what they asked, from the tool, while they wait.

To pass, the page must have taken the lead under an hour to produce without a consultant or a spreadsheet export. The manager must have understood it unaided. And the follow-up question must have been answerable from the tool in a few minutes. Fail: the numbers had to be assembled by hand, the manager needed the page explained, or the follow-up question needed a feature nobody had. This scenario is the one most trials skip because it feels like paperwork. And it is the one that catches a tool whose reports were built for a demo rather than for a Tuesday. How to write the page itself is in explaining technical things to management; the trial only needs to prove the tool can feed it.

Exit Criteria Written Before Day One

Exit criteria are statements written before the trial starts. They say what result ends it with a pass and what result ends it with a fail. Without them a trial does not end; it expires, and the decision is made afterwards by whoever remembers it most warmly. With them the last day is a reading of the sheet rather than a debate. There are three kinds, and the plan needs all three.

The first kind is the scenarios: each of the five is a pass or a fail by its written mark. A candidate that fails any one of them does not go forward unless the desk decides, in writing, that it can live with the workaround. In that case, the desk must also name who will maintain it. The second kind is the scoring sheet from evaluation criteria that actually matter. It is rescored at the end of the trial with every row marked as tried rather than read. The trial exists to turn documentation scores into tried ones. And a row that could not be tried is a question for the vendor, not a point. The third kind is the team's verdict, collected one by one and in writing. It uses a fixed question: "could you do your job in this for the next five years, and what one thing would stop you?" The written answers are kept with the plan. The quiet technician's answer is usually the one that matters.

Decide in advance who reads the exit criteria and declares the result, and make it one person with the authority to say "fail". Usually it is the desk lead, with the manager who owns the budget reading the result rather than making it. A trial whose result is declared by a committee will declare a draw, because a committee can always find one more week to run.

Running Two Candidates Fairly

If two candidates survive the scoring sheet, trial both, and make the two trials as alike as the calendar allows. Use the same two weeks if the team can bear two tools at once, which with the copied-mail approach they often can. Otherwise use consecutive fortnights, with the quieter one noted. Same person stages each scenario on the same day of the trial. Same requesters raise the same kinds of tickets. Same manager receives the same one-page report. Any difference between the two trials is a difference you cannot later separate from the difference between the tools. So write down every one you could not avoid.

Let each vendor help with setup and nothing else. Configuring departments, mailboxes and targets with a vendor's engineer on the call is reasonable and often saves a day. It is their product and they know where the settings are. Having a vendor's engineer sit with the desk while scenarios are run is not reasonable, because the engineer will drive. A scenario the vendor drove measures the vendor. Tell them the trial plan exists, tell them when it starts and ends, and tell them, kindly, that you will call if you are stuck. Most will respect it. The ones who ask for the plan so they can prepare are, in my experience, the ones whose tools do not need them to.

The table below is the five scenarios on one page, with the day, the staging and the pass and fail marks. Copy it into the plan and fill in your own days; the ones shown assume a trial that begins on a Monday. Run them in this order, because the later scenarios need tickets that the earlier ones created.

A Short War Story: The Quiet Fortnight at Bellweather Logistics

Bellweather Logistics trialled two ticketing tools in consecutive fortnights with no written plan. The second fortnight fell across the week the depots shut for stocktake. So the second tool saw about a third of the tickets the first had and none of the Monday peaks. Priya, who led the desk, reported that the second tool "felt calmer". The team agreed, and it was bought. In its first full Monday the depot mailbox's automatic acknowledgement met a courier partner's automatic reply and produced four hundred messages before lunch. The first tool had quietly handled that during its own busier fortnight without anyone noticing it had. The desk spent two days clearing the storm and a further three weeks finding tickets that had been created in the noise. That was roughly eighty technician hours in all. The courier partner's account manager sent an e-mail that Priya still describes as "memorable". The tool was not bad. It had simply never been asked the question, and a trial that does not ask the question gets the answer the calendar gives it.

The Trial Plan

The plan below is one page per candidate, written before the trial starts and signed by the person who will declare the result. The top block fixes the dates, the people and the route the real tickets will take. The middle block is the five scenarios with their days and pass marks copied from the table. The bottom block is the exit criteria and the verdict, filled in on the last day. Run it identically for every candidate and attach the completed copies to the decision record described in making the decision and writing it down. A plan that is not attached to the record is a plan somebody will remember differently.

TRIAL PLAN — ONE PAGE PER CANDIDATE
Candidate: ______________________   Trial owner: ______________   Signed: ____________
Start (Monday): ____________   End (Friday, week two): ____________

PEOPLE AND ROLES (everyone who will use the tool, in the role they will hold)
  Technicians working tickets in it: ______________________________________
  Lead (moves tickets, sets targets, runs reports): ______________
  Manager receiving the one-page report (not involved in choosing): ______________
  Requesters raising and replying (two or three, warned): ______________
  Vendor help allowed for: setup only.   Vendor present during scenarios: no.

REAL TICKETS (choose one route and record it)
  [ ] Copied mail: live mailbox forwards a copy to the trial intake; actions mirrored for ____ tickets
  [ ] One department live: ______________ department runs in the trial tool; export kept
  Sample data deleted on day one: [ ]     Minimum real tickets seen: 100 or a full fortnight

SCENARIOS (same day, same person, pass mark fixed before it is run)
  1. E-mail loop          day 2     staged by ________   pass [ ] fail [ ]   note: __________
  2. Misrouted ticket     day 3     staged by ________   pass [ ] fail [ ]   note: __________
  3. Breached target      day 4-5   staged by ________   pass [ ] fail [ ]   note: __________
  4. Reopen               day 8     staged by ________   pass [ ] fail [ ]   note: __________
  5. Report to a manager  day 10    staged by ________   pass [ ] fail [ ]   note: __________
  Any scenario failed: workaround accepted? [ ] no  [ ] yes, maintained by ________

EXIT CRITERIA (written before day one)
  All five scenarios passed, or every fail has an accepted workaround:          [ ]
  Scoring sheet rescored with every row marked T (tried); rows still D or S: __
  Each team member answered in writing: "Could you do your job in this for
  five years, and what one thing would stop you?"   Answers attached:          [ ]
  Export of everything created in the trial taken and opened:                  [ ]

VERDICT (last day, by the trial owner alone)
  [ ] Pass: goes to the decision record   [ ] Fail: because ______________________
  Differences from the other candidate's trial we could not avoid: ______________
  Signed: ______________   Date: ____________

Dev's second attempt at Harbourview used the plan, with the copied-mail route and the practice manager as the uninvolved reader. The e-mail loop passed, and the reopen passed. The breached target revealed that the tool paused its clock on hold when the clinic's own service-level document said it should not. That became a documented workaround with Dev's name on it rather than a surprise. The practice manager read the one-page report without asking what anything meant. She did ask what the oldest open ticket was about, and Dev answered from the tool in about forty seconds. That was the moment the trial had actually been for.

Frequently Asked Questions