Evaluation Criteria That Actually Matter

Tuesday, 14:30, at Stonebridge Insurance. Hannah has been handed a spreadsheet by the procurement team. It has 212 rows, one per criterion, each with a weight to two decimal places, and three candidate tools scored across the top. The totals are 847.25, 846.90 and 712.40. Procurement is pleased, because the spreadsheet is thorough. Hannah, who runs the five-person desk, reads down the rows and finds "supports attachments" weighted the same as "supports multiple languages in the portal". There is a criterion called "modern interface" with no definition. There is nothing at all about whether a store manager in one region can see a ticket raised by a store manager in another. Every score was entered by one person, in one afternoon, from the brochures. The spreadsheet is a monument. It is not an evaluation.

An evaluation criterion is a question you ask every candidate in exactly the same way, with an answer you can write on one sheet and defend to a colleague. Criteria are what the "should" band of your requirements becomes once you decide how to score it. The whole point of them is to make three tools comparable without anyone having to remember which demo they liked. This article is part of our Choosing a Ticketing System series. It is about which criteria earn a row on the sheet, and how to score them without the arithmetic becoming theatre. It also covers which criteria sound important and are not.

It covers where criteria come from and why ten is about right. Next come the ten criteria in four groups, from how work gets in to how you leave. Then it covers scoring without theatre, and the criteria that only sound important. It finishes with the criteria table, a short war story, and the scoring sheet to copy, in that order.

Criteria Are the Should List With a Scoring Rule

The requirements worksheet from writing the requirements before the demo sorted everything into must, should and nice. The musts have already done their work by the time you are scoring. A candidate that failed a must is gone. What remains are the shoulds. A should becomes a criterion the moment you attach three things to it. Those are a way to check it, a scale to score it on, and a weight that says how much it matters relative to the others. Without the three it is a wish. With them it is a row.

Ten criteria is about the right number for a desk of one to ten people, and the number matters more than it looks. With ten rows, each one carries weight, each one gets properly checked, and two people can score all three candidates in a morning. With two hundred rows, nobody checks anything. The scorer skims the documentation and guesses. The guesses average out into totals that differ by a third of a point. A long sheet does not make the evaluation more rigorous. It makes it more expensive to be wrong in the same way as before.

The ten below are the ones that decide whether a desk can do its job with the tool, and they are deliberately boring. None of them will make a room go quiet in a demo. Each of them has, at some desk I have known, been the reason a tool was quietly abandoned eighteen months after the signature. If your should list produced something not on this list, keep it; the ten are a floor, not a ceiling. If your should list produced forty things, the ten are where you start cutting.

How Work Gets In and Who Gets It

The first criterion is channels into one queue. Every door a request can come through should produce the same kind of ticket in the same list. That applies whether it is a web form, e-mail, chat, phone call logged by a technician, social message or an automated system. The tickets should have the same number, the same history and the same rules. Check it by creating one ticket through each door you use and looking at them side by side. Some tools do this plainly. QuerySolv, for one, treats a web form submission, an e-mail, a live chat, a social message and a REST API call as ordinary tickets in one queue. Others bolt a chat product onto the side with its own inbox, or treat e-mail as a separate module with separate rules. The desk then ends up watching three lists and calling it one. The question is not whether the brochure lists the channel. It is whether a ticket from that channel is a ticket like the others.

The second is routing: how a new ticket reaches the right people. For a small desk the right answer is usually a department with its own mailbox, form and permissions. That way, a message to the fleet address lands in the fleet queue without anyone touching it. A technician then claims the ticket by name. Check it by sending to each address and watching where the ticket lands. Then watch two technicians try to pick up the same ticket and see whether the tool tells the second one it is taken. Some tools offer automatic assignment by rotation or by skill. Whether you want that is a design question, covered in round robin, skill-based and claiming. The criterion here is only that the model you chose is one the tool supports without a workaround.

What the Desk Promises and What It Knows

The third criterion is service levels. Can the tool hold a target response time and resolution time per priority, measured on a business-hours schedule? Can it also show you which tickets are approaching or past their target? If the words are new, service levels in plain words defines them. For scoring, set a target for one priority and create a ticket at that priority on a Friday afternoon. Then see what the due time says on Monday. Whether the clock pauses when a ticket is waiting on the requester is a place where tools genuinely differ. Some pause, and many do not. The criterion is that the tool does what your service-level document says, whichever that is. Do not score a tool down for behaving differently from your old one. Score it against what you promised.

The fourth is the knowledge base. Can a technician write an article from a ticket, mark it for agents only or for everyone, and link it in a reply? Can a requester find it before raising a ticket? The check is to write one article during the evaluation, from a real ticket, and ask a requester to find it without help. A knowledge base that technicians cannot write to from inside the ticket will not be written to. A knowledge base requesters cannot find will not be read. Both of those are visible in an hour, and neither is visible in a brochure, where every knowledge base is "powerful".

What the Desk Can See, Who Can See It, and What Connects

The fifth criterion is reports, and the check is specific: can the tool produce, without a consultant, the handful of numbers a small desk actually uses? Volume in and out per month, backlog by age, response and resolution times against target, reopen and escalation rates, and whatever satisfaction measure you take. Run each one during the evaluation on the trial data and read it. Ignore the dashboard with the twelve widgets; score the reports a manager asked for last quarter. A tool with forty reports and none of those five scores low. A tool with six reports that include all five scores high. Count the ones you need, not the ones they have.

The sixth is permissions: who can see and do what. Requesters should see their own tickets and nothing else. Technicians should see their department's work and perhaps everyone's. Leads should be able to move tickets between departments and change targets. Managers should see reports without seeing every private note. Check it by signing in as each kind of person and trying to see something you should not. Permissions are the criterion most often left off the sheet. Nobody at the desk is thinking about the store manager who might see another region's tickets. Yet permissions are the criterion most likely to produce an urgent meeting with HR in month two.

The seventh is integration, and the right criterion is narrower than it sounds. You need the connections you can name today. For most desks those are mail intake, a way to sign in with the directory you already have, and whatever opens tickets automatically. And you need a plain way to build the connection you cannot name yet. That means an API that can create and update tickets and webhooks that tell another system when something happens. Check the named connections by configuring them. Check the API by reading its documentation and, if anyone on the desk can write a script, creating one ticket from a script. A list of two hundred integrations is not a criterion. Three working ones and an API is.

Where It Lives, What You Pay For, and How You Leave

The eighth criterion is hosting: whether the tool runs on your own server or in the vendor's cloud, and which of those your organisation needs. This is less a score than a filter. A desk whose data must stay inside the building cannot award points for a cloud tool's convenience. And a desk with no server and no one to run it cannot award points for self-hosting's control. Decide which model fits before you score, from your answer to question eight on the requirements worksheet. Then score within the model. The models and their trade-offs are the subject of hosting and licensing models in plain words. Here, the criterion is simply that the tool runs where you need it to run.

The ninth is the licensing model, which is not the same as the amount on the quote. A per-agent-per-month arrangement charges by head and by time and tends to shape how many people you give accounts to. A per-installation arrangement charges once for the installation and tends to shape how long you keep it. Neither is wrong. The criterion is whether the model fits how your desk will grow. You can only judge that once you have written down how many technicians you expect in three years. You also need to write down how many occasional users, such as a manager who approves things, will need to sign in. Score the fit, not the figure, and do the arithmetic in the hosting article's worksheet rather than on this sheet.

The tenth is the exit: how you get your tickets, attachments, articles and users out when you leave. That means a format another tool can read, with no need to ask anyone's permission or pay for the privilege. Check it during the trial by exporting everything you created and opening the result. Most tools will export tickets to a spreadsheet file. Fewer will include attachments and internal notes. Some will give you a database you can keep. This criterion feels premature when you are choosing a tool you have not bought yet. It is the only criterion whose importance grows every month you own the tool, and the only one you cannot renegotiate later.

Scoring Without Theatre

Scoring theatre is arithmetic that looks like rigour and decides nothing. It has two hundred rows, weights to two decimal places, a single scorer, and scores entered from brochures. The totals differ by less than the noise. It happens because a spreadsheet is reassuring and because nobody wants to be the person who chose a tool on judgement. The cure is not less structure. It is a smaller, stricter structure that two people can actually carry out.

Use a four-point scale, 0 to 3, with words attached. Zero: the tool does not do it. One: it does it with a workaround you would have to maintain. Two: it does it the way the desk needs. Three: it does it better than you asked, in a way you would use. Four points is enough to separate candidates and few enough that two scorers usually agree. A ten-point scale is a machine for producing arguments about whether something is a six or a seven. Weight each criterion 1, 2 or 3, in whole numbers, agreed before anyone scores anything, and never adjusted afterwards. The score for a row is weight times points; the total is the sum. That is the whole arithmetic, and a tired administrator can do it in their head.

Two people score each candidate independently and then compare. Where they differ by one, take the lower. Where they differ by two or more, one of them has misunderstood the criterion or the tool. In that case, the pair of them go and check again together. This is the step that makes the sheet honest. It is the step scoring theatre skips, because a single scorer never disagrees with themselves. The second scorer does not need to be senior. They need to have sat at the tool.

Every score carries its evidence in the next column: "documentation", "demo", "tried it", with a one-line note. A score from documentation is a claim. A score from a demo is a claim the presenter chose to demonstrate. A score from trying it is a fact. When you read the sheet at the end, discount the first two in your head. If the ranking depends on a documentation score, go and try the thing before deciding. The evidence column is also what you will be grateful for in two years. Somebody asks why a tool scored a two on reports. The answer is written down beside the number instead of lost with the person who left.

Totals rank; they do not decide. The sheet tells you which two or three candidates deserve the two-week trial described in designing a fair trial. It tells you where each candidate is weak so the trial can lean on the weakness. If two candidates are within a few points, the sheet is telling you they are close, not that the higher one is better. In that case, the honest response is to say so in the decision record. A total of 847.25 against 846.90 is not a result. It is a rounding error with a procurement team attached.

The Criteria That Sound Important and Are Not

Some criteria arrive on every sheet because they sound like the sort of thing a serious evaluation would ask. They measure almost nothing a small desk will feel. Strike them, or move them to a tie-breaker list, and give the row to permissions or the exit instead. None of the following is a flaw in a tool. Each is a flaw in the sheet.

The count of features is the first. Every brochure has a longer list than the last. The lists are not comparable because the words mean different things in each. A desk of four will use perhaps a fifth of any list. The second is the count of integrations, for the same reason. You need three and an API, and the other hundred and ninety-seven are a maintenance surface. The third is the vendor's size, or its position in some analyst's diagram. Large vendors discontinue products, change licensing models and get acquired, and small vendors do too. The criterion that protects you from all of that is the exit, which is already on the sheet.

The fourth is the look of the interface, which matters somewhat and is scored badly. That is because it is scored in a demo, on prepared data, by people who have used it for forty minutes. Score it in the trial instead, by asking the technicians at the end of week two whether anything about the screen slowed them down. The fifth is star ratings from review sites, which average the opinions of desks nothing like yours. The sixth is the roadmap: the features that are coming. A roadmap is a statement of intent by a sales organisation, and the correct weight for a feature that does not exist is zero. If it arrives, you will be pleased. If you bought the tool for it, you will be waiting.

Last, be careful with "customisable", which sounds like a criterion and is a promise that you will do the configuration. Some customisation is essential, such as a field on a department's form or a template library you can edit. Beyond that, every renamed status and redrawn workflow is a thing somebody at the desk now owns and nobody at the vendor will document. The criterion is not how much can be changed. It is whether the three things you need to change can be, and whether the rest works well enough as shipped that you will never want to.

The Criteria at a Glance

The table below is the ten criteria with the check for each and what a two looks like. That is the score that means "does it the way we need". Use the table to build your sheet, then replace the examples with your own desk's facts. A two for a three-person desk with one mailbox is not a two for a ten-person desk with four departments. The check column is the part most sheets lack, and it is the only column that stops the scores being guesses.

A Short War Story: The Half-Point Winner at Northgate Retail

Northgate Retail's procurement team scored three ticketing tools across 190 criteria and declared a winner by a margin of half a point. The desk lead, a careful man called Tomas, signed the trial agreement on the strength of the total. On the third day of the trial a regional manager in the north opened the portal. She found she could read every ticket raised by every store in the country. That included two from a store in the south about a cashier's disciplinary meeting. That happened because the sheet had no row for permissions and the tool's default was that a manager was a manager everywhere. The trial stopped, and HR asked for a written account. The desk restarted the evaluation with a ten-row sheet that put permissions second. That took six weeks and cost Tomas the spring. The tool that had come third by half a point passed the permissions check in an hour and was running by summer. A weight to two decimal places cannot rescue a criterion nobody wrote down.

A Help Desk Software Comparison Checklist

When the shortlist is down to three, the criteria above become columns. The checklist is the ten criteria as yes, no or partly questions, one row each, so that a help desk software comparison fits on a page and the half-points are visible. Fill it from the trial, not from the brochure: the answer to "does a reply from the requester reopen the ticket?" is found by replying, and the answer to "what does leaving cost?" is found by asking for the export. A row where every column says yes is a criterion that will not decide anything; a row where they differ is where the decision is.

The table is deliberately short on features and long on consequences. A feature list is what the vendor compares; the checklist is what the desk lives with. The two systems that tie on it are separated by the trial, which is the subject of Designing a Fair Trial.

The Scoring Sheet

The sheet below is one page per candidate, filled in by two scorers independently and then reconciled. Weights are agreed and written in before the first score; the evidence column is compulsory; the reconciled score is the one that counts. The total at the bottom ranks the candidates for the trial and is carried into the decision record with the sheet attached. It is deliberately small enough to fill in by hand. A sheet you can fill in by hand is a sheet you will fill in honestly.

SCORING SHEET — ONE PAGE PER CANDIDATE
Candidate: ______________________   Scorer A: ____________   Scorer B: ____________
Date: ____________   Musts checked and passed on: ____________

Scale:   0 = does not do it   1 = does it with a workaround we would maintain
         2 = does it the way the desk needs   3 = better than we asked, and we would use it
Weights: 1, 2 or 3, agreed before scoring, never changed after.
Evidence: D = documentation   S = shown in a demo   T = tried it ourselves

No.  Criterion                 Weight  A   B   Reconciled  Evidence  Note (one line)
 1.  Channels into one queue   ____   __  __     __         ___     _________________
 2.  Routing                   ____   __  __     __         ___     _________________
 3.  Service levels            ____   __  __     __         ___     _________________
 4.  Knowledge base            ____   __  __     __         ___     _________________
 5.  Reports                   ____   __  __     __         ___     _________________
 6.  Permissions               ____   __  __     __         ___     _________________
 7.  Integration (API, hooks)  ____   __  __     __         ___     _________________
 8.  Hosting                   ____   __  __     __         ___     _________________
 9.  Licensing model fit       ____   __  __     __         ___     _________________
10.  The exit                  ____   __  __     __         ___     _________________
11.  ________________________  ____   __  __     __         ___     _________________
12.  ________________________  ____   __  __     __         ___     _________________

Reconciliation rule: differ by 1, take the lower; differ by 2 or more, check again together.
Row score = weight x reconciled.   TOTAL: ______   Scores from evidence D or S only: ______

Weakest two rows (the trial must lean on these): ______________  ______________
Recommend for trial:  [ ] yes   [ ] no   because __________________________________

Hannah rebuilt the Stonebridge sheet with ten rows and two scorers, herself and the newest technician. It took them a morning each for three candidates. The totals came out 41, 38 and 29 out of a possible 60. That told them two things the 212-row sheet had hidden. The third candidate was genuinely behind, and the first two were close enough to need the trial to separate them. The row that separated them, in the end, was permissions. It had not been on the procurement spreadsheet at all.

Frequently Asked Questions