Measuring Portal Adoption: Four Numbers and What Each Means

Monday, 09:05, at Bluewater Bank. Priya has been asked by the operations director for "the portal numbers" by Thursday, and she has three. The tool's report shows tickets by channel. The web server shows how many times the portal page was loaded. That went up sharply last month because the branch network failed on a Tuesday and everyone went to look at the outage notice. And the project's closing slide shows "registered users", which turned out to be the number of people in the directory. Three numbers, all accurate, and none of them says whether a single person has chosen the portal over e-mail and then chosen it again.

Adoption is people choosing the portal when they have a need, and choosing it again the next time. It is a behaviour, not an event, which means it is measured every month rather than announced once. It is measured mostly from the tickets, because the tickets record what people actually did. Two neighbouring questions are deliberately not answered here. Whether the portal answered a question before a ticket was raised is deflection, and counting it is the subject of Measuring What Got Deflected. What the saved tickets were worth in hours and money is the return on the investment, which has its own series. This article is about whether people use the portal, and it is part of our Self-Service Portal Adoption series.

It starts with why adoption is a behaviour and not a launch. It covers the four numbers in turn, with a worked example from a fictional desk. These are share of tickets by channel, the search-to-ticket ratio, return visits, and which departments use it. Next comes what to do with each number, in a table. Then it covers the numbers that mislead and reporting adoption upward. A short war story and the adoption scorecard to copy finish the article. Every figure in the worked example is invented for the example.

Adoption Is a Behaviour, Not a Launch

A launch is a day. Adoption is what happens on the three hundred working days after it, one requester at a time, each deciding at the moment of need which door to take. A launch can be announced and celebrated; adoption can only be counted, slowly, from what people did. The desk that reports the launch has reported its own effort. The desk that reports adoption has reported the users' verdict, which is less comfortable and the only one that matters.

The counting is done monthly and from data the desk already has. Every ticket records how it arrived, provided the desk logs its walk-ups and phone calls as tickets with their own source. The desk must do this or the count is of a sample. Every ticket records who raised it and which department they belong to. The portal, in most tools, records searches and whether they found anything. That is enough for four numbers, and four is the right count. Four is few enough to read in a minute. It is still many enough that no single one can be flattered without another one giving it away.

The numbers are compared with the desk's own past, not with anyone else's present. There is no useful general figure for what share of tickets "should" come through a portal. The share depends on what kind of work the desk does and how many requesters have a desk and a browser. It also depends on how long the portal has existed. A desk compares this month with last month and with the month before the fixes, and the direction is the result. Anyone who offers you a benchmark is selling the benchmark.

Number One: Share of Tickets by Channel

The first number is the share of tickets that arrived through the portal, out of all tickets, by month. A channel is the route a request took to the desk: e-mail, phone, portal, walk-up, chat. The tool's report by channel gives the counts; the share is the portal count divided by the total. At Bluewater, in the example month, the desk logged 412 tickets: 290 by e-mail, 48 by phone, 61 through the portal and 13 as walk-ups. The portal share is 61 out of 412, which is roughly one ticket in seven. The month before the fixes, it was one in fourteen.

Read the share with the total beside it, always. A portal share that rises because portal tickets rose is adoption. A portal share that rises because e-mail tickets fell is a question, and the question is where the e-mails went. If the total fell too, requests are leaving the queue, usually for a technician's personal inbox. In that case, the portal share has improved by the worst available mechanism. I once reported a share that had nearly doubled in a month when the support address had been bouncing for a week. The share was accurate. It was the best month the portal ever had, and nobody had adopted anything.

The number to watch under this one is the e-mail count, not the portal count. The portal's job is to take requests that would otherwise have been e-mails, so adoption shows as e-mail falling while the total holds, with the difference appearing under portal. If portal rises and e-mail does not fall, the portal is attracting requests that were never raised before. That is interesting and may be good, and is not adoption. Set the three figures side by side every month: total, e-mail, portal. The story is in how they move together.

Number Two: The Search-to-Ticket Ratio

The second number is how many searches the portal handled for each ticket raised through it. It comes from the portal's own log, where the tool keeps one. Most do, some count only visits, and a desk with neither skips this number rather than guessing it. At Bluewater, the portal logged 340 searches in the month against 61 portal tickets, which is between five and six searches per ticket. The month before the fixes it was under two, because almost nobody was searching; they went straight to the form, or straight to e-mail.

The ratio measures whether people are using the portal as a place to find things. That is the half of adoption that produces no ticket at all and is therefore invisible to the channel share. A rising ratio means more visitors are trying the search before asking, which is the behaviour the portal exists to produce. What it does not tell you is whether the search worked. Five searches per ticket could be five people who found answers and left, or one person who tried five words and gave up. That question, which searches ended in an answer and which in a ticket, is deflection's. The deflection series answers it with the empty-result list and the read-then-no-ticket count. The adoption question is simpler: are they searching at all, and more than last month?

Pair the ratio with the share of searches that returned nothing, which the same log usually gives. A high ratio with a high empty share is a portal people are trying and finding empty. That is the second of the five reasons people bypass a portal. In that case, the ratio will not stay high for long, because people stop searching things that return nothing. A high ratio with a low empty share is a portal that is being used as intended. A ratio that falls while the empty share rises is a portal whose knowledge base has stopped keeping up with the questions.

Number Three: Return Visits

The third number is the share of portal users who came back. It is computed from the tickets over a quarter. Count the distinct requesters who raised at least one ticket through the portal. Then count how many of them raised two or more, and divide. At Bluewater, over the example quarter, 70 different people raised a portal ticket. Of them, 31 did so more than once, which is somewhere between four in ten and half. A quarter earlier, before the fixes, the return share was under one in five: people tried the portal once and went back to e-mail.

Return visits are the number that says whether the first visit is working, and it is the best early warning the desk has. The channel share can rise for a quarter on first visits alone, driven by a shortcut or a signature line. It can then fall back as every one of those first visitors decides not to repeat it. The return share catches that a quarter early. A low return share with a healthy count of first-time portal users means the first visit is failing somewhere in its seven moments. The rehearsal in Making the First Visit Succeed is where to look. A rising return share is adoption in the strict sense: people who chose it, and chose it again.

Compute it over a quarter rather than a month, because a month is too short for most people to need the desk twice. Read it with the count of first-time users beside it. A return share that rises because fewer new people are trying the portal is not an improvement. The healthy pattern is both rising: more people trying it for the first time, and a steady or growing share of them coming back. Where the tool cannot easily list requesters by channel, export the quarter's tickets to a spreadsheet. Filter to the portal channel and count by requester address. That does the job in ten minutes.

Number Four: Which Departments Use It

The fourth number is portal tickets per head, by department. Take each department's portal tickets for the month and divide by its headcount, then line the departments up. At Bluewater, lending, with forty people, raised twenty-two portal tickets, a little over one for every two people. Branch operations, with a hundred and twenty people, raised six, one for every twenty. Finance, with thirty, raised none. The desk's overall portal share was one in seven, and it was made almost entirely of two departments, while the largest department in the bank had barely touched it.

Per head is the only fair comparison. A department of six that raised three portal tickets has adopted the portal more than a department of ninety that raised twelve. The raw counts say the opposite. Headcount comes from HR or the directory and changes slowly, so it is looked up once a quarter. Where several departments share one help desk installation, as they can with QuerySolv, the comparison comes from one place rather than three exports. But the arithmetic is the same anywhere: one export, one headcount column, one division.

The department at zero is the finding, every time. Zero is not low adoption. It is a department the portal has not reached, for one of the five reasons. The only way to learn which is to go there and ask. The promotion plan in Promoting the Portal Without Nagging reserves an afternoon per quiet department for exactly that walk. This number is what tells the desk where to walk. A department that moves from zero to one in twenty after a visit is the clearest evidence of adoption the desk will ever get. It has a date and a cause.

What to Do With Each Number

A number earns its place on the scorecard by changing a decision, and each of the four changes a different one. For each number, the table gives the direction that is healthy and the companion figure that keeps it honest. It gives what the number means when it goes the wrong way, and the first action. Most months, the action is nothing, which is a legitimate result and should be written down as such. The months when a number turns are the months the scorecard pays for itself, because the turn is visible weeks before anyone complains.

Notice that none of the actions is "send a reminder". Each number points at something the desk can change. It might be a channel that is leaking or a knowledge base that is missing articles. It might be a first visit that fails at one of seven moments or a department that was never reached. The numbers tell the desk where to work. They do not tell the requesters anything, and were never meant to.

Numbers That Mislead

Several numbers look like adoption and are not, and most of them arrive on the desk's screen uninvited because the tool or the web server offers them. Page views are the first. A view is a person loading the page, for any reason. That includes the outage, curiosity, or the portal being the browser's home page and opening whether anyone wanted it or not. Page views measure curiosity. Tickets measure trust. The second is "registered users". In most tools, that is the number of accounts created, often by import from the directory, and therefore the number of people who exist. The third is the launch spike, a tall bar in the first week when three hundred people looked at the thing once, which is also how posters are consumed.

Subtler ones mislead from inside the ticket data. A portal share computed in a month when the total fell is a flattered share, as above. A satisfaction score taken only from portal tickets, compared with one taken from e-mail tickets, compares two groups. These are the people who chose the portal and the people who did not. The people who chose it were already better disposed. And a falling ticket count described as "deflection" is a guess dressed as a result. Tickets fall for many reasons. The deflection series insists on counting the searches that ended in a read article and no ticket, not on inferring them from an absence. A dashboard that only goes up is a dashboard that has stopped asking questions.

The test for any adoption number is the one that applies to every help desk metric: what decision would change if it moved? Page views moving changes nothing the desk can act on. Registered users cannot move. The launch spike is over. The four numbers above each point at a specific fix, and a number that points at nothing is scenery, however good it looks on the Thursday slide.

Reporting It Upward

The operations director wants one page, and the page has the four numbers on it. Each has its companion figure, this month beside the baseline month, and one sentence of what the desk did about each. Not a chart of page views. Not a celebration. A short table and four sentences, and at the bottom the one thing the desk is going to do next month and what number it expects to move. The shape of a brief that a manager can read in two minutes and act on is the subject of Explaining Technical Things to Management. The adoption scorecard is that brief applied to the portal.

Resist the request to convert adoption into money on the same page. A director who sees "one in seven tickets now arrives through the portal" will ask what that saved, and the honest answer is that adoption and savings are different measurements. The portal ticket still has to be worked. What it saved was triage questions and chasing. What the search saved was the ticket that was never raised, which is deflection's count. Converting either into hours and money is the subject of Proving It to Management in One Page. Point there, promise the money page when the deflection count exists, and keep the adoption page about behaviour. A page that mixes the two gets challenged on the money and loses the behaviour with it.

Report the bad months as plainly as the good ones. Suppose the return share falls one month. The desk runs the first-visit checklist and finds the welcome e-mail link has been changed by HR. That month makes a better report than a month where everything rises, because it shows the measurement working. Directors who see the desk find its own problems trust the desk's numbers. Directors who see only green learn, eventually, that green is the dashboard's resting colour.

A Short War Story: The Launch Spike at Acme

Acme's IT manager, Ravi, reported to the board a month after the portal launch. The page had been viewed two thousand eight hundred times, which was more than the headcount. The board congratulated the project. The following month the figure was a hundred and ninety, and the board asked what had gone wrong. Ravi had been reporting curiosity without knowing it. He had no answer, because nothing had changed except that people had finished looking. Over the same two months, portal tickets had gone from nine to eleven to twelve, and nobody had reported that figure because it was small. Dana, who ran the desk, rebuilt the report around the four numbers. The first honest version showed a portal share of about one ticket in twelve, a return share under one in five, and two departments at zero. The board found it less cheerful and far more useful, because each line came with an action. The cost was a quarter of the project's credibility and a board that had learned to distrust the IT dashboard. That took a year of plain reports to undo. A number that goes up for a month and then falls has not measured adoption; it has measured the announcement, and announcements always fall.

The Adoption Scorecard

The scorecard below is the four numbers on one page, with their companions, a baseline column for the month before the fixes began, and a line for the action taken. It is filled in on the first working day of the month from the channel report, the portal's search log and one export. It takes about half an hour once the export is set up. The "misleading numbers" box at the bottom is there to be left blank. It lists the figures that are not on the scorecard so that nobody adds them back when the page views look impressive. Keep every month's sheet, because the baseline column only means something when there are six of them in a row.

PORTAL ADOPTION SCORECARD — MONTHLY, ONE PAGE
Desk: ______________________   Month: ____________   Filled in by: ______________
Baseline month (before the fixes began): ____________

1. SHARE OF TICKETS BY CHANNEL                      baseline     this month
   Total tickets logged                             ______       ______
   E-mail                                           ______       ______
   Phone                                            ______       ______
   Portal                                           ______       ______
   Walk-up (logged as tickets with own source)      ______       ______
   Chat                                             ______       ______
   Portal share (portal / total)                    ______       ______
   Total steady or rising?  [ ] yes  [ ] no: where did the requests go? ______________

2. SEARCH-TO-TICKET RATIO (skip if the tool keeps no search log)
   Searches on the portal                           ______       ______
   Searches with no result                          ______       ______
   Portal tickets                                   ______       ______
   Searches per portal ticket                       ______       ______
   Empty-result share (no result / searches)        ______       ______
   Ten most searched terms with no result: ____________________________________

3. RETURN VISITS (this quarter to date)
   Distinct requesters with a portal ticket         ______       ______
   Of those, with two or more                       ______       ______
   Return share                                     ______       ______
   First-time portal users this quarter             ______       ______

4. PORTAL TICKETS PER HEAD, BY DEPARTMENT
   Department            headcount   portal tickets   per head   last month
   ________________      ______      ______           ______     ______
   ________________      ______      ______           ______     ______
   ________________      ______      ______           ______     ______
   ________________      ______      ______           ______     ______
   Departments at zero: ______________________________

ACTIONS
   Number that moved the wrong way: ____   Companion figure checked: [ ]
   Action taken this month: _______________________________________________
   Action for next month: _________________________   Number expected to move: ____

NOT ON THIS SCORECARD, BY DESIGN
   Page views.  Registered users.  Launch-week figures.  Portal share in a month
   when the total fell.  Savings in hours or money (see the ROI page).

Priya gave the director one page on the Thursday, with the four numbers against a baseline from the month before the fixes. A line under each said what the desk had done. The portal share was one in seven, up from one in fourteen; the return share was approaching half; finance was at zero and had an afternoon booked. The director read it in two minutes and asked one question, which was "what happens in finance?" That is the question the page was built to produce. Nobody asked about page views, because they were not on it, and they have not been on it since.

Frequently Asked Questions