Measuring and Improving Onboarding: On-Time Rate and Defects

Thursday, 14:00, in the quarterly operations meeting at Oakridge Motors, a dealership group with six sites and a desk of three. The HR director turns to Sofia, who runs the desk, and asks, in a friendly way, "How's onboarding going these days?" Sofia says "pretty well, I think," and hears herself say it. The desk did twenty-three onboardings last quarter. She knows two of them went badly, because the managers told her. She does not know how many were ready on the morning or how many starters found something missing on day three. She does not know how much notice HR gave on each, or what any of the twenty-three thought of it. "Pretty well" is what a desk says when it has done the work and kept no score.

Measuring onboarding means recording four numbers for every starter on the parent ticket when it closes. Those are whether they were ready on time, how many defects were found, how many working days' notice the desk had, and what the starter said two weeks in. Improving it is the loop that reads those numbers every quarter with HR, runs a short retrospective after a bad one, and changes one thing each time. That could be a line on the checklist, a line on a profile card, or a line in the agreement. This article is part of our Employee Onboarding series, and it is the last one. The numbers only mean something once the form, the checklist, the profiles, the lead-time agreement and the first-week routine exist to be measured.

It covers the four numbers and why only four, then the on-time rate and its counting rules. Next come defects per starter and the list that matters more than the count, then the two-week starter survey. A worked example in ranges from a fictional desk, the retrospective after a bad one, and the quarterly review with HR and the line that goes upward follow. A short war story and the onboarding scorecard to copy come next, in that order.

The Four Numbers, and Why Only Four

Four numbers describe an onboarding well enough to improve it, and each is written on the parent ticket at close by the technician who owned it. On time is yes or no: the ready test passed the day before and the starter could work from the first hour without a blocking gap. Defects is a count and a list. It records every item on the day-one checklist or the profile card that was missing on the morning or found at the day-three check-in, with who found it. Lead time is the working days between the complete request arriving and the start date, which Lead Time defines and HR owns. The starter's answer is one rating and one comment, collected two weeks in.

Four, not fourteen, because every number costs the technician a minute at close and costs the reader a minute every quarter. A scorecard with fourteen lines is a scorecard nobody reads past line six. The desk's time to complete an onboarding is tempting and misleading. That is because a well-run onboarding is spread across a fortnight in quiet half-hours and the elapsed time says nothing. The number of tickets the starter raised in week one is interesting and belongs in the defect list, not beside it. Per-technician on-time rates are a coaching conversation, not a line on the card. They are not read at all below a couple of dozen onboardings per person, which on most desks means never. The discipline of defining each number so that two people get the same one is in Defining Each Metric Exactly. The four below are defined that way on purpose.

Write the four on the parent ticket's closing note in the same order every time, so that a quarter of them can be read in a column. "On time: yes. Defects: 2 (printer mapping, found by desk; team list, found by starter). Notice: 11 working days. Survey: yes, 'all good except the second screen'." That line takes forty seconds to write and is the entire data entry for the whole system. There is no spreadsheet to keep during the quarter. There is a closing note, and a read of the closing notes at the end.

The On-Time Rate

The on-time rate is the count of onboardings that were ready on the morning over the count of onboardings closed in the quarter. Report it as two numbers, "seventeen of twenty-three", never as a percentage on its own. The denominator is every onboarding the desk closed in the period: new hires, transfers, contractors and returns from leave. Each was a person who needed to work on a date. Exclude nothing for having been hard. The onboarding that arrived with two days' notice and produced a partial day one was not on time. The right place for the explanation is the lead-time column beside it, not an asterisk on the rate.

The single most useful thing to do with the rate is to split it by notice. Onboardings with ten or more working days' notice are the ones the desk had every chance with. The on-time rate among those is the desk's own performance. Onboardings with fewer than five days' notice are the ones the agreement failed. The on-time rate among those is what the desk can do with a spare kit and a morning. Reported together, "fourteen of fifteen with full notice; three of eight with under five days", the two numbers say in one line what a hundred words of explanation would. They put the second number where it belongs, which is in front of HR.

Resist softening the yes-or-no. "Ready apart from the badge" is a no. "Ready by ten" is a no. The temptation to count a near miss as a yes is the same one that inflates every rate on a desk. The cure is the same: write the rule down once and apply it whether or not it flatters anyone. A desk that counts "ready by ten" as on time will have a rate that goes up while the starters' first hours do not change. It will have learned nothing it can act on. The honest rate is lower and more useful, which is true of most honest rates.

Defects per Starter

A defect is a thing the process was supposed to deliver and did not. The list of defects is worth more than the count of them, because the list says what to fix. "1.3 defects per starter" is a number that goes on a slide. "Printer mapping, seven times; team distribution list, five times; dock untested, four times" is a to-do list for the checklist. That list is the mechanism by which the checklist in The Day-One Checklist gets better. Each quarter, the top defect by count becomes a line on the checklist or the ready test, or a line on a profile card. The following quarter it should be near the bottom of the list. If it is still at the top, the line was added and not followed, which is a different problem and a shorter conversation.

Mark who found each defect. A defect found by the desk at the ready test and fixed before the morning cost an hour. The same defect found by the starter on day three cost the starter two days of working around it. The first kind shows the ready test working. The second kind, the escaped defect, is the one the starter experienced, and it is the number to watch. A desk whose defects are mostly found by the desk has a good test and an imperfect process. A desk whose defects are mostly found by starters has no test, whatever the checklist says.

Group the defects by the seven headings on the checklist: account, mailbox, device, access, phone, badge and desk. Add "other" for the things that were not on the list at all. The "other" group is the interesting one, because an "other" defect is a thing the list did not know to deliver. Three of them with the same name in a quarter is a new line. The desk at Oakridge found "headset for the call team" in the other column four times in one quarter. The headset has been a line on the office profile card since. The method for cutting a pile of tickets by what they have in common, and finding the thing that generates most of them, is in Finding Hotspots in the Ticket Data. The defect list is the same idea applied to a quarter of closing notes.

The Two-Week Starter Survey

The survey is one question with three answers and one comment box, sent two weeks after the start date by the technician who owned the onboarding. It asks "Did you have what you needed to do your job on your first day?", with yes, mostly and no. "What was missing or confusing?" goes underneath. Two weeks, not one, because the first week is too close to the noise of being new. Do not wait four, because by then the memory has faded into "it was fine, I think". Two weeks is when the starter can still name the thing that was missing and has stopped feeling that naming it is a complaint.

One question, because a starter in week two is being surveyed by HR, their manager and probably the pension provider. A six-question form from IT joins the pile unanswered. The one question is the one the whole process exists to answer. Everything else the desk wants to know is in the comment box, and the comment box is where the useful answers are. "Mostly, the second screen came on Thursday" is a defect with a name and a date. "Yes, but nobody told me about the shared mailbox" is an "other" that may be a profile line. The design of a survey that people answer is in Surveys People Actually Answer. That article also covers the sense in which a short survey gets truer answers than a long one. The two-week question is that advice reduced to its minimum.

Read every comment and average nothing. Twenty-three starters in a quarter, sixteen of whom answered, is not a sample from which a score means anything. The people who did not answer are not a random slice of the ones who did. The starters who were let down are both the most likely to say so and the most likely to decide it is not worth it. The way to read around that is in Response Bias and How to Read Around It. For the two-week survey it comes down to reporting the counts: "eleven yes, four mostly, one no, seven did not answer". Read the "no" and the "mostly" comments aloud at the review. The rating that rides on the resolving reply in many tools, QuerySolv among them, measures how the onboarding ticket closed, not how the first day went. So it is not a substitute for the two-week question. Use both if you have both and keep them apart.

A Worked Example in Ranges

The numbers below are from a fictional desk, Oakridge's, over one quarter. They are given as what a desk of that size and that stage might see, not as targets. Twenty-three onboardings closed. Seventeen were on time, "seventeen of twenty-three". Split by notice: fifteen had ten or more working days, and fourteen of those were on time; eight had fewer than five, and three of those were. The one full-notice miss was a device build that failed on the Thursday and was not retested in time. The five short-notice misses were partial day ones with the honest message sent on the first day. That split is the whole story of the quarter in two lines. It says the desk's process works when it is given time. It also says the agreement with HR is being kept on about two starters in three.

Defects: thirty-one across the twenty-three, a little over one per starter, nineteen found by the desk at the ready test and twelve found by the starter. The top three by name were printer mapping, seven; the team's distribution list, five; and the dock untested at the desk, four. The "other" column held the call-team headset, four times. Lead time: a median of nine working days; eight onboardings under five; three at two or fewer, all three from one site whose manager hired directly and told HR afterwards. Survey: sixteen of twenty-three answered; eleven yes, four mostly, one no. The no was a starter whose profile had been confirmed as office when the role was finance, which took four days to put right.

What should a desk expect to see? The ranges give only a rough shape. A desk that has just put the form, checklist and profiles in place often starts with something like half to two thirds on time. It often has one to two defects per starter, most of them found by starters. A desk a year in, with the ready test habitual and two or three quarters of fixes behind it, is often in the region of four in five on time. It often has under one defect per starter, most found by the desk. The short-notice on-time rate stays low on every desk, because two days is two days. The number that moves it is the share of short-notice requests, which is HR's line. Your numbers will differ. The direction of travel should not.

Hours are the currency when the numbers go upward, and the arithmetic is short. On this desk a defect found by the desk cost about half an hour. A defect found by the starter cost the desk about an hour and the starter between half a day and a day of working around. Twelve escaped defects in a quarter is therefore somewhere between six and twelve days of starters' time, which is a line a finance director understands better than a rate. The method for turning counts into hours without inventing anything is in Deflection Math Without Fooling Yourself. State the ranges as yours, from your tickets, and never as anyone else's.

The Retrospective After a Bad One

A retrospective is a thirty-minute conversation, within a week of a bad onboarding. It asks what was missing, when it was first known, which line should have caught it, and what one thing changes as a result. It is triggered by a "no" on the morning, by three or more defects on one starter, or by a "no" on the survey. The people in the room are the technician who owned the parent, the desk lead, the hiring manager, and HR if the notice was short. The first rule is that the conversation is about the process and not about any of them. The manager who hired directly and told HR afterwards is not the subject. The fact that the organisation has no line in its procedure for managers who hire directly is.

Ask four questions in order and write the answers on the parent ticket. What was missing on the morning, in the starter's words if possible. When did the desk first know it would be missing, and when could it have known? Which line on the checklist, which line on the profile card, or which line in the lead-time agreement should have caught it, or is there no such line? And what one thing changes: a line added, a line reworded, a date moved, a trigger renamed. One thing. A retrospective that produces six actions produces six things half-done. One that produces one line on the checklist produces a line on the checklist, and the next quarter's defect list says whether it worked.

Keep it short and keep it blameless, because the alternative is that nobody reports a bad onboarding and the number goes up while the first days do not. The craft of finding a cause without a committee, and without a search for someone to be wrong, is the subject of Root Cause Analysis Without a Committee. The onboarding retrospective is that craft applied to one person's first morning, and it is over in half an hour. The desk at Oakridge ran four in a quarter and changed four lines. Three of the four defects did not appear in the following quarter. The fourth was the dock, which is a hardware problem wearing a process problem's clothes. It went to the lemon list instead.

The Quarterly Review With HR, and the Line Upward

The quarterly review is thirty minutes with the head of HR, the scorecard on the table, and three cuts in front of both of you. Those are on-time by notice, the top three defects by name, and the late requests by cause. The first cut shows the desk's half and HR's half of the result on one line. The second is the desk's to-do list, and HR sees it so that they know the desk is fixing its own side. The third is HR's to-do list. The causes are nearly always the same three: a start date brought forward, an offer accepted verbally and not written down, and a manager who hired without HR. Each cause has a fix in the agreement, and the review is where the fix is agreed and dated.

Leave with three actions and no more. One is for the desk, usually a checklist or profile line from the defect list. One is for HR, usually a trigger or an intake date from the late-request causes. And one is shared, usually a conversation with one site or one manager. Write the three on the scorecard with a name and a date, and open the next quarter's review by reading them out. The format of a quarterly review that reads trends rather than months is in The Quarterly Review: Trends, Not Months. It also explains why a quarter is the smallest unit worth reviewing on a desk with twenty-odd onboardings in it. The onboarding review is a smaller version with two departments in the room instead of one.

The line that goes upward, in the desk's monthly or quarterly report to leadership, is one sentence with three numbers in it: "Seventeen of twenty-three starters were ready on their first morning; fourteen of the fifteen we had full notice for; the short-notice ones are being worked on with HR." That is a result, a cause and an action, and it is the shape leadership reads. The longer version, with the defect list and the survey comments, stays with the desk and HR. The one-page report the sentence sits in is laid out in The Monthly Report They Will Actually Read. The onboarding line is one of the few in it that HR will also read, which is a good reason to make it accurate.

Counting the quarter takes an hour if the closing notes were written. If you run QuerySolv, every parent ticket arrived through the onboarding form, so a quarter's worth of them is one search. In that setup, the four numbers are a read of their closing notes into four columns. In any tool, the search is by the form, the type, or whatever marks an onboarding as one, and the reading is the same. The hour is spent reading. The decade of desks I have watched try to do this without the closing note spent the hour reconstructing, from memory, which starters had been late, and got it wrong.

A Short War Story: The Survey Nobody Had Sent at Pennywell Timber

Pennywell Timber, a builders' merchant with nine branches and a desk of two, had never measured its onboarding because nobody had complained about it. The HR manager, Ruth, had taken the silence as a result. Then a branch manager mentioned at a quarterly meeting that his last two starters had both nearly left in their first month. He said one of them had said, in passing, that "the computer thing" had been part of it. Ruth asked the desk for the numbers and there were none. So the desk's lead, Anil, sent the two-week question to every starter from the previous six months, forty-one people. He got twenty-six answers: nine yes, eight mostly, and nine no. The comments named the same three things over and over. Those were a till account that did not work on the first morning, a branch printer that nobody had mapped, and a mailbox that arrived on day four. The cost, as far as anyone could count it, was nine starters who had each spent most of a first week working around their own kit. Two of those had resigned inside three months citing a bad start among other things. There was also a recruitment cost for their replacements that Ruth could put a figure on and the desk could not. Anil put the four numbers on every parent ticket from that week. The till account became the first line of the branch profile card, and the next quarter's survey came back with two "no" answers out of nineteen. Nobody complains about a bad onboarding, because by the time they could, they are either too new to risk it or too gone to bother. They simply remember it, and the survey is the only way the desk gets to hear what they remember.

The Onboarding Scorecard

The scorecard below is one page per quarter, filled in from the closing notes in about an hour and taken to the review with HR. Section A is the four numbers as counts, with the on-time rate split by notice. Section B is the defect list by name, with who found each, which is the desk's to-do list. Section C is the late requests by cause, which is HR's. Section D is the survey counts and the comments worth reading aloud. Section E is the three actions from the last review, with whether they were done, and the three for this one. Keep the previous quarter's card beside it. The comparison is the point, and one quarter on its own is a snapshot of a desk with a trend it cannot yet see.

ONBOARDING SCORECARD — ONE PAGE PER QUARTER
Desk: ______________   Quarter: ______________   Prepared by: ______________   Reviewed with HR on: __________

A. THE FOUR NUMBERS (counts, not percentages)
   Onboardings closed this quarter: ____   (new hires ____  transfers ____  contractors ____  returns ____)
   On time on the first morning: ____ of ____
     with 10 or more working days' notice: ____ of ____
     with 5 to 9 working days' notice:     ____ of ____
     with fewer than 5 working days' notice: ____ of ____
   Defects found: ____ total; ____ found by the desk at the ready test; ____ found by the starter
   Lead time: median ____ working days; ____ under five days; ____ at two or fewer
   Two-week survey: ____ answered of ____;  yes ____  mostly ____  no ____

B. DEFECTS BY NAME (the desk's list)
   Defect                              Count    Found by desk / starter    Line to add or fix
   ________________________________    ____     ____ / ____                ______________________
   ________________________________    ____     ____ / ____                ______________________
   ________________________________    ____     ____ / ____                ______________________
   "Other" items not on any list: ______________________________________________

C. LATE REQUESTS BY CAUSE (HR's list)
   Start date brought forward: ____    Offer accepted but not recorded: ____
   Hired without HR: ____              Intake dates not shared: ____    Other: ____ ______________

D. WHAT STARTERS SAID
   Comments to read aloud at the review:
   1. ______________________________________________________________________
   2. ______________________________________________________________________
   3. ______________________________________________________________________

E. ACTIONS
   From last quarter:  desk: ____________________ done [ ]   HR: ____________________ done [ ]
                       shared: ____________________ done [ ]
   For this quarter:   desk: ____________________ by __________ owner ______________
                       HR: ______________________ by __________ owner ______________
                       shared: __________________ by __________ owner ______________

Line for the leadership report:
   "____ of ____ starters were ready on their first morning; ____ of the ____ we had full notice for;
    ____________________________________________________________ (cause and action)."

Sofia put the four numbers on the next parent ticket that closed, the Friday after the meeting, and on every one after that. At the following quarter's operations meeting the HR director asked the same question. Sofia said "nineteen of twenty-six, and sixteen of the seventeen you gave us full notice on," and watched him write it down. Then he asked about the other nine, which was the question the number had been built to make him ask. They had the first review the following week. It took twenty-five minutes, which is five fewer than the article promised, and nobody in it said "pretty well".

Frequently Asked Questions