Measuring Recovery Requests and the Confidence They Give

Monday, 09:05, at Acme. Nadia runs a desk of four. On Friday the finance director, Graham, asked a question in the quarterly review that nobody in the room could answer. "Are our backups actually any good?" The infrastructure lead said the jobs were green. Graham said that was not what he had asked. Nadia said nothing, because she did not know either. Then on the train home she realised that she did. Her desk had handled thirty-one recovery requests in the quarter. Every one of them was a moment when a layer of protection had been asked to produce a real file for a real person. The layer had produced it, or had not. She has the tickets. She has never counted them.

Measuring recovery requests is keeping a small log of every ticket where a file had to be brought back. The log records where it lived, which layer produced it, whether it came back whole, and how long it took. Measuring means reading that log each quarter for two things: the confidence it gives in the organisation's protections, and the places that keep needing rescue. Recovery requests are a small category of ticket with an outsized meaning. That is because they are the only routine evidence most organisations ever collect that their backups, snapshots and version history work. This article is part of our Deleted Files & Recovery Requests series. It is the last one, because the log it describes is built from everything the earlier articles produce.

It covers why recovery requests are worth counting at all, then what to record on every ticket. The success rate and what it really says, and time to recover, by layer, follow. Next come the restore test that every request quietly performs and the folder that keeps needing rescue. The one line for the report, a worked example in ranges, a short war story, and the recovery log to copy follow, in that order.

Why Count Recovery Requests at All

Recovery requests are a small share of most desks' tickets, a few a month on a small desk. The first instinct is that a category that small is not worth a measurement. The instinct is wrong for one reason. Each of those tickets answers a question that the organisation otherwise pays an auditor, or a disaster, to ask. A backup job that reports success every night has proved that it ran. It has not proved that a file can be got out of it, by a person, in time. The only proof of that is a restore. A recovery ticket resolved from the backup is a restore, performed under real conditions, for a real person, with the time it took written down. Thirty-one of them in a quarter is a test programme that nobody had to schedule.

The second reason is that the log shows where files are being lost from. That is a different question from whether they can be got back, and a more useful one for the desk. A quarter of requests that mostly come from laptop desktops is a redirection project. A quarter that mostly comes from one share is a permissions or a training problem on that share. A quarter in which the overwrites all come from the old finance server is a migration case. None of that is visible from the volume number on the desk's dashboard, and all of it is visible from a log with six columns. The general method for counting tickets so that the counts mean something is in Counting Tickets Properly. The recovery log is that method applied to one ticket type with two extra columns.

The third reason is Graham. Somebody senior will eventually ask whether the backups are any good. One honest answer is "the jobs are green", which is not an answer. The other is "we got real files out of them N times this quarter, in this much time, and here is the one time we could not", which is. The desk is the only part of the organisation that can give the second answer. It can only give it if it wrote things down at the time. (It is also the only part of the organisation that the question is never addressed to. That is worth fixing.)

What to Record on Every Ticket

Six facts, recorded on every recovery ticket as it is resolved, make the log. Five of them are already in the restore record from The Restore Workflow in Plain Words. So the work is a transcription rather than an investigation. The first is where the file lived: the file service share, a laptop desktop, a sync folder, the old server, a personal device. The second is the shape of the loss: moved, deleted, overwritten, never saved, device wiped, not in the backup set. The third is the layer that produced the file: a quick win such as a search or another copy, version history, the recycle bin, a snapshot, the backup, or none. The fourth is the outcome: recovered whole, recovered in part with the gap named, or not recovered.

The fifth is time to recover, measured from the desk's first touch to the requester's confirmation, in working hours, with a note of how much was waiting on another team. The sixth is whether a lesson was sent and which one, from the after-rescue message. That is because the count of lessons by shape is what turns teaching into a change, as the article on teaching prevention after the rescue argues. A seventh, optional, is the requester's department, which most tools already hold and which makes the hotspot cut later possible without a lookup.

Record these in whatever the tool allows. Use a tag or a custom field per column where it has them. Use a fixed-format line at the top of the resolution note where it does not. The fixed line is less elegant and works everywhere: "RECOVERY: share / deleted / bin / whole / 0.4h / lesson: bin". A search for "RECOVERY:" at quarter end returns the log, and the words after it are the columns. What the desk must not do is rely on memory at quarter end. By then the thirty-one tickets are thirty-one resolution notes of different shapes and the counting takes a day. Six words at resolution, a minute at quarter end.

The Success Rate, and What It Really Says

The recovery success rate is the share of recovery requests in the period where the file came back whole, reported as counts beside the share. It is a number to read with care, because it is nearly always high and the interesting part is the remainder. On an ordinary desk with ordinary protections, most requests are resolved by a quick win, the bin or version history, and come back whole. The rate sits somewhere in the high range by default. A desk that reports "nine in ten recovered" has reported that it has a file service. The information is in the other tenth: how many came back with a gap, which layer they came from, and what shape the unrecoverable ones were.

Report three counts, not one: whole, partial, and not recovered, with the partials' gaps and the shapes of the losses beside them. "Twenty-seven whole, two partial, both from the backup and both missing the last day's work, two not recovered, both laptop desktops" says more than any rate. It points directly at two changes. Counts also behave better than a rate at this volume. Two tickets changing sides moves a rate built on thirty by a visible amount. A quarter-on-quarter comparison of rates at that size is noise with a story attached. The counts and the shapes are the story. Where the quarter is very small, report the half-year, and say so.

Do not cut this number by technician except for coaching. Even then, do so only when a technician has handled enough recovery tickets for a pattern to be real. On most small desks that means a few dozen comparable ones, and that means rarely. A technician whose two backup restores both came back partial has not failed. The backup runs nightly and the day was always going to be missing. What a per-person look can usefully show is the technician who never checks version history before asking for a backup. That is a conversation about the layer order, not a score. The rule that a per-person figure is for coaching and never for ranking holds here as everywhere in this library. Recovery tickets are too few and too varied to rank anyone by in any case.

Time to Recover, by Layer

Time to recover is read by layer or not at all, because the layers differ by two orders of magnitude and a single figure across them describes nothing. A quick win takes minutes. Version history and the bin take minutes. A snapshot takes an hour or a few, most of it waiting for the administrator. A backup takes a working day or two, nearly all of it waiting for the backup team's queue. The median of those together is whatever the mix happened to be that quarter. The average is dragged by the two backup restores into a figure that describes no ticket at all. Use the median within each layer and report the layers separately. The reasoning behind medians over averages for any time measure is in Median and Percentiles, Not Averages. That reasoning applies with extra force to a measure with four distinct populations in it.

Separate working time from waiting time where the ticket allows it. Take a backup restore that took two working days of elapsed time and twenty minutes of the desk's attention. It is a different thing from one that took two days because the technician visited the layers slowly. The restore record's timestamps tell the two apart. The waiting time is the number to take to the backup team, kindly, with the tickets. The working time is the number that says whether the desk knows its layers. At Acme, the two backup restores waited a day and a half each and took the desk under an hour between them. That is normal and is something Nadia had never been able to say before.

Watch the quick-win time especially, because it is the one the desk controls entirely. A quick win that takes a technician forty minutes is a technician searching without the intake answers, or a desk that does not know which bins its file service has. When the near-layer median drifts upward over two quarters, the fix is usually the one page of places and protections from the versioning article. That means the page pinned where the desk can see it, not anything to do with the files.

The Restore Test Every Request Quietly Performs

Every recovery ticket resolved from a snapshot or a backup is a restore test. It is a demonstration, under real conditions, that the layer can produce a specific file from a specific moment within a known time. Organisations schedule restore tests because auditors ask for them and because a backup that has never been restored from is a hope rather than a protection. Most organisations schedule them rarely and perform them more rarely still. The desk performs them whenever a requester needs a file, and the only thing missing is the record. The log is the record. A column that says "backup, whole, from the Tuesday night copy, one and a half days" is a restore test result with a date on it. A quarter's worth of them is a test programme.

Read the layer column for the layer that scores zero. Take a desk whose quarter shows twenty rescues from the bin and version history, three from snapshots and none from the backup. It has proved the near layers and proved nothing about the far one. The far one is the layer that exists for the day the service is lost. That is the quarter to schedule a deliberate restore from the backup, of a real folder to a side location. The restore is timed and recorded in the log with "test" in the shape column. That way the next quarter's log has a result for every layer. The deliberate test is cheap. The desk already knows how to ask for a restore, because it asks for them for requesters. The backup team would far rather be asked on a quiet Wednesday than on the Friday that matters.

The record also answers the auditor's question before it is asked, and more honestly than a test schedule does. "We restored real files from the backup on these dates, for these people, in these times, and here are the tickets" is evidence of a kind an auditor rarely sees from a small organisation. It costs the desk nothing beyond the six words at resolution. It also answers Graham. Backups are good to the extent that they have produced files on demand, and the log is the list of demands.

The Folder That Keeps Needing Rescue

The "where it lived" column, cut by place and by department, finds the sources of loss, and the sources are nearly always few. On most desks, one or two places produce most of the requests. Examples are laptop desktops on a model that was never redirected, or one shared folder with a permission tangle that lets a whole team delete each other's work. Others are a sync folder arrangement in one department that was set up by the department, or the old server where every overwrite loses a day. Each of these is a hotspot in the sense used in Finding Hotspots in the Ticket Data. That means a place or group that produces far more than its share of a ticket type. The recovery log is a ready-made hotspot cut with the hard part, the tagging, already done.

Read the hotspot with the shape column beside it, because the pair says what to do. The share with many deletions and a bin that always has them is a permissions problem, or a team that tidies aggressively. The fix is a narrower delete permission or a conversation with the team's manager, not a bigger bin. The desktop with many losses and no layer is a redirection project. The server with many overwrites and only the backup to rescue them is a migration case, and the lost days are its budget. The department with many sync deletions is a client setting and a one-line note in onboarding. The desk fixes the place, once, and the next quarter's log shows whether it worked, which is the whole feedback loop in two columns.

Resist fixing a hotspot that is not one. Three requests from one share in a quarter may be three different people having three ordinary accidents, and a quarter is a short time. Look at two or three quarters before calling a place a source rather than a coincidence. The exception is a shape that cannot be an accident, such as a share outside the backup set. That is a fix the day it is seen. The log is small. Small logs are honest about what they can and cannot show, if the person reading them is.

The One Line for the Report

The quarterly line for leadership is one sentence of counts and one of meaning. It goes in the standing report in the place where the desk's other numbers live. That is laid out in The Monthly Report They Will Read. The counts: requests, whole, partial, not recovered, and the number produced by snapshots and by the backup, with the median time for the near layers and for the backup. The meaning: what the unrecoverable ones had in common and what the desk proposes to change. "Thirty-one recovery requests; twenty-seven whole, two partial, two not recovered; snapshots produced files three times and the backup twice, in about a day and a half each; both unrecoverable files were on laptop desktops that are not redirected, and redirection for that model is proposed." Two sentences. Graham reads them.

Leave out the rate unless it is asked for, and if it is asked for, give the counts in the same breath. Leave out the per-layer detail except the two far layers, which are the ones that carry the confidence question. Leave out any number that the desk could not reproduce from the tickets. That is because the first quarter this line appears, somebody will ask to see the tickets. The second quarter they will not, provided the first answer was good. If you run QuerySolv, the standard reports already give volume and resolution times. The recovery log adds the layer and the outcome, which no help desk report knows to ask for. Those are the two columns the question about the backups turns on.

A Worked Example in Ranges

Here is Acme's quarter, as Nadia logged it from the thirty-one tickets on the Monday morning. The figures are rounded into ranges because a reader of this article will have a different desk, a different file service and a different mix. The table shows the layer, the count, the outcomes, and the median time to recover from first touch to confirmation. It shows roughly how much of that was the desk's own working time. The numbers are one fictional desk's and are here to show the shape of a log that has been read, not to be compared against.

Read as a whole, the log says four things, and Nadia wrote them under the table. The near layers handle most requests in minutes and are proven many times over. The far layers are proven, a few times each, with the waiting time belonging to other teams and reasonable. The two partials both came from the server that has no version history, which is the migration case with two more lines of evidence. And the two losses share a shape, the un-redirected laptop desktop, which is the one change to propose this quarter. The change is costed in the hours the two rebuilds took, which the requesters put at somewhere between a day and two days each. The desk's own time across all thirty-one was somewhere between twelve and twenty hours for the quarter. That is the cost of the confidence, and it is cheap.

Where the article on the file that cannot be recovered says to pick one change, this is where the change gets its evidence. That means two tickets, one shape, the rebuild hours, and the quarter's date. The log makes the case; the proposal makes the change; the next quarter's log shows whether the row for "none" has emptied. That is the entire loop, and it fits on a page.

A Short War Story: The Green Jobs at Oakhurst Veterinary

At Oakhurst Veterinary, a practice with three sites, the desk had never once needed the backup in two years. That was because version history and the bins on the file service had handled every recovery request. The backup jobs reported success every night to a mailbox that had belonged to a technician who had since left. A practice manager's account clean-up removed a share of laboratory results that had been set up with a short bin window and no snapshots. When that happened, the technician, Hamish, asked for the backup. The backup team found that the job for that share had been failing for five months with its warning mail going to the departed technician's mailbox. About five weeks of results had to be re-requested from the external laboratory, roughly thirty hours of administrative time across the three sites. Two follow-up appointments were delayed because the results were not to hand. The practice's recovery log, started the following month, has a row for the backup. The row was filled by a deliberate restore test in the first quarter rather than by a crisis. The lesson was that a layer the desk has never pulled a file from is a hope with a green light on it. The log is how you notice the layer that has never been asked.

The Recovery Log

The log below is the one Nadia keeps, as a fixed line on each recovery ticket at resolution. She also keeps it as a one-page sheet filled in at quarter end from a search for those lines. The first block is the line and its codes, short enough to type in under a minute. The second block is the quarterly sheet, with the three reads that matter: the layer that scored zero, the place that keeps appearing, and the shape the losses share. The last line is the one for the report, written from the sheet in the words leadership will read.

RECOVERY LOG — ONE LINE PER TICKET, ONE SHEET PER QUARTER

A. THE LINE (first line of the resolution note on every recovery ticket)
   RECOVERY: place / shape / layer / outcome / hours / lesson / dept
   place:   share | desktop | sync | oldserver | device | other
   shape:   moved | deleted | overwritten | neversaved | wiped | notinset | damaged | test
   layer:   quickwin | versions | bin | snapshot | backup | none
   outcome: whole | partial (name the gap) | notrecovered
   hours:   first desk touch to confirmation, working hours; add "wait" hours if known
   lesson:  place | version | bin | sync | save | name | none
   Example: RECOVERY: share / overwritten / versions / whole / 0.3 / version / sales

B. THE QUARTERLY SHEET
   Period: ____________   Requests: ______   Whole: ______  Partial: ______  Not: ______

   By layer          Requests   Whole   Partial   Not   Median hours   Of which waiting
   quick win         ______     ______  ______    ___   __________     __________
   version history   ______     ______  ______    ___   __________     __________
   recycle bin       ______     ______  ______    ___   __________     __________
   snapshot          ______     ______  ______    ___   __________     __________
   backup            ______     ______  ______    ___   __________     __________
   none              ______     ______  ______    ___   __________     __________
   Layer with zero requests this quarter: ______________  -> deliberate test booked for ____/____

   By place          Requests   Shapes seen                 Repeat from last quarter?
   ______________    ______     ______________________      yes / no
   ______________    ______     ______________________      yes / no
   ______________    ______     ______________________      yes / no
   Place that keeps appearing: ______________  Shape: ______________

   Not recovered (one line each): ticket ______ place ________ shape ________ rebuild hours ______
                                  ticket ______ place ________ shape ________ rebuild hours ______
   Shape they share: ______________   One change proposed: ______________  owner: __________

   Lessons sent by shape: place ___  version ___  bin ___  sync ___  save ___  name ___
   Any lesson sent more than ______ times -> structural fix overdue: ______________

C. THE LINE FOR THE REPORT
   "____ recovery requests; ____ whole, ____ partial, ____ not recovered;
    snapshots produced files ____ times and the backup ____ times, in about ____;
    the unrecoverable files shared ______________, and ______________ is proposed."

Nadia took the two sentences to Graham on the Wednesday, with the sheet behind them. The thirty-one ticket numbers were on a second page that he did not ask for. He asked how the backup had been proved. She said twice this quarter, for real people, in about a day and a half each. She also said that a deliberate test of the one share that had not come up was booked for the following week. He asked what the two losses had cost. She said the requesters put it at a day or two each, and that redirection for that laptop model would stop the shape. He approved the redirection in the meeting, which was not what she had expected, and asked for the line every quarter, which was. The infrastructure lead, to his credit, asked for a copy of the sheet.

Frequently Asked Questions