Failing-Hardware Signals: The Swap Before the Failure

Monday, 11:30, at Ashgrove Housing. Mo, a housing officer, phones the desk from a tenant's kitchen: "My laptop just restarted itself again, in the middle of the visit note. That's the third time this week." Tess takes the call, asks him to bring it in, and spends ten minutes with it that afternoon. The event log shows eleven unexpected shutdowns in three weeks. The drive has been reporting reallocated sectors for six. The battery holds about a third of what it was built to hold, which is why the restarts happen at lunchtime. That is when the morning's charge runs out and the laptop believes it still has forty percent. None of this was secret. The machine had been saying all of it, in three different places, for a month and a half, to nobody.

Tess swaps the drive and the battery that afternoon, and Mo's laptop is fine. The visit note is not; it went with the restart. A failing-hardware signal is a measurement a device makes about its own condition that precedes a failure by days to months. Examples include a drive's self-test counters, a battery's wear figure, a fan running flat out, and a run of unexpected shutdowns in the event log. The practice this article describes is reading those signals on a schedule and swapping the part on the desk's timetable rather than the part's. This article is part of our Proactive Remediation series. Hardware is the second signal to add, after disk space, and the one where the gap between a planned fix and an unplanned one is widest.

This article covers why hardware usually warns first. Next come disk health, which is the warning you do not argue with, and batteries, including the swollen one. Then come fans, heat and the laptop that sounds like a hairdryer, followed by the event log and the machine that reboots. The article provides a triage table for which signal means what. It covers the swap itself and the spares pool that makes it possible, then recording the fleet's hardware history. It finishes with a short war story and the hardware watch list to copy, in that order.

Hardware Usually Warns First

Most hardware failures on a desk's fleet are preceded by a measurable warning. And the warning is usually ignored because nobody is reading the place it appears. A drive keeps a set of self-monitoring counters about its own errors. These are often called its health data or SMART data (Self-Monitoring, Analysis and Reporting Technology). The drive reports a summary status that the operating system can read. A laptop battery reports its design capacity and its current full-charge capacity, and the ratio is its wear. A fan reports its speed and the sensors around it report temperatures. The operating system's event log records every unexpected shutdown, every disk error it caught, every memory error, every driver that fell over. Four sources, all local to the machine, all readable across the network in a second or two.

The reason to read them is the asymmetry of cost. A planned swap of a drive or a battery takes about an hour of a technician's time. It happens at a time the user chose, with the data copied first. An unplanned failure of the same part takes the user's day, the technician's afternoon, and whatever was on the drive and not yet on the share. There is also the conversation about why the organisation's laptops "just die". The part costs the same in both cases. Everything else costs more in the second. Reading the warning is how you buy the first case with the second case's budget.

The honest limit comes first, because the rest of the article depends on it. Hardware warnings predict most failures and not all. A drive can fail without a single reallocated sector. A battery can swell at a healthy-looking wear figure. A laptop can die of a motherboard fault that no sensor reports. The sweep described here catches the warned failures, which in my experience is most of them. It changes nothing about the need for backups, which is what catches the rest. A desk that reads hardware signals and keeps good backups loses very little data. A desk that does only the first has moved the risk, not removed it.

Disk Health: The Warning You Do Not Argue With

A drive's health status is the single hardware signal with the clearest rule. And the rule is that a drive which has started reporting trouble is replaced that week. Mechanical drives report reallocated sectors, which are patches of the disk that failed and were quietly swapped for spares. They also report pending sectors, which are patches that failed and have not been dealt with yet. A count above zero on either, or an overall status of anything other than good, means the drive has begun the process of failing and is telling you so. Solid-state drives wear differently and report a percentage of their rated life used. A figure approaching the top, or any reported media error, is the same message in different words. A drive that has started reallocating sectors is not asking for a second opinion.

The temptation is to write "monitor" next to the warning and look again next week. And it should be resisted for a specific reason: the drive's data is at its most reachable now. A drive at the first warning can usually be copied in full, calmly, during a planned hour. A drive at the third warning may already be returning errors on the files the user needs most. That is because the files they open most are the ones on the sectors that wore fastest. The cost of the swap is identical in both weeks. The cost of the delay is whatever the drive loses in between, and the drive does not publish that schedule.

Watch for the drive symptom that arrives as a different ticket. A drive that is retrying reads will make a machine slow in a way that looks like a full disk or a memory shortage. There are long pauses on opening files, a cursor that spins at odd moments, and a startup that takes three times as long as it did. When the disk-space sweep from Disk-Space Sweeps shows plenty of free space on a machine the user calls slow, the next place to look is the drive's health status. The sweep script for this article reads both in the same pass.

Batteries: Wear, Swelling and the Lunchtime Shutdown

A laptop battery has a design capacity, which is what it held when new. It also has a full-charge capacity, which is what it holds now. The second falls with every charge cycle. The ratio is the wear figure, and it is readable from the operating system without opening anything. A battery at sixty percent of its design capacity is a laptop that lasts a little over half as long as it did. The user experiences that as "it used to get me through the morning". A battery at thirty or forty percent is a laptop that shuts down without warning. That is because worn batteries report their remaining charge badly, and the operating system believes them. A worn battery reports forty percent right up to the moment it reports nothing.

The threshold for a planned swap is a desk decision. Somewhere around sixty percent of design capacity is a reasonable place to start for machines that leave the building. A lower figure is acceptable for laptops that live on a desk and a charger. The sweep for batteries is monthly, because wear moves slowly. The hit is a ticket and a message to the user offering an appointment. It is one of the few hardware swaps a user is pleased to hear about in advance, because the user already knows. They have been carrying the charger to meetings for a month.

Swelling is a different matter and gets a straight paragraph. A battery that has swollen pushes the case apart. The touchpad lifts or stops clicking, the laptop no longer sits flat, the keyboard bows, or the base bulges. This is a safety fault, not a wear fault. The machine comes out of use that day. The battery is not pressed, punctured, charged or put in a drawer with the others. And it goes out through the organisation's hazardous disposal route. Tell users what a swollen battery looks like in the one-time notice about the sweep. A user who knows will report it, and a user who does not will keep using it. The wear figure does not reliably predict swelling, which is why the sweep cannot replace the user's eyes on this one.

Fans, Heat and the Hairdryer

A fan that runs flat out all day is a signal. And the user will usually report it as a noise before anything else goes wrong. Laptops cool themselves by pulling air through a vent that fills with dust over a year or two. After that, the fan runs faster and longer to do the same work. The machine runs hotter, and the processor slows itself down to stay within its limits. The user experiences that as "slow", and the ticket says "slow". The disk is fine and the memory is fine and the technician is puzzled until they put a hand on the base. The laptop in that story was described in its own specification sheet as "whisper quiet", which it had been, once.

Temperatures and fan speeds are readable on most machines, though less uniformly than drive health. A sweep that reads them is worth having once the first two signals are routine. The simpler signal is the user's description plus the machine's age. A laptop over two years old whose user mentions noise or heat gets a clean and a check. That is twenty minutes with the right tools and the right training, and is a planned job rather than a repair. Heat also shortens a battery's life, so the hot laptop is often the worn-battery laptop a few months early. Fix the heat and you have bought the battery some time.

The Event Log and the Machine That Reboots

The operating system keeps an event log, a running record of what the system and its components reported. Three kinds of entry in it are hardware signals. Unexpected shutdowns are the system noting on startup that it did not shut down cleanly last time. Disk errors are where the system caught a read or write failing. The third kind is memory or hardware errors reported by the machine's own checks. One unexpected shutdown is a power cut or a user holding the button. Two in a month is worth a look. Five is a machine that is failing and a user who has stopped mentioning it.

A laptop that restarts itself has one of four problems: heat, power, the drive, or memory. That is roughly their order of likelihood and the reverse order of how often the desk checks them. Heat is the fan paragraph above. Power is the battery paragraph, or a damaged charger or socket. The drive and memory are the event log's other two entry types. When the shutdowns coincide with disk errors the drive paragraph applies and the week's rule with it. Read all four signals on the same machine before deciding, because a laptop with a worn battery and a dusty fan has two problems. On that laptop, swapping the battery alone fixes the lunchtime shutdowns and leaves the slowness.

The sweep script for hardware reads the four signals together, once a month, across the machine list. Like every sweep in this series it changes nothing. Here is its shape in plain words. The real one, in the shell or scripting language built into the operating system, is a page. That is because reading each signal takes a line or two. Most of the lines are about which of several places a given model reports the figure.

HARDWARE SWEEP (reads only; changes nothing)
for each name in machines.txt
    if name does not answer on the network
        write "name, offline" to hardware-report.csv
        continue with the next name
    health   = the system drive's reported health status (good / caution / failing)
    realloc  = the drive's reallocated plus pending sector count (or wear percent used)
    battery  = full-charge capacity divided by design capacity, as a percent
    crashes  = number of unexpected-shutdown events in the last 30 days
    diskerr  = number of disk error events in the last 30 days
    write "name, health, realloc, battery, crashes, diskerr" to hardware-report.csv
print the lines where health is not good, or realloc is above 0,
      or battery is under the threshold, or crashes is 2 or more, or diskerr is 1 or more

The report is a short list, and each line on it is a decision rather than a repair. Which decision depends on which signal fired, which is what the table in the next section is for. Attach the report to the month's sweep ticket whether or not it has lines on it, so that a quiet month is recorded as quiet. I once put "monitor" next to a drive warning for three weeks running. The drive did not monitor back.

Triage: Which Signal Means What

The signals differ in how fast they turn into failures and in what the desk should do. And the table below is the triage a technician reading the report applies to each line. It is a worked example from a fictional desk; adjust the thresholds to your fleet. And keep the actions, because the actions are the part that does not change. The first column of the watch list at the end of this article is the same set of signals with blanks.

The Swap Before the Failure

The thing that makes a planned swap possible is a small pool of spares, and it is the first purchase to make when the hardware sweep starts. Take a desk of sixty laptops that holds two spare laptops, three spare drives of the fleet's standard size and two spare batteries for the common model. That desk can swap on the day the report says so. A desk with no spares orders the part when the report says so, waits a week, and swaps after the failure anyway. The pool is not stock; it is lead time, bought in advance. Size it from the first quarter's reports, and expect the battery count to be the one you got wrong.

The swap itself is an appointment, offered to the user with a reason they will recognise. "Your laptop's drive has started reporting errors. It is still working, and we would like to replace it before it stops. That takes about an hour. Nothing you have saved to the share will be affected" gets a yes and a thank-you from nearly everyone. The few who put it off are told the week's rule and the reason. The message is the proactive notice from Telling Users Before They Notice with the part named. It says what was found, what will happen, and when, and it does not say "monitor".

Copy the data before anything else, and keep the old part for a week after the swap in case the copy missed something. For a drive that means a full image or a file copy to the share, done while the drive is still readable. That is the whole point of acting in the first week. For laptops, swap the machine rather than repairing in the field wherever the pool allows it. That way, the user is back at work in twenty minutes with a spare. The failing machine is repaired at the desk at leisure. Where the part is under warranty, the swap is also a claim. The article on Warranty and History at a Glance covers how the desk knows the warranty status from the ticket without an asset system to look it up in.

Recording the Fleet's Hardware History

Every hardware ticket names the machine, by its asset tag or its serial number. That one habit turns a pile of swap tickets into a fleet's medical record. Suppose a technician can search the ticket history for a machine's identifier and see "battery swapped", "drive warned", "fan cleaned" in order. That technician knows whether the next ticket on that machine is the fourth or the first. A desk that recorded "Mo's laptop" each time knows nothing, because Mo has had three laptops. If you run QuerySolv, the complete event history on every ticket means that record already exists. In that case, the discipline is only to put the identifier in a field the search can find.

The fleet view comes from reading the hardware tickets by model rather than by machine. Six battery swaps on the same laptop model in a quarter is a batch. A batch is a conversation with the supplier and a line in the refresh plan rather than six separate repairs. Three drive warnings on machines bought in the same order is the same thing. Tickets as Asset Intelligence is about reading the ticket history that way. The hardware sweep is its richest source, because every hit is a dated, model-tagged fact about how the fleet is ageing.

The last thing to record is the age at which parts failed, because that number sets the refresh schedule. If batteries on the common model reach the wear threshold at around two and a half years, a three-year refresh has a consequence. In that case, every laptop spends six months being carried with a charger. Either the refresh moves or the batteries are swapped once as planned maintenance. If drives warn at four years, the three-year refresh is paying for drives that would have lasted. A desk that has run the hardware sweep for a year can answer "how long do our laptops really last" with its own numbers. That is a better position in the budget meeting than quoting a supplier's.

A Short War Story: The Scanning PC at Stonebridge Insurance

Stonebridge Insurance's claims team scanned incoming paper claims on one shared PC. Its drive had shown a caution status on the desk's monthly hardware report for seven weeks running. Each time, Ben had written the word "monitor" beside it. He did not want to take the claims team's scanner away for an afternoon in a busy month. On the last working day of the month, about eleven hundred scanned pages sat in the local upload folder waiting for the overnight transfer. The drive stopped reading, and the folder went with it. Two members of the claims team spent the following two days finding and re-scanning the paper. A day's claims payments went out a day late. The operations manager asked why a known-faulty drive had been left in the one machine the month-end depended on. The swap, when Ben did it, took fifty minutes with a drive from the spares shelf that had been there the whole time. The cost was about four person-days and a difficult meeting; the saving from waiting had been nothing. "Monitor" had felt like a decision for seven weeks, and had been a deferral every time.

The Hardware Watch List

The template below is the desk's standing list of hardware signals, with the thresholds, the cadence, the action and the spares that make the action possible. The swap checklist that every hit goes through follows it. Fill in the thresholds from the triage table and adjust them after a quarter. Keep the spares section honest about what is actually on the shelf. Keep the whole thing to a page where the rota can see it. The swap checklist is the part to fight for; a watch list without it produces well-informed failures.

THE HARDWARE WATCH LIST — ONE PAGE
Desk: ______________   Owner (role): ______________   Last reviewed: ________
Machine list: machines.txt (owner: ________)   Sweep: monthly, ________

SIGNALS
  Drive health status
    Read: drive health data via the sweep script     Threshold: anything but good
    Also: reallocated or pending sectors above 0; wear used above ____ %
    Action: copy data, swap this week; keep old drive 7 days    Urgency: this week
  Battery wear
    Read: full-charge / design capacity               Threshold: under ____ %
    Action: offer appointment, swap battery                      Urgency: this month
  Battery swelling
    Read: the user; a technician's hands at every visit
    Action: out of use today; hazardous disposal; swap           Urgency: today
  Fan and heat
    Read: sensors where readable; user reports of noise or heat
    Action: clean and check; swap if it persists                 Urgency: this month
  Unexpected shutdowns
    Read: event log, last 30 days                      Threshold: ____ or more
    Action: read all signals on that machine; act on the one that fired
  Disk and memory error events
    Read: event log, last 30 days                      Threshold: 1 or more
    Action: drive rule above, or memory test and swap            Urgency: this week

SPARES ON THE SHELF (count them; do not estimate)
  Spare laptops, standard build: ____   Spare drives, standard size: ____
  Spare batteries by model: ____________________   Spare chargers: ____
  Reorder when below: ____   Reorder owner: ______________

SWAP CHECKLIST (one per hit; copy into the ticket)
  [ ] Ticket opened, named for the machine's asset tag or serial
  [ ] User told: what was found, what will happen, how long, what is unaffected
  [ ] Appointment agreed (or same-day for swelling and failing drives)
  [ ] Data copied or imaged; copy verified by opening three recent files
  [ ] Part swapped (or whole machine swapped from the pool)
  [ ] Machine returned; user signs in; share and mail confirmed working
  [ ] Old part kept 7 days, labelled with ticket number, then disposed of correctly
  [ ] Warranty claim raised if in warranty; reference recorded on the ticket
  [ ] Ticket records: signal, reading, action, part, model, machine age

RULES
  Any drive warning is swapped the week it appears. "Monitor" is not an action.
  Any swelling takes the machine out of use the same day.
  Every hardware ticket names the machine's identifier and its model.
  Review the list quarterly: thresholds, spares, and the age at which parts failed.

Mo's laptop came back to him that Monday afternoon with a new drive and a new battery. He rewrote the visit note from memory, which took him an hour he did not have. Ashgrove's desk added the hardware sweep to the first Friday of each month and read the first report with some alarm. It swapped four drives and nine batteries in the following six weeks, all by appointment, all in under an hour. The alarm was the backlog. The second report had two lines on it. The third had none, and Tess attached it to the sweep ticket anyway, with the note "nothing found". That is a different sentence from "nothing wrong" and the only one she was entitled to write.

Frequently Asked Questions