Testing Articles on Real Users: The Hallway Test
Thursday, 15:30, at Redbrook Engineering. Anneke has finished "How do I show my laptop on the meeting-room screen?" It has nine steps and two cropped screenshots. She wrote it with the system open, and her lead checked it. He called it the best article in the knowledge base. She takes her laptop to the kitchen, where Rafael from accounts is waiting for the kettle. She says: "Could you follow this for me? I'm not going to say anything." Step 1 is "Plug the grey cable into the laptop." He does. Step 2 is "Press the Screen key." Rafael looks at the keyboard. He looks at the article. He looks at the keyboard again, for longer. "Which one's the screen key?" Anneke says nothing, with some effort. He presses two keys that are not it. At the end of a minute he says "sorry, I don't know what that means", and reaches for his tea. Nine steps, two technicians, the best article in the knowledge base, and nobody who had not written it got past step two.
A hallway test is handing a draft article to someone who has never done the task, watching them follow it without helping, and writing down every place they stop. It takes about a quarter of an hour and needs no equipment beyond the draft and the real system. It finds the step that the writer and every technician reviewer walked straight past. That is because they all knew which key the Screen key was. This article is part of our Writing KB Articles People Can Actually Follow series. It is about the only test that tells you whether an article can be followed: watch someone try.
It covers why the writer cannot test their own article; the hallway test step by step; the discipline of watching in silence; reading the stall sheet; fixing the step rather than the tester. Next come the three-user rule; testing the published version on the real portal; a short war story; and the test sheet to copy, in that order.
Why the Writer Cannot Test Their Own Article
The writer of an article can check that it is accurate and cannot check that it is followable. The second requires not knowing the task, and the writer cannot arrange that. Every step reads as complete to the person who already does the action without thinking. "Press the Screen key" is a complete instruction to Anneke, because her hand goes to it. It is a riddle to Rafael, because the key has a small picture of a monitor on it and no word. The writer cannot see that the word is missing. There is no amount of rereading that supplies the reader's ignorance to the writer's eye. The sooner a desk accepts that, the sooner it stops publishing articles that have only been proofread.
A technician reviewer has the same blindness in a different chair. The lead who called Anneke's article the best in the knowledge base knew where the Screen key was too. He read step 2 as fine because, for him, it was. Review by someone who knows the task checks facts: the labels are right, the order is right, nothing is dangerous. That is worth doing and is the subject of The Article Review Checklist Before Publishing. It is not testing. Testing is review by someone who does not know. The two find different things, and an article needs both.
The honest position is that the instrument for measuring followability is a stranger, and there is no substitute. Not a checklist, not a readability score, not a second technician, not the writer reading it aloud. A person who has never done the task follows the article on the real system while the writer watches and says nothing. That is the test. Everything in this article is about running that test cheaply enough that it happens for every article rather than for none. Fifteen minutes in a kitchen is cheap. Thirty-one tickets about step 2 are not, and that is the only alternative on offer.
The Hallway Test, Step by Step
Pick a tester who has never done the task and is roughly the kind of person the article is for. Do not pick a technician, unless the article is for technicians. Do not pick the most or least confident person in the building. Just pick someone from the audience who happens to be near the kettle, which is where the test gets its name. Ask for a quarter of an hour. Most people say yes, because being asked to break something that IT wrote is a small pleasure, and because the alternative was making tea in silence.
Set up the real thing. The tester follows the article on the actual system, with an account that has the rights the audience has, not the writer's account. The draft is open on a second screen, on a phone, or on paper. If the task needs a meeting room, do it in the meeting room. If it needs a till, do it on a till. A test on the writer's laptop, with the writer's account, in the writer's chair, tests a situation no reader will be in. Then say the script, which is short: "Follow this as if I were not here. I will not help, even if you ask. Say what you are thinking out loud if you like. Tell me when you think you are done." And then stop talking.
Watch, and write down every stall. A stall is any moment the tester is not doing the next action. That could be a pause longer than a few seconds, a re-read, a scroll back to an earlier step, a click on the wrong thing, a question, a glance at you. Write the step number and what they did, in plain description: "Step 2: looked at keyboard, looked at article, pressed F-key, pressed another, asked which key, gave up." Do not write what you think they were thinking. Write what you saw. When they say "done", note whether they are. A tester who says "done" with the task half-finished has just told you the article has no result line. Then ask three questions: where was it hardest, what did you expect to see that you did not, and would you have phoned. That is the whole test. Fifteen minutes, one sheet of notes, and an article that is about to get better.
Watching in Silence Is the Hard Part
The writer's instinct, at the first stall, is to help, and helping ends the test. "Oh, it's the one with the little monitor on it" fixes Rafael and leaves the article exactly as it was. So the next reader, who does not have Anneke standing beside them, stalls in the same place with nobody to ask. The tester is not the thing being tested. The article is, and the article has to fail in front of you so that you can see where. Sit on your hands if necessary. I have sat on mine, and it is harder than it sounds. That is because the stalled tester looks at you, and you know the answer, and the answer is one word.
When the tester asks a question, the reply is "what would you do if I weren't here?" and nothing more. Usually they re-read, try something, and either get through, which is useful to see, or do not, which is more useful. When they are entirely stuck, let them use the article's own escape hatch. If the article says "raise a ticket" or "call the desk", let them say they would, and note it. The test has just told you two things: that the step fails and that the hatch was findable. Only when the tester has genuinely given up do you help. At that point the test is over for that step. You can carry on through the rest. But anything after the rescue is a tester who has been shown the answer to one thing and is now a little wiser than the real reader.
Write what they did, not what you have decided it means. "Step 4: clicked Cancel instead of Apply" is an observation. "Step 4: didn't read properly" is a verdict, and it is nearly always the wrong one. A tester who did not read properly is a reader who will not read properly either. The article has to work for them. Everything on the stall sheet should be the kind of thing a camera would have recorded. The interpreting happens afterwards, at your desk, with the sheet and the draft side by side. The interpretation is nearly always "the article is missing a word here".
Reading the Stall Sheet
Each stall has a shape, and the shape points at the fix. A tester who looks around the screen and cannot find the thing has met a step whose wording does not match the screen. Or they have met a control with no label that the article did not describe. The fix is the exact words, or a tight crop with one mark. A tester who reads the step twice and asks what a word means has met jargon. The fix is to use the screen's word and explain it once. A tester who does one thing where the step wanted two has met a merged step. The fix is to split it. A tester who finishes a step and hesitates has met a missing result line. A tester who gets to step 5 and cannot continue because they lack something has met a missing "before you start" item. And a tester may do something invisible to the writer, like clicking Add before Save, and only then get through. That tester has found the trap step.
The table below is the stall sheet read as a diagnosis, with the fix for each shape. Most tests produce two or three stalls of two or three shapes, and the fixes together take less time than the test did. The step-writing rules that the fixes depend on are in Step-by-Step Clarity: One Action per Step. The parts of the article that the "before you start" and result-line fixes belong to are in The Anatomy of a Followable Article. The stall sheet tells you which of those rules this article broke, and where.
| What you saw | What it means | The fix |
|---|---|---|
| Looked around the screen, could not find it | The step's words do not match the screen, or the control has no label | Use the printed label exactly; describe an unlabelled control by shape and position, or add one cropped picture |
| Re-read the step, asked what a word meant | Jargon, or a name the reader has not been taught | Use the screen's word and explain it once, in the step or the line before |
| Did one action where the step needed two | Merged step | Split into two numbered steps with a result line each |
| Finished the step, hesitated, looked at you | No result line; they cannot tell whether it worked | Add "You should see: ______" from the real screen |
| Reached a step and lacked something: a right, a file, a cable | Missing "before you start" item | Add it to the list at the top with how to check for it |
| Did something the article never mentioned, then got through | The trap step, done by instinct | Make it a numbered step and say what the reader sees if they miss it |
| Said "done" before the task was finished | No finished-result line, or the article ends early | Add "When you have finished, you should see: ______" |
| Said they would phone | The step failed hard, or the escape hatch was easier than the article | Fix the step; then check the hatch is placed after the fix, not instead of it |
Fix the Step, Not the Tester
The stall is data about the article, and the only acceptable response to it is a change to the article at that step. Do not add a note elsewhere or a footnote. Do not add a "tip" in a box at the end saying that the Screen key is the one with the monitor on it. The step itself becomes "Press the key with the small monitor picture on it, in the top row of the keyboard, usually between F4 and F6." The reader stalls at the step, so the fix lives in the step. A tip at the bottom is a fix for a reader who has already given up and scrolled, and most do not scroll. They phone.
The other response, the one that feels natural and is always wrong, is to explain the tester. "Rafael isn't very technical." "She wasn't really concentrating." "He's new." Each of those is true of a large part of the audience, and the article has to work for the audience. So each is a description of a reader the article will meet and an argument for the fix, not against it. The tester who struggles is the most valuable tester you will get. The tester who sails through has told you only that the article works for people like them. People like them were not going to raise the ticket.
Make the fix, and then test again with the next person, from the top, not from the step you fixed. A fix at step 2 changes how the reader arrives at step 3. A tester who starts at step 3 is not a reader. The second run usually finds that step 2 is fine now. It usually also finds a smaller stall at step 6 that the first tester never reached, because they never got that far. This is why the test is repeated, and why the next section has a number in it.
Remember: the writer cannot test their own article. Hand it to someone who has never done the task, watch without helping, write down where they stop, fix the step they stopped at, and test again. Three clean runs, then publish, then check the published version from a user's account.
The Three-User Rule
Test with three people, one after another, fixing between each, and publish when the third gets through cleanly. The first tester finds the biggest problem, which is usually the one that would have generated most of the tickets. They often find nothing else because they never get past it. The second confirms the fix and finds the next problem down, which the first never reached. The third tells you whether you are done. A clean run, from the top, by someone who has never done the task, on the real system, is the signal to publish. A stall on the third run means fix and test a fourth. (Nobody has ever needed a fifth, in my experience, for a task that was really one task.)
Three is not a magic number. It is the point at which the next test stops being worth more than the time it costs for a help desk article of ten steps. A long or dangerous procedure deserves more, and a two-step article may be done after one. The honest statement of the rule is that three clean-enough runs is not proof that the article is perfect. The tickets in the first month will be testers four to forty and will find the rest. Three is enough to catch the stalls that stop most readers, and that is the standard a knowledge base article has to meet. It is a help desk, not a laboratory, and the kettle is only in one place.
Mix the three where you can. One from the exact audience the article names, because they are who it is for. One from a different part of the organisation, because they bring different habits and a different vocabulary, and the article has to survive both. Pick one new starter, if there is one about, because a new starter knows nothing you have forgotten you know. A new starter is also the person most likely to use the knowledge base at all. Do not use three technicians, and do not use the same willing person every time, however good they are at it. After four tests the willing person is a trained reader, and the knowledge base is not written for trained readers.
Test the Published Version on the Real Portal
The draft that passed three hallway tests is not the article the reader will see. The published version is, and it differs in ways the draft could not show. The formatting changed on the way in and the steps are no longer numbered. The picture did not upload, or uploaded at a size that fills the screen. The link in "who to contact" points at the test portal, or at nothing. The article was published to the wrong audience, so technicians can see it and users cannot. The search does not find it for the words in its own title. None of those is a writing fault, and all of them make the article unfollowable. The only way to catch them is to go and look, from the outside.
The outside means a user's account, not yours. Sign in to the portal as an ordinary user, or stand beside one, and do four things. Search for the article using the words a user would type, not its title. Open it and read it on a phone as well as a laptop. Click every link. And follow it once more, briefly, to confirm the steps survived publishing. Where the knowledge base lets an article be visible to agents only or to everyone, which QuerySolv's does and most do, check which this one is from the user's side. The writer's account can usually see everything and will never notice that the user cannot. Five minutes, the day after publishing. The article that passes this is finished. The article that fails it was never published at all, whatever the knowledge base says.
The portal test also catches the thing that is not about this article. A portal where the search returns nothing useful for the top question, or where the article is three clicks deep behind a category nobody would guess, has failed before the article is reached. The reader's first visit, which decides whether there will be a second, has been wasted. That larger failure is the subject of Making the First Visit Succeed. The article writer's share of it is to search for their own article as a stranger would and to raise it when the stranger would not have found it. After that, the first month of tickets is the test that never stops. Every ticket that mentions the article, or that should have been answered by it, is a stall sheet with one line on it. The habit of opening the article during the ticket and fixing the one thing that was wrong is described in Refine on Reuse.
A Short War Story: The Till Update at Northgate Retail
Northgate Retail wrote "How do I install the till update?" at head office. It had the article reviewed by two technicians who had each installed the update a dozen times. It sent the article to sixty stores for the Saturday morning before opening, with the instruction to follow it before 08:30. Step 5 said "click Save". On the store tills, which ran an older build of the till software than the head office test machine, the button was labelled Apply. Before it appeared at all, a dialog nobody at head office had seen for a year asked "Trust this update?" That was because the test machine had answered it long ago and remembered. Hollie, who ran the Harbor Street store, phoned at 08:05 to say that there was no Save and there was a box asking about trust. She was not clicking anything until someone told her to. Forty-seven other stores phoned with the same two questions over the following fifty minutes. So both technicians on duty spent the whole morning on the phone saying "click Yes, then Apply". Four stores whose managers could not get through traded all day on the old software with the previous week's prices. The cost was two technician-mornings, four stores of mispriced trade, and a Monday meeting with the retail director about why IT had sent out instructions that did not match the tills. One store colleague following the draft on one real till on the Friday afternoon would have found both the dialog and the label in under five minutes. The article had been reviewed by experts and tested by nobody.
The Test Sheet
The sheet below is one hallway test on one page, with the three-user tally and the portal test at the bottom. That way, the whole life of the article's testing fits on a single sheet pinned to the draft. Fill in the top before the tester starts and the step rows while they work, in the words of what you saw. The debrief questions are asked at the end, in that order. The answers are written down before anything is discussed, because discussion changes what the tester remembers. The fixes column is filled in afterwards, at your desk.
HALLWAY TEST SHEET — ONE TEST, ONE PAGE Article: ______________________________________ Draft version: ____ Date: ________ Tester: ____________ Role: ____________ Done this task before? [ ] no [ ] once [ ] often Set-up: real system [ ] ordinary user account [ ] real location/device [ ] Draft read from: [ ] second screen [ ] phone [ ] paper Script said: "Follow this as if I were not here. I will not help. Say what you are thinking if you like. Tell me when you think you are done." Start: ____:____ STALLS: write what you saw, not what you think they thought Step What they did Stall? Said anything? ____ ___________________________________________________ [ ] ______________ ____ ___________________________________________________ [ ] ______________ ____ ___________________________________________________ [ ] ______________ ____ ___________________________________________________ [ ] ______________ ____ ___________________________________________________ [ ] ______________ Said "done" at: ____:____ Actually done? [ ] yes [ ] no: what was missing ________ Used the escape hatch? [ ] no [ ] said they would phone at step ____ DEBRIEF: ask in this order; write the answers before discussing Where was it hardest? ______________________________________________________ What did you expect to see that you did not? _______________________________ Would you have phoned? [ ] yes, at step ____ [ ] no FIXES MADE: one per stall, in the step itself Step ____: ________________________________________________________________ Step ____: ________________________________________________________________ Step ____: ________________________________________________________________ THREE-USER TALLY Tester 1: ____________ stalls: ____ Tester 2: ____________ stalls: ____ Tester 3: ____________ stalls: ____ Clean run? [ ] yes: publish [ ] no: tester 4 PUBLISHED-VERSION TEST: next day, from a user's account [ ] Found by searching the words a user would type: "__________________" [ ] Readable on a phone [ ] Steps still numbered [ ] Pictures present and sized [ ] Every link works [ ] Visible to the right audience, checked from a user's side [ ] Followed once more; it still works Tested by: ________ Date: ________
Anneke's step 2 became "Press the key in the top row with a small picture of a monitor on it, usually between F4 and F6; on most of our laptops it is F5," with a crop of the key. Her second tester, Sunita from the drawing office, got past it in two seconds and stalled at step 6. There, "choose the right input" turned out to mean pressing a button on the wall panel that the article had never mentioned. That was because Anneke's hand did that too. Her third tester, a new starter in his first week, went from cable to picture in four minutes and said "that was fine". That is the least exciting sentence in support and the one the test is for. The published-version check the next morning found that the crop of the key had not uploaded. It took a minute to fix, and it would have taken a month of tickets to find.
Frequently Asked Questions
We are a two-person desk. Who do we test on?
How long does a hallway test take, honestly?
What if the tester gets through with no stalls on the first run?
Can I test on a technician if nobody else is free?
Should the tester use their own account or a test account?
How do I test the published version if I cannot sign in as a user?
From the QuerySolv team: we build self-hosted help desk software for Windows. QuerySolv runs as one Windows service on your own network. It takes requests by web form, e-mail, live chat, social messages and REST API. It is licensed once per installation rather than per agent per month. The free trial runs the full product for 30 days.