Structured Interviews and Fairness: Same Questions, Agreed Rubric

Friday, 14:00, at Bluewater Bank. Grace runs the six-person desk and has interviewed four candidates this week with Rob from the network team as her second interviewer. They are in a meeting room to decide, and they disagree. Rob wants the second candidate: "I just had a good feeling. He reminded me of me when I started." Grace wants the fourth: "She answered the questions." Rob asks which questions. Grace realises that she did not ask the fourth candidate the same questions she asked the second, because the conversation had gone a different way. The notes in front of her say "good energy" for one and "a bit quiet" for the other and nothing much else. There is a decision to make and nothing on the table to make it with. The meeting runs forty minutes and ends in a feeling.

A structured interview is one in which every candidate is asked the same questions, in the same order, by the same interviewers. Each candidate is scored against a rubric. This is a written sheet saying what a strong, adequate and weak answer to each question looks like. It is agreed before the first candidate walks in. A structured interview is the opposite of a conversation that goes where it goes. It is both fairer and more accurate than one, for the same reason. It makes the comparison between candidates about the job rather than about which conversation went better. This article is part of our Hiring Support Staff series, and it is about getting the scores onto Grace's table.

It covers what a structured interview is and why it is fairer, then writing the questions before anyone is invited. Next come the rubric agreed before the first interview, and two interviewers, each with their own sheet. It covers bias, described in general terms, and what structure does about it. Then come the shape of the interview hour, the decision meeting with scores on the table, a short war story, and the scoring rubric to copy. Nothing here is legal advice. HR owns the rules about what may be asked and how decisions are recorded. This article owns the craft of a good interview inside those rules.

What a Structured Interview Is and Why It Is Fairer

An unstructured interview is a conversation, and conversations are won by people who are good at conversations. The candidate who is fluent, familiar, relaxed in rooms and quick with a story will have a better hour than the candidate who is careful, nervous and precise. The interviewer's memory afterwards will be of the hour, not of the job. None of that is anyone's fault. It is simply what a conversation measures, and the desk is not hiring for conversation. It is hiring for the traits in What to Screen For in a Support Hire, several of which belong to exactly the careful, nervous, precise person who had the worse hour.

Structure fixes the measurement. When every candidate answers the same eight questions, the interviewer is comparing eight answers with eight answers, not a good chat with an awkward one. When the rubric was written before the interviews, the standard for a strong answer was set by the job. It was not set by whichever candidate happened to give the first impressive one. When each interviewer scores alone before talking, the score is their own judgement rather than the room's. And when the scores are on the table at the decision meeting, the meeting is about evidence. A feeling has to find some before it is allowed to vote.

Structure is also what makes a decision defensible, which matters more than most desk leads expect until the first time a candidate asks why. "She answered the questions better" can be shown, question by question, on a sheet that was written before anyone knew her name. "He had good energy" cannot be shown to anyone, including the person who wrote it, three weeks later. Fairness and accuracy turn out to be the same project here. A process that can be explained is a process that was measuring something.

Writing the Questions Before Anyone Is Invited

Six to eight questions is enough for a forty-minute conversational half. Each one is tied to a trait on the screening sheet. That way, every trait is asked about at least once and the sheet can be checked for gaps. Two kinds of question earn their place. The first asks about the past: "tell me about a time when" something happened, which produces a story with the candidate's actual behaviour in it. The second asks about a plausible future: "what would you do if" a described situation arose, which shows how they reason when they have no story to tell. Use mostly the first kind, because behaviour predicts behaviour, and one or two of the second, because a career changer may have no desk stories yet and still reasons well.

Write the follow-up prompts down too, because the follow-up is where interviews drift. "What was your part in that?", "what did you say to them?", "how did it end for the user?" are the questions that turn a tidy story into evidence. If they are written on the sheet they get asked of everyone. The experienced candidate's pair about habits, "what would you keep from your last desk, and what would you change?", is one of the eight. Aptitude Versus Experience explains why it predicts more than the CV does. Career changers get the same question about their last workplace's way of handling problems, which is the same question in different clothes.

Two rules about what is not on the list. No trick questions, for the reasons given in Interview Exercises That Actually Predict: a question with a hidden catch measures whether the candidate trusts you, and they will not. And nothing that touches what a person cannot change or what a fair employer would not ask about, in the question or in the small talk before it. HR has a list of what may not be asked, and the list varies by place. Get it, read it, and if a question you are fond of is anywhere near it, cut the question. The interview is not the place to discover the edge of a rule.

The Rubric Agreed Before the First Interview

For each question, write one sentence describing what a strong answer contains, one describing an adequate one, and one describing a weak one. Give them the scores three, two and one. Those sentences are the rubric, and they do most of the work of the whole process. They stop two interviewers scoring the same answer differently. They stop the same interviewer scoring the first candidate and the last against different standards. Write them as what the answer contains, not as what the candidate is like. "Names their own part in what went wrong and what they would do differently" is scorable. "Shows accountability" is a feeling with a number attached.

Test the rubric before you use it. Take a technician on the desk whose work you know well, imagine their answers to the eight questions, and score them. Then do the same for a technician who struggled. If the good technician does not score well, the rubric is measuring something other than the job. If both score the same, the anchors are too vague to separate anyone. Twenty minutes of this with the second interviewer, before the first invitation, finds the questions that do not work while they are still cheap to fix.

Then freeze it. The rubric does not change between the first interview and the last, however tempting it becomes after a candidate gives an answer nobody anticipated. The answer is scored against the anchors as written, and the anchors are improved for the next round. A rubric changed mid-round means the early candidates and the late ones were measured with different rulers. You can no longer say which of them was taller. The exercise scores from the practical half are brought to the same table, scored on their own sheet. The two halves together are the evidence. The method of assessing soft skills on an existing team, in Teaching and Assessing Soft Skills, uses the same anchor discipline for the same reason.

Two Interviewers, One Sheet Each

Two interviewers, never one, and never more than three. One person interviewing alone has no check on their own impressions. A panel of five turns the hour into a performance in front of an audience, which the exercises article explains is the one condition that measures nothing. Two is enough to catch each other's shortcuts and few enough that the candidate is talking to people rather than to a row. Agree in advance who asks which questions and who takes the notes for each. The one not asking writes down what was said, in the candidate's words as far as possible. What was said is the evidence and the sheet is where it lives.

Choose the second interviewer for difference, not for agreement. A peer technician from the desk brings the view of the person who will sit next to the hire. A colleague from a department the desk serves, the office manager, the finance lead, brings the view of the people the hire will be writing to. That is a view the desk lead does not have. A friend of the lead who shares the lead's instincts brings nothing but a second signature on the same impression. Grace's mistake was not having Rob in the room. It was that Rob and Grace had no shared sheet, so two people's judgements arrived at the meeting as two feelings instead of two sets of scores.

Each interviewer scores every question alone, in silence, before either says a word about the candidate. This is the single most important mechanical rule in the process, and the easiest to break. The natural thing to do when the door closes is to turn to your colleague and say "well, what did you think?". Do not. Score first; the conversation afterwards is better for it, and the scores are yours rather than the room's. Treat the sheet like a working note on a ticket, written at the time rather than reconstructed later. It is for the same reason The Working Note Habit gives: memory is a poor witness and a confident one.

Bias, in General Terms

Everyone who interviews takes shortcuts, because the brain takes shortcuts, and the shortcuts are not a character flaw. They are the ordinary machinery of judgement, pointed at a task it was not built for. Naming the common ones is useful because each has a specific countermeasure, and every countermeasure turns out to be a piece of the structure already described. This article describes them in plain words only; there are no named models here, and none are needed to recognise yourself in the list.

Similarity: we warm to people who remind us of ourselves, and "reminded me of me when I started" is the shortcut in its purest form. The countermeasure is the rubric, which scores the answer and not the resemblance. First impression: a judgement formed in the first minutes and then confirmed by everything that follows. The countermeasure is scoring per question, so that a strong answer to question six can score three even if minute one went badly. Fluency: a smooth answer feels like a correct one. The countermeasure is anchors written as what the answer contains. The halo: one excellent answer lifts every other score. The countermeasure, again, is scoring each question on its own anchors. Contrast and recency: the candidate after a weak one looks strong, and the last candidate of the day looks like the only one. The countermeasure is scoring before the next candidate arrives and comparing scores rather than memories. And confirmation: having decided, we hear what agrees with the decision. The countermeasure is the decision meeting that reads the scores before anyone says who they prefer.

None of this is about anyone being a bad person. It is about the fact that an unstructured interview hands the decision to exactly these shortcuts, and a structured one takes it back. The structure is the bias countermeasure, and it works whether or not anyone in the room believes they needed one. (Everyone in the room believes they did not.) The rules about what may not be asked or considered are HR's, and they exist for the same reason. The shortcuts are strongest around exactly the things a fair employer does not ask about, and a rule is cheaper than a reflex.

The Interview Hour

The conversational half of the interview is an hour, and the shape below is the one I use. The four exercises are in a separate half, with a break between the two halves. Tell the candidate the shape at the start, including that both interviewers will be writing, so that the note-taking is not read as disapproval. Tell them, too, that every candidate is asked the same questions, because candidates find that reassuring and because it is true.

Keep to the order and the clock for every candidate. If a question runs long, the follow-ups are the first thing to shorten, not the later questions. A candidate who was never asked question seven cannot be scored on it. A blank on one sheet is a comparison you can no longer make. If the candidate finishes early, do not fill the time with new questions; give it to their questions about the desk. And when a candidate's answer wanders into territory on HR's list, bring it back gently and do not write it down. What is not on the sheet is not in the decision.

The Decision Meeting With Scores on the Table

The decision meeting starts with the sheets on the table and nobody saying who they want. Read the totals for each candidate, interview half and exercise half. Then look at where the two interviewers differ by two or more on a single question. Those are the places where one of you heard something the other missed or scored a feeling instead of an answer. Talk about what was said, in the words on the sheet. Change a score only when one interviewer remembers evidence the other did not record, and write beside the change why it was made. Scores that differ by one are left alone; two careful people will differ by one on most things, and the difference is noise.

Feelings are allowed in the meeting under one rule: a feeling is permitted to send you back to the evidence, and it is not permitted to overrule it. "I had a good feeling about him" is a prompt to look at his sheet and find what produced the feeling. If the sheet shows it, the scores already say so. If the sheet does not, the feeling was about the conversation. "She seemed a bit quiet" is the same prompt in the other direction. It is worth saying aloud that quiet is not a criterion on any sheet this series has produced. The habits question and the written exercise are where quiet candidates most often turn out to have been the strongest in the room. Nobody noticed because the room was measuring volume.

Record the decision and its reasons on one page: who was chosen, the totals, and the two or three criteria that separated them from the runner-up. Include the names of the people who agreed it. The record is kept as long as HR says records of this kind are kept, which varies by place. It lets you answer a candidate who asks why, or an HR colleague who asks how, without reconstructing anything. The questions, rubric and exercise pack live somewhere every interviewer can read the same version. If you run QuerySolv, the knowledge base's agents-only articles are a natural place for a desk's interview pack, beside the runbooks. A shared folder does the same job. The point is one version, dated, that nobody edits during a round.

A Short War Story: The Good Feeling at Harbourview Clinic

At Harbourview Clinic, a group of medical practices with a three-person desk, the lead, Noor, interviewed five candidates over two weeks. She had a different second interviewer each time, no fixed questions, and notes that ran to a few words per candidate. She hired the one she and the practice manager had both liked best, a technician called Felix. He turned out to be perfectly good at the job. The trouble arrived from a rejected candidate who wrote, politely, to ask why she had not been chosen and what she could do better next time. Her question went to HR when Noor could not answer it beyond "we felt another candidate was a stronger fit". HR asked for the interview notes and the scoring. There were no scores. The notes said "good fit" against Felix and "a bit quiet" against the candidate who had written. The second interviewer for her session had been a different person from Felix's. So nobody could say whether the two had even been asked the same things. The clinic's hiring was paused for a quarter while HR and Noor rebuilt the process. It cost Noor most of a week, HR considerably more, and the desk the fourth person it had been promised. The candidate who wrote received an apology and an invitation to the next round under the new process. She declined, having found a job elsewhere. Felix was a good hire. Nobody could show that he had been, and a decision you cannot explain is a decision you cannot defend, however right it was.

The Scoring Rubric

The rubric below is one sheet per interviewer per candidate. The eight questions are examples tied to the five traits plus the habits question and one future-facing question. Replace them with your own if you must, but keep one question per trait and keep the anchors written as what the answer contains. Fill in the top block once per round, before the first invitation, and show it to HR with the exercise pack. The scores are written in silence after the candidate has left. The totals go to the decision meeting beside the exercise scores.

THE SCORING RUBRIC — ONE SHEET PER INTERVIEWER PER CANDIDATE
Round: ______   Rubric version: ______   Agreed on: ________ by ____________ / ____________
Shown to HR on: ________   Frozen until the last interview of the round:  [ ]
Candidate: ______________   Date: ________   Interviewer (this sheet): ______________

Score each question 1 to 3 against the anchors. Write what was said, not what was felt.

Q1 CURIOSITY  "Tell me about a problem where the first explanation turned out to be wrong.
               How did you find out?"
   3 = describes what they checked and what changed their mind   1 = the explanation was never questioned
   Said: __________________________________________________   Score: __

Q2 CALM       "Tell me about a day when several things went wrong at once. What did you
               do first, and how did the rest of the day go?"
   3 = chose an order, said why, recovered for the next thing   1 = the day is described as a blur
   Said: __________________________________________________   Score: __

Q3 WRITING    "Describe something you wrote at work that someone else had to act on.
               What did you put first?"
   3 = reader and next step first, kept short                  1 = cannot recall writing anything acted on
   Said: __________________________________________________   Score: __

Q4 OWNERSHIP  "Tell me about a problem that was not resolved well. What was your part?"
   3 = names their own part and what they would do differently  1 = the story has no "I" in it
   Said: __________________________________________________   Score: __

Q5 I DO NOT KNOW  "Tell me about a time you were asked something you did not know.
                   What did you say, and what did you do next?"
   3 = said so plainly, then found out and followed up          1 = guessed, or cannot recall such a time
   Said: __________________________________________________   Score: __

Q6 USERS      "Tell me about someone who was hard to help. What made it hard, and what
               did you do?"
   3 = sees the person's situation; describes what they said    1 = the person is the problem
   Said: __________________________________________________   Score: __

Q7 HABITS     "What would you keep from how your last workplace handled problems, and
               what would you change?"
   3 = names both, with reasons that fit our desk               1 = would change nothing, or did not notice
   Said: __________________________________________________   Score: __

Q8 FUTURE     "A colleague has gone home and left you a ticket whose last note says only
               'waiting'. What do you do?"
   3 = reads the history, contacts the requester, writes a note  1 = leaves it until the colleague is back
   Said: __________________________________________________   Score: __

Interview total: ____/24     Exercise total (from the pack): ____/48
Notes (checked against the scores above before the meeting): _______________________
Scored alone, before conferring:  [ ]

Grace rebuilt the process over the following weekend from this sheet and the exercise pack. Rob stayed as her second interviewer, now with eight questions he had helped write and a rubric he had tested on two technicians he knew. At the next decision meeting, the candidate with the good energy and the quiet one were two columns of numbers. The numbers said the quiet one had answered the questions, which Grace had thought and could now show. Rob agreed inside ten minutes. The feeling had not gone anywhere. It had simply been asked for its evidence, and it turned out to have had some.

Frequently Asked Questions