Testing the Mail Pipeline Before You Trust It

Monday, 08:15, at Bluewater Bank. Dana opens the queue and it looks like a good weekend: three tickets from the web form, nothing urgent. Then she notices what is not there. Monday mornings bring twenty or so e-mailed tickets from the branches, and the newest e-mailed ticket in the queue is from Friday at 15:40. She opens the tool's mail log, which she has never had a reason to open before. She reads the same line repeated every few minutes since Friday at 15:52: authentication failed. The mailbox account's password expired on Friday afternoon, under the bank's own policy. The account had been created as a person's account and nobody had told the policy otherwise. Sixty-one messages are waiting unread. One of them is from the Harbour Street branch, sent on Saturday at 09:05, about a card reader that has been down all weekend.

The mail pipeline is the chain of links between a requester pressing send and a ticket appearing in the queue, and back again. The links are the sender's mail server, the desk's mail server, the mailbox, the tool's reader, the tool, the outbound server, and the requester's inbox. Every link can fail on its own, and nearly every failure is silent, because the one thing a broken pipeline cannot do is send anyone a message about it. Testing the pipeline is sending known messages through it and checking each outcome against what should have happened. Watching it is the ongoing check that it still works, which is what Dana did not have. This article is part of our Email-to-Ticket Done Right series. It is the last one, because it is the one that tells you whether the other five are true on your desk.

It covers why a mail pipeline fails silently, with a table of the usual breaks. Then come the ten test messages, reading the results, including the clock, and watching the mailbox itself. Next come the mail that stops on a Friday and how to recover, and the canary, an alert on the desk's own inbox. It ends with when to re-test, a short war story, and the test script to copy, in that order.

Why a Mail Pipeline Fails Silently

A web form that breaks fails in front of the requester: the page errors, the button does nothing, somebody phones. A mailbox that breaks fails in front of nobody. The requester's message leaves their outbox and they think of it as sent, because from where they sit it is. The desk's queue stops growing and the desk thinks of it as a quiet day, because from where they sit it is. The mail server, if it is the thing that broke, may be sending delivery failures to requesters, but delivery failures are messages people have learned not to read. Nobody in the chain has both the symptom and the ability to act on it. The silence is not a bug in any one link. It is a property of the chain.

The usual breaks are few, and the table below lists the ones I have seen stop a desk. It shows what each looks like from the desk's side and from the requester's. Read the third column with care: in most rows the requester sees nothing unusual at all. That column is the reason a pipeline needs watching rather than trusting, and the reason the watch has to be something other than "we'd notice".

Most of these happen on a Friday afternoon, and that is not superstition. Password expiries fall at the end of the working day the policy counts to. Changes to mail servers are scheduled for Friday afternoons because fewer people are working. Certificates are renewed by people who are leaving at five. And whatever breaks at four on a Friday has sixty-four hours to be silent before anyone opens the queue expecting it to be full. The pipeline does not know what day it is. The people who change it do.

The Ten Test Messages

The test is ten messages sent through the real pipeline from two ordinary mailboxes, one inside the organisation and one outside it. The results are written on the script at the end of this article. The test takes about an hour, most of which is waiting for the tool's next read of the mailbox. Run it before the desk's address is announced, again after any change to any link in the chain, and once a quarter regardless. The threading cases, replies with the number removed, forwards, replies from other addresses, have their own longer sheet in Threading Replies to the Right Ticket. This test covers one threading case and the rest of the pipeline.

Test one: a plain message from the inside address, a one-line subject and a two-line body. Expect a ticket in the right department with the sender as requester, and an acknowledgment back in the inside mailbox with the ticket number in its subject. Test two: the same from the outside address, because the inside and outside paths differ at the mail server. A desk that only ever tests from inside has tested half the chain. Test three: reply to the acknowledgment from test two. Expect the reply on the same ticket and no second acknowledgment, which is the loop brake being shown to work. Test four: a message with an attachment comfortably under the limit, a screenshot and a small document. Expect both files on the ticket, openable. Test five: a message with an attachment just over the limit. Expect whatever your tool does, and write it down, because the answer is the single most useful line on the sheet. It could be a note on a ticket, a ticket without the file, a message left in the mailbox, or nothing.

Test six: send the desk a message from a mailbox with an out-of-office reply switched on. Expect the message to become a ticket and the out-of-office reply, which arrives a minute later, to be held and not acknowledged. Test seven: a message from an address on the block list. Expect nothing in the queue and a line in the log, if the tool keeps one. Test eight: one message to every department address the desk has, from the outside mailbox. Expect each in its own department, and each acknowledgment from that department's own sending address. Test nine: on the ticket from test one, write an internal note, then a public reply. Expect nothing to arrive for the note. Expect the reply to arrive from the desk's address with the number in the subject and no trace of the note. Test ten: resolve the ticket from test two, and reply to it from the outside address. Expect it to reopen, or whatever your tool does on a reply to a resolved ticket. Either way, record it, because the triage habits depend on it.

Ten is a round number and the real list is "whatever the pipeline does that nobody has watched it do". Add a test for anything your desk relies on that the ten do not cover. That could be a monitoring system's alert address, a supplier's desk that e-mails yours, a second mailbox that forwards into the first. Remove nothing. A test that has passed twenty times in a row is not a test that can be skipped. It is a test that has told you, twenty times, that the pipeline still works, which is the only thing a test is for.

Reading the Results: The Clock Matters Too

Each test has a pass and a time, and the time is the half people forget to write down. For tests one and two, note when the message was sent, when the ticket's opening time says it was opened, and when the acknowledgment arrived back. The gap between the first two is how long the pipeline takes. That is mostly the tool's reading schedule plus whatever the mail servers add. The gap between the second and third is how long the outbound leg takes. Write both on the sheet. They are the only honest basis for any promise about how quickly the desk sees a message. A promise shorter than the measured gap is a promise the pipeline will break on the desk's behalf. Keep the completed sheets, because a gap that was two minutes at setup and is nine this quarter is an early warning that cost nothing to collect.

Look, too, at which time the ticket carries. Some tools open the ticket at the moment the message was sent, taken from the message itself; some at the moment the tool read it. The difference is small on a normal day and large after an outage, when sixty-one messages are read at once on a Monday morning. If the tool uses the read time, every one of those tickets is opened at 09:30 on Monday. In that case, the response-time report will say the desk answered a Saturday message in an hour. That is not a report anyone should rely on. Knowing which time your tool uses is what lets you annotate it honestly rather than discover it in a review.

Watching the Mailbox Itself

The tool's mail log is the first place to look when anything is odd, and most tools have one. It is a record of each attempt to read the mailbox, whether it succeeded, how many messages it found, and any error. Find out where yours is before you need it. Dana had never opened hers, and it had been saying "authentication failed" every few minutes for sixty-four hours. That is the whole argument for the canary below. A log is a record for after the fact. It is not a watch.

The mailbox can also be looked at from the server side. This is the one exception to the rule that nobody but the tool reads the desk mailbox. Through the mail system's administration console, or through a web view that does not mark messages read, the desk can see four numbers. Those are the unread count, the age of the oldest unread message, the storage used against the quota, and the size of the processed folder if the tool uses one. An unread message older than the tool's reading interval is a message the tool has not read, and the only question is why. A quota above three-quarters is a Friday waiting to happen. Glance at the four numbers weekly. Never, under any circumstances, open the mailbox in an ordinary mail program to do it. A mail program marks things read and some tools only fetch what is unread. The rule from How a Mailbox Becomes a Queue still stands: one reader, and it is the tool. Looking is not reading. Clicking is.

The processed folder deserves its own line, because it is the quota's usual cause. Where the tool moves each read message to a folder rather than deleting it, the folder holds a copy of every message the desk has ever received. In that case, it grows until something stops it. Decide who empties it and how often, and treat the folder as what it is: a second copy that is useful for a few weeks and then a weight. The ticket is the record. The folder is a habit the mail system has.

The Mail That Stops on a Friday

When the pipeline has been silent for a weekend, the recovery has an order. First, fix the link: reset the password, start the service, empty the folder, whatever the log says. Second, before touching the queue, find out what the tool does with the backlog, because tools differ. Some fetch everything waiting in the mailbox. Some fetch only messages newer than the last successful read. The second kind leaves the weekend's sixty-one messages sitting unread forever unless someone moves them where the tool will look. Watch the mailbox's unread count fall to zero. If it does not, the backlog needs a hand, and the test script's note on what your tool does is what tells you whether to expect that.

Third, triage the backlog oldest first, with the clock in mind. The Saturday message about the card reader is now two days old whatever time the ticket says, and it goes to the top. Where the tool stamps the tickets with the read time, write one internal note on each that matters: "arrived Saturday 09:05, read Monday 09:30 after mailbox outage". That keeps the record honest and lets the report be annotated. Fourth, tell the requesters once, briefly, as a group, through whatever channel the organisation uses for notices: "E-mail to the help desk was delayed from Friday afternoon to Monday morning. Everything sent in that time has now been received and is being worked, oldest first. If you are waiting, you will hear from us today." One notice, no explanation of passwords, no promise beyond today. The shape of a short, honest notice about something that went wrong is in The First Notice. A silent mailbox is an outage of exactly that kind, even though nothing was down.

Fifth, and on the same day, remove the cause from the future. The password that expired becomes a service account's password, or an expiry date on the desk's calendar with a reminder two weeks before. The folder that filled gets an owner and a schedule. The service that did not restart gets set to start with the server and added to the monitoring list. Recovery without the fifth step is a weekend that will happen again at the next expiry, which the policy has already scheduled.

The Canary: An Alert on the Desk's Own Inbox

A canary is a message sent to the desk's address automatically every working day, at the same time, with a fixed subject. Something other than the ticketing tool sends it. The message’s job is to become a ticket. If the canary ticket exists by a set time each morning, the pipeline read the mailbox today. If it does not, the pipeline is broken, and that fact is known at nine on Monday rather than at the first phone call on Tuesday. The canary is the cheapest monitoring a desk can have. It is the only kind that tests the whole chain from a real mail server to a real ticket, because that is exactly what it does.

The sender can be anything that can send a scheduled message. It could be a scheduled task on any server that has a mail client, or a monitoring system that can send test mail. Or it could be a separate mailbox with a scheduled send. The subject is fixed, "Daily mail check", so that the ticket is easy to find and easy to exclude from reports. The requester is a dedicated test address so that reports can exclude it too. The ticket is closed by whoever does the morning sweep, or by a rule where the tool has one. It should also, from the sender's side, produce an acknowledgment back in the sending mailbox, which tests the outbound leg. A canary ticket that exists but whose acknowledgment never came back is an outbound failure found before any requester noticed that replies had stopped.

The check that the canary arrived is the part that must not depend on the thing being tested. A person looking for the ticket at a fixed time in the morning sweep is the simplest check and perfectly adequate for a small desk. "Is today's canary there?" is a ten-second question. A monitoring system can ask the tool, through an interface built for the purpose, whether a ticket with that subject was created today. That is the version that works at weekends and does not depend on someone remembering. If you run QuerySolv, the REST API and webhooks are the durable way for a monitoring system to ask that question or be told the answer. Either way, the alarm when the canary is missing must reach a person by a route that is not the desk's mailbox. The desk's mailbox is the thing that just failed. A text message, a chat message, a monitoring dashboard: anything but an e-mail to helpdesk@example.com, which would be a cry for help addressed to the patient.

Three more items go on the watch list beside the canary, and none of them is a test. The mailbox account's password expiry, if it has one, goes in the desk's calendar. The tool's service goes on whatever monitoring the organisation has, so that a server restart that did not bring it back is noticed within the hour. And the mail server's quota alert for the desk mailbox goes to a person, not to the mailbox. The relationship between monitoring systems and tickets in general, and why an alert should become a ticket rather than an e-mail nobody reads, is in Why Alerts Should Become Tickets. The canary is the one alert that cannot be allowed to arrive by the channel it is checking.

When to Re-Test

The ten tests are run again after any change to any link in the chain. The list of such changes is longer than it looks. It includes a password rotation, a change to the mail server or the provider, a tool upgrade, and a new department address. It includes a new or changed screening rule, a certificate renewal, and a new gateway or filter in front of the mailbox. It also includes any storm or outage, because the fix for one break has a habit of introducing another. Put "run the mail pipeline tests" on the record of every such change, as the step that confirms the change worked, which is where it belongs. The one-page shape of such a record is in The One-Page Change Record.

Run them, too, during the trial of any new ticketing tool, before the trial is judged. A tool whose mail pipeline has not been tested with your mail server and your gateway has not been tried. The ten messages answer the questions a demo never will. A trial that does not include them is a trial of the screenshots.

And run them quarterly when nothing has changed, because something has. The provider changed something it did not announce. The mail administrator added a rule for an unrelated reason. The gateway updated itself. Quarterly is the interval at which the test is cheap enough that nobody argues. It is frequent enough that a silent change is caught before it has been silent for a year. Put it on the calendar beside the loop test from Loops and Auto-Reply Storms, which is three of the same messages, and do both in the same hour.

A Short War Story: The Processed Folder at Thornbury Veterinary Group

Thornbury Veterinary Group ran six practices from one IT desk. The desk's ticketing tool moved every message it read into a processed folder in the desk mailbox. Nobody had been told to empty the folder and nobody did. Eighteen months after setup the mailbox reached its storage limit at some point on a Friday evening. From then on every message sent to the desk was returned to its sender with "mailbox full". The practice managers, seeing an unfamiliar sender and a familiar-looking subject, deleted the failure message as spam. For eight working days the desk received no e-mail and believed it was having a quiet fortnight. Meanwhile, a practice manager called Hannah at the Mill Lane branch sent three messages about the booking laptop's disk errors. She got three bounces she did not open, and concluded IT was ignoring her. The laptop failed on the second Thursday. The branch rebooked a day of appointments by telephone, and the branch's bookings for that day were recovered from a printout. A technician called Lewis found the full mailbox in four minutes, once Hannah telephoned, and emptied the folder in ten. The desk then set up a canary, a quota alert to Lewis's phone, and a monthly emptying of the folder with a name beside it. A mailbox that is full is a desk that has closed without telling anyone. The only thing that would have noticed on the Saturday morning was a message whose absence somebody was expecting.

The Mail Pipeline Test Script

The script below is the ten tests with room for the result and the clock. It includes the canary's details, and the list of triggers that mean it must be run again. Run it with two mailboxes and the queue open. Write the observed outcome in words rather than a tick where the expected column says "record". Keep every completed copy with the mailbox setup sheet, because the trend across quarters is worth as much as any single run. The canary block at the bottom is filled in once and checked every quarter. That is the moment to confirm that the alarm still reaches a person who still works here.

MAIL PIPELINE TEST SCRIPT — ONE PAGE
Desk: ______________   Department(s): ______________   Run by: ____________   Date: ________
Inside test address: ____________________   Outside test address: ____________________

#   Send                                             Expect                                Observed         Pass
1   Plain message from inside                        Ticket in right dept; ack with number  ____________   [ ]
      sent __:__   ticket opened __:__   ack received __:__   pipeline delay ____ min   outbound ____ min
2   Plain message from outside                       Ticket; ack with number                ____________   [ ]
      sent __:__   ticket opened __:__   ack received __:__
3   Reply to the ack from test 2                     Joins the ticket; NO second ack        ____________   [ ]
4   Message with attachments under the limit         Both files on the ticket, openable     ____________   [ ]
5   Message with an attachment just over the limit   Record exactly what happens:           ____________   [ ]
      [ ] note on ticket  [ ] ticket without file  [ ] left in mailbox  [ ] nothing
6   From a mailbox with out-of-office switched on    Message ticketed; auto-reply HELD,     ____________   [ ]
                                                     not ticketed, not acknowledged
7   From an address on the block list                Nothing in queue; line in log          ____________   [ ]
8   One message to each department address           Each in its own dept; each ack from    ____________   [ ]
                                                     that dept's address
9   On ticket 1: internal note, then public reply    Note: nothing arrives.                 ____________   [ ]
                                                     Reply: from desk address, number in
                                                     subject, no trace of the note
10  Resolve ticket 2; reply from outside             Record it (reopen / new ticket / other) ____________   [ ]

Ticket opening time is:  [ ] message send time   [ ] time the tool read it
Backlog after an outage: [ ] tool fetches everything waiting   [ ] only newer than last read
Mail log location: ______________________   Last successful read shown there: [ ]
Mailbox from the server side: unread ____   oldest unread ____   quota used ____%   processed folder ____

TEST TICKETS: closed with reason "pipeline test", requester = test address, excluded from reports [ ]

THE CANARY
  Sent by: ______________________  at __:__ each working day   Subject: "Daily mail check"
  Checked by: [ ] person at morning sweep __:__   [ ] monitoring system via the tool's interface
  Alarm if missing reaches: ______________  by: [ ] text  [ ] chat  [ ] dashboard  (never by desk mail)
  Ack returns to the sending mailbox: [ ] checked
  Canary ticket closed by: ______________

ALSO ON THE WATCH LIST
  [ ] Mailbox account password expiry: ________  (calendar reminder two weeks before)
  [ ] Tool's service monitored; starts with the server
  [ ] Mailbox quota alert goes to: ______________ (a person, not the mailbox)
  [ ] Processed folder emptied by: ______________  every ________

RUN THIS SCRIPT AGAIN AFTER
  password rotation / mail server or provider change / tool upgrade / new department address /
  screening rule change / certificate renewal / new gateway or filter / any storm or outage /
  and every quarter regardless.   Next scheduled run: ________

Signed off: ____________

Dana ran the script at Bluewater on the Tuesday, once the sixty-one were triaged and the branches had their notice. Test five produced a ticket with no file and no note, which went on the attachment policy that week. Test ten reopened the ticket, which the desk had assumed and now knew. The mailbox account became a service account with its expiry in the desk calendar. The quota alert went to Dana's phone. A scheduled task on a server in the data centre began sending "Daily mail check" at 07:30 every working day. It has arrived every morning since. It is the dullest ticket on the desk, and the only one Dana looks for before her coffee.

Frequently Asked Questions