Loops and Auto-Reply Storms: Detection and the Kill Switch

Monday, 06:50, at Bellweather Logistics. Tomas is in early to clear the weekend and the queue says there are 3,180 new tickets. They are all from the same address, dispatch-noreply@example.com, which belongs to a customer's warehouse system. Each has the subject "Automatic reply: your message has been received" and a body that says the mailbox is not monitored. A quoted copy of the desk's own acknowledgment follows, then a quoted copy of the previous automatic reply, and so on down, like a hallway of mirrors. The first one arrived on Friday at 17:24. The most recent arrived forty seconds ago, while Tomas was reading. He scrolls to the bottom of the list, which takes a while, and finds the message that started it. A planner called Sofia had e-mailed the customer's dispatch address about a failed data feed and copied the desk "so IT knows". IT knows.

A mail loop is two automatic senders answering each other, each message triggering the next, with no person anywhere in the chain to notice and stop. An auto-reply storm is what a loop produces: hundreds or thousands of messages in a few hours, each one becoming a ticket, until something runs out. That could be a mailbox's storage, a mail server's sending limit, or the patience of whoever opens the queue on Monday. The help desk's own acknowledgment, the "we have received your request" that every e-mail-to-ticket setup sends, is one half of nearly every loop. That is why this article exists. This article is part of our Email-to-Ticket Done Right series. It is about the one failure in the mail pipeline that gets worse every minute you do not look at it.

It covers how a loop starts, then the three usual partners, with a table. Next come the conventions that should stop loops and sometimes do, and what loop detection actually does. The kill switch follows, which is a decision to make before you need it. Then come the morning after, preventing the next one, a short war story, and the loop prevention checklist to copy, in that order.

How a Loop Starts

A loop needs two things that reply to mail without reading it, and a help desk's mailbox is one of them by design. The acknowledgment is an automatic message. A new message arrives, and the tool creates a ticket. The tool sends a reply saying so, to whoever the message came from, without anyone deciding to. That is the right behaviour for a person's request. It is the wrong behaviour for a message that came from a machine, because the machine may reply to the acknowledgment the same way. The tool, seeing a new message from the machine, may create another ticket and acknowledge that. Each side is doing exactly what it was configured to do. Neither has any way to notice that it has done it four hundred times.

The trigger is almost always a small human act. Somebody copies the desk on a message to a machine, as Sofia did. Somebody gives the desk's address to a system as its contact address, "so the alerts go into tickets". Somebody sets an out-of-office reply and then e-mails the desk. Somebody adds the desk's address to a distribution list that the desk also sends to. None of these is a mistake anyone would recognise as one. The first few hundred messages happen before any person is in a position to recognise anything. Loops do not need a weekend. They prefer one.

The shape of the storm depends on whether the partner's replies thread. If the partner answers with a fixed subject of its own, as the dispatch system did, every round is a message with no ticket number in it. In that case, every round becomes a new ticket: that is the 3,180. If the partner replies properly, keeping the desk's subject with the number in it, every round threads onto one ticket. In that case, the storm only continues if the desk sends something in response to a reply. A tool that acknowledges only new tickets, never replies, breaks that version of the loop on the second message. That one setting, acknowledgment on new tickets only, is the cheapest loop brake there is. It is worth checking it is set before reading the rest of this article.

The Three Usual Partners

Three kinds of automatic sender account for nearly every storm I have cleaned up, and a fourth is the desk talking to itself. The out-of-office reply is the mildest, because most mail systems send it once per sender per absence. The desk acknowledges it, but the absent person's mailbox has already replied to this sender today. So the loop dies at two messages. Some systems send it every time, and some people set up their own rule that replies to everything. Then it does not die at all. Another desk's ticketing tool is the worst partner, because it is a mirror. It acknowledges every new message, and so does yours. Neither side has any reason to stop. The bounce is the sneaky one. The desk acknowledges a message from an address that does not exist, and the mail server returns a delivery failure. The tool treats the failure as a new message and acknowledges it, and the server returns another failure. Most servers send bounces from an empty return address precisely so that nobody replies to them; most tools honour that. The exceptions find each other.

The self-loop in the fourth row deserves a sentence, because it is the one people do not believe until they have seen it. Take a forwarding rule on the desk mailbox that sends a copy to a team list, where the list includes the desk's address. That is a system that e-mails itself, acknowledges itself, and e-mails itself about the acknowledgment. It takes about an hour to fill a mailbox. I set one up once, in the interests of "keeping the team informed", and the team was informed roughly nine hundred times before lunch.

The Conventions That Should Save You

Mail has had rules for this for a long time, and they work when both sides follow them. An automatic message is supposed to carry a header saying it was generated automatically, so that other automatic systems can see it and keep quiet. Bounces are supposed to come from an empty return address, so that nothing replies to them. Many systems also put a recognisable word at the front of the subject: "Automatic reply" or "Out of office" or "Undeliverable". They name their sender something like noreply or mailer-daemon or postmaster. A well-behaved system does two things with all this. It marks its own automatic messages so others can recognise them, and it refuses to reply to anything it recognises as automatic. A help desk tool is a system like any other, and it should do both.

Check that yours does, in both directions. Send the desk a message from a mailbox with an out-of-office reply switched on. Confirm that the out-of-office reply did not become a ticket and was not acknowledged. Then look at the acknowledgment the desk sends and ask whether a partner system could recognise it as automatic. Does it carry the header? Does the subject say what it is? Does it come from an address that is clearly the desk's? The honest answer for some tools is "partly". The honest response is to find out which part and lean on detection for the rest. Conventions are what the other system is supposed to do. Detection is what you do when it does not.

What Loop Detection Does

Loop detection is the set of checks a tool runs on an incoming message to decide whether it came from a machine and, if so, to refuse to answer it. The first check is recognition: the automatic-message header, the subject words, the sender names, and the empty return address. Any of these marks the message as automatic. The second is behaviour. A message that quotes the desk's own acknowledgment is a reply to a machine by a machine, and some tools spot that. The third is volume. More than a handful of messages from one sender in a few minutes is not a person typing, and the tool can hold everything after the first few. The fourth is memory. A tool that has already acknowledged a sender once in a short window does not do it again. That breaks a loop at round two even if nothing else caught it. Tools differ in how many of the four they have. Some have all of them, some have one, and some have a tick-box called "loop protection" whose behaviour nobody at the desk can describe. Find out which checks yours has by testing, not by reading the box.

What the tool does with a message it has recognised matters as much as recognising it. The best outcome is that the message is held. It does not become a ticket, and it is not acknowledged. It sits in a review queue where a person can see it and release it if the detection was wrong. The next best is that it becomes a ticket but is not acknowledged, which stops the loop at the cost of a queue full of out-of-office notices. The worst is that it is deleted unseen, because the day the detection is wrong about a real person's message, nobody will ever know. Where a tool holds screened mail for a person to review, as QuerySolv does, a storm that detection catches is a storm in a holding pen. In that case, there are a few hundred messages to discard on Monday. There are no tickets, no acknowledgments, and no partner system receiving anything to reply to. The review queue is the subject of Spam, Junk and the Review Queue. Here it is enough to say that "hold" is the right answer to "what should detection do".

Detection has a false-positive problem, and it is a real one. Take a requester whose signature contains the words "automatic reply" because they once pasted a template. Or take a supplier whose address is literally noreply@ because their mail system was set up that way. Or take a customer replying from a phone whose subject still says "Out of office" from a thread two weeks ago. Each is a person, and detection will call each a machine. That is why hold beats block, and why a person reads the held queue daily. The detection is protecting the desk from a storm. The person is protecting the requester from the detection.

The Kill Switch

The kill switch is the short list of steps that stop a storm in progress. It is written down before the storm, with the names of the people allowed to pull it. It has four steps, and the order is the whole point. First, stop the tool reading the mailbox: pause the department's mailbox in the tool, or stop the service that reads it, whichever your tool offers. Second, switch off the acknowledgment for that department, so that when reading resumes nothing goes back out. Third, block the partner's address, in the tool's screening rules or at the mail server, so that what is still arriving is held rather than processed. Fourth, go into the mailbox and move everything from the partner into a folder the tool does not read. That way, when reading resumes it does not process the four thousand messages that arrived while you were doing steps two and three. Then resume reading, and watch for five minutes.

The order matters because each step alone leaves a gap. Block the sender first and the messages already sitting in the mailbox are still processed and acknowledged. In that case, the storm continues from the mailbox for an hour after the block. Switch off the acknowledgment first and tickets are still being created, hundreds of them, from the backlog. Stop reading first, and everything else can be done calmly. That is because the desk's half of the loop is off and the partner's half is now shouting into a mailbox that nobody answers. The partner will keep sending for a while. That is the partner's problem, and the e-mail to their administrator can wait until the queue is clean.

Decide now who may pull the switch, and make the answer "anyone on the desk, at any hour, without asking". A storm at two in the morning on a Saturday cannot wait for a lead to approve stopping the mailbox. The cost of a wrong pull is a few minutes of delayed tickets, while the cost of a right one not pulled is the whole weekend. Write the four steps as a runbook, with where each setting lives in your tool and the name of the service. That way, the person on call can run it half awake. The shape of such a page is in Writing the Runbook. And practise it once, on a quiet afternoon, with a test mailbox. The first time anyone pulls a kill switch should not also be the first time anyone finds out where it is.

The Morning After

Once the loop is stopped, the queue needs cleaning, and the first job is to find everything the storm made without touching anything it did not. Filter the queue by the partner's address and the time window, from the first storm message to the last, and read the count. Then look inside that filter for the ones that are not storm. Look for the real request that arrived from the same address before the loop, Sofia's original message in Bellweather's case. Look for any ticket the partner's address genuinely raised. Then look outside that filter for the real tickets that arrived during the storm from everyone else. They are in the queue too, buried between page forty and page ninety, and they have been waiting all weekend. Those get triaged first, before any cleanup, because their requesters are people.

Then the storm tickets are closed in bulk, and how depends on the tool. Some tools can delete tickets; some cannot, and never will, because the ticket history is treated as a record. Where deletion is not available, close them with a status or a note that says "mail loop", so that every report can exclude them. In that case, accept that the ticket numbers are used. Volume for the month is now wrong by three thousand. The first-response figure shows three thousand breaches. The busiest-day chart has a spike that will be asked about in every review for a year. Annotate the report once, in plain words, and move on. A report that says "3,180 tickets excluded: automatic-reply loop, Friday to Monday, see ticket 5201" has told the truth. A report that quietly deletes the weekend has told a story.

Three more jobs finish the morning. Write to the partner organisation's administrator, with the facts and without blame, because they have the same storm in their system and may not know yet. A loop has two halves and both organisations will want to know what to change. Add the partner's address to the desk's hold rules, so that if their system wakes up again its messages wait for a person. And find the trigger, Sofia's copy in this case, and remove or redirect it. The customer's dispatch address is now in the desk's screening rules. Sofia has been asked, kindly, to tell IT about data feeds by raising a ticket rather than by copying the desk on mail to a machine. She will do that. The next person will not, which is why the rules exist.

Preventing the Next One

Prevention is a short list of settings and one address policy. The acknowledgment goes out once per new ticket, never in response to a reply. It never goes to a message recognised as automatic, and never to an address whose name says it is unattended. The desk's own acknowledgment is marked as automatic, so that a well-behaved partner keeps quiet. Detection is switched on and set to hold rather than block or delete. And the department's acknowledgment can be switched off in one place, by anyone, which is the kill switch's second step rehearsed in advance.

The address policy is that the desk's address belongs to people. It goes on no distribution list, because a list the desk also sends to is a self-loop waiting for a forwarding rule. It is no system's contact address, because a system that receives an acknowledgment will reply to it or bounce it. Monitoring and alerting systems in particular should reach the desk through an interface designed for machines rather than through the mailbox. Where they must use mail, they get their own address in a department that never acknowledges anything. The volume problem that follows is the subject of Avoiding Alert Ticket Floods. Suppliers with their own desks get an agreement. It says: your tickets about us come to a named person, and ours about you go to your desk. Neither side copies the other's desk address on anything.

Then test it, quarterly, in five minutes. Send the desk a message from a mailbox with an out-of-office reply switched on and confirm the out-of-office was held, not ticketed, not acknowledged. Send the desk a message with the automatic header set, if you have a way to do that, and confirm the same. Send a reply to an existing ticket and confirm no acknowledgment came back. Three messages, three checks, and the pipeline's loop brakes have been shown to work this quarter rather than assumed to work since setup. The same three are on the pipeline test script in Testing the Mail Pipeline Before You Trust It. The quarterly version is just those three, on their own, on a calendar reminder.

A Short War Story: The Bank Holiday at Castlemill Brewery

Castlemill Brewery's IT desk and the support desk of its packaging-line supplier both ran e-mail-to-ticket. Both acknowledged every new message. Neither marked its acknowledgments as automatic or checked incoming mail for the marker. On the Friday evening before a long weekend, a Castlemill technician called Ines e-mailed the supplier's desk about a labeller fault. She sent it from the ticket so that the reply would thread. The supplier's tool created its own ticket and acknowledged her. Castlemill's tool treated the supplier's acknowledgment as a new message and acknowledged that. The two desks spent three days thanking each other for getting in touch. By Saturday night Castlemill's desk mailbox had reached its storage limit. Every real message sent to IT for the next thirty-six hours bounced back to its sender with "mailbox full". That included the duty engineer's report that the cold-store sensor gateway had gone offline. He took the bounce to mean that IT's e-mail was down and did not chase. Ines and a colleague called Rob spent the whole of Tuesday closing 9,400 tickets and annotating a month's reports. They also spent that day explaining to the supplier's administrator why their desk had 9,400 tickets too. The gateway was fixed on Wednesday, two days after it could have been. The kill switch had existed, in the sense that stopping the mailbox was possible. But the only account that could do it belonged to someone on holiday. Either desk honouring the automatic marker would have stopped the loop at the second message. A kill switch nobody on duty can reach is a description of a switch, not a switch.

The Loop Prevention Checklist

The checklist below is in three parts. First are the settings and address rules to put in place before anything happens. Next is the kill switch card to fill in with your tool's own names and keep where the on-call person can find it. Then comes the morning-after list for when a storm gets through anyway. Fill in the blanks for each department that has a mailbox. The kill switch card in particular must name the real places in your tool. That is because "pause the mailbox" is an instruction only to someone who already knows where that is. Review the whole sheet quarterly, when you run the three-message test.

LOOP PREVENTION — ONE PAGE PER DEPARTMENT
Department: ______________   Mailbox: ______________@example.com   Reviewed: ________

BEFORE: SETTINGS
  [ ] Acknowledgment sent for new tickets only; never for replies
  [ ] Acknowledgment not sent to messages recognised as automatic
  [ ] Acknowledgment not sent to unattended addresses (noreply, mailer-daemon, postmaster)
  [ ] Acknowledgment marked as automatic in its own headers (if the tool allows)
  [ ] Loop or automatic-mail detection switched on; outcome is HOLD, not block or delete
  [ ] Held mail is read daily by: ______________
  [ ] Rate limit or repeat-sender limit, if the tool has one: set to ________

BEFORE: ADDRESSES
  [ ] Desk address is on no distribution list
  [ ] Desk address is no system's contact or alert address
  [ ] Monitoring and alerts arrive by interface, or to a department with acknowledgments off
  [ ] No forwarding rule on the desk mailbox sends to anything that can send back
  [ ] Supplier desks that e-mail us: ________________  Agreement on direction: [ ]

THE KILL SWITCH CARD (keep with the on-call notes)
  Who may pull it: anyone on the desk, at any hour, without asking
  1. Stop the tool reading the mailbox
     Where: ___________________________________  Account needed: ______________
  2. Switch off the acknowledgment for this department
     Where: ___________________________________
  3. Block the partner address
     In the tool: ______________________   or at the mail server: ______________________
  4. Move the storm out of the mailbox into a folder the tool does not read
     Mailbox access: ______________________   Folder name: ______________
  5. Resume reading; watch for five minutes; confirm no new storm tickets
  Practised on: ________  by: ______________

AFTER: THE MORNING AFTER
  [ ] Storm tickets identified by sender and time window: count ______
  [ ] Real tickets from the same address found and kept: ______
  [ ] Real tickets from everyone else during the storm triaged first
  [ ] Storm tickets closed in bulk with reason "mail loop" (or deleted, if the tool allows)
  [ ] Reports for the period annotated; service-level figures excluded
  [ ] Partner administrator told, with the facts
  [ ] Partner address added to hold rules
  [ ] Trigger found and removed: ______________________________________
  [ ] Mailbox storage and processed folder checked
  [ ] One-page write-up filed; this sheet updated

QUARTERLY TEST (five minutes)
  [ ] Out-of-office reply to the desk is held, not ticketed, not acknowledged
  [ ] Message with the automatic header is held, not acknowledged
  [ ] Reply to an existing ticket gets no acknowledgment

Tomas pulled the switch at 06:58, in the order above. He had found the steps in the on-call notes where a previous lead had written them after a smaller storm. By 08:30 the 3,180 were closed with a reason. The eleven real tickets that had arrived over the weekend were triaged and owned. Sofia's original message about the data feed had become ticket 5201 with the planner herself as requester. The customer's administrator had a short, friendly message explaining what their dispatch system had been doing since Friday. The month's report carries one footnote. Tomas wrote it before anyone asked.

Frequently Asked Questions