Triple Threat

What Went Wrong: A Massive Newsletter Crash

Imagine a quiet day at a company’s email server. The Mitel Small‑Business Enterprise (SME) server, a familiar workhorse for many small‑to‑mid‑size businesses, is humming along, handling routine traffic and sending out status updates. Suddenly, a single message arrives that is as bulky as a brick: a 34‑megabyte newsletter from newsletter@sillyplace.com. The server accepts the mail, because the qmail subsystem - though not designed for such large bursts - has no built‑in restriction on attachment size. The sender, perhaps an automated system that never anticipated a packet that big, believes the mail was delivered and schedules a resend every twenty minutes in case it was lost. This harmless‑looking loop becomes the first domino in a cascade of problems.

The SME server itself has enough disk space and bandwidth to absorb the extra load, so initially there is no visible impact on the server’s performance. The trouble begins on the client side. The recipient, using Microsoft Outlook Express on a Windows machine, starts to choke as the program attempts to download a new 34‑MB attachment every twenty minutes. Outlook Express, known for its limited memory handling, quickly becomes sluggish and eventually crashes. The user’s inbox fills with duplicate copies, and every new delivery stalls the next, creating a feedback loop that feels like a digital avalanche.

At the same time, the user has configured forwarding from the SME address to a home email account. The home ISP has its own limits, often capping inbound message size at around 10–20 MB. When the 34‑MB newsletters hit the home server, the ISP rejects them with a bounce message. According to the SMTP standard, that bounce is returned to the original SME address, which then attempts to resend the same oversized message. The SME server, obeying its own rules, forwards again, and the cycle repeats. Each bounce adds another copy of the monstrous newsletter to the SME queue, and the system now carries a growing stack of identical messages all vying for the same outbound bandwidth.

This chain reaction - auto‑resend, heavy outbound traffic, repeated bounces, and a sluggish client - creates a death spiral that threatens the stability of the SME server itself. The server’s network interface saturates, outgoing connections pile up, and the internal qmail queue fills with thousands of identical entries. Administrators begin to notice a drop in throughput for other legitimate mail, and the once reliable server is now a bottleneck. The only way out of this mire is to break the loop at its source and clear the queue before the server becomes overwhelmed. The story illustrates a classic case of a single misconfigured newsletter causing a cascade of failures across a small business’s email ecosystem.

Halting the Mail Loop: Blocking the Sender

The first step to restoring order is to stop the server from accepting new copies of the offending newsletter. The Mitel SME architecture uses a front‑end SMTP daemon called smtpfront‑qmail, which sits in front of the underlying qmail delivery system. While the qmail subsystem handles the heavy lifting of delivering messages, smtpfront‑qmail is responsible for the initial connection from the Internet. Because smtpfront‑qmail respects qmail’s control files, we can leverage that mechanism to block the sender without touching the deeper layers of the mail stack.

The control file that lists prohibited senders lives at /var/qmail/control/badmailfrom. By adding the offending address newsletter@sillyplace.com to this file - ensuring the address is in all lower‑case to match smtpfront‑qmail’s matching rules - we tell the front‑end to reject any subsequent connection attempts from that address. Before making the change, it is essential to stop smtpfront‑qmail so that the new file is read on startup. In the SME environment, services are managed by the svc command from the daemontools toolkit, which replaces the traditional inittab for service control. To stop smtpfront‑qmail, run either svc -d /service/smtpfront-qmail or the init script /etc/rc.d/init.d/smtpfront-qmail stop. Both commands perform the same function: they send a SIGTERM to the service and wait for it to exit cleanly.

Once the service is stopped, edit the badmailfrom file using a text editor such as vi or nano, add the line newsletter@sillyplace.com, and save the file. Restart the service with svc -u /service/smtpfront-qmail or /etc/rc.d/init.d/smtpfront-qmail start. As soon as smtpfront‑qmail reloads, any connection from newsletter@sillyplace.com is rejected with a 554 SMTP error. The sender’s mail server receives the bounce and ceases to attempt retransmissions. This action cuts off the incoming flood at the very first point of contact, preventing further copies from reaching the SME queue.

With the sender blocked, the twenty‑minute resend cycle stops almost instantly. However, the SME server already has a backlog of large messages waiting to be forwarded or delivered. If left unattended, those queued items will continue to bloat the system. Therefore, the next phase involves purging the queue of the accumulated newsletters, freeing disk space and restoring normal mail flow. The procedure is a bit more involved because qmail’s queue is distributed across many subdirectories, but the process is straightforward once the service is stopped. By following this two‑step approach - blocking the sender and clearing the queue - you eliminate both the source of new problems and the existing pile‑up that threatens system performance.

Removing the Accumulated Mail from the Queue

After stopping smtpfront‑qmail, the next task is to tackle the qmail queue. The queue resides under /var/qmail/queue, split into subfolders for temporary data (/var/qmail/queue/tmp), the main message store (/var/qmail/queue/mess), and other auxiliary files. Each queued message occupies a unique identifier directory inside /var/qmail/queue/mess, often nested within numbered subdirectories to keep the filesystem from becoming overloaded. For example, a message might be stored at /var/qmail/queue/mess/21/1713843, where “1713843” is the queue ID. The actual email data, including headers, is in the directory named after that ID.

Because the SME’s qmail implementation is driven by the svc command, we stop the core mail delivery process with svc -d /service/qmail. Once qmail-send has been halted, the queue becomes editable. The next step is to locate all messages that match the newsletter pattern. A quick way to do this is to search the queue ID directories for the string “Newsletter” in any of the header files. Using a combination of find, grep, and a temporary list file, you can compile the full paths of each offending message. For instance, running

cd /var/qmail/queue/mess

will generate a file /tmp/these containing all file paths that contain the keyword. Once you have that list, you can iterate over each entry and remove the corresponding message directory. A safe loop might look like this:

cd /var/qmail/queue

Be careful with rm -rf; it will delete the entire message tree, so double‑check that the IDs in /tmp/these are indeed the ones you want to remove. After purging the queue, restart the qmail service with svc -u /service/qmail and then bring smtpfront‑qmail back up with svc -u /service/smtpfront-qmail. At this point the SME queue should be empty of the offending newsletters, and the server’s disk usage will drop accordingly.

You may also want to verify that no stray temporary files remain in /var/qmail/queue/tmp. Occasionally, aborted deliveries leave orphaned files that can clog the queue. A quick clean‑up of that directory, with a command such as find /var/qmail/queue/tmp -type f -delete, will ensure the system is fully refreshed. By systematically stopping the delivery process, locating the offending IDs, and removing them, you eliminate the backlog that was consuming bandwidth and disk space, restoring the SME’s ability to handle legitimate mail normally.

Confirming the Fix and Final Cleanup

With the sender blocked and the queue cleared, the final step is to confirm that the system behaves as expected. Inspecting the smtpfront‑qmail log file gives you immediate feedback on whether new newsletters are still attempting to arrive. The current log is usually located at /var/log/smtpfront-qmail/current. Using the tai64nlocal utility to translate timestamps into human‑readable format, run:

tail -n 100 /var/log/smtpfront-qmail/current | tai64nlocal

This command shows the last 100 entries with timestamps in UTC+0, making it easy to spot any recent 554 rejection codes or unexpected 250 success codes for newsletter@sillyplace.com. If you see only the expected rejection messages and no successful delivery attempts, the block is working.

Next, verify that the SME queue is empty. Run

ls -R /var/qmail/queue/mess | wc -l

or, more simply,

find /var/qmail/queue/mess -type d | wc -l

Both commands should return a number that matches the count of root directories (typically 0 or just the base subfolders). If the queue is clear, you can safely delete the badmailfrom file or leave it as a permanent guard against future abuse. Finally, restart smtpfront‑qmail one more time to ensure the changes are fully in effect:

svc -u /service/smtpfront-qmail

At this point, the SME server no longer accepts the oversized newsletter, the backlog has been purged, and the recipient’s Outlook Express should resume normal operation. The system’s bandwidth is freed, and legitimate mail can flow without interruption. This case study illustrates the importance of monitoring queue sizes, setting proper size limits on incoming mail, and knowing how to intervene quickly when a single misbehaving source overwhelms a small‑business mail environment. The steps above - blocking the sender, stopping services, clearing the queue, and verifying logs - provide a practical playbook that can be replicated on any SME or qmail‑based server encountering similar problems.