Invasion of the Email Snatchers

How Email Harvesters Operate and How to Spot Them on Your Site

When a new visitor lands on a web page, most browsers quickly hand off the request to the server, which then sends back the requested files. Search‑engine crawlers follow the same path, but they do so with a very clear purpose: to index content for future search results. Email harvesters are a subset of these crawlers that have a darker motive. They roam the internet looking for any publicly exposed e‑mail addresses, then collect them and sell or redistribute the data. Their presence on your site can result in a sudden spike in spam, or worse, a breach of privacy for your users. Understanding how they work is the first step to defending your site.

Harvesters usually begin by sending a request for the homepage of your domain. If the site contains any visible e‑mail addresses - be it in the footer, a contact form, a comment section, or even in a JavaScript variable - they will capture that information. The process repeats: the harvester follows every hyperlink it finds, drilling deeper into nested directories and database‑driven pages. Because most websites have many internal links, a harvester can traverse an entire site, gathering data from guestbooks, message boards, forums, and database exports that have been inadvertently made public. The sheer volume of addresses collected in a short time can be overwhelming. The harvested data is often sold on the black market or used by spammers to launch targeted campaigns.

Because the goal of a harvester is to accumulate as many addresses as possible, it does not care about whether the visitor is human or robot. The only thing that sets a harvesting bot apart from a regular crawler is its user‑agent string, the text that identifies itself in the HTTP header. Some of the most common user‑agents include “EmailSiphon,” “Crescent Internet Tool Pack v1.0,” “Cherry Picker,” “Email Collector,” and “Libwww-perl 1.0.” If you’ve ever seen a page on your server logs filled with requests from these agents, you’ve just witnessed a harvester in action.

Checking your logs is the most reliable way to confirm whether your site has been targeted. If your hosting provider offers a web‑stats dashboard, look for any of the user‑agent strings mentioned above in the “Browser” or “Agent” reports. If you have access to the raw access logs, you can download the file and open it in a text editor that supports searching - most modern code editors do this. A simple search for “EmailSiphon” or “Crescent” will reveal the number of times the bot accessed your pages. If you find that a particular bot accessed a large number of pages, it is almost certain that it was harvesting e‑mail addresses.

Even if you don’t have a full‑featured stats system, you can still spot harvesters manually. Look for patterns that differ from typical human browsing: a user‑agent string that contains the word “bot” or “spider,” requests that come from a single IP address in rapid succession, or access that occurs at odd hours. Harvesters often hit hundreds of pages in a matter of minutes, whereas a normal visitor will only traverse a few links during a session. By flagging these patterns, you can create a list of suspect IP addresses and block them at the firewall level.

It is also worth noting that some harvesters disguise themselves by using legitimate browser user‑agents. In those cases, the only reliable indicator is the volume and frequency of the requests. A single user’s navigation pattern rarely looks like a bot’s; if you see dozens of requests from the same IP address in a short window, the likelihood that it’s a bot increases.

When you know that a harvester has visited your site, the next step is to take action. While detection itself is straightforward, prevention can be tricky because the technology and tactics used by harvesters evolve constantly. In the following section we will explore the practical defenses you can deploy, ranging from simple file placements to server‑side rule sets that actively block or mislead these unwanted visitors.

Protecting Your Site from Email Snatchers: Practical Measures You Can Implement

Defending against email harvesters requires a multi‑layered approach. No single tactic will guarantee 100 % protection, but combining several strategies will significantly reduce the likelihood that your site becomes a data source for spammers. Below we outline a series of actionable steps, from setting up a robots.txt file to customizing .htaccess rules, that you can implement to keep unwanted bots at bay.

The first and most basic measure is to create a robots.txt file in the root of your web directory. This file is part of the Robots Exclusion Protocol and informs well‑behaved crawlers which areas of your site they may index. For example, you can place the following lines in your robots.txt to block the most common harvester user‑agents:

Although most search engines respect this protocol, harvester bots often ignore it because they have no incentive to follow a polite rule set. Therefore, while robots.txt offers a layer of deterrence, it is not foolproof.

Another tactic is to deploy a “honeypot” script that actively misleads harvesters. This approach involves creating a page that contains a large list of bogus e‑mail addresses, such as random strings followed by “@example.com.” When a harvester accesses this page, it will harvest thousands of fake addresses. The resulting bounces and spam complaints may make the bot’s operator realize that the target site is not worth the effort, prompting them to move on. One downside of this method is that it still collects the legitimate addresses that are present on the page, so you should use it only on pages that are not critical for user contact. You can find sample scripts at reliable CGI resources sites; just be sure to read any legal disclaimer associated with the code before use.

For smaller sites that host only a few e‑mail addresses, you can use a script that hides your real addresses from bots while leaving them visible to human visitors. Common techniques include wrapping the address in JavaScript that only executes in browsers that support it, or breaking the address into pieces and concatenating them on the client side. While these methods are easy to implement, they can still be bypassed by advanced harvesters that parse the page source directly. Use them as part of a layered defense rather than as a standalone solution.

A more aggressive, server‑side defense involves using the .htaccess file to block requests from known harvester user‑agents. The following snippet illustrates how to deny access to several notorious bots. Add it to your .htaccess file in the root directory of your site:

The rule redirects any request that matches one of the listed user‑agent strings to a harmless page called badspammer.html. You can create that page with a simple message such as “Access denied.” This technique forces the bot to terminate its request early, saving your bandwidth and reducing the chances it will continue to harvest your data. If your hosting provider does not allow direct edits to .htaccess, reach out to them and ask whether they can add these rules for you; many providers are happy to help customers protect their sites.

Beyond blocking specific bots, you can also limit the number of requests a single IP address can make in a given period. Most web servers support a “mod_qos” or “mod_evasive” module that throttles traffic. By setting a threshold - say, no more than 10 requests per minute - you can automatically drop traffic from aggressive bots without affecting normal visitors. This approach is particularly useful when dealing with bots that disguise themselves using legitimate user‑agents; the volume of requests remains a reliable indicator.

Another useful practice is to periodically scan your public pages for exposed e‑mail addresses. Tools such as EmailHunter can crawl a website and report any addresses that could be harvested. If you find any unprotected contact forms or user‑generated content that accidentally displays e‑mail addresses, re‑format them to use a form or a JavaScript obfuscator. Reducing the number of visible addresses is the most effective way to blunt a harvester’s payoff.

Finally, staying informed about new harvesting techniques and tools is essential. Communities such as the