Xblog

Some fun stats:

• Since I created the jslopez@xman.org account in Google Apps, it has received over 920,000 e-mails.
• The total size of the e-mail that has been routed to the gmail account is 3.75GB. Fortunately, I have a 25GB quota, but at this pace I can expect to exceed the quota given to normal gmail users by the end of the week!
• Meanwhile, my old mail server continues to receive some jslopez@xman.org, although the rate of delivery has tapered off significantly. At its peak I was processing on the order of 500 jslopez@xman.org e-mails per second, and now it is more like two or three per minute.
• My old mail server logs show 550,000+ e-mail delivery attempts to jslopez@xman.org. That is over and above all the e-mails sent to Google Apps.
• My logs were totally overwhelmed by the deluge of spam and so they only go back to the afternoon of the 25th… in other words this is all pretty much after I had created the Google Apps account.
• This means I’ve received roughly 1.5 million e-mails probably around 5GB in total ever since I first started publishing SPF records which made it trivial to prove that the messages were forgeries. I published the SPF records immediately after adding the MX records for Google Apps, so the nearly 1 million messages that have been sent to the Google Apps account in particular have no excuse for being there.
• I conservatively estimate another 400,000 or so rejects that were lost in my logs. I expect by the end of the day today, jslopez@xman.org will have received on the order of 2 million bounces in total, representing approximately 8 GB of bounce messages.
• Most bounce messages are terser than the original messages, so I suspect this means the total for the original messages that got bounced is measured in tens of gigabytes.
• I’d like to think most spam delivery attempts don’t result in bounces, either because they get through (otherwise, why bother?) or are rejected/swallowed without a bounce (surely some MTA’s are correctly configured). This one attack probably represents hundreds of gigabytes if not terabytes of e-mail bouncing all around the Internet.
• Had this bandwidth not been used for of spamming the Internet, the spammer could have used all this bandwidth for a good cause: like stealing a half a million songs, or torrenting a thousand movies or watching Internet porn 24/7 for a year.

It’d be fun to do some more stats, like estimating how many watts this one deluge of spam likely consumed, just so I can come up with some convoluted way of demonstrating that spammers are “with the terrorists”, but I’ll stop now, because it just makes me want to cry.

All this is making me think that small mail servers need a very efficient way to discard e-mails sent to an invalid recipient. I still haven’t made an embedded database of valid e-mails for my domain, but that is the logical next step. I need to make sure the check is done very early in my e-mail pipeline: before grey listing, before domain verifications, baysian filtering, virus checks, etc. Packages like postfix should have a setting that will allow them to automatically build a cdb database of e-mail addresses and hosted domains whenever they are presented with an LDAP/SQL backend for their datastore.

I’m also increasingly thinking I should perhaps change my e-mail config: have my VPS server just serve to filter out invalid spam, and then forward the good stuff to my server at home. It’s insane, but if spamming economics don’t change, I suspect hosting mail for even a small domain may require fairly significant computing resources and bandwidth.

I did some more tweaking, and concluded that my best moves were the following tweaks:

• Reduce the # of slave processes for the MTA to 2.
• Set up an explicit access rule for jslopez@xman.org that causes an immediate rejection and a nice little “don’t be an idiot and bounce forged return path’s” public service message.
• Get the accept queue depth as deep as possible for the slave processes.
• Reject any messages without a proper e-mail address in the FROM: envelope.

The killer solution was Google Apps for Domains though. I have registered for the service, updated my MX records, and once that information propagates through the Internets all my domain’s e-mail will get routed to Gmail, which has exactly one registered account: jslopez@xman.org. Gmail is configured to route any e-mails to an unknown address to my mail server. The net effect is that all this backscatter will get swallowed by the Gmail black hole, and everything else will remain outside the event horizon and hopefully get delivered to my mail server at something approaching the speed of light.

The other lesson learned from this is that openldap is slow, so one shouldn’t using it for accessing one’s MTA configuration. I intend to set up a cron job that will periodically dump the contents of LDAP in to files and then have postfix just read those files directly. This should prove to be infinitely more scalable and efficient, at the cost of updates being somewhat delayed.

The net effect of all this was to completely tie up my mail server and for the most part prevent any mail delivery. I’ve now tweaked the server a bit so I do eventually get mail, but it is still rather grim. So far, I’m killing connections to clients after two errors, I’ve trimmed my accept queue depth, and dramatically increased the number of simultaneous connections I will process. The overall effect has been pretty taxing on my mail server, and I still see significant delays in delivery times, so I’m all ears to any brilliant suggestions on how to address this problem.

If you are a mail admin and are wondering why your queues are backed up with tons of jslopez@xman.org e-mail, please, please kill it. I suspect thought that most of my mail is coming from bots, so I’ll probably need to start adding immediate filtering at connect time that drops suspected bots.

Is this happening to anyone else?