Xblog

Some fun stats:

• Since I created the jslopez@xman.org account in Google Apps, it has received over 920,000 e-mails.
• The total size of the e-mail that has been routed to the gmail account is 3.75GB. Fortunately, I have a 25GB quota, but at this pace I can expect to exceed the quota given to normal gmail users by the end of the week!
• Meanwhile, my old mail server continues to receive some jslopez@xman.org, although the rate of delivery has tapered off significantly. At its peak I was processing on the order of 500 jslopez@xman.org e-mails per second, and now it is more like two or three per minute.
• My old mail server logs show 550,000+ e-mail delivery attempts to jslopez@xman.org. That is over and above all the e-mails sent to Google Apps.
• My logs were totally overwhelmed by the deluge of spam and so they only go back to the afternoon of the 25th… in other words this is all pretty much after I had created the Google Apps account.
• This means I’ve received roughly 1.5 million e-mails probably around 5GB in total ever since I first started publishing SPF records which made it trivial to prove that the messages were forgeries. I published the SPF records immediately after adding the MX records for Google Apps, so the nearly 1 million messages that have been sent to the Google Apps account in particular have no excuse for being there.
• I conservatively estimate another 400,000 or so rejects that were lost in my logs. I expect by the end of the day today, jslopez@xman.org will have received on the order of 2 million bounces in total, representing approximately 8 GB of bounce messages.
• Most bounce messages are terser than the original messages, so I suspect this means the total for the original messages that got bounced is measured in tens of gigabytes.
• I’d like to think most spam delivery attempts don’t result in bounces, either because they get through (otherwise, why bother?) or are rejected/swallowed without a bounce (surely some MTA’s are correctly configured). This one attack probably represents hundreds of gigabytes if not terabytes of e-mail bouncing all around the Internet.
• Had this bandwidth not been used for of spamming the Internet, the spammer could have used all this bandwidth for a good cause: like stealing a half a million songs, or torrenting a thousand movies or watching Internet porn 24/7 for a year.

It’d be fun to do some more stats, like estimating how many watts this one deluge of spam likely consumed, just so I can come up with some convoluted way of demonstrating that spammers are “with the terrorists”, but I’ll stop now, because it just makes me want to cry.

All this is making me think that small mail servers need a very efficient way to discard e-mails sent to an invalid recipient. I still haven’t made an embedded database of valid e-mails for my domain, but that is the logical next step. I need to make sure the check is done very early in my e-mail pipeline: before grey listing, before domain verifications, baysian filtering, virus checks, etc. Packages like postfix should have a setting that will allow them to automatically build a cdb database of e-mail addresses and hosted domains whenever they are presented with an LDAP/SQL backend for their datastore.

I’m also increasingly thinking I should perhaps change my e-mail config: have my VPS server just serve to filter out invalid spam, and then forward the good stuff to my server at home. It’s insane, but if spamming economics don’t change, I suspect hosting mail for even a small domain may require fairly significant computing resources and bandwidth.

Critical for this was getting AIO to work with poll(2), because the ddsnapd daemon follows the tried-and-true “poll then do something” loop that allows for efficient, scalable, and relatively simple Unix server design. Unfortunately (or fortunately, if you are familiar with POSIX ;-), Linux’s native AIO doesn’t follow the POSIX AIO spec, and instead implements it’s own event queue for notification of completion of IO operations. This event queue isn’t exposed as a file, so you can’t poll it. So, I hacked together a library that spawns a separate thread which does nothing but read in events and copy them out to a pipe, so that the main thread can poll said pipe just like any other file descriptor. Ugly? Yes. Wasteful? Yes. Easier to work with than the apparent alternatives? Yes.

I got most of the way through the process. I discovered what appears to be some kind of race condition in AIO where the vast majority of the time I was losing completion events if I submitted multiple IO requests at once. I still haven’t tracked it down, but while looking for possible sources of the problem, I discovered a heretofore unknown (well, by those of us on the project at least) syscall: eventfd(2).

eventfd does for AIO what signalfd(2) does for signals. In other words: it does the obvious thing that we wanted in the first place but were too mentally challenged to find. The moral of the story: even if you think a Linux API (AIO in this case) sucks, expect it to suck less, and question why when it doesn’t.

I did some more tweaking, and concluded that my best moves were the following tweaks:

• Reduce the # of slave processes for the MTA to 2.
• Set up an explicit access rule for jslopez@xman.org that causes an immediate rejection and a nice little “don’t be an idiot and bounce forged return path’s” public service message.
• Get the accept queue depth as deep as possible for the slave processes.
• Reject any messages without a proper e-mail address in the FROM: envelope.

The killer solution was Google Apps for Domains though. I have registered for the service, updated my MX records, and once that information propagates through the Internets all my domain’s e-mail will get routed to Gmail, which has exactly one registered account: jslopez@xman.org. Gmail is configured to route any e-mails to an unknown address to my mail server. The net effect is that all this backscatter will get swallowed by the Gmail black hole, and everything else will remain outside the event horizon and hopefully get delivered to my mail server at something approaching the speed of light.

The other lesson learned from this is that openldap is slow, so one shouldn’t using it for accessing one’s MTA configuration. I intend to set up a cron job that will periodically dump the contents of LDAP in to files and then have postfix just read those files directly. This should prove to be infinitely more scalable and efficient, at the cost of updates being somewhat delayed.

The net effect of all this was to completely tie up my mail server and for the most part prevent any mail delivery. I’ve now tweaked the server a bit so I do eventually get mail, but it is still rather grim. So far, I’m killing connections to clients after two errors, I’ve trimmed my accept queue depth, and dramatically increased the number of simultaneous connections I will process. The overall effect has been pretty taxing on my mail server, and I still see significant delays in delivery times, so I’m all ears to any brilliant suggestions on how to address this problem.

If you are a mail admin and are wondering why your queues are backed up with tons of jslopez@xman.org e-mail, please, please kill it. I suspect thought that most of my mail is coming from bots, so I’ll probably need to start adding immediate filtering at connect time that drops suspected bots.

Is this happening to anyone else?

The 127 MB download took forever (like 45 minutes despite our pricey DSL connection), and then the installer popped up a dialogue saying that it needed at least 3GB of disk space on the main Windows partition and wouldn’t install until I made room. I’m trying to figure out how a rational person could think that a 127MB download might need 3GB of disk space to complete (yeah, I can see contrived situations where it’d be possible, but come on!). I dutifully cleared away like 15GB of downloads that were just sitting around on my drive, ran the install, dutifully rebooted the system as one would expect from Windows, and then confirmed that the entire install had consumed ~800MB of disk space (I’m guessing a 2:1 compression ratio, binary diffs, plus backing up the old files for rollback pretty much explains all of that).

You know, it is sad enough watching those progress bars behave like meandering drunks stumbling to the next bar during the later hours of a pub crawl, but really, is it so hard to have a reasonably accurate estimate of how much disk space is needed for an install?

Upon reflection, I don’t think this was a particularly amazing or eloquent speech. In a lot of ways, it was very calculated and a repetition of campaign speeches going back as much as a year (and in some ways even echoing Obama’s 2004 Convention speech). It still brought a tear to my eye. Why? Because this is the kind of response I’ve been hungering for in reaction to the usual “gotcha journalism”, holier than though punditry, politics of division, reducing lifetimes to one unfortunate sound bite or image, focus on the horse race not the challenges, political ADD, and dammit-we-won’t-stop-this-insanity-until-someone’s-career-is-over, campaign against some talking head rather than your opponent, yellow journalism masquerading as political correctness, idiocy that has plagued us for so long, and has basically owned the 2008 presidential campaign process almost before it started.

Seriously, I’d only have been prouder if he’d just walked up to the podium with a sign behind him saying something along the lines of, “A Message For Those Concerned By Ferraro, Wright, etc.”, waited for silence, and then with both hands, emphatically performed the Trudeau salute, then silently stepped down from the podium. Unfortunately, that path leads to the PMO in Ottawa rather than 1600 Pennsylvania Avenue in D.C.

…plus, this was a bit more uplifting.