Xblog

Some fun stats:

• Since I created the jslopez@xman.org account in Google Apps, it has received over 920,000 e-mails.
• The total size of the e-mail that has been routed to the gmail account is 3.75GB. Fortunately, I have a 25GB quota, but at this pace I can expect to exceed the quota given to normal gmail users by the end of the week!
• Meanwhile, my old mail server continues to receive some jslopez@xman.org, although the rate of delivery has tapered off significantly. At its peak I was processing on the order of 500 jslopez@xman.org e-mails per second, and now it is more like two or three per minute.
• My old mail server logs show 550,000+ e-mail delivery attempts to jslopez@xman.org. That is over and above all the e-mails sent to Google Apps.
• My logs were totally overwhelmed by the deluge of spam and so they only go back to the afternoon of the 25th… in other words this is all pretty much after I had created the Google Apps account.
• This means I’ve received roughly 1.5 million e-mails probably around 5GB in total ever since I first started publishing SPF records which made it trivial to prove that the messages were forgeries. I published the SPF records immediately after adding the MX records for Google Apps, so the nearly 1 million messages that have been sent to the Google Apps account in particular have no excuse for being there.
• I conservatively estimate another 400,000 or so rejects that were lost in my logs. I expect by the end of the day today, jslopez@xman.org will have received on the order of 2 million bounces in total, representing approximately 8 GB of bounce messages.
• Most bounce messages are terser than the original messages, so I suspect this means the total for the original messages that got bounced is measured in tens of gigabytes.
• I’d like to think most spam delivery attempts don’t result in bounces, either because they get through (otherwise, why bother?) or are rejected/swallowed without a bounce (surely some MTA’s are correctly configured). This one attack probably represents hundreds of gigabytes if not terabytes of e-mail bouncing all around the Internet.
• Had this bandwidth not been used for of spamming the Internet, the spammer could have used all this bandwidth for a good cause: like stealing a half a million songs, or torrenting a thousand movies or watching Internet porn 24/7 for a year.

It’d be fun to do some more stats, like estimating how many watts this one deluge of spam likely consumed, just so I can come up with some convoluted way of demonstrating that spammers are “with the terrorists”, but I’ll stop now, because it just makes me want to cry.

All this is making me think that small mail servers need a very efficient way to discard e-mails sent to an invalid recipient. I still haven’t made an embedded database of valid e-mails for my domain, but that is the logical next step. I need to make sure the check is done very early in my e-mail pipeline: before grey listing, before domain verifications, baysian filtering, virus checks, etc. Packages like postfix should have a setting that will allow them to automatically build a cdb database of e-mail addresses and hosted domains whenever they are presented with an LDAP/SQL backend for their datastore.

I’m also increasingly thinking I should perhaps change my e-mail config: have my VPS server just serve to filter out invalid spam, and then forward the good stuff to my server at home. It’s insane, but if spamming economics don’t change, I suspect hosting mail for even a small domain may require fairly significant computing resources and bandwidth.

I did some more tweaking, and concluded that my best moves were the following tweaks:

• Reduce the # of slave processes for the MTA to 2.
• Set up an explicit access rule for jslopez@xman.org that causes an immediate rejection and a nice little “don’t be an idiot and bounce forged return path’s” public service message.
• Get the accept queue depth as deep as possible for the slave processes.
• Reject any messages without a proper e-mail address in the FROM: envelope.

The killer solution was Google Apps for Domains though. I have registered for the service, updated my MX records, and once that information propagates through the Internets all my domain’s e-mail will get routed to Gmail, which has exactly one registered account: jslopez@xman.org. Gmail is configured to route any e-mails to an unknown address to my mail server. The net effect is that all this backscatter will get swallowed by the Gmail black hole, and everything else will remain outside the event horizon and hopefully get delivered to my mail server at something approaching the speed of light.

The other lesson learned from this is that openldap is slow, so one shouldn’t using it for accessing one’s MTA configuration. I intend to set up a cron job that will periodically dump the contents of LDAP in to files and then have postfix just read those files directly. This should prove to be infinitely more scalable and efficient, at the cost of updates being somewhat delayed.

The net effect of all this was to completely tie up my mail server and for the most part prevent any mail delivery. I’ve now tweaked the server a bit so I do eventually get mail, but it is still rather grim. So far, I’m killing connections to clients after two errors, I’ve trimmed my accept queue depth, and dramatically increased the number of simultaneous connections I will process. The overall effect has been pretty taxing on my mail server, and I still see significant delays in delivery times, so I’m all ears to any brilliant suggestions on how to address this problem.

If you are a mail admin and are wondering why your queues are backed up with tons of jslopez@xman.org e-mail, please, please kill it. I suspect thought that most of my mail is coming from bots, so I’ll probably need to start adding immediate filtering at connect time that drops suspected bots.

Is this happening to anyone else?

So, now the new thing appears to be some bozo trying to create a MySpace page account thinks that they know their e-mail address. I will give MySpace the thumbs up for at least sending a confirmation e-mail, but their cancellation process is tedious to say the least. I have to click a link, which then promptly says: “you need to login”. I login. I am them asked to enter a comment and click on one of two buttons. Click on the wrong one and the account doesn’t get canceled. I then get a confirmation e-mail for the cancelation. I click on a link in that e-mail, which takes me to another page which asks me to type in my e-mail address and then once again I have to click on one of two buttons to get the account canceled. I kind of appreciate the caution in their process, but given that they haven’t even received a confirmation of the e-mail address yet, isn’t that a bit too much?

Oh, and the best part: it takes up to 48-hours to delete the account. I started the delete process on one account and was about one hour in before another account was created with my e-mail address. At this point I’m starting to wonder if the best solution is for me to create my own MySpace account with my e-mail address in hopes that this will lock out other attempts to use my e-mail address.

Despite the best efforts of anti-spam software, e-mail is precipitously close to going the way of USENET.

This phenomenon seems to have really become a lot worse in the last couple of years, probably because e-mail has become so mainstream that friends and company use your e-mail address more than they use your phone number. I have tried, repeatedly, to send people e-mails clarifying that someone has given them the wrong address, but more often than not, it simply results in more confusion.

Over the last couple of years, I have been signed up for Kodak’s photo repository (which apparently fails to collect any contact information beyond an e-mail address and fails to validate that e-mail address before letting you upload pictures). I’ve been able to see some lovely wedding photos and what I suspect are some honeymoon photos (I didn’t want to see that!) that were not intended for the general public. I’ve been signed up for free Microsoft Office trials. I’ve been put on to mailing lists for military personnel in Georgia. I’ve been contacted by domain registrars about configuration updates to various domain names. I’ve been put on three different PTA mail lists. I’ve received shipping notices and order invoices from Dell and HP, going to people all over the US as well as a Christian missionary group in Texas that proselytizes by teaching English. I’ve received multiple correspondences about wedding and honeymoon plans (I’m guessing related to the photos) from planners, hotels, restaurants, and… Victoria’s Secret (really, I didn’t want to know). I’ve received real billing notices from the BofA, with lots of account information included. The worst part is that I’ve been put on so many spammer’s lists it is insane (the overlap between people who don’t know their own e-mail address and people who don’t recognize phishing scams is… significant).

I have tried to be nice about this. I’ve carefully unsubscribed or disabled most of the stuff set up with my e-mail address. I’ve tried to contact the senders and recipients of e-mails to let them know that this is the wrong e-mail address. I’ve tried looking up whatever contact info is available to track people down. I haven’t actually sent snail mail, but that is what I’m going to do next.

No more Mr. Nice Guy. I’m going to go public. If you still don’t get it, I’m going to start signing people up for obnoxious services as retribution. Today’s latest is:

Carolyn Smith 34991 Hamilton Ct. Farmington Hills MI 48334

Carolyn claims to work for a company called ACN. I tried looking up a phone number for that address, but apparently there is none. I’ll be sending snail mail, but I suspect it is going to take weeks for that to work. Carolyn, your e-mail address on gmail is NOT what you think it is. If you’ve been wondering why you haven’t obtained your free Microsoft Office trials as well as all the Microsoft junk mail you’ve signed up for, this is the reason. I’ve disabled the subscription multiple times, but you seem to keep signing me up. Please, take a look at what your e-mail address is. If anyone reads this thinks they know Carolyn, please contact her and tell her to send an e-mail to what she thinks is her e-mail address. Then maybe we can sort this all out.

And finally, as a PSA for all those folks managing mail lists and using e-mail addresses for usernames. Please, do yourself and your customers a favour and verify e-mail addresses before signing them up.

First of all, Typo filtering just doesn’t seem to work how I’d expect it to. It may all be user error, as I haven’t looked at the code for it. I’ll look at it more closely in the near future, but for now impression is that it doesn’t work the way it needs to to be effective.

The Akismet support for Typo also seems to be lacking. Enabling it mostly seemed to result in me getting timeouts. I went to look at the Typo site to see how other people were tackling this and discovered… the site was temporarily down. Nice that.

What blows me away is that Typo’s “bulk operations” is still tied to the same synchronous “wait for it to complete or we’ll time out” interface as everything else. What’s clearly called for here is some primitive queuing up of bulk tasks. I’m still learning Rails, but my bet is Rails doesn’t have an easy way to do such a thing (although it seems like it’d be easy to do in Ruby).

Ultimately, I ended up going with the tried and true approach: just blocking port 80 access to IP’s and subnets that were clearly running comment bots. Not an ideal solution if you have a broad audience, but given that I probably have a typical audience of a handful of people, the odds of me accidentally knocking off someone who’d actually read my blog are slim to none. Since I started firewalling off people’s IP’s, the torrent of comment spam that I normally see has diminished to a perfectly manageable level.

So keep that in mind: firewalls are your best frend when it comes to blocking comment spam.

I’m hoping it works a bit like graffitti: if you are aggressive enough in removing it, they decide their efforts are better spent elsewhere (as a public service, people should setup up blogs that do nothing but collect blog spam ;-). I will be setting up Akismet this evening in order to avoid waking up to thirty odd comments that need deleting.