Sometimes a Picture Is Worth More Than 1000 Words
Normally I’m not a big fan of Valleywag, but days like today are the ones they really suit up for. Without further ado, let me summarize today’s tech news:
Back in the Land of the Living
Well, our server crashed today. Weirdest bug I ever saw: we got a kernel oops when smartd tried to get health information from the drives in the 3ware RAID array. One of the drives appears to have malfunctioned, so perhaps that is related. The fragility was possibly caused by running a fairly up to date smartd on a fairly out of date kernel with SKAS patches… but it is far from clear. I need to test this out more to be sure of what the magic sequence was, but needless to say… it’s been an experience.
Valleywag hasn't gone downhill, News has
I can’t believe anyone in the tech community is still covering the events at JavaOne, but sure enough, we-troll-for-hitsValleyWag was there to capture Neil Young’s appearance yesterday. Now, I remember when Douglas Adams showed up for the Keynote on the last day of the conference, and that made sense. It was the last day of the conference and everyone was fried –if they hadn’t left town already. Douglas, true to form, provided some great entertainment and geek cred to start off the last day push. But Neil Young is to Java as the Smurfs are to the Iraq War. Could Sun make a more profound statement about how JavaOne jumped the shark long ago than to have an aging rocker whose seminal moments occurred before Java was ever invented keynote on the second day of the event? Best quote from the whole experience goes to Dan Farber’s blog entry, where after carefully promoting BluRay, Java, the PS3, and most importantly his Archive project, we read: “…As an artist I try to remove myself from the business,” Young said. “I steer myself away from that…”.
The previous article captures how Mark Kirk has skillfully managed to create controversy in order to get media attention during an election year. “Online porn” doesn’t quite drag voters attention away from all the other election year theatrics, and “online child predator” is so yesterday’s news, but “rape rooms” is a sure fire hit. Is there any trick from Hussein’s regime that politicians won’t copy and/or trivialize?
Darl McBride Does His Iraqi Minister of Intelligence Imitation
ArsTechnica was there to catch CEO SCO describing an interesting variant of reality. Highlights include objectively verifiable claims that books on how to program Linux don’t exist, that there is no difference between Linux and Unix, and directly contracting his own SVP’s earlier testimony that they have evidence that System V Unix is in Linux. Don’t be shocked if he later claims Shakespeare copied System V, that Linus assassinated JFK, and that Poland was never dominated by the Soviet Union.
In Your Face ComScore
Man, I so wanted to say something when ComScore’s initial report came out, but my insider status (barely insider really) makes it dangerous. So, it is with great joy that I let c|net do the talking for me.
Gentoo lirc problems
If you are using Gentoo amd64 for MythTV like me, you have probably noticed some problems building lirc. This has been driving me nuts for the last week. I found the bug behind it. I make a note of it simply to spare others the pain and suffering.
Blowback From the War On Spam 1
So, the deluge of spam blowback continues. The problem seems widespread enough at this point that I feel like contacting the authors of major anti-spam software and suggest that they just immediately drop all e-mail with a jslopez@xman.org return path forever. I have added an SPF record to the domain’s DNS in the hopes that this will help other MTA’s realize that the e-mail is forged and not to send a bounce message, but I haven’t seen much in the way of impact.
Some fun stats:
- Since I created the jslopez@xman.org account in Google Apps, it has received over 920,000 e-mails.
- The total size of the e-mail that has been routed to the gmail account is 3.75GB. Fortunately, I have a 25GB quota, but at this pace I can expect to exceed the quota given to normal gmail users by the end of the week!
- Meanwhile, my old mail server continues to receive some jslopez@xman.org, although the rate of delivery has tapered off significantly. At its peak I was processing on the order of 500 jslopez@xman.org e-mails per second, and now it is more like two or three per minute.
- My old mail server logs show 550,000+ e-mail delivery attempts to jslopez@xman.org. That is over and above all the e-mails sent to Google Apps.
- My logs were totally overwhelmed by the deluge of spam and so they only go back to the afternoon of the 25th… in other words this is all pretty much after I had created the Google Apps account.
- This means I’ve received roughly 1.5 million e-mails probably around 5GB in total ever since I first started publishing SPF records which made it trivial to prove that the messages were forgeries. I published the SPF records immediately after adding the MX records for Google Apps, so the nearly 1 million messages that have been sent to the Google Apps account in particular have no excuse for being there.
- I conservatively estimate another 400,000 or so rejects that were lost in my logs. I expect by the end of the day today, jslopez@xman.org will have received on the order of 2 million bounces in total, representing approximately 8 GB of bounce messages.
- Most bounce messages are terser than the original messages, so I suspect this means the total for the original messages that got bounced is measured in tens of gigabytes.
- I’d like to think most spam delivery attempts don’t result in bounces, either because they get through (otherwise, why bother?) or are rejected/swallowed without a bounce (surely some MTA’s are correctly configured). This one attack probably represents hundreds of gigabytes if not terabytes of e-mail bouncing all around the Internet.
- Had this bandwidth not been used for of spamming the Internet, the spammer could have used all this bandwidth for a good cause: like stealing a half a million songs, or torrenting a thousand movies or watching Internet porn 24/7 for a year.
It’d be fun to do some more stats, like estimating how many watts this one deluge of spam likely consumed, just so I can come up with some convoluted way of demonstrating that spammers are “with the terrorists”, but I’ll stop now, because it just makes me want to cry.
All this is making me think that small mail servers need a very efficient way to discard e-mails sent to an invalid recipient. I still haven’t made an embedded database of valid e-mails for my domain, but that is the logical next step. I need to make sure the check is done very early in my e-mail pipeline: before grey listing, before domain verifications, baysian filtering, virus checks, etc. Packages like postfix should have a setting that will allow them to automatically build a cdb database of e-mail addresses and hosted domains whenever they are presented with an LDAP/SQL backend for their datastore.
I’m also increasingly thinking I should perhaps change my e-mail config: have my VPS server just serve to filter out invalid spam, and then forward the good stuff to my server at home. It’s insane, but if spamming economics don’t change, I suspect hosting mail for even a small domain may require fairly significant computing resources and bandwidth.
Standards, Standards Bodies, and "complete, utter, unadulterated bullshit"
If you haven’t read it already, run over to Tim Bray’s blog about the ISO BRM around the OOXML standard. He is a quiet, staid, technical expert, representing quiet, staid, Canada. Despite this, he was so disgusted by what happened that he wrote the following:
What Was Bad · The process was complete, utter, unadulterated bullshit. I’m not an ISO expert, but whatever their “Fast Track” process was designed for, it sure wasn’t this. You just can’t revise six thousand pages of deeply complex specification-ware in the time that was provided for the process. That’s true whether you’re talking about the months between the vote and when the Responses were available, the weeks between the Responses’ arrival and the BRM, or the hours in the BRM room. As the time grew short there was some real heartbreak as we ran out of time to take up proposals; some of them, in my opinion, things that would really have helped the quality of the draft. This was horrible, egregious, process abuse and ISO should hang their heads in shame for allowing it to happen. Their reputation, in my eyes, is in tatters. My opinion of ECMA was already very negative; this hasn’t improved it, and if ISO doesn’t figure out away to detach this toxic leech, this kind of abuse is going to happen again and again.
Blowback 1
So, with a bit more investigation, it is now clear what exactly was going on with my mail server. It appears that some spammer has decided to send out massive numbers of spams with a forged return path, and said forgery pointed to jslopez@xman.org. As per usual, there are still massive numbers of domains that will bounce such messages, and on top of that there are mlm’s and vacation programs that automatically respond to the return path of anything they get, so my MTA has been consumed by the blowblack/backscatter.
Awesome.
I did some more tweaking, and concluded that my best moves were the following tweaks:
- Reduce the # of slave processes for the MTA to 2.
- Set up an explicit access rule for jslopez@xman.org that causes an immediate rejection and a nice little “don’t be an idiot and bounce forged return path’s” public service message.
- Get the accept queue depth as deep as possible for the slave processes.
- Reject any messages without a proper e-mail address in the FROM: envelope.
The killer solution was Google Apps for Domains though. I have registered for the service, updated my MX records, and once that information propagates through the Internets all my domain’s e-mail will get routed to Gmail, which has exactly one registered account: jslopez@xman.org. Gmail is configured to route any e-mails to an unknown address to my mail server. The net effect is that all this backscatter will get swallowed by the Gmail black hole, and everything else will remain outside the event horizon and hopefully get delivered to my mail server at something approaching the speed of light.
The other lesson learned from this is that openldap is slow, so one shouldn’t using it for accessing one’s MTA configuration. I intend to set up a cron job that will periodically dump the contents of LDAP in to files and then have postfix just read those files directly. This should prove to be infinitely more scalable and efficient, at the cost of updates being somewhat delayed.
Still Standing
For those of you who still care about Vista, I “upgraded” to SP1 and didn’t encounter any of the problems that a lot of others have had.
The 127 MB download took forever (like 45 minutes despite our pricey DSL connection), and then the installer popped up a dialogue saying that it needed at least 3GB of disk space on the main Windows partition and wouldn’t install until I made room. I’m trying to figure out how a rational person could think that a 127MB download might need 3GB of disk space to complete (yeah, I can see contrived situations where it’d be possible, but come on!). I dutifully cleared away like 15GB of downloads that were just sitting around on my drive, ran the install, dutifully rebooted the system as one would expect from Windows, and then confirmed that the entire install had consumed ~800MB of disk space (I’m guessing a 2:1 compression ratio, binary diffs, plus backing up the old files for rollback pretty much explains all of that).
You know, it is sad enough watching those progress bars behave like meandering drunks stumbling to the next bar during the later hours of a pub crawl, but really, is it so hard to have a reasonably accurate estimate of how much disk space is needed for an install?