Xblog: Category Computers

The New Hardware

Christopher Smith — Sun, 28 Sep 2008 06:28:00 +0000

So, after much theatrics, our little co-op managed to get it together and install new hardware. We ended up going with Silicon Mechanics Rackform nServe A266. It’s drool worthy hardware, so I thought I’d detail what a wonderful little box this is for our needs.

A bit about Silicon Mechanics. They are a SuperMicro system builder who’ve always made their presence felt in the local Los Angeles Linux community. They show up at SCALE each year without fail, often donating one of their servers as a raffle prize (and you don’t even have to give them your business card to get it!). They clearly care a great deal about their hardware and about the Linux community, so they seemed the right kind of people for our project (and to their credit, they were very patient and helpful with us, particularly considering how small an order we were).

The A266 appealed to us for a lot of reasons, but probably the biggest one was its power footprint and how easy Silicon Mechanics makes it to figure out what your power draw is really going to be. Right there as you are configuring your hardware they estimate your power consumption in very precise terms, so you can know your power footprint before you ever see hardware, and more importantly you can easily figure out what kind of adjustments to make to get the best bang for the watt. This is a brilliant reaction to data center concerns shifting away from rackspace and towards heat and power consumption. I really hope other vendors adopt this themselves. The system uses Opteron cores, which aren’t exactly legendary for their power footprint (AMD is apparently about to bring out some real miserly Opterons towards the end of this year, but until then Intel clearly has the edge), but by going with 2 quad core, high efficiency Opteron 2347 HE’s, we were able to get significant processing power without burning through our entire power budget.

The real win on the power consumption front though was with the RAM. We’re doing server virtualization, so RAM is our most valuable resource. Accordingly, we loaded up on 32GB of RAM. Now, if you have an Intel based system, you end up with FB-DIMMS, which suck power and generate heat like a hair dryer. They have low voltage FB-DIMMS out now, but I have yet to see anything to suggest that this competes well with the DDR2 memory we got in our system. The power savings on the RAM were so great, they totally overcame any excess heat from the CPU. While the memory is by no means the fastest you can get, AMD’s architecture, with HyperChannel and it’s 3 levels of cache, tends not to be as sensitive to such things, and for our needs, more RAM is way more valuable than faster RAM.

As we expected from SiliconMechanics, the system’s construction is first rate (I’ll try to get pictures up at some point). When we installed it at the colo, we got numerous positive comments from the ops who passed by. These guys see boxes all the time, and while they tend to be hardware junkies, they also tend to be blazé about the usual fare. About the only criticism I could make is that the box is definitely noisy, but this is most likely on account of the extensive cooling efforts (our hard drive temperatures appear significantly lower than with our old 4U box).

We got the system with SuperMicro’s IPMI 2.0 card with full KVM over LAN support. This really improves our ability to manage the system remotely, which is a big concern for us. I guess we could have bought a network KVM of our own, but I like having it all integrated in with the system. Unfortunately, I managed to seize up our LAN at one point, which is probably one of the main scenarios where the IPMI card isn’t going to save you.

Our storage needs.are kind of weird. We need space, but we also need fairly decent IOPS. While each individual member doesn’t test the storage system much, collectively you can end up with a lot seeks going on at the same time. We ended up going with a fairly interesting strategy. We got 4 500GB drives and hooked them up to a 3Ware 9650SE RAID controller with built in battery backed cache. Normally I’m not a huge fan of hardware RAID or storage caches, but in this instance it makes a lot of sense. With all the RAM we have, we can actually expect excellent filesystem caching performance, but the one thing Linux’s filesystem cache can’t help with is a write that needs to be flushed to disk. This was particularly painful as we were going with a RAID-5 configuration, which doesn’t exactly give you great write IOPS. Piling on to this was our selection of high density 7200rpm drives instead of VelociRaptors or low latency SAS drives. The battery backed cache is a big game changer in this regard. It supports a variety of modes of operation that trade off between performance and reliability, but we went with the “balanced” mode, where it journals writes in the cache, and then signals to the OS that they are complete, while completing the actual write to the RAID at a later point. The net effect is that we can handle bursts of write IO’s very quickly and our peak write IOPS is much higher than it otherwise would be. When we initially set up the system, it did still seem kind of sensitive to high IO loads, but after some tuning it seems to be much more efficient. For our drives we went with Seagate’s Barracude ES.2’s, whose firmware seems particularly good at handling multiple in flight IO’s. We could have gone with Western Digital’s slightly cheaper and much more energy efficient RE2-GP drives, but their latencies are so much worse than the Seagate’s, and thanks to the RAM we had plenty of room in our power budget. The case has 8 hot-swap drive bays, so drive failures can be handled by the colo’s ops without so much as a hiccup for the system (and 3ware’s 3DM2 software allows you to manually flash a particular drive’s LED so the ops can be sure to pull the right one). Knowing this, we deliberately under did our drive order, with the idea being that we’d simply order new drives on an as needed basis, hopefully benefiting from cutthroat evolution of the hard drive market, such that future drives would be denser, faster, and cheaper.

This whole thing is powered by a 90% efficiency redundant 700w PSU (which is another key part of keeping our power budget down). Our previous system didn’t have a redundant power supply, and while it never failed on us, I lived in fear of getting that midnight call. I fear not now.

So far, our experience with the system has been pretty amazing. Horribly abusive emerge’s inside my virtual instance fly by like it is no big deal. This blog, despite still being Typo 4, is so much zippier than its previous instantiation. It’s hard to know how much credit ought to go to our new software platform (more on that another time), but it is clear that at the very least a huge chunk of it belongs to this new hardware.

Sometimes a Picture Is Worth More Than 1000 Words

Christopher Smith — Thu, 12 Jun 2008 17:59:00 +0000

Normally I’m not a big fan of Valleywag, but days like today are the ones they really suit up for. Without further ado, let me summarize today’s tech news:

Back in the Land of the Living

Christopher Smith — Tue, 20 May 2008 03:45:00 +0000

Well, our server crashed today. Weirdest bug I ever saw: we got a kernel oops when smartd tried to get health information from the drives in the 3ware RAID array. One of the drives appears to have malfunctioned, so perhaps that is related. The fragility was possibly caused by running a fairly up to date smartd on a fairly out of date kernel with SKAS patches… but it is far from clear. I need to test this out more to be sure of what the magic sequence was, but needless to say… it’s been an experience.

Valleywag hasn't gone downhill, News has

Christopher Smith — Wed, 07 May 2008 08:30:00 +0000

I can’t believe anyone in the tech community is still covering the events at JavaOne, but sure enough, ~~we-troll-for-hits~~ValleyWag was there to capture Neil Young’s appearance yesterday. Now, I remember when Douglas Adams showed up for the Keynote on the last day of the conference, and that made sense. It was the last day of the conference and everyone was fried –if they hadn’t left town already. Douglas, true to form, provided some great entertainment and geek cred to start off the last day push. But Neil Young is to Java as the Smurfs are to the Iraq War. Could Sun make a more profound statement about how JavaOne jumped the shark long ago than to have an aging rocker whose seminal moments occurred before Java was ever invented keynote on the second day of the event? Best quote from the whole experience goes to Dan Farber’s blog entry, where after carefully promoting BluRay, Java, the PS3, and most importantly his Archive project, we read: “…As an artist I try to remove myself from the business,” Young said. “I steer myself away from that…”.

The previous article captures how Mark Kirk has skillfully managed to create controversy in order to get media attention during an election year. “Online porn” doesn’t quite drag voters attention away from all the other election year theatrics, and “online child predator” is so yesterday’s news, but “rape rooms” is a sure fire hit. Is there any trick from Hussein’s regime that politicians won’t copy and/or trivialize?

Darl McBride Does His Iraqi Minister of Intelligence Imitation

Christopher Smith — Thu, 01 May 2008 14:40:00 +0000

ArsTechnica was there to catch CEO SCO describing an interesting variant of reality. Highlights include objectively verifiable claims that books on how to program Linux don’t exist, that there is no difference between Linux and Unix, and directly contracting his own SVP’s earlier testimony that they have evidence that System V Unix is in Linux. Don’t be shocked if he later claims Shakespeare copied System V, that Linus assassinated JFK, and that Poland was never dominated by the Soviet Union.

In Your Face ComScore

Christopher Smith — Thu, 17 Apr 2008 17:23:00 +0000

Man, I so wanted to say something when ComScore’s initial report came out, but my insider status (barely insider really) makes it dangerous. So, it is with great joy that I let c|net do the talking for me.

Gentoo lirc problems

Christopher Smith — Fri, 04 Apr 2008 15:03:00 +0000

If you are using Gentoo amd64 for MythTV like me, you have probably noticed some problems building lirc. This has been driving me nuts for the last week. I found the bug behind it. I make a note of it simply to spare others the pain and suffering.

Blowback From the War On Spam

Christopher Smith — Mon, 31 Mar 2008 17:14:00 +0000

So, the deluge of spam blowback continues. The problem seems widespread enough at this point that I feel like contacting the authors of major anti-spam software and suggest that they just immediately drop all e-mail with a jslopez@xman.org return path forever. I have added an SPF record to the domain’s DNS in the hopes that this will help other MTA’s realize that the e-mail is forged and not to send a bounce message, but I haven’t seen much in the way of impact.

Some fun stats:

Since I created the jslopez@xman.org account in Google Apps, it has received over 920,000 e-mails.
The total size of the e-mail that has been routed to the gmail account is 3.75GB. Fortunately, I have a 25GB quota, but at this pace I can expect to exceed the quota given to normal gmail users by the end of the week!
Meanwhile, my old mail server continues to receive some jslopez@xman.org, although the rate of delivery has tapered off significantly. At its peak I was processing on the order of 500 jslopez@xman.org e-mails per second, and now it is more like two or three per minute.
My old mail server logs show 550,000+ e-mail delivery attempts to jslopez@xman.org. That is over and above all the e-mails sent to Google Apps.
My logs were totally overwhelmed by the deluge of spam and so they only go back to the afternoon of the 25th… in other words this is all pretty much after I had created the Google Apps account.
This means I’ve received roughly 1.5 million e-mails probably around 5GB in total ever since I first started publishing SPF records which made it trivial to prove that the messages were forgeries. I published the SPF records immediately after adding the MX records for Google Apps, so the nearly 1 million messages that have been sent to the Google Apps account in particular have no excuse for being there.
I conservatively estimate another 400,000 or so rejects that were lost in my logs. I expect by the end of the day today, jslopez@xman.org will have received on the order of 2 million bounces in total, representing approximately 8 GB of bounce messages.
Most bounce messages are terser than the original messages, so I suspect this means the total for the original messages that got bounced is measured in tens of gigabytes.
I’d like to think most spam delivery attempts don’t result in bounces, either because they get through (otherwise, why bother?) or are rejected/swallowed without a bounce (surely some MTA’s are correctly configured). This one attack probably represents hundreds of gigabytes if not terabytes of e-mail bouncing all around the Internet.
Had this bandwidth not been used for of spamming the Internet, the spammer could have used all this bandwidth for a good cause: like stealing a half a million songs, or torrenting a thousand movies or watching Internet porn 24/7 for a year.

It’d be fun to do some more stats, like estimating how many watts this one deluge of spam likely consumed, just so I can come up with some convoluted way of demonstrating that spammers are “with the terrorists”, but I’ll stop now, because it just makes me want to cry.

All this is making me think that small mail servers need a very efficient way to discard e-mails sent to an invalid recipient. I still haven’t made an embedded database of valid e-mails for my domain, but that is the logical next step. I need to make sure the check is done very early in my e-mail pipeline: before grey listing, before domain verifications, baysian filtering, virus checks, etc. Packages like postfix should have a setting that will allow them to automatically build a cdb database of e-mail addresses and hosted domains whenever they are presented with an LDAP/SQL backend for their datastore.

I’m also increasingly thinking I should perhaps change my e-mail config: have my VPS server just serve to filter out invalid spam, and then forward the good stuff to my server at home. It’s insane, but if spamming economics don’t change, I suspect hosting mail for even a small domain may require fairly significant computing resources and bandwidth.

Standards, Standards Bodies, and "complete, utter, unadulterated bullshit"

Christopher Smith — Sat, 29 Mar 2008 00:59:00 +0000

If you haven’t read it already, run over to Tim Bray’s blog about the ISO BRM around the OOXML standard. He is a quiet, staid, technical expert, representing quiet, staid, Canada. Despite this, he was so disgusted by what happened that he wrote the following:

What Was Bad · The process was complete, utter, unadulterated bullshit. I’m not an ISO expert, but whatever their “Fast Track” process was designed for, it sure wasn’t this. You just can’t revise six thousand pages of deeply complex specification-ware in the time that was provided for the process. That’s true whether you’re talking about the months between the vote and when the Responses were available, the weeks between the Responses’ arrival and the BRM, or the hours in the BRM room. As the time grew short there was some real heartbreak as we ran out of time to take up proposals; some of them, in my opinion, things that would really have helped the quality of the draft. This was horrible, egregious, process abuse and ISO should hang their heads in shame for allowing it to happen. Their reputation, in my eyes, is in tatters. My opinion of ECMA was already very negative; this hasn’t improved it, and if ISO doesn’t figure out away to detach this toxic leech, this kind of abuse is going to happen again and again.

Blowback

Christopher Smith — Tue, 25 Mar 2008 20:08:00 +0000

So, with a bit more investigation, it is now clear what exactly was going on with my mail server. It appears that some spammer has decided to send out massive numbers of spams with a forged return path, and said forgery pointed to jslopez@xman.org. As per usual, there are still massive numbers of domains that will bounce such messages, and on top of that there are mlm’s and vacation programs that automatically respond to the return path of anything they get, so my MTA has been consumed by the blowblack/backscatter.

Awesome.

I did some more tweaking, and concluded that my best moves were the following tweaks:

Reduce the # of slave processes for the MTA to 2.
Set up an explicit access rule for jslopez@xman.org that causes an immediate rejection and a nice little “don’t be an idiot and bounce forged return path’s” public service message.
Get the accept queue depth as deep as possible for the slave processes.
Reject any messages without a proper e-mail address in the FROM: envelope.

The killer solution was Google Apps for Domains though. I have registered for the service, updated my MX records, and once that information propagates through the Internets all my domain’s e-mail will get routed to Gmail, which has exactly one registered account: jslopez@xman.org. Gmail is configured to route any e-mails to an unknown address to my mail server. The net effect is that all this backscatter will get swallowed by the Gmail black hole, and everything else will remain outside the event horizon and hopefully get delivered to my mail server at something approaching the speed of light.

The other lesson learned from this is that openldap is slow, so one shouldn’t using it for accessing one’s MTA configuration. I intend to set up a cron job that will periodically dump the contents of LDAP in to files and then have postfix just read those files directly. This should prove to be infinitely more scalable and efficient, at the cost of updates being somewhat delayed.