Xblog

A bit about Silicon Mechanics. They are a SuperMicro system builder who’ve always made their presence felt in the local Los Angeles Linux community. They show up at SCALE each year without fail, often donating one of their servers as a raffle prize (and you don’t even have to give them your business card to get it!). They clearly care a great deal about their hardware and about the Linux community, so they seemed the right kind of people for our project (and to their credit, they were very patient and helpful with us, particularly considering how small an order we were).

The A266 appealed to us for a lot of reasons, but probably the biggest one was its power footprint and how easy Silicon Mechanics makes it to figure out what your power draw is really going to be. Right there as you are configuring your hardware they estimate your power consumption in very precise terms, so you can know your power footprint before you ever see hardware, and more importantly you can easily figure out what kind of adjustments to make to get the best bang for the watt. This is a brilliant reaction to data center concerns shifting away from rackspace and towards heat and power consumption. I really hope other vendors adopt this themselves. The system uses Opteron cores, which aren’t exactly legendary for their power footprint (AMD is apparently about to bring out some real miserly Opterons towards the end of this year, but until then Intel clearly has the edge), but by going with 2 quad core, high efficiency Opteron 2347 HE’s, we were able to get significant processing power without burning through our entire power budget.

The real win on the power consumption front though was with the RAM. We’re doing server virtualization, so RAM is our most valuable resource. Accordingly, we loaded up on 32GB of RAM. Now, if you have an Intel based system, you end up with FB-DIMMS, which suck power and generate heat like a hair dryer. They have low voltage FB-DIMMS out now, but I have yet to see anything to suggest that this competes well with the DDR2 memory we got in our system. The power savings on the RAM were so great, they totally overcame any excess heat from the CPU. While the memory is by no means the fastest you can get, AMD’s architecture, with HyperChannel and it’s 3 levels of cache, tends not to be as sensitive to such things, and for our needs, more RAM is way more valuable than faster RAM.

As we expected from SiliconMechanics, the system’s construction is first rate (I’ll try to get pictures up at some point). When we installed it at the colo, we got numerous positive comments from the ops who passed by. These guys see boxes all the time, and while they tend to be hardware junkies, they also tend to be blazé about the usual fare. About the only criticism I could make is that the box is definitely noisy, but this is most likely on account of the extensive cooling efforts (our hard drive temperatures appear significantly lower than with our old 4U box).

We got the system with SuperMicro’s IPMI 2.0 card with full KVM over LAN support. This really improves our ability to manage the system remotely, which is a big concern for us. I guess we could have bought a network KVM of our own, but I like having it all integrated in with the system. Unfortunately, I managed to seize up our LAN at one point, which is probably one of the main scenarios where the IPMI card isn’t going to save you.

Our storage needs.are kind of weird. We need space, but we also need fairly decent IOPS. While each individual member doesn’t test the storage system much, collectively you can end up with a lot seeks going on at the same time. We ended up going with a fairly interesting strategy. We got 4 500GB drives and hooked them up to a 3Ware 9650SE RAID controller with built in battery backed cache. Normally I’m not a huge fan of hardware RAID or storage caches, but in this instance it makes a lot of sense. With all the RAM we have, we can actually expect excellent filesystem caching performance, but the one thing Linux’s filesystem cache can’t help with is a write that needs to be flushed to disk. This was particularly painful as we were going with a RAID-5 configuration, which doesn’t exactly give you great write IOPS. Piling on to this was our selection of high density 7200rpm drives instead of VelociRaptors or low latency SAS drives. The battery backed cache is a big game changer in this regard. It supports a variety of modes of operation that trade off between performance and reliability, but we went with the “balanced” mode, where it journals writes in the cache, and then signals to the OS that they are complete, while completing the actual write to the RAID at a later point. The net effect is that we can handle bursts of write IO’s very quickly and our peak write IOPS is much higher than it otherwise would be. When we initially set up the system, it did still seem kind of sensitive to high IO loads, but after some tuning it seems to be much more efficient. For our drives we went with Seagate’s Barracude ES.2’s, whose firmware seems particularly good at handling multiple in flight IO’s. We could have gone with Western Digital’s slightly cheaper and much more energy efficient RE2-GP drives, but their latencies are so much worse than the Seagate’s, and thanks to the RAM we had plenty of room in our power budget. The case has 8 hot-swap drive bays, so drive failures can be handled by the colo’s ops without so much as a hiccup for the system (and 3ware’s 3DM2 software allows you to manually flash a particular drive’s LED so the ops can be sure to pull the right one). Knowing this, we deliberately under did our drive order, with the idea being that we’d simply order new drives on an as needed basis, hopefully benefiting from cutthroat evolution of the hard drive market, such that future drives would be denser, faster, and cheaper.

This whole thing is powered by a 90% efficiency redundant 700w PSU (which is another key part of keeping our power budget down). Our previous system didn’t have a redundant power supply, and while it never failed on us, I lived in fear of getting that midnight call. I fear not now.

So far, our experience with the system has been pretty amazing. Horribly abusive emerge’s inside my virtual instance fly by like it is no big deal. This blog, despite still being Typo 4, is so much zippier than its previous instantiation. It’s hard to know how much credit ought to go to our new software platform (more on that another time), but it is clear that at the very least a huge chunk of it belongs to this new hardware.