Xblog

• The GPL was primarily conceived as a means of protecting software freedom itself, not individual projects. Indeed, the GNU foundation acknowledges that it imposes limitations that can limit a project, but in exchange further the their cause.
• The LGPL was primarily conceived in recognition that for certain circumstances, the GPL was unable to further the cause of freedom and was therefore would do nothing beyond harming a project itself, which in effect would hurt the cause of software freedom.
• X11/BSD style licenses have always been the licenses that placed concerns about software/code ahead of the cause of software freedom. If that lines up with your concerns, that’s the kind of license you want.

While the defensive nature of GPL and LGPL licenses might strike people as overkill and ultimately harmful, it is helpful to keep in mind the well documented circumstances that brought about their origins. It is possible for a key contributor to a project to turn on you. One might argue that in the modern world open source is so much more successful that such events are less likely to happen, and you’d be right, but you can even find modern examples of the issues involved (contention around MySQL AB’s licensing practices comes to mind).

In short: licensing isn’t likely to change who your key contributors are, but that is far from the only issue one need contemplate when licensing.

Let’s try to break things down a few different ways:

• Cheap drives do lose a lot of performance to error correction… but that is merely a firmware setting that costs $0 to change… if you can get access to the firmware. Enterprise class drives expect to run in a RAID config where it is better to just give up and hope that other drives have the parity data. If you are buying a lot of hard drives (i.e. you are building an EDW), you can no doubt negotiate a firmware config change.

• Faster rotation, smaller media, more heads, etc. are all great for random seeks. They are pointless for throughput (or more accurately counter balanced by greater media density). Being bound by seeks is a great way to have a visit from the fail whale.

• The vibration issue is a real one. Solution: dampen vibrations before they get to the drive. Padding, mount suspensions, etc., etc. Even if you build it in to the drive, this is actually very cheap to do right, but enterprise drives charge a premium for it.

• Command queuing support is now a commodity feature. You can get some improvements by having better drive firmware for this, and moderate improvements by providing faster processors and more memory for deeper queues, but unless you clog the pipe with random seeks (read: bad), even commodity drives will deliver.

• The “electronic features” section is basically is talking about the performance characteristics of the node that have NOTHING TO DO WITH THE DRIVE. If you want to see cheap drives with massive CPU & RAM available to them, go check out your local Hadoop cluster.

• The drive industry continues to price and market drives based on their interfaces far too much, and this article makes the same mistake. Storage interfaces are increasingly similar in terms of their performance capabilities. To the extent that one has better bandwidth or latency than the other, you pay for it in $’s. While storage interfaces have unique characteristics that make them better or worse suited for certain types of solutions, we really need to move past the notion that say having a Fibre interface means you’ll have better iops or for that matter throughput even if the SAS drive has faster rotation rates, shorter strokes, more heads, density, etc., etc.

• Talking about SSD’s being a 100x performance win over disk drives is once again showing a focus on iops. SSD’s are awesome for iops, and you can use cache and RAID techniques to get their throughput up to impressive levels, but I’m sorry they aren’t anywhere close to 100x better at throughput. It’s nice that they fill in for an area where magnetic disk continues to fall behind, but the bottom line is that even SSD’s are only a short term fix that ignore the larger trend: if your software is iops bound, it is increasingly going to fall behind relative to sequential scan (even dumb sequential scan) software.

• In general, it shouldn’t be news to anyone that expensive drives have better iops performance than cheap ones. That’s been the story of the storage industry for at least the last two decades.

Here’s a simpler summary of what is really going on here: seeks suck. Seeks have not gotten much better over time, though throughput has improved tremendously. Enterprise drives, which traditionally target the OLTP market, have the best seek performance you can get from magnetic disc media, because with OLTP all that matters is iops/$, and in many cases just iops. It turns out though that the only area where we can really move the needle on iops is if you have parallel, independent I/Os (very much the kind of thing that scatter/gather AIO is all about). So, you can’t improve seek latency much (Savvio 15K.2’s are at best 5x better), but you can improve how many seeks you can do at once… to a point. Here’s the thing though: at a certain point, sequential throughput becomes sooo much faster than random seeks, that it can be as fast to do the logical equivalent of a table scan than to do hundreds of parallel indexed lookups. Good software will recognize this and do the right thing.

Here’s the thing: assuming you are being smart in your software, a single Savvio is going to beat the snot out of a single Barracuda drive, but the Savvio is going to cost a lot more. For the same price you could buy several Barracuda drives, and then the Barracuda’s have such a huge win on sequential scan performance. You have to ask yourself: “how often are my indexes eliminating so many records for me that the Savvio’s better iops still trumps the awesome scan performance of this chunk of Barracuda array?” But here’s the real killer: since parallel, independent I/O’s are where the Savvio’s are going to really show their stuff… couldn’t you instead have your software split the I/O’s over the Barracuda drives? How many iops does that get you? ;-)

The trend in data warehouses is to build highly parallelized clusters that bring the processing power closer and closer to the storage (moving away from the SAN model), and to have the software treat each node as a highly sophisticated IOP, a IOPU (to play on the GPU concept) if you will, that simultaneously scans and analyzes the data on each node, and then aggregates the results. To the extent that the drive and the IO controller try to get clever with their own software, they actually get more and more in the way over time. At some point, it might theoretically be better to have a hundreds of really cheap, really dense single platter, single head drives with firmware that doesn’t try any error recovery or command queuing, and leave the software running on the individual “IOPU’s” to do all the cleverness along with the actually analytical work at hand.

I did find myself more in agreement with Carson on the non-storage tidbits (which I’m sure indicates just how wrong I am about the storage stuff). 10GigE seems ready to wipe Infiniband off the map, and Nehalem seems to have taken out any remaining advantages AMD had over Intel for OLAP work.

…I get pings from people I’ve worked with, people I barely remember, and even recruiters. The latter show a shocking amount of compassion/desperation. I actually had a recruiter call me up today and offer to hand me a resume even though he knew we don’t work with outside agencies to keep our costs down, and so there could not be any compensation in it for him (and yes, this gesture will undoubtedly pay dividends when this mess comes to an end, but he has to know that may be a long way off and I’m sure he is as desperate to bring home the bacon right now as anyone). I’ve never seen a recruiter do something like that before. Not even during the .com bust years. THAT is how bad it is.

Hang on to your horses everyone, this is going to be a very bumpy ride… since we’re in LA you may choose to think of it as an economyquake. ;-)

The Abysmal Status Quo

If you’ve worked in C or C++ (or Java for that matter) for a while, particularly if you’ve also had exposure to scripting languages, you have probably come to recognize how limited the built-in literals are. You have literals for the built-in types. You also have initializers for arrays and structs, which were so limited that the C99 committee felt compelled to improve them (see designated initializers). The Java world, after a great deal of reflection which I presume involved the use of powerful hallucinogens, has concluded XML is the best language around for defining data and there is no way another programming language can match it. So Java has simply chosen an idiom where what one would normally think of as a fine opportunity to use literals is instead a fine opportunity to define a new DTD and/or XML Schema and then write out some verbosely tagged data (always being careful to insert XML entities instead of > or < signs).

When one looks at Boost’s Program Options Library, one can’t help but marvel at the syntactic trickery employed to do something that is so simple and straightforward in, say, Perl. (Look mom! I used “Perl” and “simple” in the same sentence!). What I find most disturbing though, is that user-defined types can’t have literals associated with them. The closest you can get is a constructor that takes primitive types (which do have literals) as arguments. So, for example, if you are working with Unicode strings, you invariably end up writing something like:

UnicodeString("this is a Unicode string", "UTF-8")

Now, in their infinite wisdom, the C++Ox committee has addressed the Unicode issue separately from user defined types, so now you can do something like:

u8"this is a Unicode string"

The existence of this particular literal extension to C++0x if anything demonstrates literals are important. This is particularly true as the committee, in what can be described as a philosophical feud with the C99 folks, has not provided a corresponding built in set of literals for the complex number type (“see, we can do complex numbers entirely as a library, thereby keeping our syntax simple…” —just try to say that with a straight face).

To Boldly Go Where No C++ Compiler Has Gone Before

Into this mess, the C++0x committee has brashly bravely charged. The result is the extensible literals (i.e. user-defined literals) proposal. As with all good things in C++, there turns out to be a fair bit of complexity to the matter, but it all makes sense once one thinks about the hoops one is having the compiler jump through. The new literals mechanism is built off of suffixes (apparently, to use prefixes for user defined literals would invoke a computer apocalypse of sorts… I don’t understand the details, but I heard mumblings about California’s governor going back in time in starkers and Google’s distributed computer calling itself “Skynet”), allowing for things like 123km to translate literally in to some object representing 123 kilometers, which I have to say seems rather cool and… straightforward at first glance. That’s the simple concept from which the inevitable complexity begins.

Two Dancers Alternating Through Double Hops, One in Black and One in Yellow, Various Other Pairs of Dancers, a Guy with a Truly Impressive Falsetto, and a Guy In Stripes Who Is Definitely Not A Dancer

It turns out, to be able to express all the wonderfulness that should be C++ literals, the proposal introduces two distinct types of literals: The Raw And The Cooked.

Editor’s Note: For those of you who didn’t enter the workforce until after the original C++ standard was introduced, you may have to contact one of your elders to properly understand the cultural reference. For those of you wondering what this link has to do with the dichotomy between the natural and artificial world… I can only say that most of us at the time didn’t get the video either, but during that era the cool thing to do was have an abstract, obtuse video that completely went over the heads of most of the audience —that’s the way it was, and we liked it!

The raw literal is defined as the raw sequence of characters that form a literal. It’s the raw bytes of the literal before the compiler has had its way with it (although after the preprocessor has expanded any macros and any string literal concatenations have been done… just so that we can’t have a completely simple definition and the C preprocessor can continue to be the bane of all C++ developer’s experience).

The cooked form is defined as the typed value that the literal string represents (before all the magical user-defined literal processing happens). In particular, this allows one to be able to have that user defined literal in the “123km” example operate on the integer value “123” rather than have to first transform { ‘1’, ‘2’, ‘3’, ‘\0’ } in to a useful binary value.

My God… It’s Full of Operators

C++ operator overloading is simultaneously one of its most useful and most abused features. In what will undoubtedly ensure that everyone will either love or hate extensible literals, the proposal follows the C++ standards tradition and adds… more operators to overload in to the language. Bravely eschewing the C++/CLI’s approach of overloading the meaning of yet another symbol (I seriously could have imagined something like Foo::$Foo()), instead we have some new operators. For raw literals, we have the form:

T operator "suffix" (char const*)

Where T is the type of the literal, and “suffix” the magical suffix that identifies the user defined literal. So, when the compiler sees: 123km it interprets that (if the appropriate function is defined) as a call to long long operator"km"("123") (just like a constructor it has the option of throwing an exception… that one is going to keep the exception safety nuts up for months).

So far, though, things aren’t complicated enough to really meet the usual standard, multi-paradigm mischief that we’ve come to expect from C++. I mean, it’s barely OO, and doesn’t really tie in to the whole generic/functional programming world C++ developers have come to know and love. Fortunately, this proposal leaves no stone unturned. Indeed, this particular stone has been turned… and kicked around a fair bit afterwards, in the form of this:

template<char...> T operator "suffix" ();

Yup, that’s not just a templated version, but a variadic templated version. For those of you playing this at home, Variadic Templates are also part of the C++0x standard. They are essentially a way of turning LISP-y looking Typelists in to C-y looking vararg functions. So, our 123km example, if someone had defined an extensible literal operator like this:

template <char...> long long operator"km"();

then the compiler interprets that as a call to:

operator"km"<'1', '2', '3'>()

Note the lack of null at the end there? I’m sure that was put in there to ensure inconsistency with the other form. ;-)

Anyway, the primary advantage of the variadic template form is that if you tag the function with the new “constexpr” keyword, the entire thing can be evaluated at compile time like all good template metaprogramming foo. One could argue a sufficiently smart compiler might be able to determine when to do compile time evaluation of the former form in cases where it was possible. However, if there is one myth that the C++ committee considers heresy, it must by the myth of the Sufficiently Smart Compiler (one of life’s little ironies is how this view directly results in C++ compilers having to be the most sophisticated, complex, and nuanced compilers known to man.. but I digress).

I hope your head isn’t spinning and spewing forth pea soup right now, because we’re just firing up the barbie to move on to “cooked form” extensible literals.

Looking at the proposal, cooked literals seem actually quite straight forward and the rational way to handle the 123km example I keep bandying about. According to the proposal, one would define:

Kilometer operator"km"(unsigned long long);

which would then be invoked with 123. There is no special templated form, and cooked literals wisely take precedence over raw literals. There are similar forms for doubles and the various forms of C-style strings that now exist in C++0x (strings prefixed with “u” and “U” both get their special form), although surprisingly the functions are length terminated, so for example, one could make a convenience literal for std::strings such that:

"this is an std::string\n"s

creates an std::string by defining a literal operator as follows;

std::string operator"s"(const char* s, size_t length) { return string(s, length); }

While at first this might seem counter intuitive, it does provide a nice way to distinguish between cooked literals and non-templated raw literals.

I’ve Come To Bury Extensible Literals, Not to Praise Them

The one thing I didn’t see in the proposal is how negative integers are handled (it seems odd that the proposal would imply defaulting to unsigned integers, but perhaps I’m missing something). Unfortunately, the proposal also doesn’t really address a simple way to define hierarchical/structured literals like Boost’s Program Options library desperately cries out for. While one could define literals for maps and hash maps, I just don’t see them even remotely approaching the elegance you find in scripting languages, and it still seems like there isn’t a convenient way to have a literal composed of compound expressions. Then again, it’d be hard to distinguish between the latter and a sufficiently clever constructor and literals for the constructor’s type parameters.

There are lots of cool uses one can foresee for this proposal. Obviously one could make a units/measurements system that was much more seamlessly integrated through the use of literals for constructing measurement instances. Having an agreement upon literal syntaxes for string objects would bring them one step closer to first class status beside their C-style predecessors. One particularly cool example from the proposal is to have literals for internationalization efforts, such that “foo”_i18n might translate to: ‘lookup the key “foo” in the appropriation i18n table and use the appropriate value’, which might reduce i18n friction enough that developers would adopt sane i18n development practices without first having several sessions on the rack. One can see some interesting abuses, as well. I have to wonder at the extent to which they can be employed recursively. I can’t see a reason why one couldn’t create literals with side effects, although hopefully this practice would be viewed as in poor taste.

Despite the tongue-in-cheek comments found throughout this article, and some of the shortcomings in the concept and the specifics of the proposal, I actually quite like the design of extensible types and hope that a somewhat more polished version of it will make it into the standard and the top tiered C++ compilers quickly. While a complex solution for what at first glance seems like a simple problem, like a lot of C++ features, most of the complexity is pushed to the library designer side (i.e. the person writing the extensible literal), while using the feature seems likely to be simple and straight forward in the most common cases. The former is, in my opinion, intrinsic to C++’s nature, forgiveable, and arguably a feature. Designing code for reuse is always difficult, and sometimes languages which make it deceptively easy encourage very poor designs that would effectively be stillborn in C++. If, however, you can make using said code fairly straightforward, you make it easier for less sophisticated developers to leverage the skills of the masters. In the end, this proposal strikes me as emblamatic of the language itself: yes, it is complex under the hood; yes, it has a face only a mother could love; yet, beneath all that is is both powerful, pragmatic, and cleaner to work with than the typical hackery C and C++ programmers tend to come up with to address this issue.

So starting with the outer bits of the code first, it is pretty clear that the bulk of the message is escaped Javascript which is fed in to eval. Not exactly a good sign, but not necessarily a bad thing. The first step I took was to unescape the relevant data (without performing the eval of course), which yielded this.

The source was clearly obfuscated, so I went through it, cleaned up the formatting and attempted to assign more meaningful names to variables and functions. This yielded this. The transform1 function included a use of eval(someString.replace(/blah/, ”)), which was clearly just a way of obfuscating how myCallee was computed, so I performed the replace and substituted the result, yielding this much clearer source, which shows that myCallee is actually the function itself!

So, it now becomes clear that the obfuscation of transform1 was not entirely random in nature, as the source itself was used as part of the key to decode the “payload” (the string passed in to transform1 after its definition). The easiest way to accurately decode this was to create both the original fpTu function in all its obfuscated glory, then set myCallee to fpTu in transform1, and then see what we got back from transform1 when we passed it the payload.

Not surprisingly, we have another somewhat obfuscated chunk of Javascript. After reformatting it (thank you Steve Yegge for js2-mode and providing some more meaningful function and variable names, the payload now looks like this.

There are still a bunch of places where the replace(/blah/, ”) idiom is being used to obfuscate things, and another place where a variant is used where all alphanumeric values are being replaced with a period, and then all instances of multiple periods are replaced with a single one. After unraveling those, the intent of the code becomes clear and we can attach more meaningful names to the functions and variables. Thus I ended up with this.

We can now see that the code points to 3traff.cn (don’t you feel safer already? ;-), and it cleverly embeds an iframe pointing to 3traff.cn in to each document as it is loaded. I’m not up on my browser exploits, but it looks like the intent of all this is to at the very least track users as they go off site, and also might be hoping to confuse a browser about which domain a document came from, and therefore potentially cause cookies to leak to 3traff. Either way, this isn’t the kind of code I like running in my browser.

The killer insight that I was lacking was the notion that people reading the books would focus on the wrong thing. I was blinded by what I found to be very clear statements by the authors of the book about the benefits and purposes of patterns, how one should select patterns, how they differed from other reuse techniques, and indeed even the title itself. It seemed clear to me that it was all about design, all about practical solutions to common problems encountered in OO designs (and yes, the authors explicitly state that these solutions are nothing new, but rather well established), and most importantly: not about the code.

What the Design Patterns book teaches is for people to think in the box instead of outside of it; just as I did on that exam. Pattern thinking has now permeated and perverted peoples’ thinking to the extent where patterns are perceived as being an ends; something you need to use to correctly solve problems.

First, I don’t get how that principle related to “the Plank”, because it seems clear from the statement that he knew how to solve the problem using a different one of the cookie-cutter solutions he’d learned. Rushing to judgment before understanding a problem doesn’t strike me as “thinking in the box”. Normally, people “thinking in the box” spend most of their effort on careful consideration of how a problem resembles and differs from problems they’ve seen before, giving up if it is too different from anything they’ve worked with before. Secondly, he jumps from talking about “Design Patterns” the book to “Patterns” in general, apparently failing to recognize that Design Patterns was just one book detailing commonly encountered design techniques of OO developers and comes far short of representing all patterns (indeed, the PLoP books detail new patterns presented each year), they were just the low-lying fruit. Even if you forgo this misnomer, it somehow misses the very logical conclusion one should reach in not finding a pattern that fits your problem: your problem isn’t a commonly recurring one. It boggles the mind to think one would instead think to apply a pattern to a problem it was explicitly not designed for.

Now I understand why people gripe about the Singleton all the time: they’ve too often run in to developers who use it because they can, not because they need to, developers who think of the Singleton solution instead of the kinds of problems the pattern addresses.

Here’s the problem, if one is the sort of person who tries to figure out the minimal amount of studying to get through one’s statistics class (ironically applying approaches to achieving this that could have been derived from statistical principles ;-), one probably skimmed (or worse skipped) over the beginning of the book that takes the time to explains how to use the patterns. One probably went straight to the patterns themselves, and after looking at several of them, one probably decided the real meat of the patterns was in the code itself, because one’s main use for programming books was to look for code samples and apply them to your work without having to think so carefully to avoid making mistakes.

To paraphrase JWZ: now they have two problems.

Thinking in patterns is exactly the wrong thing to do! It makes you think in terms of the solution instead of in terms of the problem! Pattern thinking makes you try to fit a round, square, or oval hole (your choice of patterns) to the triangular peg (your problem). When all you have is a hammer…

A designer who goes around thinking about design in terms of trying to shove a bunch of pre-built solutions in to a problem is screwed whether they’ve read Design Patterns or not. A designer’s first focus should always be on what problems they have to solve, and only think about solutions as a consequence of addressing said problems. If you focus on the solutions instead, you will inevitably be wandering around with a hammer, thinking all the problems you face are pegs.

What was curious to me when first reading the quoted text above was that I thought it was odd to think of Patterns as solutions looking for a problem. The book quotes about the specific way that patterns are documented, and right there at the beginning of each pattern is talk about the kinds of problems the pattern is appropriate before. Heck, the pattern catalog was even grouped by the kinds of problems each pattern solved! It seemed obvious that the proper way to organize your thinking about patterns was in terms of the problems they solved (indeed, I’d often forget details of specific aspects of designs or implementation techniques for patterns I didn’t use much, but I tended to remember their applicability quite clearly). It’s odd to me that one would think one had to organize your thinking in terms of the solution, particularly since the solution was inevitably so much more complex than the problems it solved. What I was missing of course, was that to someone trying to find the “meat” of a pattern without reading and understanding the whole thing, they’d tend to of course focus on where the biggest sections of the pattern description were, and where the complexity was. So now you’ve compounded the problem: you’ve dramatically increased the cognitive load necessary to select appropriate patterns and you are focused on the wrong thing for coming up with a good design.

But wait, it gets worse!

For example, before dumb and dumber decided to call it the “visitor pattern” every programmer worth his salt would just call it a “map” operation (as in the LISP functions “map”, “mapcar”, “maplist”, etc). That’s just, oh, something like 1994-1960(?) = 34 years of previously established terminology! Twits.

Yup, he just completely missed that the visitor pattern is primarily about double dispatch, and even talks about how CLOS does this natively. It talks about how Smalltalk’s do: (the moral equivalent of map/mapcar/maplist) won’t get you double dispatch support. Most importantly, it even talks about how putting traversing behavior in the Visitor, as done in the sample code, only really makes sense if the logic for determining what to traverse depends on results of the algorithm itself (i.e. when map/mapcar/maplist won’t work).

So, by focusing on the code, rather than the discussion of the problem, it’s various facets, and how they effect possible solutions, one not only misses out on understanding the problem, but also in understanding the solution.

It really is a poor craftsman who blames his tools. The problem is being focused on solutions, and that doesn’t come from Design Patterns, it was there before hand (as demonstrated by The Plank).

In retrospect, I think the real mistake some pattern advocates may have made was in encouraging people who just didn’t get it, or didn’t want to get it, to use patterns anyway. To really mix up some metaphors: once you’ve brought the horse to the fountain of knowledge, if it doesn’t drink, just shoot it and put it out of its misery. If you force it, it’s just going to pretend to drink while secretly gargling, and you’re going to find yourself stuck in the middle of the desert with a dehydrated horse. ;-)