The tl;dr version: Arguably all interesting advances in computer science and software engineering occur when a resource that was previously scarce or expensive becomes cheap and plentiful.

The longer version:

This particular thought was provoked by a series of exchanges on blogs and in Twitter yesterday. It started with a piece at Information Week in which Joe Emison bemoaned the fact that Netflix was holding back progress in cloud computing. The Clouderati jumped all over this, and Adrian put together a detailed response which he also posted to his blog. By the time I got around to responding, IW had closed comments on the original piece, and so I followed up on Adrian’s blog.

Joe’s criticism was based on two points:

Netflix’s cloud architecture[...] is fundamentally (a) so intertwined with AWS as to be essentially inseparable, and (b) significantly behind the best *general* open options for configuration management and orchestration.

Point (a) is pretty silly: Netflix is a business, not a charity. Of course they’re going to work with the best of breed. But it was Joe’s second point that really bugged me. I responded (and here’s where the “Thought for the day” comes in):

Amazon and Netflix are dramatically ahead of the curve, not behind it. The configuration management pattern you seem to prefer – just-in-time customization using Chef or Puppet – was pretty old school when Sun acquired CenterRun and built out N1 and Grid Engine. It’s incredibly inefficient compared with early-bound EBS-backed AMIs.

Arguably all interesting advances in computer science and software engineering occur when a resource that was previously scarce or expensive becomes cheap and plentiful. We’ve seen it with graphical user interfaces, interpreted languages, distributed storage, and SOA. Traditional late-bound configuration management treats machine images and VM instances as expensive; AWS and Netflix invite you to imagine the possibilities if they’re effectively free. Welcome to the real Cloud 2.0…

In a subsequent Twitter exchange, I said:

@adrianco We used to talk about “specific excess MIPS” driving change. Now it’s “specific excess VMs”

… to which Adrian replied:

@geoffarnold with SSD excess IOPS can be used in interesting ways

One Response to “Thought for the day”
  1. Stu says:

    I think the important thing to takeaway from Adrian’s note is “just in time configuration management” is still needed in two cases:

    1. at build time, to assemble a new AMI [how else do you manage package & config dependencies?]
    2. when your app can’t necessarily handle dynamic discovery of new instances “appearing in the cloud” and requires a config management agent for registration/advertisement

    Naturally the second case is the result of an economic tradeoff between continuing with older architectures vs. rebuilding with newer architectures. Both are not inefficient at all, and are good reasons to still have these sorts of agents.

    That said, yes, in my experience Netflix has it right – spinning up a new servers from a snapshot after Puppet/Chef/BladelogicSA has done its work is generally faster and more consistent than doing it from the ground up every time.

    Stu

  2.  
Creative Commons Attribution-ShareAlike 3.0 Unported
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported.