The tl;dr version: Arguably all interesting advances in computer science and software engineering occur when a resource that was previously scarce or expensive becomes cheap and plentiful.
The longer version:
This particular thought was provoked by a series of exchanges on blogs and in Twitter yesterday. It started with a piece at Information Week in which Joe Emison bemoaned the fact that Netflix was holding back progress in cloud computing. The Clouderati jumped all over this, and Adrian put together a detailed response which he also posted to his blog. By the time I got around to responding, IW had closed comments on the original piece, and so I followed up on Adrian’s blog.
Joe’s criticism was based on two points:
Netflix’s cloud architecture[…] is fundamentally (a) so intertwined with AWS as to be essentially inseparable, and (b) significantly behind the best *general* open options for configuration management and orchestration.
Point (a) is pretty silly: Netflix is a business, not a charity. Of course they’re going to work with the best of breed. But it was Joe’s second point that really bugged me. I responded (and here’s where the “Thought for the day” comes in):
Amazon and Netflix are dramatically ahead of the curve, not behind it. The configuration management pattern you seem to prefer – just-in-time customization using Chef or Puppet – was pretty old school when Sun acquired CenterRun and built out N1 and Grid Engine. It’s incredibly inefficient compared with early-bound EBS-backed AMIs.
Arguably all interesting advances in computer science and software engineering occur when a resource that was previously scarce or expensive becomes cheap and plentiful. We’ve seen it with graphical user interfaces, interpreted languages, distributed storage, and SOA. Traditional late-bound configuration management treats machine images and VM instances as expensive; AWS and Netflix invite you to imagine the possibilities if they’re effectively free. Welcome to the real Cloud 2.0…
In a subsequent Twitter exchange, I said:
@adrianco We used to talk about “specific excess MIPS” driving change. Now it’s “specific excess VMs”
… to which Adrian replied:
@geoffarnold with SSD excess IOPS can be used in interesting ways