Last week, at our regular Thursday wine & cheese, we had a little celebration. January marked 5 years for me at NuoDB and March 14 was the five year anniversary for my awesome colleague Oleg. It’s pretty amazing to think back on everything we’ve done. In our industry, five years is a long time.
Of course if you go back another five years from March 14, we’re also celebrating 10 years of Amazon Web Services. Understandably, the Internet has been all a-buzz with fond retrospectives. Early participants like Jeff Barr, James Hamilton and Werner Vogels have written some great thoughts on what’s happened since AWS’ first major announcement ten years ago.
Another retrospective? Of a sort, yes.
Ten years ago, I was working at Sun Microsystems. We had spent the last few years pushing hard on this idea called “horizontal scaling,” using open source software along with commodity hardware to focus on scale-out execution instead of big iron servers. It’s also when I started having debates in earnest about whether the word “cloud” was just marketing hype or something deeper. Students of history know we weren’t in time to change the future of Sun as an independent company but we were still on the leading edge of a fundamental transformation.
What else happened in 2006? For one, Google published their “Big Table” paper at OSDI. The project, started in 2004, had been talked about in public for more than a year but that paper sparked a lot of interest as a web-scale service built on the Google File System and Map-Reduce. Inspired a few years earlier by those same papers, 2006 also saw the evolution of an early Apache project into the first version of Hadoop. One year later, in 2007, Amazon added to this list of seminal events with its publication describing Dynamo. But, that’s getting ahead of things.
In 2006 Sun released its Utility Compute Grid (later renamed to include the word “cloud”). This was, as the name suggests, a large cluster of servers available to the public as a utility. It was built to run Solaris applications and priced based on CPU-hour of work. In other words, Sun Grid was focused on users who needed to run traditional, massive, parallel tasks. The thought of having all those cycles available was really exciting, but even from the inside looking out the question a lot of us asked was how you’d get data into a cluster like this and which tasks wouldn’t create a lot of data that at the end you’d want to get back out of the grid.
When Amazon announced S3 it was exactly the opposite. S3 was an obviously valuable way to store large amounts of Internet-accessible data, and an intriguing and exciting new kind of primitive, but who would accept those latencies and want to re-write their applications against a custom API? And where would those applications run that needed so much remote storage? A few months later, EC2 was announced answering that question in many people’s minds.
The thing is, however, what Amazon really did is what they’ve been doing ever since. Rather than taking existing assumptions about data-center design and just charging by the hour, they broke things down into novel, simple primitives that gave users just enough to re-assemble the pieces to solve their new tasks at scale. S3 was a great place to store your data on EC2, but it was just as powerful as a remote service for tasks running anywhere in the world because S3 did just enough: it had a simple API, provided a clear (and default) security model, made monitoring & billing simple and then got out of the way of developers rather than assuming any one end-to-end use of the service. This wasn’t a service for EC2 compute, it was a general service for storage that let us re-think how the storage & service layers were, independently, composed.
Movement to the cloud
In my last few years at Sun I worked on distributed, scale-out systems that separated the in-memory service layer from durability & management of data at-rest. Then I went on to Nokia to help build a private cloud with the same model. Then from there I joined NuoDB five years ago to work on a SQL database with the same design goals at its heart. It actually wasn’t until I got to NuoDB that I shifted from spectator to active participant on AWS, but the way that Amazon has been changing everything about systems design still influenced my work for the last decade.
Today, I spend a lot of my time talking with enterprise architects and development teams that are making the “migration to cloud.” That doesn’t always mean public cloud, but even on-premise what they’re really describing are always (at least) two things: a move to distributed computing models and a need for the commoditized, composible service model like the ones that AWS pioneered in 2006. AWS has obviously changed the landscape by empowering startups and disrupting data center build-out, and I’m sure that a lot of people reading this use AWS in practice. Whether public or on-premise, however, if you do anything in the “cloud” space Amazon should get a lot of the credit for shifting our mindset as an industry away from client-server models and into the distributed architectures that we (mostly) enjoy today.
Is Amazon the only thought-leader that’s made these changes happen? Definitely not, but by actually building the tools for others to explore & grow they’re at the forefront. They’ve also done something that’s often challenging in our industry: iterate service offerings to help users evolve instead of just showing up with brilliant new ideas that aren’t something we understand yet how to use. This evolution has, in a lot of ways, helped people learn what “the cloud” is all about and how to migrate their applications. It’s certainly why traditional enterprises, sometimes slower to evolve their IT models, are now making the move to AWS en mass.
Where do you see this? Going back to S3 as a simple primitive, that was the value. It wasn’t something wildly complex, but it was the right next step to help users start their evolution. I’ll give you two more examples. One is RDS, which has for a long time offered traditional relational databases as managed services. These offerings don’t scale on-demand nor do they have the same “cloud” design that S3, Dynamo and others have, but they’re still core requirements for so many systems. As users who need SQL got comfortable with other elastic services in AWS, Aurora rolled out as something familiar that helped users take that next step. Obviously I’m biased, working on a cloud-scale SQL database designed for the continued evolution of these services, but I believe transactional data management is one of the most important primitives that systems can build on as their core platform. AWS has kept that concept first-class, rather than only offering web-scale key-value services, and in doing so has ensured that the platform as a whole is viable for enterprise migration into cloud architectures.
The second example? At Re:Invent in 2014 Amazon announced Lambda. As a design model Lambda has gained a lot of popularity and eventually the team behind AWS decided it was time to embody this server-less model as a service. The first version was limited in functionality and integration options but as use increased, and as customers started to understand what they could do with it as a new kind of primitive, so too did its functionality and capabilities. I was lucky enough not just to have been at the conference for the announcement in 2014, but to have been on a “CTO Fireside Chat” with Werner himself. When he asked me what announcement got me most excited it was Lambda: not because I could see myself building anything on it immediately but because of all the questions it made me start asking and all the great ways it let me challenge existing “truths” in systems architectures.
It’s been a great five years architecting data services for the enterprises of the world who are moving to cloud. It’s been an even better time because of the way Amazon Web Services has expanded everyone’s view of what we can and should be building. Thank you to everyone at Amazon who has done so much to get us where we are. Here’s to the next ten years of evolution, fun challenges and systems at scales we’re not even thinking about today!