NewSQL and NuoDB
Matthew Aslett, a senior analyst at the 451 Group research firm, just over a month ago single-handedly launched a new database category by coining a great name for it, NewSQL. This is a nice play on the NoSQL name as it is meant to encompass vendors of relational databases who are tackling the thorny issues related to horizontal scalability, elasticity and performance in the cloud. As testament to this new heightened level of innovation in the database market the NewSQL market was quickly split into three sub-categories; storage engines (primarily for MySQL), transparent sharding, and entirely new solutions. A large incumbent RDBMS vendor has already jumped in putting to rest any doubt of the demand for elastic scalable relational databases. The reality is that SQL is so ingrained in everything from the enterprise to the web that if there is a good solution, everyone wants it.
NuoDB/NimbusDB is a distributed relational data management system with support for SQL and ACID transactions. It is based on a new application of multi-version concurrency control and systematic messaging between collaborating peer nodes. NuoDB/NimbusDB fits into the “entirely new” category of NewSQL, we’ve thrown out the design assumptions that come with existing architectures. Most architectures for distributed relational data management start with a common file or files representing the current consistent state, and not surprisingly this is exactly where most RDBMS systems fall short in a distributed environment. Despite great work by academics and commercial companies to find efficient ways (from logical operations to log records) to replicate data using this outdated but de-facto standard model not a single solution has scaled to meet today’s data storage requirements without requiring the database design and functionality be severely limited. This is the reason that NoSQL solutions arrived on the scene. They tossed out the gating assumptions and found that if you’re willing to sacrifice a lot of “must-have” features common in relational databases then you can in fact get the performance required by today’s environment. NimbusDB is also putting aside the standard approach to data management and the limitations that come with it, but we are keeping the requirement for SQL and ACID transactions. To do this we needed to start with a clean slate that reached further into database internals than any MySQL engine would be allowed to go.
What does that look like? Well, it’s impossible to describe in detail in this post, but there are some key things to understand at a high level. The consistent state of a database in NuoDB/NimbusDB is really the state of many collaborative nodes at any given time. These nodes are either processing queries, or archiving data to durable storage. Brokers direct client connections to idle query processing nodes. Those nodes then examine the SQL request, optimize it, and plan its execution given the current state of the collaborating nodes. Then the data required for the query is gathered from peer nodes into the operational context of the transactional node. As operations are performed changes are echoed out to peers in real-time operating optimistically and asynchronously while listening to hear if there are conflicts, constraints or deadlocks to be mitigated. By the time an operation is nearing end all the necessary orchestration for an ACID commit has been pre-negotiated between peers and recorded by one or more collaborating archive nodes in a pending state. The commit is then simply an announcement to and acknowledgement from a coterie of peers that the work should now be considered complete. Because all negotiation and pre-commit work was done piecemeal during query processing the work required at commit is minimal and so the overhead in an OLTP setting is small enough to enable a large cluster to scale transaction throughput as the number of nodes increases.
We’re excited to have a new market and a new approach to database design that addresses the needs of this market. We believe that NuoDB/NimbusDB will allow all those who want SQL for its utility and ubiquity but also want NoSQL for all its ease of use and scalability to find what they want. Simply Scaling SQL, that’s what we’re all about.