Learn about how the client-side technologies used to implement the NuoDB Automation Console make a distributed database usable.
(Seth Proctor): Hey everyone, welcome to another edition of “NuoDB Unplugged.” I’m Seth Proctor, and I have with me today Luke.
(Luke): Hi guys.
(Seth Proctor): Luke is an awesome member of our engineering team who works on a lot of what makes it actually possible to use our software. Helps us actually visualize things, see things, and work with what is a very complicated product, and try to actually make it simple. And so, we’re going to be talking today about a lot of things that happen in the browser window, which I admit, like that’s the part of our software I know nothing about, so I’m looking forward to hearing what Luke has to say. We’re coming to you from our fabulous new Cambridge office. If you’re used to dialing into our webinars, you’re used to seeing like a window back here and stuff, that’s because we’ve expanded. We’ve totally almost moved into this new space, it’s pretty exciting. But hopefully the network will work with us. If you’re in the Cambridge area, stop by, hang out, we’d love to see you. So, today Luke is going to talk about the great work we have in the console. Rest interfaces, management components, have some different interesting things. Luke, you want to tell the folks at home a little bit about kind of who you are, how you came to us, before we dive in?
(Luke): Awesome, yeah. So previous to NuoDB, I founded a company called Locally, based in the greater Philadelphia area. Basically we were trying to find a way to create a great, hyperlocal tool to find events in real time. And it was a really great project, and previous to that, I worked at a company called S2. So, really awesome companies worked for, did some really interesting web stuff in both those companies, respectively. So, yeah.
(Seth Proctor): And so, on those projects you were both a creator of kind of that front end technology, and also a consumer of the back end technology?
(Luke): Indeed, yes, I did full stack.
(Seth Proctor): You did full stack, so this is a good context for what we’re doing here today.
(Seth Proctor): So as I mentioned, Luke’s focus here is on our front end tools, both in terms of visually, the user experience of the tools, but then also kind of the stack event structure that gets information from our core database, out into a usable product. And that’s especially important when you think about a distributed system like NuoDB, where we’ve got lots of pieces floating around all over the place, and it can be very hard to work with. And so, Luke’s job is to make sure that it’s not hard to work with. And that comes in a couple of flavors, that’s tools, that’s some work we’re doing around automation. And that’s rest interfaces and kind of APIs and other languages to help work with that. Luke, why don’t you tell us a little bit about kind of what the admin console is, why we’re building this, and then why don’t you give us an example?
(Luke): Sure, sure. So basically, if you think about NuoDB, NuoDB’s a distributed database. And with that, there’s a lot of complexity in that. So, the goal behind the automation console is really to create a very easy to use web app that can come in, log in with their credentials, they can create a managed database instantly. This is built on a rest interface that, as Seth indicated, that really makes provisioning these databases extremely simple.
(Seth Proctor): So, do you want to show us, just walk us through a little bit?
(Seth Proctor): I’ve got to hit the share my screen button here. Through the awesome power of technology, we can show everyone at home. Yeah, go for it.
(Luke): Great, here we go. So hopefully everybody can see this. But here we have our automation console loaded. And we’re presented with the dashboard here. And here, you have a lot of widgets that we created, that we thought were useful, they’re configurable, you can see different things happening on the system at any point in time. Right now we have three databases running, we have a load running against these databases, and it’s really simple to see and monitor what’s going on.
(Seth Proctor): And so, when we’re looking at it, like those -- I see host, CBU host memory and also commits per second, what we’re seeing here is we’re seeing information both about the databases that are running, and also about the hosts, the Linux instances in this case that are actually supporting the database, right?
(Luke): Right, indeed, yes. And, you know, we want to not only just focus on the database itself, but the domain and the hosts that are actually running these systems as well. So give you a good perspective of what’s going on in your domain.
(Seth Proctor): Right. So this is -- you’re saying this is kind of a simple visualization for being able to see things broken down by database, by host, by physical region where things are running, and then kind of correlate kind of systems administration, detailed with a kind of DBA level detail. So this is kind of a one-stop shop for operations.
(Luke): Yes, indeed. And again, like I said it’s configurable to whatever you need. So you can add different widgets here as you need to. So it’s completely customizable to what your use case might be.
(Seth Proctor): Neat. And just to be clear, like so this is also something then that lets you, if I want to drill down, if I’m looking at things and everything looks great, and suddenly, you know, one of those databases doesn’t seem to be behaving the way I expect,.
(Luke): Yeah, you can just kind of drill into the correct database that has some interest. Like this Facebook database, and you can inspect and see what’s going on. So Facebook is running one region, maybe it was supposed to be running two regions, who knows? You know, the processes that you can inspect, so we took a top down approach to this, and we’re trying to hide the details of a particular database and just really show you, we want to roll this (inaudible) database in a very simple view. But you can easily drill down and do the triage, what’s going on.
(Seth Proctor): Right, right. So this is kind of -- you can see things at a high level of your multi-tenant deployment, or a large scale distributed system. What you want is, you want kind of high level metrics, is everything running well? And when it’s not, you can kind of quickly identify where the issues might be, and then you can see, I see on the bottom here (inaudible) got the ability to see SQL queries, you can see commit rates, you can see where processes are running, you kind of ask the right questions about what’s happening.
(Luke): Indeed, so, yes.
(Seth Proctor): Nice. And so for folks who are following along at home, you can download the software. This is, we’ve got information on our docs, and in our tech blog about how to enable a system basically that has our full stack of kind of these preview features, so you can just start running and you can start exploring. So, feel free to try it out, and feel free to send us some questions if you’re trying along at home, and you’re curious about how to do some things.
(Luke): Yeah, yeah. It’s really exciting, really cool technologies. So, maybe we just want to talk about the technologies that we’re using for this.
(Seth Proctor): Yeah, for sure. So why don’t we go back to the... There we go, and we’re back.
(Seth Proctor): So this is like those widgets, for example, we’re seeing on the (overlapping dialogue; inaudible). You can modularize those, we can take them and drop them in different places, but then you can also take them into isolation and test them, so you can really know that that individual piece works really well.
(Luke): Yeah. The widget, even the whole app itself, everything you see on the client side, you know, from the log-in to every interaction, you can actually really try that. You can script against it, you can test it. Which is great, you know, this is, it makes the code much more maintainable and usable, so.
(Seth Proctor): And so, you mentioned CoffeeScript is one of the things in there.
(Seth Proctor): So, talk a little bit about -- because this was something I had to learn about. So, talk to the folks on the call about what that is.
(Seth Proctor): And what is it, I mean --
(Seth Proctor): So this is kind of an abstraction --
(Seth Proctor): Very cool.
(Seth Proctor): So we’ve got a couple questions already coming in. And I’m going to get to one of them in a minute. But one of them deals with the rest interface that you’re building here. So, part of what we’ve done, right, is we’ve got this pretty browser that you’re building. And it’s got all this CoffeeScript and other modern technologies embedded in the browser. But it’s talking over essentially a common API that we’re building for doing management, doing monitoring, kind of doing administration on top of the systems. Do you want to talk a little bit about kind of what we’re building? What we’re trying to provide?
(Luke): Sure, sure. So basically, you know, the rest interface, this is like the, you know, (inaudible) API, that’s the interface that people can interact with, is extremely beneficial, for a number of reasons. It’s language-agnostic. All it requires is an HTTP client to connect to two of the -- put posts at various end points, and you can grab information. So, it provides a very simple way to interact with the NuoDB system without having to have a language that might be specific to it. So with our (inaudible) we’re basically wrapping a lot of our internal entity logic to then provide that in an easy to use format for people to consume on the client.
(Seth Proctor): So this is like, if anyone watching has used our command line tools, or has used previous browsers, you’ll see kind of [primitives?] around being able to query for a set of provision hosts, where processes are running, drill down into each process and see it (inaudible) get statistics associated with a given host, or a given process, and then you have the ability to start processes to change configuration. And so, that’s kind of part of what’s in this API, right?
(Luke): Yeah, yeah.
(Seth Proctor): And so, this is something that we’ll be creating not just for our tools, but we’ll be exposing the next kind of revision of the software as a state (inaudible) right?
(Luke): Indeed, yes.
(Seth Proctor): And, one of the things it uses is also the reason that we saw a demo called the auto consoles for automation. It can build on something we call templates. Do you want to talk a little bit about how you used that asset?
(Luke): OK, yeah. So I think we kind of alluded to it. But this whole notion of a managed database is, so we have these templates, and these templates are kind of policies that govern how the database is going to be run on the NuoDB system. So, previous to what we had to do, we had unmanaged databases where the user was required to create processes, transaction engines, storage (inaudible) on the post, and basically handled that. But well, this derivative is really a managed approach where the policy will actually govern how the database is going to be constructed. So, as a really simple example, you know, if you have a domain, US East, US West, and you have a database that scales across those, with a policy that says have a transaction engine on US -- have a transaction and storage manager in US East and in US West, the main database will monitor those processes and make sure that that database meets that criteria. If anything were to happen to one of the processes, it will restart the process (inaudible).
(Seth Proctor): So from the rest point of view, I could have a template that declared a description of one of these SLAs. What is it I need? And it’s a single rest call, right? To just say essentially, start a database that is started against that requirement.
(Seth Proctor): Right, the single -- what is it, (overlapping dialogue; inaudible)?
(Luke): -- that database’s end point and you just, they have a database provision on those.
(Seth Proctor): So that’s pretty cool. And I got those templates into the system, because we shipped with some, but then there’s also a rest call to --
(Luke): Yeah, there’s a rest call (overlapping dialogue; inaudible) create a new template that, you know, the user defines. (overlapping dialogue; inaudible).
(Seth Proctor): So as a user, I can define a requirement that said I’ve got, as you said, two regions running, so the East Coast and the West Coast, sorry the middle of the country, not that we’re leaving you out, it’s just an easy example. So I’ve got the East Coast and the West Coast, and I’ve got a template, a definition that says I want to run kind of a minimally redundant state, so that there are multiple processes on both coasts, so there’s a full copy of my data on both coasts.
(Luke): Indeed, yes.
(Seth Proctor): So you can describe that in this thing, you can install that template through one rest call. And through a second rest call, you can say start a database that meets that requirement that we start it up, and then we’ll monitor it.
(Luke): Indeed. Indeed, yeah.
(Seth Proctor): So that’s, I mean I think that’s pretty cool.
(Luke): I think that’s really cool. That’s really (overlapping dialogue; inaudible).
(Seth Proctor): It means, yeah, it means you can do a lot with very few calls. And then the rest API then also lets me monitor statistics associated with this, right?
(Luke): Yes. Yes indeed, yeah. So we provide a number of statistics. We’re only exposing about a dozen in the front end, but we have over 100 right now that can be monitored on the client side, that we will expose eventually. And basically we have an end point that is restful, and you can just kind of query for this service. In the future, we’re actually going to be providing a soft API, which (inaudible) which is going to be push driven, so that as events occur on our -- are (inaudible) by that service side, we can push it to our client. And it will be a much more efficient way of monitoring. So typically for one (inaudible) solutions, for real-time, if there’s a real-time component on a web app, it’s better to have the server side push technologies, like [Socket?] I/O or (inaudible). So.
(Seth Proctor): So this is like, I’ve got software distributed across several regions, lots of posts, lots of databases running.
(Luke): Lots of databases running.
(Seth Proctor): And I’m keeping track of commit rates, and client connections, and bike counts, and I’m keeping track of CPU and memory and watching disc usage. I’ve got all these different steps floating around, and I want to write (inaudible) single client that sits there and says, you know, I don’t care where all the information is coming from, get it to me in a secure fashion so I can do stuff with it. That kind of push interface, that web stock interface, (overlapping dialogue; inaudible).
(Luke): That would make it simpler. And it would make it more scalable. Because, you know, right now at the current incarnation where we’re doing a pulley mechanism, but soon we’re going to have a socket API that, you know, there’s push in it. So, it’s less, you know, less overhead in terms of, you know, opening connections, sending the (inaudible), you know, if there’s an SSL, terminate the SSL. So these are things that slow down and inhibit the performance of a web app when, you know, in case the socket (inaudible).
(Seth Proctor): All right, cool. Well, so we’ve got a couple questions, and since we’re talking about kind of collecting statistics anyway, we have a question from Natalya, who asks which metrics do you think are most important to track with the automation console? So you mentioned that internally in our system we track just a pile of statistics, but for kind of the first version of the web and kind of visualization, or basically we picked a few, what have we picked, and why?
(Luke): I think the most important ones that we care about, probably arguably the most important one is commits. And that’s just because, that just kind of gives us what’s happening on the system right now. That gives you an instant feedback as to what’s going on, you know, is this database active or not? You know, what’s going on in that respect. We also have rollbacks, and then a host of other metrics that we care about, like CPU, memory, network in, (inaudible) coming in on the network, going out on the network. So, these are all metrics that we provide that provide some use cases. But, I’d say that the commits are probably the most (inaudible).
(Seth Proctor): So these are kind of -- it sounds to me, these are kind of statistics that they’re very high level, give you a pulse on both the database and the provision (overlapping dialogue; inaudible). They kind of tell you something about the rate at which things are working, but they also tell you something about kind of the health. Are you overprovisioned or underprovisioned?
(Luke): Indeed, indeed, yes.
(Seth Proctor): Very cool. And as Luke says, there’s a lot of data that we keep track of internally that’s necessary to be able to kind of evolve the product, it’s necessary for the automation work we’re doing. So there’s a lot of data that we do track. Part of what we’re trying to do with this higher level is figure out how to make that information easier to work with. Whether it’s through these kinds of raw statistics, whether it’s in current events, you know, information, looking at several statistics, and deciding that certain events have happened. So that’s kind of all stuff that we’re working on now, if folks out there kind of are interested in being able to ask specific kinds of questions of a running database, we’d love to hear, we’d love to know. So thanks Natalya, great question. Another question we got here, kind of on the question of the rest interface, Riley asks, can you tell me about some of the challenges you’re facing while building this rest API? So what are some of the, you know, I mean everyone does rest APIs, but everyone does it differently. (overlapping dialogue; inaudible).
(Luke): Yeah. So, previously coming from like a node Python background, it was a little harder to kind of initially come up, come over to the Java side of the world and kind of get that up and running. So I think most of the challenges for me specific to getting the rest API, the biggest challenge for me was for just understanding how to get the Java thing working. So, I came across this library called Drop Wizard, which is actually a really, really awesome library, so this is what we’re using to kind of build our Java rest API in. And so, but outside of that, I’m not -- there’s not too many more specifics in terms of challenges that I’d say. I think it all just comes from getting it up and running.
(Seth Proctor): So this is, in a lot of ways, you think kind of a pretty standard, like if someone came to this rest interface, they wouldn’t say oh, there’s something kind of crazy, new, and different we’re doing here, this is --
(Luke): No. No. No, I mean if they’re looking at the actual code, I think it’s pretty standard, what we’re doing. Industry standard, so.
(Seth Proctor): OK. Now you mentioned -- thanks Riley, good question. So Luke, you mentioned that kind of your company (inaudible) doing this in Java in part because that’s kind of a common stack we’re using --
(Seth Proctor): -- for other tools. But you mentioned you came I think kind of through a previous background of Python and node.
(Seth Proctor): So those are actually technologies you also worked on here, right?
(Seth Proctor): So we have a node driver, we have a Python driver, we’ve done Django integration. Do you want to talk a little bit about kind of those parts of the system as well? And kind of how they relate?
(Seth Proctor): Because those are like, some of the functionality we’ve been talking about is available there as well. So if a user likes a rest interface, great. If the user prefers to do something from a Python point of view, also good. And then as you’re writing around locations, we get drivers at the second level, as well.
(Luke): Yeah, yeah. So I mean, it’s great to, you know, the benefit of having drivers, it really gives you the flexibility to speak in whatever language that you want. So, you know, we want to have -- we have drivers in all the really popular languages, but the ones that I focused on was the node.js driver, the Python driver, and the SQL (inaudible) and Django driver. So, each one of those provides an easy to use way to (inaudible) NuoDB. The Django driver in specific is kind of abstract in a way, so you know you can kind of get it up and running using NuoDB with Django very simply. It’s nothing like, there’s no magic there. But the NuoDB Python driver, you know, is using our SQL protocol, it’s actually talking our buddy code to our NuoDB systems that you can actually do queries, inserts, etc., with that driver.
(Seth Proctor): So that’s a native Python driver (overlapping dialogue; inaudible).
(Luke): It’s not layered on anything. Unlike, well our node.js driver is actually leveraging our C library to interact with our NuoDB system, which isn’t necessarily talking (inaudible) SQL protocol.
(Seth Proctor): And those both speak kind of standard interfaces, is that right?
(Luke): They speak standard, yes, they do, yeah. Yeah.
(Seth Proctor): Yeah, very cool. And from the Python point of view, I can do SQL but I can also do kind of some of the same operations we were talking about for like, what’s in those rest interfaces, right?
(Luke): Indeed, yeah. You can wrap, you know, actually the original incarnation of the rest API that we have today in the automation console wasn’t built in Python. Wrapping our Python driver, which is talking to the SQL protocol. So yeah, you can wrap a lot of the drivers, and actually provide a nice interface, whether it’s the Python node, or Ruby, for that matter, and provide the same information to (overlapping dialogue; inaudible).
(Seth Proctor): Right. And just so folks at home know, what’s interesting about the Python implementation is it has both that SQL driver, but it also has a whole management API. So all the things we’re talking about for rest’s point of view, the ability to do process management, work with templates, and kind of do provisioning and do data monitoring and all these other things, are available there as well. So, that’s kind of a one stop, if you want to be able to do kind of both, you know, write your application, do DBA work, and do kind of systems administration managing and monitoring, that’s all available kind of in those pipeline APIs. And both of those projects are (inaudible) right? So (overlapping dialogue; inaudible) --
(Luke): Yeah, I mean we --
(Seth Proctor): -- get started with them.
(Luke): Yeah, and definitely try it out, and they’re out there, so, you know, (inaudible).
(Seth Proctor): Very cool. So, you’ve been working hard on a bunch of different tools on that console front end, on the rest interfaces. As you look forward kind of, as we start to get all of this into the stable version of the product out there, are there kind of directions that you’re particularly excited about where we’re heading, in terms of the usability of the products?
(Luke): Indeed. Yeah, I think that, you know, we’re going to make it much -- we’re going to focus on this managed approach that we’re -- in the automation console. We’re going to make that a focus, but the thing that’s really (inaudible) is not only the socket technology, but this whole kind of inventing service is going to be coming about in (inaudible) release, and it’s going to facilitate just more ease of use in the system. Easier to manage the system, more -- to take the burden off the user having to manage the system, so. Yeah, that’s (inaudible).
(Seth Proctor): Cool. Very cool. Well Luke, thank you for joining us here today. I hope folks out there enjoyed this conversation. If you want more, anything more about what it is we’re building, the tools, the rest interfaces, if you’re really dying to dig in, please download the software, give it a shot, feel free to send us emails, to jump onto the forums, to find us on Twitter, kind of wherever it is that you want to ask us about it. We’d love to -- we’d really love to hear feedback. This is part of the product that we’re intentionally evolving based on customer experience. So we want to make sure we’re solving your problems. Luke here would love to hear feedback about how people are using it, and how we can keep building out new systems. So until next time, thanks again for watching “NuoDB Unplugged,” and please send us suggestions you have for our next topic. And it’s been fun. Talk to y’all later, bye.