You are here

Running NuoDB in Docker - Part II

A demonstration of Scale-out, Continuous Availability, and Visual Monitoring

RUNNING SQL TRANSACTIONS AT SCALE AND WITH CONTINUOUS AVAILABILITY

In a blog post earlier this year I outlined how to deploy the NuoDB database in Docker containers. To follow up on that, let's put NuoDB to the test. Your mileage will vary based on the available compute and disk storage used, but our Community Edition on a single host will demonstrate NuoDB's OLTP performance capabilities. If you’d like a full demonstration or want to conduct your own proof of concept test in your environment using our Enterprise Edition, just contact us

For this demonstration using the NuoDB Community Edition (CE), follow the instructions in my previous (part I) blog post to start your NuoDB database in Docker on a single host with one NuoDB Admin process, one NuoDB Storage Manager (SM), and one NuoDB Transaction Engine (TE). Confirm your database is running by running the following command:

$ docker exec -it nuoadmin1 nuocmd show domain
 
server version: 4.0-1-b947d0fa06, server license: Community
server time: 2019-09-11T20:01:16.959, client token: 7d1101146e2dda772ca6dddbdbad8808e60e167a
Servers:
  [nuoadmin1] nuoadmin1:48005 (LEADER, Leader=nuoadmin1) ACTIVE:Connected *
Databases:
  test [RUNNING]
    [SM] test-sm-1/172.18.0.3:48006 (Default) [sid = 0] [server = nuoadmin1] MONITORED:RUNNING
    [TE] test-te-1/172.18.0.4:48006 (Default) [sid = 1] [server = nuoadmin1] MONITORED:RUNNING

Next, we'll need to install: 

  1. A visual monitoring tool. For monitoring, we’ll use our own NuoDB Insights.
  2. A SQL workload generator. For a SQL workload generator, we’ll use YCSB, the Yahoo Cloud Serving Benchmark tool. 

Step 1: Install NuoDB Insights

The use of NuoDB Insights is optional, but highly recommended. The tool will collect time series performance metric data and send it to the NuoDB Insights portal running in the AWS public cloud. The data collected will be maintained privately and is only accessible by using your unique Subscriber ID, which will be provided to you during the install process. To opt in and install NuoDB Insights, run the following commands:

$ docker exec -it nuoadmin1 nuoca register insights --insights-url https://insights.nuodb.com/api/1 --enable-insights
$ docker exec -dit nuoadmin1 nuoca start nuoca --insights

Note: The output display will include your personal and private link (with Subscriber ID) to your database performance monitoring data. For example, the output display after starting Insights will look like this:

Insights Subscriber ID: YJ2JD33240
 
NuoDB Insights is now enabled. To access your personalized dashboard, visit: https://insights.nuodb.com/YJ2JD33240/
 
Instructions for customers running legacy Admin (nuoagent)
==========================================================
If the nuoagent service is running, NuoDB Insights metrics collection will automatically begin within 5 minutes.  If the nuoagent service is not running, or to start collecting immediately, you can (re)start the nuoagent with the command: 
 
  "/opt/nuodb/etc/nuoagent" restart
 
Instructions for customers running NuoDB Admin (nuoadmin)
==========================================================
If you are running nuoadmin with TLS enabled (which is the default) you must manually provision a TLS key for NuoDB Insights and copy it into /etc/nuodb/keys/nuodb_insights.pem'. NuoDB Insights metrics collection will automatically begin within 5 minutes.
Insights Registered. Subscriber Id YJ2JD33240

Note: If you prefer to use NuoDB Insights installed directly on your host machine rather than connecting to the NuoDB Insights portal hosted on AWS, just contact NuoDB support at support@nuodb.com to obtain the installation instructions.

Step 2: Install and start the YCSB SQL workload benchmark tool

The first command will take 20 seconds or so to pull the image into your local Docker repo.

$ docker pull nuodb/ycsb:latest
$ docker run -dit --name ycsb-1 \
            --hostname ycsb \
            --network nuodb-net \
             --env PEER_ADDRESS=nuoadmin1 \
             --env DB_NAME=test \
             --env DB_USER=dba \
             --env DB_PASSWORD=goalie \
            nuodb/ycsb:latest /driver/startup.sh

By default, the YCSB container will create 10 connections to your database and each connection will run 10K SQL statements with a mix of 95% reads and 5% updates, then disconnect, and start a new connection, and so on. 

Scaling Transactional Throughput

In this example below, on a Macbook with a single SSD drive, a single NuoDB TE and a single YCSB app running, NuoDB is processing 4K SQL transactions per second (TPS) and the TE is running at about 40% CPU as reported by NuoDB Insights.

NuoDB Insights on a MacBook

Next, by following the previous examples for starting NuoDB TEs and YCSB sample SQL apps, add one more TE and one more YCSB app (which will double the SQL workload). Here we learn that our single machine is actually disk I/O bound and with one TE we have already reached our max SQL TPS of 4K for this configuration, but Insights shows the two TEs are now equally processing the YCSB SQL workload, 2000 SQL TPS each.

Balancing the YCSB workload

We have run the same benchmark in cloud environments in a single AWS region across two availability zones using EBS storage and scaled up both the YCSB sample apps and the NuoDB TEs to achieve up to 50K TPS sustained!

Note: In these examples each SQL transaction is a single SQL statement.

Continuous Availability

Finally, to show NuoDB's resilience to failure events and NuoDB's continuous availability, go ahead and remove a running transaction engine (TE) container to simulate a hardware, software, or network failure event. It's ok, you still have one remaining TE! Pick any one and the SQL load will redistribute to the one remaining TE. This type of resilience is also available for the Admin Services and Storage Managers in the NuoDB Enterprise Edition. To remove a TE engine, just type this command at the prompt:

$ docker rm -f test-te-1

Notice in the screenshot below that the TE count has updated to one and the lower left panel shows the remaining one TE still processing SQL transactions. The lower center panel shows that the aggregate TPS remained the same and the application keeps running even though NuoDB lost one of its TEs.

TPS Remains the same despite losing a transaction engine

For improved resiliency and fault-tolerance, we recommend running NuoDB in container management platforms, such as Red Hat OpenShift or any managed Kubernetes platform like GKE, AKS, EKS, etc. In these systems, process orchestration will manage the desired number of each process type. Thus, if a process is lost for any reason, the container management system will restart the process in just a few seconds! For more information about running NuoDB in a container orchestration system, check out this blog, follow our step-by-step instructions to try it yourself, or reach out to us.

-------

Stay tuned for our next Docker blog installment where we will demonstrate how easy it is to setup a universal GUI database management and analysis tool (DBVisualizer) that simplifies running SQL on NuoDB — the ideal SQL workbench tool for developers and DBAs when using NuoDB!

 

Add new comment