You are here

NuoDB Backups and Hotcopy

Every release, NuoDB gets easier to use, and release 2.3 has added a preview feature that promises to make the life of a systems administrator much easier: hot copy.  (The use of preview features is encouraged in development projects but not yet supported in deployments.)

Hot copy uses one of the existing Storage Managers (SM) or Snapshot Storage Managers (SSM) to create a transactionally consistent copy of the archive and journal of a database with no interruption of service.  The database continues running and transactions keep processing, even in a single host database.

In previous versions of the product, backing up a running database and getting a consistent, usable snapshot of the data posed a few challenges.  Your choices were:

  • Create a point-in-time snapshot of your archive using an advanced filesystem like ZFS or BTRFS (If you're running on them)
  • Create a point-in-time snapshot of your whole machine using your virtualization tools (If you have them)
  • Spin up a storage manager to generate a static archive.

Each one has its challenges.  The first two require that you be running on specific infrastructure, whether it's a virtual machine or a filesystem, and not everyone does.  The third poses a whole stack of issues.

Spinning up a dedicated SM to generate an archive from a running database uses a lot of resources and requires some additional logic to make sure the archive is fully synced.  Once it's synced, you need to do a clean shutdown of the SM and then you can point your backup system at the output.  Starting up a dedicated SM uses extra resources as well, consuming CPU, RAM, and disk IO.  I'm not a huge fan of this approach, but I found it necessary to use since I was running on ext3 and ext4.

Enter NuoDB 2.3 and hot copy.

My cron job to pull a backup image has gone from this:

nuomgr () {
    /opt/nuodb/bin/nuodbmgr --user "$DBA_USER" --password "$DBA_PASS" --broker "$HOST" --command "$@"
start_backup_sm () {
  export PID=`nuomgr "start process sm archive $1 database $2 host $HOST waitForRunning true options '--mem 512m' initialize yes" | sed -re 's/^.*?(pid = [0-9]+).*$/\1/' | awk '{ print $3 }'`
mkdir -p $ARCHIVE_DIR
chown -R nuodb:nuodb $ARCHIVE_DIR
for DB in `nuomgr "show domain databases" | grep -v nuodb_system | awk '{print $1}'`; do
  echo "Starting up SM to make backup archive..."
  start_backup_sm $ARCHIVE_DIR/$DB $DB
  echo "Waiting for SYNC to complete for database $DB..."
  while [ `nuomgr "show process status host $HOST pid $PID" | grep -c 'RUNNING'` -ne 1 ]; do
    if [ `nuomgr "show process status host $HOST pid $PID"` = "At line 1: show: Unknown process" ]; then
      echo "SM failed to start for database $DB" && continue 2
    sleep 5
  echo "Shutting down SM pid $PID for database $DB..."
  nuomgr "shutdown process host $HOST pid $PID graceful true allowNonDurable false" > /dev/null
  echo "Saving archive to /var/opt/nuodb/backup-archives/$BACKUP..."
  tar -czpPf $ARCHIVE_DIR/$BACKUP.tar.gz $ARCHIVE_DIR/$DB && rm -rf $ARCHIVE_DIR/$DB/*
  echo "Backup of database $DB complete!"

To this: 

nuomgr () {
    /opt/nuodb/bin/nuodbmgr --user "$DBA_USER" --password "$DBA_PASS" --broker "$HOST" --command "$@"
get_pid () {
  export PID=`nuomgr "show database processes database $DB" | grep SM | grep $HOST | sed -re 's/^.*?(pid = [0-9]+).*$/\1/' | awk '{ print $3 }'`
for DB in `nuomgr "show domain databases" | grep -v nuodb_system | awk '{print $1}'`; do
# get the pid for the SM and make sure the temporary archive destination is there
  echo "Backing up database $DB"
  mkdir -p $ARCHIVE_DIR/$DB
  chown -R nuodb:nuodb $ARCHIVE_DIR/$DB
# Run the hotcopy off of the existing SM
  nuomgr "hotcopy database $DB destinationArchiveDirectory $ARCHIVE_DIR/$DB host $HOST pid $PID"
#  Create the backup archive and clean up the temp directory
  echo "Creating tarball"
  tar -czpPf $ARCHIVE_DIR/$BACKUP.tar.gz $ARCHIVE_DIR/$DB && rm -rf $ARCHIVE_DIR/$DB/* && echo "Backup of database $DB completed."

There are no nested loops, and no funky "wait and see if it's done" logic.  It's just "Run my command" and when it's done, it's done.  It goes from 37 lines of funky script to 26, plus 4 lines of comments.  It makes the whole process of taking a backup of your databases easier, faster, and more reliable, all things sysadmins appreciate.

Add new comment