1. TwinDB Backup Tool Release 2.15.7

TwinDB Backup Tool Release 2.15.7

Some time ago, our team released a new version of TwinDB Backup Tool that contains minor fixed and some major improvements. In this article I want to highlight major features in this release.

Status refactoring

The tool stores backups metadata in a JSON file that I will refer further as the status. The status is stored on a destination along with backup copies and it contains necessary information to restore from incremental copies, for example. Sometimes during the network transfer the status file had been silently corrupt. That led to bugs like #87. In the 2.15.7  release we added a status integrity check. From now on the status is protected by a checksum. It helps to identify the status integrity and inform a user if something goes wrong. In future releases we plan to enforce stricter consistency checks, add crash recovery, and ensure the status is updated by an atomic write.

Xtrabackup tool included in twindb-backup package

TwinDB Backup Tool uses Xtrabackup for MySQL backups. It was declared as an external dependency in our twindb-backup package. There was a problem though. The percona-xtrabackup package installs version 2.3 on CentOS and Debian/Ubuntu, however we needed the 2.4+ version because it supports MySQL 5.7. The 2.4 version comes in a package percona-xtrabackup-24 which conflicts with percona-xtrabackup. Same time, some other packages (for example Percona XtraDB Cluster) depend on the old percona-xtrabackup, so it was kind of impossible to cleanly install everything.

Another unnecessary step was a requirement to have two repos – TwinDB’s and Percona’s – to install twindb-backup. That wasn’t very much user friendly and we fixed that by shipping Percona Xtrabackup with the twindb-backup package.

TwinDB Backup Tool is built using Omnibus. The idea was to build and include Xtrabackup into the Omnibus package and declare Xtrabackup’s dependencies as the external dependencies of the Omnibus package. The xtrabackup software is available in our omnibus-software repository:

name "xtrabackup"
default_version "2.4.9"

skip_transitive_dependency_licensing true

source url: "https://s3.amazonaws.com/twindb-release/percona-xtrabackup-2.4.9.tar.gz",
       md5: "fca658cb3b004d2a2df342a136125f84"

relative_path "percona-xtrabackup-2.4.9"
whitelist_file /.*/

workers = 1

build do
    env = with_standard_compiler_flags(with_embedded_path)
    command "cmake -DBUILD_CONFIG=xtrabackup_release " \
          "-DWITH_MAN_PAGES=OFF -DDOWNLOAD_BOOST=1 " \
          "-DWITH_BOOST=#{install_dir}/libboost " \
          "-DWITH_SSL=system " \
          "-DINSTALL_BINDIR=#{install_dir}/embedded/bin", env: env

    make "-j #{workers}", env: env
    make "install", env: env
end

Please note we build Xtrabackup in one thread – sometimes there were intermittent compiler errors with default settings.
The next step is to add xtrabackup as a dependency of the twindb-backup package:

#
# Copyright 2016 TwinDB LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# require "./lib/ostools.rb"

name 'twindb-backup'
maintainer 'TwinDB Packager (TwinDB packager key) <packager@twindb.com>'
homepage 'https://twindb.com'

# and /opt/twindb-backup on all other platforms
install_dir '/opt/twindb-backup'

build_version '2.15.7'

build_iteration 1

description 'Backup and recovery tool for MySQL
 TwinDB Backup tool for files, MySQL et al.'

# ------------------------------------
# Generic package information
# ------------------------------------

# .deb specific flags
package :deb do
    vendor 'TwinDB Packager (TwinDB packager key) <packager@twindb.com>'
    license 'Apache License Version 2.0'
    section 'database'
    priority 'optional'
end

# .rpm specific flags
package :rpm do
    vendor 'TwinDB Packager (TwinDB packager key) <packager@twindb.com>'
    dist_tag ''
    license 'Apache Software License 2.0'
    category 'Applications/Databases'
    priority 'extra'
    if ENV.has_key?('RPM_SIGNING_PASSPHRASE') and not ENV['RPM_SIGNING_PASSPHRASE'].empty?
        signing_passphrase "#{ENV['RPM_SIGNING_PASSPHRASE']}"
    end
end

# ------------------------------------
# OS specific DSLs and dependencies
# ------------------------------------

# Creates required build directories
dependency 'preparation'

# twindb-backup dependencies/components

dependency 'python'
dependency 'pip'
dependency 'libffi'
runtime_dependency 'libtool'
runtime_dependency 'nmap'
dependency 'xtrabackup'

# Debian
if debian?
    dependency 'libyaml'
    runtime_dependency 'openssh-client'
    runtime_dependency 'cron'
    runtime_dependency 'libev-dev'
    runtime_dependency 'libgcrypt11-dev'
    runtime_dependency 'libaio-dev'
end

# RHEL/CentOS
if redhat?
    runtime_dependency 'openssh-clients'
    runtime_dependency 'initscripts'
    runtime_dependency 'cronie'
    runtime_dependency 'logrotate'
    runtime_dependency 'libaio-devel'
    runtime_dependency 'libgcrypt-devel'
    runtime_dependency 'libev-devel'
    runtime_dependency 'libcurl-devel'
end

# twindb-backup
dependency 'twindb-backup'

# Version manifest file
dependency 'version-manifest'

exclude '**/.git'
exclude '**/bundler/git'

While this approach solved the problems there is a drawback, too. The package obviously got bigger, now it’s 180MB and when installed TwinDB Backup uses almost a gigabyte of disk space in /opt. It is definitely something to consider when provision a backup server.

Previous Post Next Post