1. Building RPM On Travis-CI In Docker Containers

Building RPM On Travis-CI In Docker Containers

monument

Travis-CI is a crucial component of Continuous Integration/Continuous Deployment. We use it a lot to run unit tests and build/upload Python modules.

Recently I had to solve a problem of building RPMs on Travis-CI with Docker containers. In this post I will describe step-by-step how to do that.

We distribute our backup tool as RPM packages for CentOS 6 and 7. But Travis-CI slaves run Ubuntu trusty. In theory, I could use a tool like fpm and build RPMs on Ubuntu (never tried that, but should be possible). However, I like to have a .spec file. It gives me full flexibility on how RPM behaves, and it’s easier to maintain it.

Travis-CI supports Docker containers, so I decided to go that way.

At a high level the process is following. A Travis-CI slave starts a docker daemon. A script starts a docker container, and in the container it starts a build job. How do RPMs get onto the host machine (=travis-ci)? The container mounts a volume where the build script puts the packages. So, after the build is done the RPMs are available on the travis-ci slave.

travis-ci-runs-container

All travis-ci configuration is done in the .travis.yml file, it’s stored in a root directory of a source code repo.

screenshot-2016-12-09-20-40-23

Enable Docker

By default, a travis-ci slave doesn’t run the docker service. It has to be explicitly enabled.

$ cat .travis.yml
...

services:
  - docker
...

Running Centos In Container

To run two instances of travis-ci slaves, we need to define an environment matrix.

$ cat .travis.yml
...

env:
  matrix:
    - OS_TYPE=centos OS_VERSION=6
    - OS_TYPE=centos OS_VERSION=7
...

OS_VERSION=6 means that there will be an environment variable OS_VERSION with the value 6 on a travis-ci slave. We will use it to decide which CentOS to run.

The docker container is started by a script. The script also does all the preparations of the container for the build, installs required packages, and builds the RPMs.

$ cat .travis.yml
...

script:
  - make docker-rpm
...

Let’s see what make docker-rpm does.

$ cat Makefile

docker-rpm: ## Build rpm in a docker container
        @sudo docker run \
            -v `pwd`:/twindb-backup:rw centos:centos${OS_VERSION} \
            /bin/bash /twindb-backup/support/bootstrap-docker.sh

So, the script starts a container and runs bootstrap-docker.sh in it.

bootstrap-docker.sh‘s tasks are:

  • Installing required packages
  • Starting MySQL (we need it for integration tests)
  • Running the tests
  • Building the RPM

UPDATE: Things are moving fast. We build RPM/DEB packages with Omnibus now. However, here’s the bootstrap-docker.sh content.

#!/usr/bin/env bash

set -eux
yum -y install epel-release
yum -y install http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm


PACKAGES="
gcc
python-devel
zlib-devel
openssl-devel
rpm-build
make
python-setuptools
python-pip
Percona-Server-server-56
Percona-Server-devel-56
percona-xtrabackup
"
for i in $(seq 5)
do
    yum -y install ${PACKAGES} && break
done

mysql_install_db && mysqld --user=root &

timeout=300
while [ ${timeout} -gt 0 ] ; do mysqladmin ping && break; sleep 1; timeout=$((${timeout} - 1)); done

cp -Rv /twindb-backup /tmp/
pip install /tmp/twindb-backup

make -C /tmp/twindb-backup test test-integration rpm

cp -R /tmp/twindb-backup/build /twindb-backup/

Once the build script successfully finishes, the next and last steps are to upload packages.

Upload RPM to PackageCloud

We use PackageCloud as our YUM repo provider.
One and decisive reason why we went with PackageCloud is that it integrates very well with Travis-CI.

The relevant part of .travis-ci.yaml goes next.

deploy:
  provider: packagecloud
  repository: main
  username: twindb
  token:
    secure: ***
  dist: el/${OS_VERSION}
  skip_cleanup: true
  local-dir: build/rpmbuild

This means that Travis-CI will look for packages in build/rpmbuild, will upload everything it finds to a repo called main under el/${OS_VERSION}, that belongs to the PackageCloud user twindb and will use an API key from a encrypted secure string.

This way, Travis-CI automates all CI/CD steps starting from pulling new source code revision, to running tests, building packages, and landing them into a packages repository.

Previous Post Next Post