1. Automating Terraform workflow with Travis-CI

Automating Terraform workflow with Travis-CI

While DevOps engineers are grasping benefits of Infrastructure as Code, there is still no consensus on how particular tool fits into the whole picture. For example, people compare Ansible, Chef, Puppet and Terraform like they’re somewhat equivalent. Of course, they are not. There is overlap, indeed, between their features, but one has to understand they are solving completely different problems and using completely different approaches.

Ansible uses imperative language, it tells what to do. While Puppet, Chef and Terraform use declarative language – a piece of code in these languages tells what you want to achieve. Code execution of declarative language is idempotent, that’s why it’s more appropriate for environments leveraging Continuous Deployments.

Chef and Puppet are tools to provision a server – to install a package, to start a service. Terraform works on a level below. It’s like Puppet for infrastructure. The Terraform’s job is to create an EC2 instance and the Puppet’s job is to provision it.

There is even more confusion when it comes to organizing workflow of changes with infrastructure tools.

Terraform for example, doesn’t offer mechanisms of sharing and locks the state out of box. It gives the freedom of specific implementation to a user, but how would the user know what’s the right way?

After some time working with Terraform and Chef I developed a way of automating Terraform workflow. Not necessary the right way, or the only way. And definitely not the best way. But the way that works reasonably well. It allows conveniently and safely make changes, fully leveraging Infrastructure as Code principles.

To prove it works I opensourced TwinDB infrastructure repository, so everything I’ll be talking about will be illustrated by the real case. I plan to keep it public from now on, all future improvements will be public, too. The code is published under the Apache Software License 2.0, feel free to reuse it for your needs.

Why automate Terraform workflow

Before jumping into solution details let us formulate the problem first:

  • All infrastructure bits have to be represented in code.
  • All changes to the infrastructure have to be in a form of a git pull request.
  • It has to be possible to do the code review of a suggested change.
  • Whatever implemented in the code has to be continuously deployed.

As I mentioned before, the community Terraform doesn’t offer out-of-box solution for sharing and locking the state. Fortunately, sharing the state is easy, Terraform can store the state in S3 and thus it can be shared. Locking is more difficult. When multiple players are about to make a change there must be some locking mechanism to coordinate the changes. But the locking problem disappears when there is only one changing player. I will delegate this function to Travis-CI.

So the plan is: developers suggest changes via pull requests, their colleagues review and approve the changes and Travis-CI executes the changes.

Workflow

The workflow starts with a human preparing a change. The developer forks a branch “feature-X” from the master branch.

When the change is ready the developer creates a pull request to merge the “feature-X” branch into the master branch.

Travis-CI detects a new pull request, runs “terraform plan” and publishes its result as a comment to the pull request.

Now, another developer reviews the pull request. At the moment of the code review the reviewer has a) test result, b) terraform plan summary and details, and c) the changes itself.

If the change doesn’t look good the reviewer requests changes. The author of the change fixes problems and sends the change for the review again.

If the change looks good the reviewer approves the change and merges the “feature-X” branch into the master.

Travis-CI detects a new commit in the master branch and runs “terraform apply” that reflects the new infrastructure code into the reality.

GitHub configuration

Some time ago I wrote about how to manage GitHub with Terraform. Let’s see how it is configured for our case.

There is a Terraform module twindb-github-repo. It implements a simple repo with one master branch:

The master branch has the branch protection. The branch protection ensures a) nobody can push commits to the branch directly – only via pull requests, b) Travis-CI checks must bless the pull request.

Also, the repository has a webhook to let Travis-CI know about the changes:

Travis-CI configuration

A Travis_CI worker does three things:

  1. Decrypts secrets – AWS keys, GitHub, CloudFlare tokens, Chef certificates and keys.
  2. Installs the “infra-support” Python package (it comes with the infra repo) and terraform client.
  3. Runs a Python script “ci-runner” that in its turn runs “terraform plan” (or “apply” if it’s a master branch) for each Terraform module, parses output and publishes a comment back to GitHub.

This all is defined in the .travis-ci.yml:

Same way as you run puppet agent periodically on a server I configured Travis-CI to periodically run “terraform apply”. That way Travis-CI ensures the real infrastructure matches to what is defined in the infra repository.

Instead of conclusion

So here is a nice and robust Terraform workflow implemented on GitHub with Travis-CI. In future however I will migrate to some hosted CI system (most likely GitLab, but will see). The reason for that is that the CI system should be able to bootstrap EC2 instances without public IP (like a MySQL database). External service like Travis-CI obviously cannot do it.

Meanwhile I encourage you to star/watch the TwinDB infrastructure repo. Eventually I will move all services TwinDB uses to Terraform, so it will be a unique example of how to integrate that altogether.

Previous Post Next Post