1. How we manage WordPress website with Terraform, Chef and GitHub

How we manage WordPress website with Terraform, Chef and GitHub

water drop falling

Our website twindb.com is built on WordPress software and has always been. A while ago we decided the website needs a better look. But not only that. Managing our old website was quite laborious and manual process which goes strongly against our culture at TwinDB to do things right. Few weeks ago we migrated to the new website and I’m so proud to share with the community how we provision a fully automated WordPress website with Terraform, Chef and GitHub.

Requirements

We run our stuff in AWS. We wanted to have better control of our AWS infrastructure. Most of old infrastructure was created manually in AWS console which is probably the worst way to manage AWS. We needed Terraform to implement Infrastructure as Code and keep it in Github.

We needed to provision several web servers and use load balancers for load share.

We needed reliable and repeatable way to provision a web server instance without even SSH-ing to it.

The website lives a long life, but it’s not static. We needed to update plugins versions, install new ones. WordPress itself also must be kept up-to-date.

We needed a way to work with a third party who wrote the code and keeps maintaining it – fixing bugs, adding new features. We needed better grasp of what is being changed on our website.

This post is going to describe how we solved each of these problems. Let’s start with Terraform.

Terraform

If you don’t know, Terraform is a kind of Chef/Puppet for a cloud. It implements so called Infrastructure As Code approach. You define the desired state – what VPC, subnets, routing tables, security groups, instances, load balancers you need, and Terraform converges your cloud to that state. Terraform manages everything in your cloud – from user accounts to network, instances and S3 buckets. It’s quite important to manage all entities with Terraform, otherwise bugs and unpredicted states are guaranteed. It’s like with cars – you can drive either right side or be Britain but not mix.

That’s why we created a brand new account that is fully managed by Terraform and migrated all services to the new account.

For the website we created web server instances, one database instance, and a load balancer.

There are two instance for web servers:

resource "aws_instance" "web_app" {
  # CentOS 7
  # https://aws.amazon.com/marketplace/pp/B00O7WM7QW
  ami = "ami-46c1b650"
  instance_type = "m4.large"
  subnet_id = "${aws_subnet.web_subnet.id}"
  vpc_security_group_ids = [
    "${aws_security_group.web_app_sg.id}"]
  key_name = "deployer"

  ebs_block_device {
    device_name = "/dev/sdb"
    volume_type = "gp2"
    volume_size = 30
    delete_on_termination = true
  }

  tags {
    Name = "web_app_${count.index}"
  }
  count = "2"

}

There is one load balancer. It also terminates HTTPS.

resource "aws_elb" "twindb_https_elb" {

  name = "twindb"
  security_groups = ["${aws_security_group.web_app_sg.id}"]
  subnets = ["${aws_subnet.web_subnet_public.id}"]

  "listener" {
    instance_port = 80
    instance_protocol = "http"
    lb_port = 443
    lb_protocol = "https"
    ssl_certificate_id = "arn:aws:acm:us-east-1:100576059692:certificate/xxx"
  }

  "listener" {
    instance_port = 80
    instance_protocol = "http"
    lb_port = 80
    lb_protocol = "http"
  }

  health_check {
    healthy_threshold   = 10
    unhealthy_threshold = 2
    timeout             = 5
    target              = "HTTP:80/index.php"
    interval            = 30
  }

  instances                   = [
    "${aws_instance.web_app.*.id}"
  ]
  cross_zone_load_balancing   = true
  idle_timeout                = 60
  connection_draining         = true
  connection_draining_timeout = 60

  tags {
    Name = "twindb-https-elb"
  }
}

And one database instance:

resource "aws_instance" "db01" {
  # CentOS 7
  # https://aws.amazon.com/marketplace/pp/B00O7WM7QW
  ami = "ami-46c1b650"
  instance_type = "t2.medium"
  subnet_id = "${aws_subnet.default_private.id}"
  vpc_security_group_ids = [
    "${aws_vpc.default.default_security_group_id}"]
  key_name = "deployer"

  ebs_block_device {
    device_name = "/dev/sdb"
    volume_type = "gp2"
    volume_size = 30
    delete_on_termination = true
  }

  tags {
    Name = "db01"
  }
}

Now, if I want to replace one instance I will simply terminate it, then run terraform apply – it will launch a fresh instance. When Chef provisions the instance the load balancer will add it into rotation. I can also add remove web servers depending on load on the website.

Chef

To provision instances we use Chef. When Chef bootstraps a new web server it performs these steps:

  1. Configures software repositories.
  2. Installs packages.
  3. Installs and configures Apache + PHP.
  4. Configures backups.
  5. Installs latest WordPress.
  6. Installs the website from GitHub.
  7. Installs the website configs.
  8. Syncs the media library with a shared S3 bucket.

From then on Chef regularly pulls latest website changes from GitHub and syncs the media library.

Software Repositories, Apache and PHP

Let’s start with the repos. We will need TwinDB, Percona, MySQL Community, and Epel repositories. It’s pretty straightforward.

# TwinDB repo
packagecloud_repo 'twindb/main'

# Percona repo
rpm_package 'percona-release-0.1-4.noarch.rpm' do
  source 'http://www.percona.com/downloads/percona-release/redhat/0.1-4/percona-release-0.1-4.noarch.rpm'
  action :install
end

# MySQL repo
rpm_package 'mysql57-community-release' do
  source 'https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm'
end

# EPEL repo
package 'epel-release'

Let’s see now what we need to install Apache and PHP.

include_recipe 'php'
include_recipe 'apache2'
include_recipe 'apache2::mod_php'

# This is needed to enable PHP. On some reason 'apache2::mod_php' recipe doesn't do it.
link "#{node['apache']['dir']}/mods-enabled/php.conf" do
  to '../mods-available/php.conf'
  notifies :reload, 'service[apache2]', :delayed
end

web_packages =['php-mbstring', ...]

web_packages.each { |pkg|
  package pkg do
    action :install
    notifies :reload, 'service[apache2]', :delayed
  end

}

Configure TwinDB Backups

To enable backups on the host we need to install a twindb-backup package and its configuration file.

package 'twindb-backup' do
  action :upgrade
end

aws_secrets = data_bag_item('web_app', 'secrets')

template '/etc/twindb/twindb-backup.cfg' do
  source 'twindb-backup.cfg.erb'
  variables(
      aws_access_key_id: aws_secrets['access_key_id'],
      aws_secret_access_key: aws_secrets['secret_access_key']
  )
  owner 'root'
  group 'root'
  mode '0600'
  sensitive true
end

Why do we need backups if the instance can be safely discarded and rebuilt, you may wonder? The main reason for that is to preserve a file cache, so when the instance is put into rotation it’s already warmed up. And Phoebe recommends doing so.

WordPress

Chef installs the latest WordPress only one time – at the instance bootstrap. If I need to upgrade wordpress version I will terminate this instance and bootstrap a new one. We also keep plugins in the website repo, so we remove akismet plugin that comes with WordPress. That’s it, no black magic here.

tar_extract node['wordpress']['url'] do
  target_dir node['wordpress']['dir']
  creates File.join(node['wordpress']['dir'], 'index.php')
  user node['wordpress']['install']['user']
  group node['wordpress']['install']['group']
  tar_flags [ '--strip-components 1', '--no-same-owner' ]
  not_if { ::File.exists?("#{node['wordpress']['dir']}/index.php") }
  notifies :run, 'execute[remove_akismet]', :immediately
end

document_root = node[:web_app][:http_root]

execute 'remove_akismet' do
  command "rm -rf '#{document_root}/wp-content/plugins/akismet'"
  action :nothing
end

Website

The website is the most interesting part. We had to solve many problems here.

We keep the website code on GitHub. In the repo we store plugins and TwinDB theme. Amazing Fruiful Code team created the design and developed code for TwinDB.com. Thank you guys!

$ cd website
$ ls
plugins themes
$ ls themes/
index.php   twindb

Chef clones this repo when it bootstraps an instance and pulls changes when the instance is operational.

execute 'git clone' do
  command "rm -rf #{document_root}/wp-content; git clone #{node[:web_app][:website_repo]} #{document_root}/wp-content"
  not_if "test -d #{document_root}/wp-content/.git"
end

execute 'git pull' do
  command 'git reset --hard; git pull'
  cwd "#{document_root}/wp-content"
  only_if "test -d #{document_root}/wp-content/.git"
end

So, when I want to update a plugin version I put the new code into the repo, push it and in 30 minutes the new version is deployed on all web servers.

When Fruitful Code makes changes they commit them to the master branch. For extra security we might require a PR before the code gets into the master.

Then Chef restores a media library and cache from a backup.

bash 'restore_backup' do
  code <<-EOH
    if ! test -d #{document_root}/wp-content/uploads
    then
      recent_basename=$(twindb-backup ls | grep #{backup_prefix}_wp-content_uploads | awk -F/ '{ print $NF}' | sort | tail -1)
      backup_copy=$(twindb-backup ls | grep ${recent_basename} | head -1)
      tmp_dir=$(mktemp -d)
      twindb-backup restore file ${backup_copy} --dst $tmp_dir
      mkdir -p #{document_root}/wp-content/uploads
      mkdir -p #{document_root}/wp-content/cache
      cp -r $tmp_dir/#{document_root}/wp-content/uploads/* #{document_root}/wp-content/uploads
      cp -r $tmp_dir/#{document_root}/wp-content/cache/* #{document_root}/wp-content/cache
      rm -rf $tmp_dir
    fi
  EOH
  user 'root'
end

Then Chef determines the current database host and saves it in a file. The WordPress config reads that file to define DB_HOST. It is a kind of poor man high availability :).

execute 'detect_db_host' do
  command "db_host=$(AWS_ACCESS_KEY_ID=#{AWS_ACCESS_KEY_ID} \
AWS_SECRET_ACCESS_KEY=#{AWS_SECRET_ACCESS_KEY} \
AWS_DEFAULT_REGION=#{AWS_DEFAULT_REGION} aws ec2 describe-instances \
  --filters 'Name=tag:#{tag},Values=#{tag_value}' | \
  /usr/bin/jq -r .Reservations[0].Instances[0].PrivateIpAddress | \
  /bin/tr -d '\n'); echo $db_host > #{document_root}/.db_host"
  sensitive true
end

template("#{document_root}/wp-config.php") do
  owner 'root'
  group 'root'
  mode '0644'
  source('wp-config.php.erb')
  variables(
      documentroot: document_root,
      db_name: secrets['wp_db_name'],
      db_user: secrets['wp_db_user'],
      db_password: secrets['wp_db_password']
  )
  sensitive true
end

And the last step is to sync the media library. The matter is it’s a bad practice to store media files in git. Same time I didn’t want to deal with any kind of shared volumes. So, instead, Chef syncs media library with an S3 bucket every 30 minutes.

bucket = node[:web_app][:uploads_bucket]

execute 'sync_uploads_local_s3' do
  command "AWS_ACCESS_KEY_ID=#{AWS_ACCESS_KEY_ID} \
AWS_SECRET_ACCESS_KEY=#{AWS_SECRET_ACCESS_KEY} \
AWS_DEFAULT_REGION=#{AWS_DEFAULT_REGION} aws s3 sync '#{document_root}/wp-content/uploads' s3://#{bucket}"
  sensitive true
end

execute 'sync_uploads_s3_local' do
  command "AWS_ACCESS_KEY_ID=#{AWS_ACCESS_KEY_ID} \
AWS_SECRET_ACCESS_KEY=#{AWS_SECRET_ACCESS_KEY} \
AWS_DEFAULT_REGION=#{AWS_DEFAULT_REGION} aws s3 sync s3://#{bucket} '#{document_root}/wp-content/uploads' "
  sensitive true
end

Conclusion

Our new website is live few weeks. So far I like the result. Since then we installed new plugins, upgraded the plugins and wordpress itself, made numerous changes to the website itself, wrote couple of posts. This architecture proved to be viable.

Probably, in future we will configure staging environment and integration tests before putting changes in production. But so far, so good.

Previous Post Next Post

Comments are closed.