Encrypting MySQL Backups
Encryption is an important component of secure environments. While being intangible, property security doesn’t get enough attention when it comes to describing various systems. “Encryption support” is often as detailed as it gets, if you ask how secure the system is. Other important facts are often omitted, but, as we know, the devil is in the details. In this post, I will describe how we secure backup copies at TwinDB.
What’s Bad Encryption?
See the picture. This is what happens when encryption is used incorrectly. The encryption algorithm can be perfect, but poor choice of the mode results in a barely readable encrypted image. This mode is called “Electronic Code Book”, avoid it by all means.
Another bright example of improper encryption use was illustrated in Venona project.
Finding the Perfect Solution
There is an encryption algorithm mathematically proven to be unbreakable. That means you cannot decrypt a cipher text, even with a brute force attack and having unlimited computing power and time. It’s One-Time Pad. It’s very simple and fast – it’s an XOR of a plain-text message and a key. The only problem with the algorithm is that it requires a key as large as the plain text message. But if you can securely transfer the key, why would you need the encryption at all? That’s the reason why One-time Pad isn’t very popular. But it’s still usable. For example, you can generate a really large key, much larger than a typical plain text message. Then, use a secure channel to transfer the key, and then use parts of the key to send encrypted messages.
Basically, this is what Soviets did to spy in the USA. They generated a bunch of keys and used them to encrypt messages to a Moscow recipient. Because Soviets did everything backwards, they either ran out of keys or some lazy-bone didn’t destroy used keys, and at some point they reused the keys to encrypt more than one message. That broke their encryption and the USA managed to decipher many messages.
Thus, there is a fundamental rule. Never use or design home backed encryption algorithms.
What We Do At TwinDB
The public key of the dispatcher is hard-coded in the agent (I’ll refer to it as email@example.com). The agent uses the public key firstname.lastname@example.org to send encrypted messages to the dispatcher. When a user registers a new MySQL server at TwinDB, the agent generates its own RSA keys pair, it’s referred as <UUID>@twindb.com, for example email@example.com .
gpg -K /root/.gnupg/secring.gpg ------------------------ sec 2048R/17B2D796 2014-12-08 uid Backup Server id e456b18e-11eb-49ff-93f9-5341656799fe (No passphrase) <firstname.lastname@example.org> ssb 2048R/604969DA 2014-12-08
Here’s how you can generate a pair of GPG keys in Python non-interactively:
server_id = "e456b18e-11eb-49ff-93f9-5341656799fe" email = "email@example.com" gpg_cmd = ["gpg", "--batch", "--gen-key"] gpg_script = """ %%echo Generating a standard key Key-Type: RSA Key-Length: 2048 Subkey-Type: RSA Subkey-Length: 2048 Name-Real: Backup Server id %s Name-Comment: No passphrase Name-Email: %s Expire-Date: 0 %%commit %%echo done """ % (server_id, email) p = subprocess.Popen(gpg_cmd, stdin=subprocess.PIPE) p.communicate(gpg_script)
The agent sends its public key to the dispatcher and from this point on, the dispatcher can communicate securely with the agent. It’s nice that a gpg can sign messages, so the dispatcher can authenticate the agents, too.
Encrypting MySQL Backups
When the agent takes a backup, it streams the backup copy to a gpg process and it encrypts the stream with its own public key.
innobackupex --stream xbstream ./ | gpg --encrypt --yes --batch --no-permission-warning --quiet --recipient firstname.lastname@example.org
Hence, one of the biggest features of TwinDB I’m proud of. The agent encrypts the backup copy with its own key. So we at TwinDB cannot decrypt user backups! We did it for two reasons. First, we want our users to trust TwinDB. “You can get much farther with a kind word and encryption than you can with a kind word alone” (c) almost Al Capone. Second, it’s kind of protection for us. If someone breaks into TwinDB (I have no illusions, there are smarter hackers out there than us) the burglars won’t gain access to user data.
The private key is generated locally and never leaves the server where it was generated.
But how can we restore a database if the server is destroyed with a private key?
We ask users to generate their own key pair and give us the public key. Users should keep the private key in a secure place, print it out and put in a bank deposit box.
If the public key is available, the dispatcher schedules a job for the agent and asks it to encrypt its private key with the user’s public key and send it back to the dispatcher. If the server is completely out of order the user can get the encrypted server’s private key, decrypt it and decrypt the backup copy to restore the database.