Fixing Drupal Fast - Using Ansible to deploy a security update on many sites

Earlier today, the Drupal Security Team announced SA-CORE-2014-005 - Drupal core - SQL injection, a 'Highly Critical' bug in Drupal 7 core that could result in SQL injection, leading to a whole host of other problems.

While not a regular occurrence, this kind of vulnerability is disclosed from time to time—if not in Drupal core, in some popular contributed module, or in some package you have running on your Internet-connected servers. What's the best way to update your entire infrastructure (all your sites and servers) against a vulnerability like this, and fast? High profile sites could be quickly targeted by criminals, and need to be able to deploy a fix ASAP... and though lower-profile sites may not be immediately targeted, you can bet there will eventually be a malicious bot scanning for vulnerable sites, so these sites need to still apply the fix in a timely manner.

In this blog post, I'll show how I patched all of Midwestern Mac's Drupal 7 sites in less than 5 minutes.

Hotfixing Drupal core - many options

Before we begin, let me start off by saying there are many ways you can apply a security patch, and some are simpler than others. As many have pointed out (e.g. Lullabot, you can simply download the one line patch and apply it to your Drupal codebase using patch -p1.

You could also use Drush to do a Drupal core update (drush up drupal), but you'll still need to do this, manually, on every Drupal installation you manage.

If you have multiple webservers with Drupal (or multiple instances of Drupal 7 on a single server, or spread across multiple servers), then there are simpler ways of either deploying the hotfix, or upgrading Drupal core via drush and/or version control (you are using Git or some other VCS, right?).

Enter Ansible, the Swiss Army Knife for infrastructure

Ansible is a powerful infrastructure management tool. It does Configuration Management (CM), just like Puppet or Chef, but it goes much, much further. One great feature of Ansible is the ability to run ad-hoc commands against a bunch of servers at once.

After installing Ansible, you need to create a hosts file at /etc/ansible/hosts, and tell Ansible about your servers (this is an 'inventory' of servers). Here's a simplified overview of my file:

[mm]
jeffgeerling.com drupal_docroot=/path/to/drupal

[servercheck-drupal]
servercheck.in drupal_docroot=/path/to/drupal

[hostedsolr-drupal]
hostedapachesolr.com drupal_docroot=/path/to/drupal

[drupal7:children]
mm
servercheck-drupal
hostedsolr-drupal

There are a couple quick things to note: the inventory file follows an ini-style format, so you define groups of servers with [groupname] (then list the servers one by one after the group name, with optional variables in key=value format after the server name), then define groups of groups with [groupname:children] (then list the groups you want to include in this group). We defined a group for each site (currently each group just has one Drupal web server), then defined a drupal7 group to contain all the Drupal 7 servers.

As long as you can connect to the servers using SSH, you're golden. No additional configuration, no software to install on the servers, nada.

Let's go ahead and quickly check if we can connect to all our servers with the ansible command:

$ ansible drupal7 -m ping
hostedapachesolr.com | success >> {
    "changed": false,
    "ping": "pong"
}
[...]

All the servers have responded with a 'pong', so we know we're connected. Yay!

For a simple fix, we could add a variable to our inventory file for each server defining the Drupal document root(s) on the server, then use that variable to apply the hotfix like so:

$ ansible drupal7 -m shell -a "curl https://www.drupal.org/files/issues/SA-CORE-2014-005-D7.patch | patch -p1 chdir={{ drupal_docroot }}"

This would quickly apply the hotfix on all your servers, using Ansible's shell module (which, conveniently, runs shell commands verbatim, and tells you the output).

Fixing core, and much more

Instead of running one command via ansible, let's make a really simple, short Ansible playbook to fix and verify the vulnerability. I created a file named drupal-fix.yml (that's right, Ansible uses plain old YAML files, just like Drupal 8!), and put in the following contents:

---
- hosts: drupal7
  tasks:
    - name: Download drupal core patch.
      get_url:
        url: https://www.drupal.org/files/issues/SA-CORE-2014-005-D7.patch
        dest: /tmp/SA-CORE-2014-005-D7.patch

    - name: Apply the patch from the drupal docroot.
      shell: "patch -p1 < /tmp/SA-CORE-2014-005-D7.patch chdir={{ drupal_docroot }}"

    - name: Restart apache (or nginx, and/or php-fpm, etc.) to rebuild opcode cache.
      service: name=httpd state=restarted

    - name: Clear Drupal caches (because it's always a good idea).
      command: "drush cc all chdir={{ drupal_docroot }}"

    - name: Ensure we're not vulnerable anymore.
      [redacted]

Now, there are again many, many different ways I could've done this. (And to the eagle-eyed, you'll note I haven't included my test for the vulnerability... I'd rather not share how to test for the vulnerability until people have had a chance to update all their sites).

I chose to do the hotfix first, and quickly, since I didn't necessarily have time to update all my Drupal project codebases to Drupal 7.32, then push the updated code to all my repositories. I did do this later in the day, however, and used a playbook similar to the above, replacing the first two tasks with:

- name: Pull down the latest code changes.
  git:
    repo: "git://[mm-git-host]/{{ inventory_hostname }}.git"
    dest: "{{ drupal_docroot }}"
    version: master

Using Ansible's git module, I can tell Ansible to make sure the given directory (dest) has the latest commit to the master branch in the given repo. I could've also used a command and run git pull from the drupal_docroot directory, but I like using Ansible's git module, which provides great reporting and error handling.

Summary

This post basically followed my train of thought after hearing about the vulnerability, and while there are a dozen other ways to patch the vulnerability on multiple sites/servers, this was the way I did it. Though I patched just 9 servers in about 5 minutes (from the time I started writing the playbook (drupal-fix.yml) to the time it was deployed everywhere), I could just as easily have deployed the fix to dozens or hundreds of Drupal servers in the same amount of time; Ansible is fast and uses simple, secure SSH connections to manage servers.

If you want to see much, much more about what Ansible can do for your infrastructure, please check out my book, Ansible for DevOps, and also check out my session from DrupalCon Austin earlier this year: DevOps for Humans: Ansible for Drupal Deployment Victory!.