Benchmarking PHP 7 vs HHVM - Drupal and Wordpress

[Multiple updates: I've added results for concurrencies of 1 and 10, results on bare metal vs. VMware instances, tested Drupal 8 vs Drupal 7 vs Wordpress 4.4, and I've also retested every single benchmark at least twice! Please make sure you're read through the entire post prior to contesting these benchmark results!]

tl;dr: Always test your own application, and trust, but verify every benchmark you see. PHP 7 is actually faster than HHVM in many cases, neck-in-neck in others, and slightly slower in others. Both PHP 7 and HHVM blow PHP ≤ 5.6 out of the water.

Skip to benchmark results:

• Drupal 8 Results (concurrency 10)
• Drupal 8 Results (concurrency 1)
• Drupal 8 Results ('bare metal', concurrency 1)
• Wordpress 4.4 Results ('bare metal', concurrency 1)
• Drupal 7 Results (concurrency 10)

Introduction and Methodology

As PHP 7 became a reality through this past year, there were scores of benchmarks pitting PHP 7 against 5.6 and HHVM using applications and frameworks like Drupal, Wordpress, Joomla, Laravel, October, etc.

One benchmark that really stood out to me (in that it seemed so wrong for Drupal, based on my experience) was The Definitive PHP 7.0 & HHVM Benchmark from Kinsta. Naming a benchmark that way certainly makes the general PHP populace take it seriously!

The results are pretty damning for PHP 7:

In the comments on that post, Thomas Svenson mentioned:

Standard installation for Drupal 8 has cache on as default. If you did not turn that off, then it is probably a reason to why the PHP 7 boost isn't bigger.

Would be interesting to see the result comparing the benchmark with/without caching enabled in Drupal 8. Should potentially reveal something interesting.

This was my main concern too, as there wasn't enough detail in the benchmarking article to determine what exactly was the system under test. Therefore, I'll submit my own PHP 7 vs HHVM benchmark here, using the following versions:

• Ubuntu 14.04
• Drupal 8.0.1
• Nginx 1.4.6
• MySQL 5.5.46
• PHP 5.6.16, PHP 7.0.1, or HHVM 3.11.0

All tests were run using Drupal VM version 2.1.2 with VMware Fusion 8.1.0, on my mid-2013 MacBook Air 13" 1.7 GHz i7 with 8GB of RAM. Using the above notes, you can exactly replicate this benchmarking environment should you desire. All tests were run five times, the first two results were discarded (because they often reflect times when some caches are still warming), and the latter three were averaged.

After installing Drupal 8.0.1 with the standard installation profile (this is done automatically by Drupal VM), I logged in as the admin user (user 1), then grabbed the admin user's session cookie, and ran the following two commands:

Drupal 8 results (concurrency 10)

...and some graphs of the above data:

PHP 5.6, PHP 7, and HHVM running Drupal 8.0.1, cached

PHP 5.6, PHP 7, and HHVM running Drupal 8.0.1, uncached

Drupal 8 results (concurrency 1)

Sometimes, the use of concurrency (-c 10 in the above case)(to simulate concurrent users hitting the site at the same time, can cause benchmarks to be slightly inaccurate. The reason I usually use a level of concurrency is so the benchmark more closely mirrors real-world usage, and tests the full stack a little better (because PHP by itself is nice to benchmark, but very few sites are run on top of PHP alone!).

Anyways, I re-ran all the tests using -c 1, and am publishing the results below:

In all my benchmarking, I care more about deltas and reproducibility than measuring raw, clean-room-scenario performance, because unless a result is absolutely reproducible, it's of no value to me. Therefore if I can prove that there's no particular difference to testing with certain concurrency levels, I typically move the benchmark to a level that mirrors traffic patterns I actually see on my sites :)

Absolute numbers mean nothing to me—it's the comparison between test A and test B, and how reproducible that comparison is, that matters. That's why I enjoy benchmarking on the incredibly slow Raspberry Pi model 2 sometimes, because though it's much slower than my i7 laptop, it sometimes exposes surprising results!

Drupal 8 Results ('bare metal', concurrency 1)

Some people argue that running benchmarks in a VM is highly unreliable and leads to incorrect benchmarks, so I've also sacrificed a partition of a Lenovo T420 core i5 laptop (it has 3 SSDs inside, so I just formatted one, installed Ubuntu desktop 15.10, then installed PHP, MySQL, and Nginx exactly the same as with Drupal VM (same settings, same apt repos, etc.), and re-ran all the tests in that environment—so-called 'bare metal', where there's absolutely no overhead from shared filesystems, the hypervisor, etc.

After running these benchmarks with an identical environment on 'bare metal' (e.g. a laptop with a brand new/fresh install of Ubuntu 15.10 running the same software, with 8 GB of RAM and an SSD), it seems HHVM for some reason performed even worse than PHP 5.6.16 for Drupal 8.

Since this result is wildly different than the Kinsta post (basically the opposite of their results for Drupal 8), I decided to test Wordpress 4.4 as well.

Wordpress 4.4 Results ('bare metal', concurrency 1)

For Wordpress, I ran the test using the exact same Lenovo T420 environment as the test above, and tested an anonymous user (no cookie value) hitting the default home page, and an admin logged in (using a valid session cookie—actually all five of the cookies wordpress uses to track valid sessions) visiting the admin Dashboard page (/wp-admin/index.php).

...and some graphs of the above data:

PHP 5.6, PHP 7, and HHVM running Wordpress 4.4, anonymous home page

PHP 5.6, PHP 7, and HHVM running Wordpress 4.4, admin dashboard

These results highlight to me how much the particular project's architecture influences the benchmark. Wordpress still uses a traditional quasi-functional-style design, while Drupal 8 is heavily invested in OOP and a bit more formal data architecture. While I'm not as familiar with Wordpress's quirks as I am Drupal, I know that it's no speed demon, and also benefits from added caching layers in front of the site! It's interesting to see that PHP 7 and HHVM are practically neck-and neck for front-facing portions of Wordpress (and FAR faster than 5.6), while HHVM runs even a little faster than PHP 7 for administrative tasks.

Drupal 7 Results (concurrency 10)

I also benchmarked Drupal 7 on Drupal VM for another point of comparison (using -c 10):

For these tests, I went to the Performance configuration page prior to running the tests, and enabled anonymous page cache, block cache, and CSS and JS aggregation (to make D7 match up to D8 cached anonymous user results a little more evenly).

Some people point out benchmarks like these and say "Drupal 8 is slow"... and they're right, of course. But Drupal 8 trades performance for better architecture, much more pluggability, and the inclusion of many more essential 'out-of-the-box' features than Drupal 7, so there's that. Having built a few Drupal 8 sites, I don't ever want to go back to 7 again—but it's nice to know that PHP 7 can still accelerate all my existing D7 sites quite a bit!

Summary

tl;dr: For Drupal 7 and Drupal 8 at least, PHP 7 takes the performance crown—by a wide margin.

After running the benchmarks, I scratched my head, because almost every other benchmark I've seen either puts HHVM neck-and-neck with PHP 7 or makes it seem HHVM is still the clear victor. Maybe other people running these benchmarks didn't have PHP's opcache turned on? Maybe something else was missing? Not sure, but if you'd like to reproduce the SUT and find any results different than the above (in terms of percentages), please let me know!

I ran the HHVM benchmarks three times with fresh new VM instances just because I was surprised PHP 7 stepped out in front. PHP 5.6's performance is as expected... it's better than 5.3, but that's not saying much :)

The moral of the story: Trust, but verify... especially for benchmarks which compare a plethora of totally different applications, each result can tell a completely different story depending on the test process and system under test! Please run your own tests with your own application before definitively stating that one server is faster than another.

Installing HHVM in Drupal VM

Just for posterity, since I want people to be able to reproduce the steps exactly, here's the process I used after using Drupal VM's default config.yml (with Ubuntu 14.04) to build the VM:

1. Log into Drupal VM with vagrant ssh
2. $ sudo su
3. # service php5-fpm stop
4. # apt-get install -y python-software-properties
5. # curl http://dl.hhvm.com/conf/hhvm.gpg.key | apt-key add -
6. # add-apt-repository http://dl.hhvm.com/ubuntu
7. # apt-get update && apt-get install -y hhvm
8. # update-rc.d hhvm defaults
9. # /usr/share/hhvm/install_fastcgi.sh
10. # vi /etc/nginx/sites-enabled/drupalvm.dev.conf and inside the location ~ \.php$|^/update.php block:
    1. Clear out the contents of this configuration block.
    2. Replace with include hhvm.conf;
11. # service hhvm restart
12. # service nginx restart

Visit the /admin/reports/status/php page after logging in to confirm you're running HHVM instead of PHP.