benchmarking

I got this Rosewill RC-20001 PCIe 2.5 Gbps Network Adapter working on the Raspberry Pi Compute Module 4.

Right after I got the card working, though, I tested it in an external powered PCI Express riser, and that test released the card's magic smoke. Oops.

Here's a dramatic re-enactment that's actually pretty accurate to what it looked like in real life:

Luckily, buying a replacment wasn't too bad, since the card is less than $20. But to get it to work on my spiffy new ten gigabit network, I also had to buy a new SFP+ transceiver that was compatible with 1, 2.5, 5, and 10 Gbps data rates, and that cost $60!

Raspberry Pi OS isn't really built to be a server OS; the main goals are stability and support for educational content. But that doesn't mean people like me don't use and abuse it to do just about anything.

In my case, I've been doing a lot of network testing lately—first with an Intel I340-T4 PCIe interface for 4.15 Gbps of networking, and more recently (yesterday, in fact!) with a Rosewill 2.5 GbE PCIe NIC.

And since the Pi's BCM2711 SoC is somewhat limited, it can't seem to pump through many Gbps of bandwidth without hitting IRQ limits, and queueing up packets.

In the case of the 2.5G NIC, I was seeing it max out around 1.92 Gpbs, and I just wouldn't accept that (at least not for a raw benchmark). Running atop, I noticed that during testing, the IRQ interrupts would max out at 99% on one CPU core—and it seems like it may be impossible to distribute interrupts across all four cores on the BCM2711.

Something I've long been meaning to benchmark, but never really got around to, is benchmarking the amount of time it takes on a Raspberry Pi to open a browser, load a page, and quit.

This is a relatively decent thing to benchmark, compared to other raw performance metrics, because it's something that probably 99% of Raspberry Pi users who use it with a GUI will do, with some frequency (well, probably loading more than one page before quitting, but still...).

So I asked on Twitter:

So... if I want to measure how long it takes to:

1 Open Chrome/Chromium
2 Load a web page

And I want to do this in Linux (Debian), and have an objective measure of that time (until the page finishes loading)... is there any way to do that?

CLI opens then sits until you quit.

At this point, I've showed you how you can use the Turing Pi as a Kubernetes cluster to run different things. I barely scratched the surface of what's possible with Kubernetes, but I'm planning on doing another series exploring Kubernetes itself later this year. Subscribe to my YouTube channel if you want to see it!

In this post, I'm going to talk about the Turing Pi's performance. I'll compare it to a more traditional Raspberry Pi cluster, my Pi Dramble, and talk about important considerations for your cluster, like what kind of storage you should use, or whether you should run a 32-bit or 64-bit Pi operating system.

As with all the other work I've done on this cluster, I've been documenting it all in my open source Turing Pi Cluster project on GitHub.

Video version of this post

This blog post has a companion video embedded below:

[Multiple updates: I've added results for concurrencies of 1 and 10, results on bare metal vs. VMware instances, tested Drupal 8 vs Drupal 7 vs Wordpress 4.4, and I've also retested every single benchmark at least twice! Please make sure you're read through the entire post prior to contesting these benchmark results!]

tl;dr: Always test your own application, and trust, but verify every benchmark you see. PHP 7 is actually faster than HHVM in many cases, neck-in-neck in others, and slightly slower in others. Both PHP 7 and HHVM blow PHP ≤ 5.6 out of the water.

Skip to benchmark results:

[Update 2015-08-25: I reran some of the tests using two different settings in VirtualBox. First, I explicitly set KVM as the paravirtualization mode (it was saved as 'Legacy' by default, due to a bug in VirtualBox 5.0.0), which showed impressive performance improvements, making VirtualBox perform 1.5-2x faster, and bringing some benchmarks to a dead heat with VMware Fusion. I also set the virtual network card to use 'virtio' instead of emulating an Intel PRO/1000 MT card, but this made little difference in raw network throughput or any other benchmarks.]

My Mac spends the majority of the day running at between one and a dozen VMs. I do all my development (besides iOS or Mac dev) running code inside VMs, and for many years I used VirtualBox, a free virtualization tool, along with Vagrant and Ansible, to build and manage all these VMs.

I currently own or use a variety of Macs, and am approaching the end of a 'cycle' of Mac usage, where I need to decided what Mac I'd like to purchase next. Currently, I'm using a 27" iMac at work, an 11" MacBook Air (from work) for travel, and a 24" iMac at home. They're all great computers in their own right, and using Dropbox, MobileMe, and a couple other helper services, I can operate simultaneously on all three Macs, without any hiccups.

So, I'm thinking about getting a new Mac for hardcore development work (web and app), some graphic design, and possible portability. I have an iPad for lighter computing (reading, browsing, email, videos...), so even though the MacBook Air is probably the best thing to happen to a laptop in a very long time, I'm shying away from it as my primary personal computer.

I just discovered (after asking about it in the #drupal IRC channel) the wonderful little program ab, included in an Apache installation. This little nugget does one thing, and does it well: It beats the heck out of your server, then tells you how your server did in terms of page serving. I tested a few different configurations on a dedicated, 4-core, 4 GB RAM server from SoftLayer, and used the following two commands:

1. Download the specified URL 1,000 times, with KeepAlive turned off (each request gets a new http connection):

ab -n 1000 -c 5 http://ip.address.of.site/path-to-page.php

2. Downlaod the specified URL 1,000 times, with KeepAlive turned on (thus allowing the connection to be maintained for as many http downloads as you have set in your httpd.conf file):

ab -n 1000 -kc 5 http://ip.address.of.site/path-to-page.php

I ran these tests a few different ways, and here are the results of the tests I ran with KeepAlive on, with the number of pages per second ab reported listed after the method:

• Drupal - normal page caching turned on, css/js aggregation, 55kb page – 12.5 pages/sec
• Joomla - no page caching (disabled due to buggy 1.x caching), 65kb page – 8.2 pages/sec
• Drupal - boost module enabled, serving up the boost-cached file – 3,250 pages/sec
• Joomla - custom page caching system enabled, serving static html file – 2,600 pages/sec

Speed boost due to caching: ~250x faster!