nvme

With the Raspberry Pi 5's exposed PCI Express connector comes many new possibilities—which I test and document in my Pi PCIe Database. Today's board is the Geekwork X1011, which puts four NVMe SSDs under a Raspberry Pi.

Unlike the Penta SATA HAT I tested last month, this carrier uses thinner and faster NVMe storage, making it a highly-compact storage expansion option, which has the added benefit of freeing up the top of the Pi 5 for other HAT expansion options.

In my video about the brand new Raspberry Pi 5, I mentioned the new external PCIe port makes it possible to boot the standard Pi 5 model B directly off NVMe storage—an option which is much faster and more reliable than standard microSD storage (even with industrial-rated cards!).

Enabling NVMe boot is pretty easy, you add a line to /boot/firmware/config.txt, modify the BOOT_ORDER in the bootloader configuration, and reboot!

Last year, after I started a search for a good out-of-the-box all-flash-storage setup for a video editing NAS, I floated the idea of an all-M.2 NVMe NAS to ASUSTOR. I am not the first person with the idea, nor is ASUSTOR the first prebuilt NAS company to build one (that honor goes QNAP, with their TBS-453DX).

But I do think the concept can be executed to suit different needs—like in my case, video editing over a 10 Gbps network with minimal latency for at least one concurrent user with multiple 4K streams and sometimes complex edits, without lower-bitrate transcoded media (e.g. ProRes RAW).

For many years, I've maintained some scripts to do basic disk benchmarking for SBCs, to test 1M and 4K sequential and random access speeds, since those are the two most relevant tests for the Linux workloads I run on my Pis.

I've been using this script for years, and it uses fio and iozone to get the metrics I need.

And from time to time, I would test a number of microSD cards on the Pi, or run tests on NVMe SSDs on the Pi, Rock 5 model B, or other SBCs. But my results were usually geared towards a single blog post or a video project.

In 2021 James Chambers set up PiBenchmarks to move to a more community-driven testing dataset.

You can run the following command on your SBC to test the boot storage and upload results directly to PiBenchmarks.com:

This is a simple guide, part of a series I'll call 'How-To Guide Without Ads'. In it, I'm going to document how I set up bcache on a Raspberry Pi, so I could use an SSD as a cache in front of a RAID array.

Getting bcache

bcache is sometimes used on Linux devices to allow a more efficient SSD cache to run in front of a single or multiple slower hard drives—typically in a storage array.

In my case, I have three SATA hard drives: /dev/sda, /dev/sdb, and /dev/sdc. And I have one NVMe SSD: /dev/nvme0n1.

I created a RAID5 array with mdadm for the three hard drives, and had the raid device /dev/md0.

I then installed bcache-tools:

$ sudo apt-get install bcache-tools

And used make-bcache to create the backing and cache devices:

Late last year, an engineer at Broadcom sent me some hardware and offered some help getting Broadcom's MegaRAID card working on the Raspberry Pi. It took some time, but eventually we were able to get the card and a demonstrator 'UBM' backplane working on the Pi, and it culminated in my posting about Hardware RAID on the Pi, and on a livestream, getting 16 hard drives working on a Pi.

The one thing I couldn't test in those earlier videos was the backplane and storage card's 'Tri-mode' support, allowing PCI Express NVMe drives—like KIOXIA's CM6—to work in the same slot as the SATA and SAS drives I was used to testing.

So after some conversation with reps at KIOXIA, I was able to get a PM6 and three CM6 drives on loan to test them:

When the Compute Module 4 was released (see my CM4 review here), I asked the Pi Foundation engineers when we might be able to boot off NVMe storage, since it was trivially easy to use with the exposed PCIe x1 lane on the CM4 IO Board.

The initial response in October 2020 was "we'll see". Luckily, after more people started asking about it, beta support was added for direct NVMe boot just a couple weeks ago.

Just posting to the blog for reference; I posted this video on YouTube recently, in which I built (what I believe to be) the world's tiniest NVMe SSD RAID array, using the Raspberry Pi Compute Module 4 and three diminutive WD SN520 NVMe drives (which are M.2 2230 size, which makes them each about the size of a quarter):

I ran some benchmarks in RAID 5 and RAID 0, as well as one drive by itself, and found one surprising thing: the Pi's overall IO bandwidth is already saturated by just one drive, so putting NVMe disks in RAID doesn't really help with performance, like it does with slower spinning hard drives.

Ever since the Pi 2 model B went to a 4-core processor, disk IO has often been the primary bottleneck for my Pi projects.

You can use microSD cards, which aren't horrible, but... well, nevermind, they're pretty bad as a primary disk. Or you can plug in a USB 3.0 SSD and get decent speed, but you end up with a cabling mess and lose bandwidth and latency to a USB-to-SATA or USB-to-NVMe adapter.

The Pi 4 actually has an x1 PCI Express gen 2.0 lane, but the USB 3.0 controller chip populates that bus on the model B. The Compute Module 4, however doesn't presume anything—it exposes the PCIe lane directly to any card it plugs into.

And in the case of Oratek's TOFU, it's exposed through an M.2 slot, making this board the first one I've used that can accept native NVMe storage, directly under the Pi:

A few months ago, I posted a video titled Enterprise SAS RAID on the Raspberry Pi... but I never actually showed a SAS drive in it. And soon after, I posted another video, The Fastest SATA RAID on a Raspberry Pi.

Well now I have actual enterprise SAS drives running on a hardware RAID controller on a Raspberry Pi, and it's faster than the 'fastest' SATA RAID array I set up in that other video.

A Broadcom engineer named Josh watched my earlier videos and realized the ancient LSI card I was testing would not likely work with the ARM processor in the Pi, so he was able to send two pieces of kit my way: