Trying KIOXIA CM6 and PM6 Enterprise SSDs on a Raspberry Pi

Late last year, an engineer at Broadcom sent me some hardware and offered some help getting Broadcom's MegaRAID card working on the Raspberry Pi. It took some time, but eventually we were able to get the card and a demonstrator 'UBM' backplane working on the Pi, and it culminated in my posting about Hardware RAID on the Pi, and on a livestream, getting 16 hard drives working on a Pi.

The one thing I couldn't test in those earlier videos was the backplane and storage card's 'Tri-mode' support, allowing PCI Express NVMe drives—like KIOXIA's CM6—to work in the same slot as the SATA and SAS drives I was used to testing.

So after some conversation with reps at KIOXIA, I was able to get a PM6 and three CM6 drives on loan to test them:

I tested both drives inside the experimental 'Elrond' backplane, which is a demonstrator for Universal Backplane Management (SFF-TA-1005), connected through the Broadcom MegaRAID 9460-16i card, and as promised, I was able to hot-swap SATA, SAS, and NVMe drives using the same slots.

Video

There is a video to go along with this post, which goes into a little more detail:

Why SFF NVMe?

Before getting to the benchmarks, I figured I should explain a little more about the 'U.3' (SFF-TA-1001) NVMe drive I tested, the CM6.

Most people are familiar with NVMe drives in the M.2 form factor:

But in enterprise storage, most servers have storage backplanes that offer slots with multiple SFF 2.5" drive bays. Traditionally, the bays would accept SATA or SAS drives, but with the rise in popularity of PCI Express-based storage, Intel soon created the U.2 standard to allow NVMe storage using the same connector—but with a different and incompatible pinout.

Therefore server vendors would have to decide to cable ports on their drive bays for either SATA/SAS, or NVMe. Thus the need for a new protocol—SFF-TA-1001, sometimes referred to as 'U.3'—which allows the same connector and the same cabling to support all three drive types.

And in both U.2 and U.3's case, the basic idea is to have a larger volume to support hot swap, higher capacities, and better cooling, making a much more enterprise-ready package than the diminutive M.2 form factor.

Benchmarks

Now, getting to the meat of this post, I tested both of these drives against each other, and against some other drives I had on hand.

First, comparing the PM6 and CM6 on the Pi:

The CM6 beats out the PM6 in random write performance, but otherwise displays that the underlying flash storage's similarity between the two drives results in roughly equivalent numbers. I should note that these drives are capable of 4 to 7 GiB/sec of performance. The Raspberry Pi's x1 Gen 2 bus really is the limiting factor (besides the anemic CPU performance), and results in the Pi getting a 17x performance penalty compared to a nice, new PCI Express Gen 4 server.

I also compared the CM6 standalone to the CM6 in RAID 0, and found it was very slightly faster, but mostly I think I'm running into the Pi's CPU bottlenecks (in addition to the bus limitations):

As a final test I wanted to see how the CM6 compared to the cheaper consumer Kingston SSDs I was testing on the Pi:

And it's good to know these blazing-fast enterprise SSDs are, indeed, faster than the $30 off-the-shelf bargain bin SSDs. But that's not the only reason you'd spend the extra money for 'Enterprise'-grade SSDs—these SSDs are rated for at least 1 DWPD (so they should last much longer in write or read-heavy workloads), they have better cooling, they have better warranties—and that's all before mentioning they use an interface allowing for 10-20x more throughput than the SATA on the consumer SSD!

Thanks to KIOXIA for loaning me these drives to test, and thanks again to Broadcom for sending their UBM enclosure and a MegaRAID card, plus the engineering time to help get their driver patched for the Pi!