I built a special PCIe card to test GPUs on the Pi

I partnered up with Mirek (of Mirkotronics / @Mirko_DIY on Twitter) to build the Pi4GPU (or 'PiG' for short):

This journey started almost three years ago: almost immediately after the Raspberry Pi Compute Module 4 was launched, I started testing graphics cards on it.

First I tried some low-spec cards like the Zotac Nvidia GT 710 and the VisionTek AMD Radeon 5450. They kept locking up regardless of the driver and Linux versions.

Over the next couple years, I kept testing more and more cards—over 14 at the time of this writing. The reason? Each card (really, each generation of each vendor's cards) had quirks that made it more or less likely to run on the Pi.

Why's that? Well, the Pi has problems with cache coherency on the PCI Express bus beyond 32-bits. And many (well, all nowadays) of the drivers expect that to function. That wasn't the first problem, though—early on, there were issues with the BAR (Base Address Register) space allocated on the Pi's OS. Luckily this could be worked around by re-compiling a DTB file.

This whole endeavor is what inspired me to create the Raspberry Pi PCIe Device Database, which documents (often in excruciating detail) the travails bringing up various PCI Express devices on a Raspberry Pi—and now, on other ARM SBCs like the Radxa Rock 5 and the Pine64 SOQuartz.

Making a Custom PCB

Last year I floated the idea of a custom PCB to 'plug a computer into a graphics card' to Mirek. He didn't immediately say 'no', so I started pursuing the idea, coming up with this quite basic post-it note illustration:

Miraculously, through a series of emails, we refined that concept into a working PCB design. Mirek had it printed by JLCPCB, soldered on a bunch of SMD components, and shipped the PCBs (along with some metal brackets he had his friend Adam fabricate) by February.

In the midst of that journey, I had another major surgery, and wound up working on a so-far-still-secret-project that soaked up the entire month of March 2023.

So here we are today: between spare hours in March and a few weeks' time finally testing this thing with all the cards this month, I've uncovered one or two minor quirks with the build, designed a 3D-printed base (pictured above) that supports up to a 4090-sized behemoth PCIe card (pictured below), and documented everything in our open source Pi4GPU repository.

The card has a standard standard PCIe x4 (physical) edge connector, and it plugs into a special PCIe-to-PCIe adapter board, which fits neatly into the recessed part of a 3D printed base.

The graphics card slots into the x16 (physical, only x1 pins are connected) slot on the adapter board, and if needed, you use an external power supply to power beefier GPUs.

The Pi4GPU board itself can be powered via USB-C or 6-pin PCIe (internal edge), or via 12v barrel plug (external edge). It also has 2 USB 2.0 ports, a full-size HDMI port, and a 1 Gbps LAN port (all through the rear PCIe bracket):

It can physically slot inside a computer in a motherboard, but that is not recommended, as I haven't tested that configuration...

External GPUs on a Raspberry Pi (or other Arm SBCs)

So we have this card, and I can plug it into a variety of graphics cards... do any of them work? Or for avid readers of this blog—has anything changed since last year's update?

Well... it's still a bit grim. So far we only have some older AMD cards working with a kernel patch for the radeon graphics driver, and the SM750 GPU which is used on ASRock Rack's M2_VGA, using this patch to the sm750 driver.

In some positive news, though, Nvidia's proprietary driver no longer hard-locks-up when it hits the memory errors on the Pi. Now it will attempt to load, but then error out. That would make debugging easier... if the driver source were fully open and available.

Unfortunately, it's not.

And on AMD's side (as well as other vendors for other PCI Express cards, like Google's Coral TPU), there's no desire to either maintain a fork of their drivers or spend time hacking their driver to work on the buggy PCIe bus on the Pi.

And the Pi isn't alone—it seems other SBC SoCs like the Rockchip RK3566 and RK3588 have a slightly broken PCIe implementation as well. 'Cache coherency' is the problem—these ARM SoCs that have their heritage in TV boxes and embedded devices don't have fully working PCI Express implementations.

Other ARM chips do, however, like this Ampere Altra Development Platform that just arrived courtesy of Ampere:

It has a fully functional PCI Express implementation, and also has 128 lanes of PCI Express Gen 4... which is about a zillion times more bandwidth than the single PCIe Gen 2 lane on the Pi and similar-era SoCs.

Check out my initial review of the Ampere system: Testing a 96-core Ampere Altra Developer Platform.

Apple's M-series chips might have even more bandwidth (per lane), but there's no easy way to get at the PCI Express expansion on them. Maybe the upcoming Mac Pro will make it happen, but I'm not holding my breath.

Video

Check out my video with even more detail about the project: