In my video about the brand new Raspberry Pi 5, I mentioned the new external PCIe port makes it possible to boot the standard Pi 5 model B directly off NVMe storage—an option which is much faster and more reliable than standard microSD storage (even with industrial-rated cards!).
Enabling NVMe boot is pretty easy, you add a line to /boot/firmware/config.txt, modify the BOOT_ORDER in the bootloader configuration, and reboot!
Of course, you'll also need to get Pi OS onto the NVMe, and there are a few ways to do that—I'll walk you through my favorite method below.
Note: Raspberry Pi announced they are developing an official Raspberry Pi M.2 HAT, but there is no word on a launch date for it yet. I am also tracking other PCIe HATs for Raspberry Pi 5 here.
Enable the external PCI Express port
Note: If using a HAT+-compliant NVMe adapter (like Raspberry Pi's own NVMe HAT), you do not need to enable the external PCIe port—it will be enabled automatically. But you can still force PCIe Gen 3 speeds using the option below.
First, enable the external PCIe port on the Raspberry Pi 5. Edit /boot/firmware/config.txt and add the following at the bottom:
# Add to bottom of /boot/firmware/config.txt
dtparam=pciex1
# Note: You could also just add the following (it is an alias to the above line)
# dtparam=nvme
# Optionally, you can control the PCIe lane speed using this parameter
# dtparam=pciex1_gen=3
I have the pciex1_gen=3 part commented out above because Raspberry Pi allows you to tweak the bus speed (you can choose Gen 1 for 2.5 GS/s, Gen 2 for 5 GS/s, and Gen 3 for 8 GS/s), but the port is only rated for up to PCIe Gen 2 speeds.
In practice, I have been able to run multiple NVMe SSDs at Gen 3.0 speed (getting up to 900 MB/sec) on my alpha Pi 5, but YMMV—PCIe can be very fickle, depending on the quality of the FFC cable and connections on your own setup.
Set NVMe early in the boot order
The PCIe connection should work after a reboot, but your Pi won't try booting off an NVMe SSD yet (at least, if there are any other boot devices present!). For that, you need to change the BOOT_ORDER in the Raspberry Pi's bootloader configuration:
# Edit the EEPROM on the Raspberry Pi 5.
sudo rpi-eeprom-config --edit
# Change the BOOT_ORDER line to the following:
BOOT_ORDER=0xf416
# Add the following line if using a non-HAT+ adapter:
PCIE_PROBE=1
# Press Ctrl-O, then enter, to write the change to the file.
# Press Ctrl-X to exit nano (the editor).
Read Raspberry Pi's documentation on BOOT_ORDER for all the details. For now, the pertinent bit is the 6 at the end: that is what tells the Pi to attempt NVMe boot first!
Reboot your Raspberry Pi 5 to make the change take effect.
NVMe boot won't work unless you have the external PCI Express port enabled, and there's a working NVMe drive with a valid boot partition! If you don't have that (e.g. you used Raspberry Pi Imager with an external USB NVMe adapter to flash Pi OS to an NVMe drive from another computer), then follow the steps in the next section to clone your existing Pi OS install to an NVMe SSD.
Flash the SSD with Raspberry Pi Imager
To get the NVMe SSD to boot your Pi, it needs to have an OS. One option would be to clone an existing installation to it using rpi-clone or some other tool (see below), but my preferred option is to flash a fresh Pi OS install using Raspberry Pi Imager.
- Install Pi Imager and open it
- Plug your NVMe SSD into your computer using a USB to NVMe adapter
- Choose an OS to install
- Choose the drive (connected through your adapter) to flash
- Click write (and set any options you'd like)
Once flashing is complete, pull the NVMe drive, attach it to your Pi 5, and it should boot off it (with or without a microSD card inserted)—assuming you have the bootloader up to date and set the BOOT_ORDER appropriately!
Clone your microSD boot volume to an NVMe SSD
Assuming you already have Raspberry Pi OS on a microSD card that is booting your Raspberry Pi 5 internally, and the NVMe SSD is connected and visible (check if you see a device /dev/nvme0n1 after running lsblk), you can use rpi-clone to clone the internal microSD boot volumes to your NVMe SSD:
# Install rpi-clone.
git clone https://github.com/geerlingguy/rpi-clone.git
cd rpi-clone
sudo cp rpi-clone rpi-clone-setup /usr/local/sbin
# Clone to the NVMe drive (usually nvme0n1, but check with `lsblk`).
sudo rpi-clone nvme0n1
Note: I'm using my fork of
rpi-clone, because the official version has gone unmaintained—see NVMe support. Huge thanks to@billw2for buildingrpi-clone!Note 2: You may want to wipe all disk partitions before cloning:
sudo umount /dev/nvme0n1p? sudo wipefs --all --force /dev/nvme0n1p? sudo wipefs --all --force /dev/nvme0n1 sudo dd if=/dev/zero of=/dev/nvme0n1 bs=1024 count=1
NVMe behind a PCIe bridge / switch
Currently the Raspberry Pi 5 only exposes one PCIe lane externally—though there are four more lanes taken up by the RP1 chip. Typical PC motherboards have a number of lanes to play with, so you often find two, three, or even four M.2 NVMe slots on high-end motherboards.
Even there, some motherboards have PCI Express switches (or 'bridges') which allow multiple PCIe devices to share the same lane or lanes, in a similar way an Ethernet switch can allow multiple computers to share a single network connection.
On the Compute Module 4, bootloader space constraints prevented NVMe boot if you used a switch, but I wonder if that restriction is lifted on the Raspberry Pi 5—and if so, is it already implemented?
As of now, no. I can see and use an NVMe SSD through a PCIe switch, but I am not able to boot the Raspberry Pi 5 from it, unless it is directly connected (as the lone PCIe device on the bus).
I've opened an issue to ask about this feature in the Raspberry Pi firmware repo: Can't boot Pi 5 via NVMe behind PCIe switch / bridge.
Comments
Well done Jeff, I don't know where you find the time! Hopefully we will see some decent Pi5 nvme NAS hardware if PCIe switching is reliable.
My Pi5 4GB arrived yesterday, so retail has started shipping - at least in the UK.
I am seeing some spurious nvme boot disk issues (the same nvme to USB adapter and SSD from my Pi4). I'm investigating root cause and will open an issue if I can't fix it.
Nice to hear your Pi5 has turned up. My 8GB is still sitting at “waiting to be picked” at Pimoroni. I was hoping it would at least ship on the 23rd.
I wonder if it is possible to bridge two raspberry pi 5s together via their pcie buses?
The purpose would be to have one running Android TV and the other running Hyperbian? That might avoid having to connect an external video decoder/encoder for ambient style lighting for the LED strip around the rim of the TV.
nice work bro
Hi Jeff,
can you suggest me, please, what I have to buy to attach an SSD NVMe to the Raspberry Pi (and boot the OS from the NVMe)? What kind of device are using you?
Regards
Massimiliano
I think Jeff is using a dev board from the Pi foundation. There will be actual PCIe HATs available in the future though, the foundation had a picture of one of their M.2 PCIe HATs in one of their blog posts about the Pi5
Nice one!
I'm currently using an SSK M.2 NVME SATA Adapter plugged into the USB 3.0 port with a WD Blue SA510 2TB M.2 SATA SSD (Prime day sale purchase). It was not happy at first, but after a little trial and error it is working perfectly. The USB speeds are amazing, copying files of my Pi 4 takes hours to a USB memory stick, then literally seconds or minutes to copy from the drive to my Pi 5.
I can't wait for an official PCIe M.2 adapter, hopefully they are wallet friendly. Currently transferring some of my old Docker containers from the Pi 4 to the Pi 5 (mainly for the extra storage and speed for Jellyfin and NextCloud, my Pi 4 has a 500GB M.2 SSD in the Argon ONE M.2).
If anyone intersted, you can run the rpi5 at max performance without a PD psu, only a +- cable and 5v 5A+ psu is needed and this row in the eeprom config file:
PSU_MAX_CURRENT=5000
Fresh and crusty info from rpi support.
Excellent! I was wondering how we might be able to bypass the current restrictions in cases where we bypass the power supply
If you'd like to add an M.2 NVMe SSD to your RPi5, you can order one here:
RASPBERRY-PI SC1166 RPI 5 M.2 HAT
https://www.newark.com/raspberry-pi/sc1166/rpi-5-m-2-hat/dp/82AK3964
0 In stock
Non-Cancelable / Non Returnable
Price for each: $37.63
Seems like this would preclude the use of the fan for active cooling of the BCM chip, no?
Supposedly the official M.2 HAT would be high enough the active cooler could still run and get enough airflow underneath. Supposedly
Any reason not to use an M,2 to USB adapter card in one of the USB3 connectors while waiting for the official M.2 HAT? It may not be as fast, but this might save a few steps down the road.
Yep! That's definitely an option, it just means a bit more of a mess on the desk until something in HAT form comes :)
what is the bootime on nvme ?
Similar to microSD, around 6-8 seconds with Pi OS. The boot time is highly optimized, and the NVMe's speed advantage isn't as great there as other storage-related tasks (like file copies, installs, launching heavy apps, that sort of thing).
Hi Jeff
Would be grateful if you can give some advise on what types of NVMe M.2 SSD drives would be compatible. Would Gen4 x 4 PCIe work, or do I need to stick with Gen3 x 4 PCIe?
PCIe devices are usually very good about backwards compatibility / slower speeds—so you can generally take any drive (even PCIe Gen 5) and drop it into even PCIe Gen 1 x1 and it will work, just a bit slower.
Thanks very much Jeff. Would you generally say go for PCIe Gen5 SSD over earlier gen version for raspberry Pi 5, or would there generally be no benefit given it will downgrade?
I would go for a Gen 3 or Gen 4 SSD for now—Gen 3 especially is getting cheaper as more people move on to Gen 4, so for example, TeamGroup seems to be doing a lot of sales on their older drive versions, which are still very fast on Gen 3 systems.
Thanks very much Jeff. Really appreciate the advice. Still waiting for my Pi 5 on delivery but I'm told it should be soon!
Hi Jeff,
This is great stuff, thanks for sharing! For booting off a switched NVME, would it work to have /boot on an SD card, and / on the NVME? That way the RPi boot loader isn't hampered by the switch, but by the time Grub is running off the SD, it should be able to mount and load the rest of the file system from the NVME. Would obviously need to run a full installer instead of flashing an image, but that's not the most awful process.
Thanks for this post, Jeff! Instead of waiting for the hats and adapters, I use NFSv4 to share ZFS pools and file systems on the Pi with an HP Proliant Microserver which has 4 SATA bus bays. Speed over a wired intranet connection isn’t that much of an issue for me, I’m mainly interested in mirrored redundancy, and capacity. I run Ubuntu 23.10 on the HP, and Bookworm-based RPiOS on my Pi. Both with ZFS installed.
Hello Jeff,
First off I love your videos and blog. They are so helpful and full of knowledge. I was wondering what the board is called that you are connecting the flex pci cable to in your pictures that allows you to use the pcie port? Thanks in advance
That is a board Raspberry Pi designed for testing and evaluation of the PCIe port on the Pi 5. Unfortunately they are not yet selling it or anything like it. Hopefully a third party will!
Jeff,
I have always been using USB3 to NVMe adapters on my Pi4's and Pi3's without issue. I tried the same while waiting for the Pineberry board with my Pi5, and it will occasionally try to boot. Once it does, its just a matter of minutes before it comes up with a write error and "file system is read-only" error. booting from SD card and using the NVMe as a storage drive, it eventually does the same. Have tried different NVMe drives in the adapter, same issues. these same drives/Adapter work fine with other Pi's with Bookworm. Any ideas?
Which power supply are you using? If it's not the official 5A 5V power supply, it could be supplying inadequate power to the USB ports.
I am having the same issue - using an official power supply, an Orico M2 NVME enclosure, and a Firecuda 530. Seems to partially boot and then fail.
Using an SD card the Pi works fine
I'm having the same issue with a Samsung SSD and USB 3.0 adapter. Will boot from the SSD but very quickly I get a blank screen with the wastebasket only displayed, no menu bar. Using the official Raspberry Pi5B power suplly. I've also tried to use a powered USB 3 hub but I have the same issues. The same setup works just fine on my RPi4 systems using the same OS image. So, I'm wondering if there is an underlying issue with the USB ports on the RPi5!
My Pi5, Crucial P3 4TB and Pineberry HAT Bottom are all on their way...
Have you any recommendations on SD Card boot/NVME vs no SD Card and only boot from NVME?
I'm wondering whether it is better to use the SD Card for boot and logs (e.g. Pi Hole logs) to avoid wearing out the more expensive NVME drive (which in reviews seems to have a downside of low(ish) number of writes. Extra benefits would also be that any OS corruption could be resolved with a replacement SD card leaving files on the NVME drive untouched and the ability to have a backup copy of OS while adding any new features. I intend to use the Pi 5 as a NAS file server, media centre, Pi Hole. I'm sure I'll find some other services to run on it (NTP? personal cloud server?). I'm hoping that the increase in Pi 5 performance will be suficient and don't expect to need the fastest drive speed, it's more about the ability to store all the files.
Hi and thanks for the amazing content.
I got my Pineberry M2 Hat today and thought I would share my experience for those like who don't have a M2 usb adapter.
I had a working install on an SD card, I booted on it with the hat and a new drive on it (with the config.txt dtparam=pciex1)
Ran SD Card Copier and started copying the content of my sd to my drive.
And right there I made 3 huge mistakes:
1) I didn't tick the New Partition UUID
2) Realising I didn't have the rpi-eeprom-config installed, I installed it, then though it came to me that I changed the content of the SD while copying it to the new drive, so I interupted the copy.
3) Went on with changing the boot order and rebooted, thinking it would fail over to the SD card.
But no the change worked, it tried to boot from the half copied Drive and Kernel Panicked
I could still boot on the SD but only if the hat or drive was disconnected.
But if the Pi was booted without the Drive, I couldn't get it to hot plug and re-appear on the system.
So I changed the boot order back not to boot on the drive but the SD, while the drive was connected. But this doesn't work it would always pick the drive and kernel panic.
I finally understood while the boot order was not respected: I didn't change the partition UUID. And the boot loader uses decides to use the SD, take's its UUID assign it has the one to boot on, then pick the drive instead as it has the same UUID.
Fortunately I had another SD card I cloned my system on and ticking Generate new partition UUID this time, booted on that one with the drive connected to it and clone again to the drive, changed the boot order to use the drive first, rebooted and it works fine now.
Long story short, if you have a system on an SD:
1) just plug the hat and drive,
2) use SD Copier, tick new partition UUID
3)Wait until the copy is over
4)Change the boot order
5)Remove or not your SD and reboot
6) Enjoy your drive
I had to install rpi-eeprom first:
sudo apt install rpi-eeprom
Hi Jeff ... Would doing the steps out of order corrupt the Eeprom? I seem to have done that with the Pineberry pi hatdrive. I changed the boot order in the Eeprom now it seems to be freezing up after about a minute and I am not able to enable the pcie port. It also seems to be overheating and shutting the fan off. I'm a bit stumped and figured you might have some insight.
I bought the same Pineberry Pi hat and have the same issue as you have if I take out the SD card. I noticed that however I changed boot order to BOOT_ORDER=0xf416 and added the line PCIE_PROBE=1 then I keep looking for the SD card during boot henge I inserted the SD card again. And then it actually boot.
I installed Raspbian on the SD card and Stellarmate OS on the NVMe SSD. And as it looks now. I have the Raspbian desktop picture underneath the Stellarmate OS.
I'm no Linux shark. So its just make me confused why works like this.
I had same issue. I tried to reboot from the OS and nothing worked. I have now figure out the if you reboot from OD you get the exact behaviour as you describe. But if you shutdown, then then disconnect the power cord and plug it back again or you reboot and let it stall in the no-booting loop and then disconnect/connect the power cord. The the instructions actually works
I’ve been struggling with finding a way to make this work with a 4TiB SSD. Using the imager or rpi-clone converts the 3.6TiB GPT partition back to 2.0TiB MBR. Should I create a 3rd partition, clone the micro SD to p2, then convert p3 to GPT?
I also have the same problem, cannot get root partition beyond 2TB:
nvme0n1 259:0 0 3.6T 0 disk
|-nvme0n1p1 259:1 0 512M 0 part /boot/firmware
`-nvme0n1p2 259:2 0 2T 0 part /
I encountered this issue too. pi images are mbr based and therefore are limited to 2TB. You can use gdisk to convert the created mbr based paritions to gtp based, but this also changes the GUIDs, so as a minimum you would have to update the GUID referenced in the cmdline.txt file and also fstab. This is probably possible if you are running from an sd card image and mounting the nvme drive temporarily to make these adjustments.
There may be other changes needed to, and I ended up keeping the pi booting from an sd card and just mounting the nvme drive purely as a data disk, this gets me the full 4tb, but I waste an sd card and get slower boot times.
I have this working now, although my solution was suboptimal and I did a lot of it manually, and made quite a few mistakes on the way.
I'm running a Crucial P3 4Tb which I picked up from Amazon for £154, with a 52PI N04 HAT in a Geekpi in a KKSB 'HAT' case. I wouldn't recommend the latter though as it's far too fiddly and without a removable lid impossible to change the NVMe without totally dismantling it.
rpi-clone may do some of what you need but I suspect not the >2Tb use case - it didn't work for me and I focussed on the task in hand not what that tool needed to do. I wish I had looked at it more closely as it has the correct rsync syntax - you need to exclude /mnt/clone for example else you end up recursively copying files. I had written over 850Gb to my nvme before I realised something was wrong.
It might be just a case of changing the script to use gdisk rather than fdisk. I'm not sure. The good news is that the NVMe formatted with GPT works fine and I now have one big 3.5Tb partition to work with. Whether this was a good idea or not (maybe I should have split this into two partitions) I don't know but it's working fine now.
Hey Jeff, I'm completely new to Raspberry Pi world. I ordered my Raspberry 5 8G and will have it soon. My question is could I flash Pi OS to M.2 from PC then plug it in with a hat. Could I boot my Raspberry 5 this way? (without MicroSD card.)
Hey Jeff, quick question: any news on the official top/bot NVMe hat+ for pi 5? I mean, from the RPi Foundation? Have you heard anything about a possible release date?
Thanks!
Been asking about it. The plan is still "soon" — I'm interpreting that to mean in the next month or so. I know they've had the design ready (they have pictures of it!), they've probably been working on getting the production and documentation spun up. There are a few other options that will be shipping in Jan/Feb (Pineberry is already shipping now!).
Thanks for your reply Jeff! Well, i know pineberry and a few other are already up and running but i got a feeling i should stick with the official, especially because i have a custom case in my pi 5 (geekworm P511-B) and i think none of the nvme adapter is 100% compatible. I think either an L-shape nvme adapter or a bottom adapter might cut it, depending on the length of the pcie cable. And i suspect that the official adapter will have better support and functionality in the long run. Keep us posted mate, great work!!
There are a couple of thing I haven't been able to determine out while learning more about NVMe in relation to the PI5.
1) Is there a size limit?
2) Are GenX devices forward and backward compatible?
I would like to make a JellyFin media server using a PI5 and a Geekworm model: X1001 NVMe hat. I would like to put the highest capacity NVMe on it that I can get, which is around 4 terabytes.
Thanks in advance
Walt Williams
Technically the size limit for an NVMe is very high—higher than any M.2 drive available right now (for any price). However, there's currently an issue trying to flash Pi OS to any M.2 SSD larger than 2 TB, see: Supporting Larger Drives (GPT).
Hey Jeff,
I just got a raspberry pi 5 (my first pi). I have a geekworm X1002 pcie m.2 Nvme ssd bottom hat. I tried following your guide, and I can’t get the nvme ssd to show up in the terminal with “lsblk” (after having enabling pcie external connector and rebooting. Have you tested this hat? Or know anything about it?
Hello Jeff, thank you for your fork of rpi-clone which helped me to setup my Pimoroni NVME Base. After running the cloning process, the Pi5 did not boot from the NVME drive. After some diagnosis, I discovered that the PARTUUID in the CMDLINE.TXT contained the value of the microSD card. After editing the file with the correct PARTUUID value, the PI5 booted from the NVME drive. Maybe this is the reason that the rpi-clone script does not always work and can help some others.
Ah, good catch! That makes sense.
And looking in the rpi-clone issue queue, it looks like there are a couple issues open for this:
I'm creating another issue on my fork to track fixing this... Bookworm needs tweak for /boot/firmware directory move
Save yourself a couple bucks... on my rpi5b 8GB, i installed, on the official ubuntu 22 image, Nextcloud, Plex Media Server, Airsonic, webmin, postfix, samba, nfs, mdadm, lvm2, and a few other tricks, and with my rpi5, under MAX load serving three transcoded plex streams, two mp3 music streams (1 airsonic, and 1 nextcloud), navigating nextcloud, and navigating all over webmin all simultaneously, the rpi5 barely touches 1GB of real RAM usage. Heck, i can do most of this on my banana pi m3, and my banana pi m64 also (except no transcoding), and they don't get over 1GB RAM usage either. Save your money on 4GB and 8GB models if You're building little NAS/RAID servers... it won't use it.
I have the pineberry bottom HAT, its pugged in and the green light is on the board is lit, I assume this is working.
But when I run lsblk I do not see the nvme.
I may wait for the Official Hat.
The green light just means the board has power—you also need to enable the PCIe port in config.txt and reboot, and if you want to boot off the NVMe, you will need to modify the EEPROM as well (and reboot!).
Does anybody knows how to set the PCIe lane speed when using Ubuntu 23? There is no /boot/config.txt only automatically generated kernel config file. If create config.txt or add mentioned settings to existing config file nothing happen :(
Does Ubuntu 23 have a
/boot/firmware/config.txtfile?No it doesn't. How can I fix this issue
Awesome technical writing Sir, came here after watching several Youtube videos you have produced - I just love all the info I can get.
Now, I thought I heard or read somewhere a "list" of NVMe drives that work better than others and some that don't work well at all for the RPi5 ... do you have that info somewhere already - OR - can you maybe provide it somehow please ?
I just want to get the most for my money, a 1Tb for a bottom mount 2280 ...
Great article as always on the pi! Just to let others know, I am using a Waveshare PCIe to M.2 Hat and have the same problem some others are having with their hats, where in lspci and lsblk, neither the hat or NVMe drive show up. I have enabled the PCIe port in the config.txt and the hat even lights up its power light that it has power. There most be some additional tweaking that needs to be done with these third party m.2 hats to get them to work.
https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/2052861 Think the Ubuntu 23.10 kernel needs updating...
Hi Jeff,
Thanks for the write-up!
I've had no issue getting the nvme detected after updating the eeprom firmware (needed the desktop version of pi-os to get the latest). However I'm unable to boot from the nvme after cloning. doing an lsblk I have this:
mmcblk0 179:0 0 14.9G 0 disk
├─mmcblk0p1 179:1 0 512M 0 part /boot/firmware
└─mmcblk0p2 179:2 0 14.4G 0 part /
nvme0n1 259:0 0 931.5G 0 disk
├─nvme0n1p1 259:1 0 512M 0 part
└─nvme0n1p2 259:2 0 931G 0 part
I've tried adding the PARTUUID to cmdline.txt (it was not even in there) but to no avail. How do I move the /boot partition from the SD to the NVME?
Have you tried unplugging the microSD card too? If you do, and it still doesn't boot off NVMe, double check the EEPROM setting for boot order (6 must be at the end), and then try
rpi-cloneagain—make sure you're using the very latest version, because there were some Pi OS changes recently that broke it.Will this possibly work if i have booted from a hdd instead of the ssd? (I’m using a pi 4 and want to migrate to a ssd)
Does the Pineberry hat bottom support ubuntu? It works using Raspberry PI OS 64 bit but I can't get Ubuntu Desktop 23.10 64 bit to see it.
I have similar issue. The Pimironi hat is seen with NVME drive (came with the base as a supported drive). I run Ubuntu Desktop and wish to clone this to boot from NVMe. Sadly it's not detected with Ubuntu (lsblk does not show any NVMe connected) However RaspiOS does show the device. I've edited the /boot/firmware/config.txt, I have tried updating the eeprom from from RaspiOS as it shows one much later than Ubuntu does. (Jan 05, 24 vs Sept 25, 23 this is after updating it!). I guess Ubuntu is just behind with this. I'm probably missing something simple, but an initial scrape for info on this hasn't provided anything obvious. Any suggestions Jeff (or even Red Shirt Jeff') would be greatly appreciated.
I also have this issue. The best answer I found is here:
https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/2052861
Apparently a bug was introduced in the 6.2 kernel that only got fixed in 6.6.x (I'm on the latest for 23.10, which is 6.5.0, so broken!). I guess the only way is to wait for 24.04, due in late April. Even if I could upgrade the kernel on my 23.10 to something later than 6.6, how could I create a bootable NVMe if there's no Ubuntu image out there with the kernel fix?
Jeff, with your amazing video and blog guide, I've successfully got my Raspberry Pi 5 to boot off an NVMe drive - using a base rather than a HAT board.
I'd appreciate you advice on whether there is any way to also add a USB Boot option on top of this - without killing the whole setup.
The aim is to have a way to configure the Pi so that it will always try booting from an OS on an SSD drive plugged into one of the USB 3.0 ports, and if not there, boot from the NVMe connected to the PCIe bus. I've seen "USB BOOT" only videos that mention using the Raspberry Pi imager to download the USB BOOT image to a spare SD Card and flash the Pi device with that. But I'm not sure if that will completely undo the NVMe boot I've got working.
Would welcome any guidance on how to have both USB BOOT and NVMe boot options enabled on the device.
Yes—you would need to modify the boot order in the EEPROM to have the USB bit last in the string, then NVMe after that.
I getting my 8gb Pi-5 soon. I just finished watching your FINALLY! NVMe SSDs on the Raspberry Pi. I plan to use a PCI hat and install ans MVMe board on it. I want the hat to install under the Pi-5 for better cooling and I will be using an Active cooler on the PI. Ideally I would like a hat that will take two MVMe drives that I can have one boot from and the other that I can mount and occasionally use SD Copier to backup the active drive to the second drive. I also need a case that will accept the configuration.
Do you have any recommendations?
If you have any recommendations and would like to email me the information, I can be reached at [email protected]
Love your videos and thanks for doing what you do.
After hours of troubleshooting, it appears required to add dtparam=pciex1_gen3 to config.txt when using Geekworm X1002 base with Crucial T500 PCIe Gen 4 NVMe M.2 SSD 1TB. Without it, boot times were slow (anywhere from ~35 seconds to ~240 seconds) and dmesg was littered with pcieport and nvme errors.
Thank you,
I just got my hat w/NVME working thanks to this.... (I was missing the PCIE_PROBE doh!)
Hello guys, I need help.
I bought a Kingston NV2 NVMe PCIe 4.0 M.2 2280 SSD, to this day I still cant change the boot from my sd card to my ssd. Does I need to change the nve to sda when I do the steps that Jeff posted?
lsblk:
sda 8:0 0 465,8G 0 disk
├─sda1 8:1 0 512M 0 part
└─sda2 8:2 0 465,3G 0 part
mmcblk0 179:0 0 119,1G 0 disk
├─mmcblk0p1 179:1 0 512M 0 part /boot/firmware
└─mmcblk0p2 179:2 0 118,6G 0 part /
Hi Jeff,
Just a quick message to let you know that I'm using a 1TB Lexar NM790 SSD on a Pimoroni NVMe Base and getting fantastic results in Gen 3!
Read and write speeds suggest that it's as close to saturating the PCIe bus as makes no difference, and the pibenchmarks score is around 58,000. Also, journalctl -b | grep pcieport has shown zero errors during sessions of 24 hours or more.
Maximum power consumption is only 3.7W for long sequential writes, so it's well within the 1A (5W) limit of the Pi5's PCIe connector.
The SSD also has a rated endurance of 1,000TBW, so it's a no-brainer for anyone with the space for a 2280.
This 8GB Pi5 absolutely flies as a low-power desktop machine. I couldn't be happier!
There is one quirk to this... if you leave the Micro SD card in the slot... the Raspberry Pi boot loader is not smart enough to NOT mount the Micro SD card as '/boot', for example, before clone:
# lsblk
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 1 0B 0 disk
sr0 11:0 1 1024M 0 rom
mmcblk0 179:0 0 29.5G 0 disk
├─mmcblk0p1 179:1 0 512M 0 part /boot/firmware
└─mmcblk0p2 179:2 0 29G 0 part /
nvme0n1 259:0 0 1.8T 0 disk
After clone, change of boot loader change and reboot:
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 1 0B 0 disk
sr0 11:0 1 1024M 0 rom
mmcblk0 179:0 0 29.8G 0 disk
├─mmcblk0p1 179:1 0 512M 0 part /boot/firmware
└─mmcblk0p2 179:2 0 29.3G 0 part
nvme0n1 259:0 0 1.8T 0 disk
├─nvme0n1p1 259:1 0 512M 0 part
└─nvme0n1p2 259:2 0 29.3G 0 part /
Notice /boot sourced from the Micro SD card, but the root filesystem (/) sourced from the NVME. Talk about a split-brain issue! Believe this happens because the cmd.line.txt for both NVME and Micro SD card is the same? Changing the cmdline.txt and NVME with a unique root UUID might avoid the issue? But don't see an easy way to do this, once you have booted via the NVME. Boot loader bug? It should find boot file system ONLY on the PCIE bus, right? Even if there is a duplicate UUID?
The "split brain" problem you cited is (from my experience) non-deterministic. This was at least the case when booting CM4 instances on a Super6c board. It was "Most apparent" as each "reboot" provided different results, for each of the 6 CM4s, and different results each time I booted. The "boot_order preference" was clear, but the only way to "get what I demanded" was to interrogate all CM4s, and then ONLY reboot the one(s) that "got unlucky," rinse and repeat, until I had a full set of properly booted CM4s. When I reboot them all at once, there was only about a 1 in 8 chance that ALL 6 CM4 would boot properly, and mount all partitions correctly (i.e., mount NVME exclusively). It is unacceptable to leave a cluster in such a state... and reboots happen :-/
I waxed poetic about the issue in my comments at the end of:
https://www.jeffgeerling.com/blog/2022/6-raspberry-pis-6-ssds-on-mini-i…
In the end, my conclusion is that although the Pi appears to try to USUALLY follow the BOOT_ORDER setting, it is probably the case that "sometimes" one or more of the selected sources (listed early in BOOT_ORDER) will time-out (my wild guess), resulting in use of "Less preferred" sources of boot material .
My only work-around was to ensure that the alternatives were "no longer bootable." For me, with my CM4 having on-board MMC storage (that I was using to boot from BEFORE I installed NVME storage), this meanh using "parted" to repartition the the MMC, destroying the bootable partition. [Note: It is NOT sufficient to "toggle the bootable flag," as that doesn't seem to play a role in bootability selection].
For you, with a Pi 5, it may be sufficient to remove the unused MMC, or USB boot device.
This nondeterminacy is perhaps also at the root of some other posts I've seen about folks having a hard time booting from USB... but I have no hands-on proof that such is the case.
...and it would be nice to fix the apparent BOOT_ORDER selection bug. It might be as simple as a larger time-out... or it could involve better logic in the testing of each device for bootability, and possibly including some retry logic.
@jim, you are right, I continued to do more testing, and I discovered as you did the bootloader seems.... to get confused. Not only does this happen on the Pi, but it also happens (a bit different) but comparable on SBCs that use U-Boot, such as the IndieDroid Nova. I discovered how inconsistent the boot process is when I cloned my NVME setup to a Micro SD card, keeping the UUIDs the same, backing out the Pi EEPROM directive of 0xf416 to the oriignal 0xf41, which explicitly sets the boot order to be Micro SD. And what happened... Pi 5 booted via NVME, completely ignored the Micro SD card, and apparently ignoring the explicit 0xf41 setting. Nuts What I wanted was to boot NVME but have a ready backup on the Micro SD card. But that was not as easy as I thought it might be. The real solution... I believe to two steps, or features... one make the device selection consistently deterministic. And establish a boot menu option (comparable to say grub?) that would let the operator select the boot device, if not explicit default defined. U-boot has a simple menu system, but it ONLY is displayed on the (real) console. But since (as far as I know) Pi devices don't use U-Boot, the fact that there seems to be no common bootloader solution, is another key issue as well.
It would be cool to see the problem deterministically resolved. ...but until then...
There is a possibility that the timeout could be varied as a work-around, and although I didn't try it, there might be some possible work-arounds with listing the same (potential) boot source more than once in the setting!
If you are lucky, you could at least reduced the "probability" of missing a good boot source. For example, though I have not tried this... what would happen if you set:
BOOT_ORDER=0xf1666
There is a bit of a pun here, with the 666 as a devilish way to try to mitigate this problem... as I'm hopeful the errant (racey?) boot_order code was written simply using short iterative loop (so it is willing to try the same method more than once!).
Since the spec says(?) that you can provide 8 hex digits, you could even try:
BOOT_ORDER=0xf1666666
That would try numerous times to get the high-performance NVME boot... but eventually give up and go MMC. ... and if it couldn't get the MMC to work, it would try again.... rinse and repeat (with 6 times as many shots at the NVME source).
Please post what you learn... as others will surely be interested.
Is there a reason why you can't run it at x2 lanes of Gen 3? like "dtparam=pciex2_gen=3"
The "slot" is physically only has x1 lanes.
There must be something up with the latest firmware.
I went to Microcenter and bought an RPi5 8GB and Crucial P3 Plus M.2 to experiment with.
I used Balena Etcher to image Ubuntu Desktop 23.10 on the M.2 stick.
I installed the drive in a ZDE ZP510U NVME bottom mount board.
I booted the system and it worked out of the box with no edits whatsoever.
Life was good through a few reboots right up until I rebooted after doing sudo apt get update and sudo apt get upgrade.
It literally failed to boot, black screen and no activity.
I can boot from SD card with the same Ubuntu Desktop 23.10 image.
I used RPi Imager to make an firmware update SD card in hopes of getting back to the OG firmware but no luck.
Now if I try and NVME boot I get: https://i.imgur.com/1X3ewtM.jpeg
PCIe timeout: 0x0001e08f
Failed to open device: nvme
I have tried yet another drive on a new Raspberry Pi 5 and Geekworm X1002 this time, and I think I've narrowed down the problem.
What appears to cause the problem is a hang on reboot after editing the boot order in the eeprom.
So far I've not been able to recover the ability to boot from NVME once this happens. Not even if I boot from uSD and change the boot order back to stock.
I'm tired of ruining Raspberry Pi boards though, someone let me know when this is fixed.
@Dan
I have the same issue. After making the recommended changes, the Pi 5 boots to the net-install screen every time. I can hit to see the boot messages, and I get the same text shown in your image, except this comes up four times before it continues to boot normally. Total boot time is 90 seconds, which is incredibly slow. I can't find any way to work around this. SD card boot is fine and I have reinstalled RaspiOS Bokkworm three times with the same result.
Great. For every questions that comes up, the answer is here yet. Thank you Jeff💪
I added NVMe to 3 PI5's and both of the 8G units WLAN interfaces shut down after +/- 10min, but this behaviour is new they worked fine for 2 and 3 weeks respectively.
The 4G PI5 is not having this issue. I have swapped around the MVNe drives and base boards and the problem stays only on the 8G PI5's independent of the base board or drive on the boards... I have reset them to boot from the SD cards (because some posters said the problem was only when booting off the MVNe board) and this has had no affect. I am using the PI-5 27w supplies anyone else see this problem?
This post saved my sanity. I've been using ubuntu since 2009 and reflexively chose it on all my raspberry pis because there is often great support for the software I want to run. When it comes to pcie/nvme recently they've had bugs and it meant while I could get the image in place and I could seemingly make the settings changes, they would not take and I could not boot from my nvme drive.
Thanks for calling out that you prefer the more native raspian/pios, that worked for me!
Thank you Jeff! With your video and website and rpi-clone help, I managed to get my Pi 5 HAT and an NVMe SSD installed and cloned. Cloning took 11 minutes and I am happy as f*** now!
ahh yes, yet another "tutorial" for linux users only. Why oh why authors of all blogs/webpages (or almost all) that treat about rpi thinks that person who looks for help on their webpage are linux savvy? Im not, im trying to find the answer how to turn on that bloody ssd drive on my rpi5 (that ment to be emulation station based on batocera) and every single webpage i visited up for now have almost the same thing. Type this there, save it, should work, well done. Why theres no tutorial for idiots? is it really that hard to write step by step? do i really need to spend 3 weeks learning that shit? Im trying to get on to linux for last 20 odd years but i always fail, becasue all "help" on the internet think that i know linux. No i dont, i want to, but theres no user friendly tutorials like for windows, step by step.
so i think they added support now
i have the NVMe Base Duo for Raspberry Pi 5
and with the firmware update 17/05/24 it seems to works
If like you me you are attempting to do this with an NVMe drive that is larger than 2TB, salvation comes in the form of: https://github.com/mcpat-it/NVMe
Nice article - i took a differen aproach.
1. power up with nvme connected but no SD card and plugged into Ethernet port
2. press shift to network boot when prompted
3. install pi5 OS 64bit
thats it.
done - can still tweak boot.txt settings, but no need to mess about witg SD card etc
Any ideas??
I ran in to the same issue:
I am using a Pi 5 with a NVMe Base Duo for Raspberry Pi 5 board from Pimoroni and followed your directions and it worked flawlessly with a WD_BLACK M.2 2230 1TB drive. Thanks for the guide.
Category Test Result
HDParm Disk Read 373.94 MB/sec
HDParm Cached Disk Read 414.64 MB/sec
DD Disk Write 273 MB/s
FIO 4k random read 99417 IOPS (397669 KB/s)
FIO 4k random write 88658 IOPS (354632 KB/s)
IOZone 4k read 162682 KB/s
IOZone 4k write 152949 KB/s
IOZone 4k random read 72055 KB/s
IOZone 4k random write 165134 KB/s