Note: My home lab consists of 4 HP Z6 G4 workstations with 192G ECC RAM and a Xeon Gold 6138 in each, as well as a horde of other much smaller machines running in conjunction as a Kubernetes cluster. I run Rook Ceph in the cluster and have multiple NVMe drives (though limited to gen3 PCIe speeds) on each of the HP nodes, as well as on several other random nodes. I also have a very small, low-power Proxmox cluster. My network is primarily UniFi hardware, though I have a Dell PowerConnect 8024F I use for 10gE LACP connections to many of the nodes/servers. I have a Mellanox SX6036 coming soon so I can go full 40gE across the cluster(s).
So, I picked up a bunch of parts for cheap during the recent Prime Days sale in order to fill out a case I had lying around and make another Ceph OSD node.
Other parts I had lying around include an ATI FirePro 2450 X1 card and dongle. Having something like this is crucial as you need video for this motherboard to boot, but not occupy the main, CPU-backed PCIe X16 slot. That's for the riser card!
The motherboard has a chipset-backed, PCIe gen3 M.2 slot and a CPU-backed PCIe gen4 M.2 slot. As I intend to use this for Ceph OSDs, I put a crummy old 512G NVMe in the chipset-backed M.2 slot as my boot drive, and will be using the gen4 M.2 slot, and the 2 gen4 M.2 slots on the riser for Ceph OSDs, and the X16 slot on top of the riser started with a low-profile Intel X710 DA-2 dual 10gE SFP+ card (gen3 X8).
So... does it work? And if so, how well?
TL;DR — Yes! It works!
The X16 slot can be split to give you more NVMe drives at PCIe bus speeds, as well as have X8 lanes for a low profile NIC (ostensibly much faster than onboard 1gE/2.5gE).
Installation
You can't put the riser card in until you configure the BIOS to recognize the video card in the second X16 (chipset-backed) slot and configure the primary X16 slot to use bifurcation — x8 + x4/x4 in my case.
If you try to put it all in at once, it won't POST because it appears to be looping, looking for a video card in the primary X16 slot. Once I removed the riser (with NVMe drives and NIC), it booted without complaint. After I configured the BIOS to turn on bifurcation as indicated, I let it boot once without the riser to ensure the BIOS was fully updated. I then shut it down and put the riser card, complete with 2 NVMe drives and NIC in and it booted right up!
Performance
I booted into an Ubuntu 22.04 server image I had on USB. I got things to a space where I could open multiple additional terminal windows at once (Alt F#s), and installed fio so I could do some performance testing. My primary concern was not to be particularly thorough, but to see what the uncached read performance in direct mode would do if I tried to read from all 4 NVMes (including the boot drive) concurrently.
The result was illuminating. I was easily able to get between 4200MB/s and 4400MB/s from both the CPU-backed motherboard M.2 slot and the primary M.2 slot on the riser card. I think the bifurcation lost a little something in translation because the performance on the secondary M.2 on the riser card was better than PCIe gen3, but not the same 4200-4400MB/s I got from the other two. It ended up running between 3400MB/s to 3700MB/s — not a perfect score or a match for the other two, but definitely better than I get from pure gen3 PCIe M.2 slots, which tend to peak between 3200MB/s and 3300MB/s in the same test for me. I presume that makes this something more like an x8 + x4/x3.5 bifurcation, but it's good enough for me.
I was unable to run an iperf3 test concurrently with this, but my experience has these bifurcated risers giving full speed to the first and second "ports" in a bifurcated slot, and the final port (or two, depending) tends to run slower than the full slot count would, if truly evenly divided, so I'm inclined to believe that the NIC will do just fine.
Future
The next steps will be to try a Mellanox ConnectX-3 Pro at 40gE speeds in the riser slot on this Ryzen machine. I will also be trying to add the same riser to one of my other desktop-grade motherboards. It will be Intel-based, so the available PCIe lane count is lower (20 vs 24), but that should still work: 4 to the on-board M.2, and 16 to the X16 slot. If the bifurcation works as expected, this will allow me to have more OSDs on a node than before, and still have the 10gE or potentially 40gE speeds.
Is anyone else doing any crazy riser-card contortions like this?
So, I picked up a bunch of parts for cheap during the recent Prime Days sale in order to fill out a case I had lying around and make another Ceph OSD node.
Other parts I had lying around include an ATI FirePro 2450 X1 card and dongle. Having something like this is crucial as you need video for this motherboard to boot, but not occupy the main, CPU-backed PCIe X16 slot. That's for the riser card!
The motherboard has a chipset-backed, PCIe gen3 M.2 slot and a CPU-backed PCIe gen4 M.2 slot. As I intend to use this for Ceph OSDs, I put a crummy old 512G NVMe in the chipset-backed M.2 slot as my boot drive, and will be using the gen4 M.2 slot, and the 2 gen4 M.2 slots on the riser for Ceph OSDs, and the X16 slot on top of the riser started with a low-profile Intel X710 DA-2 dual 10gE SFP+ card (gen3 X8).
So... does it work? And if so, how well?
TL;DR — Yes! It works!
The X16 slot can be split to give you more NVMe drives at PCIe bus speeds, as well as have X8 lanes for a low profile NIC (ostensibly much faster than onboard 1gE/2.5gE).
Installation
You can't put the riser card in until you configure the BIOS to recognize the video card in the second X16 (chipset-backed) slot and configure the primary X16 slot to use bifurcation — x8 + x4/x4 in my case.
If you try to put it all in at once, it won't POST because it appears to be looping, looking for a video card in the primary X16 slot. Once I removed the riser (with NVMe drives and NIC), it booted without complaint. After I configured the BIOS to turn on bifurcation as indicated, I let it boot once without the riser to ensure the BIOS was fully updated. I then shut it down and put the riser card, complete with 2 NVMe drives and NIC in and it booted right up!
Performance
I booted into an Ubuntu 22.04 server image I had on USB. I got things to a space where I could open multiple additional terminal windows at once (Alt F#s), and installed fio so I could do some performance testing. My primary concern was not to be particularly thorough, but to see what the uncached read performance in direct mode would do if I tried to read from all 4 NVMes (including the boot drive) concurrently.
The result was illuminating. I was easily able to get between 4200MB/s and 4400MB/s from both the CPU-backed motherboard M.2 slot and the primary M.2 slot on the riser card. I think the bifurcation lost a little something in translation because the performance on the secondary M.2 on the riser card was better than PCIe gen3, but not the same 4200-4400MB/s I got from the other two. It ended up running between 3400MB/s to 3700MB/s — not a perfect score or a match for the other two, but definitely better than I get from pure gen3 PCIe M.2 slots, which tend to peak between 3200MB/s and 3300MB/s in the same test for me. I presume that makes this something more like an x8 + x4/x3.5 bifurcation, but it's good enough for me.
I was unable to run an iperf3 test concurrently with this, but my experience has these bifurcated risers giving full speed to the first and second "ports" in a bifurcated slot, and the final port (or two, depending) tends to run slower than the full slot count would, if truly evenly divided, so I'm inclined to believe that the NIC will do just fine.
Future
The next steps will be to try a Mellanox ConnectX-3 Pro at 40gE speeds in the riser slot on this Ryzen machine. I will also be trying to add the same riser to one of my other desktop-grade motherboards. It will be Intel-based, so the available PCIe lane count is lower (20 vs 24), but that should still work: 4 to the on-board M.2, and 16 to the X16 slot. If the bifurcation works as expected, this will allow me to have more OSDs on a node than before, and still have the 10gE or potentially 40gE speeds.
Is anyone else doing any crazy riser-card contortions like this?