Get it done my man! I know it's a slog...lots of F-bombs and "ahhh screw it, I'll do it tomorrow" type of reactions, but once done, it's...glorius.
Two racks...lost of cursing while getting them completed. Here's a funny story. In the compute rack (right rack in the picture above), I thought I had "finalized" my hardware config, built the sleds, mounted them in the rack, was gonna use iSCSI boot for everything (which did work as such), but...due to many different things, plans changed...
I ordered a whole bunch of those adapters that can hold one m.2 sata SSD and one NVME, so I can have SATA boot and one NVME in each sled. Then...had to unmount ALL of the sleds...one by one...add the adapter with SSD and sata cable...mount the sled back...
And there's a shit load of them...fun fun fun...
I've given up on Starwind. I realized I didn't really
need high availability per se, a 15 minute rsync is more than enough to give me the data security I need (This is a mirror of my prod, so the data is
already duplicated per se). So, once that requirement went away, I realized, well, why Starwind?
And it still doesn't do NVME-oF with HA, actually even without HA as well, still is stuck on the 20.04 kernel...no feature parity between different flavors (bare metal vs Proxmox VM vs Windows VM vs ...).
Moved to Truenas and it seemed to fit the bill...but then their RDMA restrictions kicked in...and the frustration began anew...
Been testing RHEL (specifically Almalinux) with Cockpit and surprisingly it works great! The installer is flawless, configuration is straightforward, the documentation is top notch, every flavor of RDMA under the sun, NVME-oF works OOTB...

I might stick with this, once I've put everything through its paces.