I just wanna say that as of 22.12 release of TrueNAS SCALE, RDMA is doable in a few steps.
1. Add RDMA support to the system - I've went with OFED package from
Nvidia : Mellanox OFED (MLNX_OFED) Software: End-User Agreement
- add kernel support, enable nvmf and (optional) nfsover rdma (I'd like to further test proxmox as an initiator to see if there are any benefits). Main initiator "target" for this is vmware - post 7.0.2 which has al lthe bels and whistles to support rdma with iser and nvmeof.
results here should render the system rdma capable, have a nvme target rdma capable, iser target module etc.
2. rebuild scst - I did this with the lastest git (3.7.0 - same as what's already in truenas) ; Check the install script to make sure you are starting the services with the new binaries (/etc/init.d/scst is the script used to start the service wrapped under the "goodness" of systemd /etc/systemd/system/scst.service.d)
3. reboot and if everything is doing what's supposed to, see your iscsi targets sessions with RDMAExtensions Yes in the logs.
4. On the vmware side (if that's your poison, proxmox/kvm should be even easier)
esxcli rdma iser add - yeah, no gui afaik
add the relevant config to match vmkernel to rdma enabled adapters
add initiators, target config etc. - the usual stuff
I didn't pay too much attention to the vmware setup, but I think you have to have vsan enabled on the vmkernel adapter - and only one active physical adapter - there are way better guides around here so I won't insist on this side.
5. Quirks and other relevant info :
- I've yet to test nvme rdma target - will get back.
- So far I have iser enabled for some targets and test vm's running smoothly for a few weeks - no issues whatsoever. iser interaction with reboots for either initiator or target looks as smooth as possible. Mind you, in true rdma fashion (for those who played with infiniband if you remember) there is no traffic shown on the disk or the network monitoring in vmware
- Used adapters were cx4 and cx5 with no mods to the firmware
- yeah, I'm glad the days of modding cx3 and cx3pro's and hoping I wouldn't hit any random crashes seem to be behind.
- Switch used has only 10g ports so that won't allow for numbers to write (servethe)home
- There are a few things that could get tuned - one would be the iscsi target that only has 32q depth by default - but this is fine tuning in my book
- I only did a few benchmarks on a low performance zvol (spinning rust with flash log) just to see that we're not getting any crashes/lockups with high usage or network congestion.
- on the Truenas side, making apt executable should get you started for missing dependencies.
Let me know if I've missed any important piece and happy modding