FreeNas/TrueNas RDMA Support (FR for voting)

[tsteine]

@BoredSysadmin This is very interesting.

I've personally abandoned Truenas Scale in the homelab in favor of running openzfs on my linux distro of choice, there is no reason why I can't simply test this, and see whether iSER has tangible benefits for iSCSI, or nfs over rdma.

 

[Rand__]

Lol, nah I am trying to understand and not to go pointing fingers

 

[Bjorn Smith]

Lets be honest - in a homelab its probably not really relevant with RDMA - but we are all nerds and want the best performance, so that is why we want this

 

[i386]

in a homelab its probably not really relevant with RDMA

When I copy files via robocopy from my workstation to my windows file server the server cpu stays at ~2.4GHz (it's a 1620v3 with 3.5GHz clock), so it probably saves some watts here and there

 

[Rand__]

A TrueNAS and iXsystems Questions & Answers Interview [with elaboration request on RDMA answer]

Hey guys, I just found this https://nascompares.com/news/truenas-ixsystems-nas-qa-answering-your-questions-in-2022/ There are some good answers for new users who are looking into NAS systems and are considering TrueNas as option, I like it I actually stumbled over this while searching for news...

www.truenas.com

If anyone has positive (or negative ) experience or expectations regarding RDMA then now might be a good time to chime in ...

He explicitly asked for it, even on non TN systems, so ...

 

[tsteine]

@Rand__

Judging from the the two posts you've gotten out of them, it seems to me like they haven't actually bothered testing RDMA themselves.

 

[Rand__]

Well I am not sure they didnt test it, it sounds rather like they did test with a certain setup and it did not yield significant improvement (which is o/c exactly what they say).
Now whether that was a best or worst case scenario test (or if its was even relevant to the potential improvements RDMA could bring) i don't know .

 

[efschu3]

Couldn't let it go

A TrueNAS and iXsystems Questions & Answers Interview [with elaboration request on RDMA answer]

Hey guys, I just found this https://nascompares.com/news/truenas-ixsystems-nas-qa-answering-your-questions-in-2022/ There are some good answers for new users who are looking into NAS systems and are considering TrueNas as option, I like it I actually stumbled over this while searching for news...

www.truenas.com

I could Not let it Go too. Especialy when they Switched to Debian, the "extra" Work to be done would only be a few minutes. But I dont have the testlab for doing this by myself right now.

But I told them and pointed them on how less Work IT would be.

Hope they finally understand the benefit of RDMA.

 

[efschu3]

Well, it seems we must write them a step by step guide for dummies how to Install rdma-core package and enable iser Portal on target.

I dunno why they are so lazy to "activate" an already existing and already integrated Feature.

When I get back my testlab I will write them a "Patch"

 

[i386]

I dunno why they are so lazy to "activate" an already existing and already integrated Feature.

Probably because of their customers? I think ix customers are the same people that would use qnap or synology stuff, but want (or need) it to be "free" (like freedom, not like free beer ). And these people don't need/know about rdma

But maybe I'm wrong...

 

[PigLover]

@i386 the lack of demand may be part of it (though I'd argue most of IX main customer base is much more savvy than you suggest).

That said, the biggest reason is likely related to testing, tools and delivery practices. Once IX lists RDMA as 'supported' they have a responsibility to make sure it works. While turning on the existing upstream code may be easy the effort to integrate testing and validation into their CICD tool chains is non trivial. And actually testing RDMA right requires quite a bit of investment in test target equipment and tools.

Lastly, RDMA is of limited value to the masses - but almost limitless value to the small set of customers that really need it (need, not want...). This is exactly the kind of feature a vendor may want to hold back for a paid feature with later upsell opportunities.

 

[BoredSysadmin]

Probably because of their customers? I think ix customers are the same people that would use qnap or synology stuff, but want (or need) it to be "free" (like freedom, not like free beer ). And these people don't need/know about rdma

But maybe I'm wrong...

I disagree, my company went with two iX X20-HA systems for secondary storage due to a combination of support for HA, cost, and support. Qnap and Synology STILL don't have 24/7 support and without it, I'd personally NOT consider ANY storage system in an enterprise.

 

[BoredSysadmin]

Lastly, RDMA is of limited value to the masses - but almost limitless value to the small set of customers that really need it (need, not want...).

I am curious to learn more about these cases, I know that the NEED for RDMA is likely to be driven by the need to have ultra-low latency, but what are specific use cases?

 

[i386]

I am curious to learn more about these cases, I know that the NEED for RDMA is likely to be driven by the need to have ultra-low latency, but what are specific use cases?

No matter what uses cases I think of it always ends at persisting data -> my answer would be "fast" storage (low latency, high throughput) for databases, message brokers and stuff like that

 

[bilbo1337]

I'm trying to figure this out right now only because I found a super cheap $120 25GbE card that can do iSER. In TrueNAS's latest stable update, I still only see regular iSCSI with no options for RDMA. I feel like people would want something like this especially if they're using their system as a virtualized desktop/workstation, no? The lower latency would make everything feel snappier.

 

[Bjorn Smith]

The lower latency would make everything feel snappier.

Indeed - but it is a niche product - and thats probably why they haven't bothered. iSCSI is good enough for 99% of usage scenarios - it is rare that you required multi gigabyte transfer rates to a single host.

Not that I think its a bad idea if they supported it, since it would lower CPU usage on both ends - but as they say - cores are cheap.

 

[XeonSam]

I dunno... RDMA is great, love it. But for what TrueNAS is, people still just see it as a free NAS solution. If they were to invest precious R&D time into a answer that large corporations/science & public institutions need (require) it would be a surprise to me. I've done a few setups trying to get RDMA to perform well enough to justify the cost and it just wasn't easy. And the time spent due to compatibility issues were a nightmare. I have a friend that is a kungfu master w/IB and even he gets frustrated... and he works in R&D for a science lab where he has to cluster GPU nodes.

If iX ends up adding it, props to them... but I'll bet only <1% of the users will actually praise this addition to their (great) solution. I use all flash for my home TrueNas box and I just don't need "near local storage" latency even when i connect a few virtualization hosts to it. I just make a datastore with async off and it does the job.

 

[tsteine]

I dunno... RDMA is great, love it. But for what TrueNAS is, people still just see it as a free NAS solution. If they were to invest precious R&D time into a answer that large corporations/science & public institutions need (require) it would be a surprise to me.

That's the thing. The amount of effort to include this is very low, shockingly so.

TrueNAS scale uses SCST as their iscsi target, and SCST doesn't require changes to its configuration file to allow iSER connections. Meaning it's not necessary to change the middleware to support this.
This is literally just a matter of including the packages for ISER from the debian package repo.

I think the bigger job would actually be to extend the middleware with support for setting up PFC/DCBX for lossless ethernet networks, which has applications beyond RDMA.

 

[efschu3]

For LVL2 based RoCEv1 you only need to enable Globalpause on Switches and TX/RX Pause on the Ehrernet Interface.

 

[Gabriel Mateiciuc]

I just wanna say that as of 22.12 release of TrueNAS SCALE, RDMA is doable in a few steps.

1. Add RDMA support to the system - I've went with OFED package from Mellanox Nvidia : Mellanox OFED (MLNX_OFED) Software: End-User Agreement - add kernel support, enable nvmf and (optional) nfsover rdma (I'd like to further test proxmox as an initiator to see if there are any benefits). Main initiator "target" for this is vmware - post 7.0.2 which has al lthe bels and whistles to support rdma with iser and nvmeof.
results here should render the system rdma capable, have a nvme target rdma capable, iser target module etc.

2. rebuild scst - I did this with the lastest git (3.7.0 - same as what's already in truenas) ; Check the install script to make sure you are starting the services with the new binaries (/etc/init.d/scst is the script used to start the service wrapped under the "goodness" of systemd /etc/systemd/system/scst.service.d)
3. reboot and if everything is doing what's supposed to, see your iscsi targets sessions with RDMAExtensions Yes in the logs.

4. On the vmware side (if that's your poison, proxmox/kvm should be even easier)
esxcli rdma iser add - yeah, no gui afaik
add the relevant config to match vmkernel to rdma enabled adapters
add initiators, target config etc. - the usual stuff
I didn't pay too much attention to the vmware setup, but I think you have to have vsan enabled on the vmkernel adapter - and only one active physical adapter - there are way better guides around here so I won't insist on this side.

5. Quirks and other relevant info :
- I've yet to test nvme rdma target - will get back.
- So far I have iser enabled for some targets and test vm's running smoothly for a few weeks - no issues whatsoever. iser interaction with reboots for either initiator or target looks as smooth as possible. Mind you, in true rdma fashion (for those who played with infiniband if you remember) there is no traffic shown on the disk or the network monitoring in vmware
- Used adapters were cx4 and cx5 with no mods to the firmware - yeah, I'm glad the days of modding cx3 and cx3pro's and hoping I wouldn't hit any random crashes seem to be behind.
- Switch used has only 10g ports so that won't allow for numbers to write (servethe)home
- There are a few things that could get tuned - one would be the iscsi target that only has 32q depth by default - but this is fine tuning in my book
- I only did a few benchmarks on a low performance zvol (spinning rust with flash log) just to see that we're not getting any crashes/lockups with high usage or network congestion.
- on the Truenas side, making apt executable should get you started for missing dependencies.

Cheers

Let me know if I've missed any important piece and happy modding