FreeNas/TrueNas RDMA Support (FR for voting)

tsteine

Active Member
May 15, 2019
124
62
28
@BoredSysadmin This is very interesting.

I've personally abandoned Truenas Scale in the homelab in favor of running openzfs on my linux distro of choice, there is no reason why I can't simply test this, and see whether iSER has tangible benefits for iSCSI, or nfs over rdma.
 
  • Like
Reactions: BoredSysadmin

i386

Well-Known Member
Mar 18, 2016
3,121
997
113
33
Germany
in a homelab its probably not really relevant with RDMA
When I copy files via robocopy from my workstation to my windows file server the server cpu stays at ~2.4GHz (it's a 1620v3 with 3.5GHz clock), so it probably saves some watts here and there :D
 

Rand__

Well-Known Member
Mar 6, 2014
6,135
1,499
113
If anyone has positive (or negative ) experience or expectations regarding RDMA then now might be a good time to chime in ...

He explicitly asked for it, even on non TN systems, so ...
 

tsteine

Active Member
May 15, 2019
124
62
28
@Rand__

Judging from the the two posts you've gotten out of them, it seems to me like they haven't actually bothered testing RDMA themselves.
 

Rand__

Well-Known Member
Mar 6, 2014
6,135
1,499
113
Well I am not sure they didnt test it, it sounds rather like they did test with a certain setup and it did not yield significant improvement (which is o/c exactly what they say).
Now whether that was a best or worst case scenario test (or if its was even relevant to the potential improvements RDMA could bring) i don't know .
 

efschu3

Member
Mar 11, 2019
95
30
18
Couldn't let it go;)

I could Not let it Go too. Especialy when they Switched to Debian, the "extra" Work to be done would only be a few minutes. But I dont have the testlab for doing this by myself right now.

But I told them and pointed them on how less Work IT would be.

Hope they finally understand the benefit of RDMA.
 
Last edited:
  • Like
Reactions: RageBone and Rand__

efschu3

Member
Mar 11, 2019
95
30
18
Well, it seems we must write them a step by step guide for dummies how to Install rdma-core package and enable iser Portal on target.

I dunno why they are so lazy to "activate" an already existing and already integrated Feature.

When I get back my testlab I will write them a "Patch" o_O:rolleyes::oops::eek::cool:
 

i386

Well-Known Member
Mar 18, 2016
3,121
997
113
33
Germany
I dunno why they are so lazy to "activate" an already existing and already integrated Feature.
Probably because of their customers? I think ix customers are the same people that would use qnap or synology stuff, but want (or need) it to be "free" (like freedom, not like free beer :D). And these people don't need/know about rdma :D

But maybe I'm wrong...
 

PigLover

Moderator
Jan 26, 2011
3,071
1,375
113
@i386 the lack of demand may be part of it (though I'd argue most of IX main customer base is much more savvy than you suggest).

That said, the biggest reason is likely related to testing, tools and delivery practices. Once IX lists RDMA as 'supported' they have a responsibility to make sure it works. While turning on the existing upstream code may be easy the effort to integrate testing and validation into their CICD tool chains is non trivial. And actually testing RDMA right requires quite a bit of investment in test target equipment and tools.

Lastly, RDMA is of limited value to the masses - but almost limitless value to the small set of customers that really need it (need, not want...). This is exactly the kind of feature a vendor may want to hold back for a paid feature with later upsell opportunities.
 

BoredSysadmin

Not affiliated with Maxell
Mar 2, 2019
821
310
63
Probably because of their customers? I think ix customers are the same people that would use qnap or synology stuff, but want (or need) it to be "free" (like freedom, not like free beer :D). And these people don't need/know about rdma :D

But maybe I'm wrong...
I disagree, my company went with two iX X20-HA systems for secondary storage due to a combination of support for HA, cost, and support. Qnap and Synology STILL don't have 24/7 support and without it, I'd personally NOT consider ANY storage system in an enterprise.
 

BoredSysadmin

Not affiliated with Maxell
Mar 2, 2019
821
310
63
Lastly, RDMA is of limited value to the masses - but almost limitless value to the small set of customers that really need it (need, not want...).
I am curious to learn more about these cases, I know that the NEED for RDMA is likely to be driven by the need to have ultra-low latency, but what are specific use cases?
 

i386

Well-Known Member
Mar 18, 2016
3,121
997
113
33
Germany
I am curious to learn more about these cases, I know that the NEED for RDMA is likely to be driven by the need to have ultra-low latency, but what are specific use cases?
No matter what uses cases I think of it always ends at persisting data -> my answer would be "fast" storage (low latency, high throughput) for databases, message brokers and stuff like that
 

bilbo1337

Member
Sep 18, 2020
79
45
18
Florida
I'm trying to figure this out right now only because I found a super cheap $120 25GbE card that can do iSER. In TrueNAS's latest stable update, I still only see regular iSCSI with no options for RDMA. I feel like people would want something like this especially if they're using their system as a virtualized desktop/workstation, no? The lower latency would make everything feel snappier.
 

Bjorn Smith

Well-Known Member
Sep 3, 2019
520
274
63
48
r00t.dk
The lower latency would make everything feel snappier.
Indeed - but it is a niche product - and thats probably why they haven't bothered. iSCSI is good enough for 99% of usage scenarios - it is rare that you required multi gigabyte transfer rates to a single host.

Not that I think its a bad idea if they supported it, since it would lower CPU usage on both ends - but as they say - cores are cheap.
 

XeonSam

Active Member
Aug 23, 2018
108
53
28
I dunno... RDMA is great, love it. But for what TrueNAS is, people still just see it as a free NAS solution. If they were to invest precious R&D time into a answer that large corporations/science & public institutions need (require) it would be a surprise to me. I've done a few setups trying to get RDMA to perform well enough to justify the cost and it just wasn't easy. And the time spent due to compatibility issues were a nightmare. I have a friend that is a kungfu master w/IB and even he gets frustrated... and he works in R&D for a science lab where he has to cluster GPU nodes.

If iX ends up adding it, props to them... but I'll bet only <1% of the users will actually praise this addition to their (great) solution. I use all flash for my home TrueNas box and I just don't need "near local storage" latency even when i connect a few virtualization hosts to it. I just make a datastore with async off and it does the job.
 

tsteine

Active Member
May 15, 2019
124
62
28
I dunno... RDMA is great, love it. But for what TrueNAS is, people still just see it as a free NAS solution. If they were to invest precious R&D time into a answer that large corporations/science & public institutions need (require) it would be a surprise to me.
That's the thing. The amount of effort to include this is very low, shockingly so.

TrueNAS scale uses SCST as their iscsi target, and SCST doesn't require changes to its configuration file to allow iSER connections. Meaning it's not necessary to change the middleware to support this.
This is literally just a matter of including the packages for ISER from the debian package repo.

I think the bigger job would actually be to extend the middleware with support for setting up PFC/DCBX for lossless ethernet networks, which has applications beyond RDMA.
 
  • Like
Reactions: Patrick

efschu3

Member
Mar 11, 2019
95
30
18
For LVL2 based RoCEv1 you only need to enable Globalpause on Switches and TX/RX Pause on the Ehrernet Interface.