TrueNAS network tuning

[Bjorn Smith]

Hi,

I am just setting up my new TrueNAS server - and I have a ConnectX-5 100Gbps card that I have installed.

Machine have 128GB RAM, a Xeon E5-1650v3, SuperMicro X10SRi-F.

Network is set to 9k jumbo frames.

It is working as it should - but doing a iperf from my esxi box to the TrueNAS server I can "only" get around 77-82 Gbps

Is this as expected or is there any tuning I can do in TrueNas to make it run faster.

Client connecting to 10.100.10.202, TCP port 5001
TCP window size:  516 KByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  8] local 10.100.10.184 port 13844 connected with 10.100.10.202 port 5001
[  4] local 10.100.10.184 port 59671 connected with 10.100.10.202 port 5001
[  5] local 10.100.10.184 port 50657 connected with 10.100.10.202 port 5001
[  3] local 10.100.10.184 port 16646 connected with 10.100.10.202 port 5001
[  7] local 10.100.10.184 port 10172 connected with 10.100.10.202 port 5001
[  6] local 10.100.10.184 port 24179 connected with 10.100.10.202 port 5001
[ ID] Interval       Transfer     Bandwidth
[  8]  0.0-10.0 sec  15.7 GBytes  13.5 Gbits/sec
[  4]  0.0-10.0 sec  12.7 GBytes  10.9 Gbits/sec
[  5]  0.0-10.0 sec  15.7 GBytes  13.5 Gbits/sec
[  3]  0.0-10.0 sec  15.7 GBytes  13.5 Gbits/sec
[  7]  0.0-10.0 sec  14.1 GBytes  12.1 Gbits/sec
[  6]  0.0-10.0 sec  15.9 GBytes  13.7 Gbits/sec
[SUM]  0.0-10.0 sec  89.8 GBytes  77.2 Gbits/sec

I have turned on autotune and rebooted the server, which gave me the following settings:




I know there are tuning for 10Gbps, but I wonder if my machine is simply too slow for 100GBps? My CPU usage goes to around 75% when running the iperf test.

Thanks in advance for any responses.

I have tuned a little according to: FreeBSD Network Performance Tuning @ Calomel.org - and my CPU dropped to around 50% and I gained on average 5 Gbps - so not a lot increase, but better CPU usage.

I wonder if I would gain the little extra I "need" to get to 100Gbps if I had a 8 core CPU? Like the E5-1680 v4

 

[Laugh|nGMan]

Maybe memory bandwidth bottleneck?
According to Anandtech:”.. v3 has access to 15-21% more bandwidth than the v2.


Single socket v2 Ivy hitting 54 GB/s, so v3 maybe max. 65 GB/s. With 2400 memory ~76 GB/s ?!

Prety sure u need six or better octal ch. system for single socket. If i remember correctly W32XX was first true 100-120GB system suited for workstation needs

 

[Bjorn Smith]

Maybe memory bandwidth bottleneck?

Possibly - but my memory should have a bandwidth of around 13GB/s so "plenty" - and that is on both boxes - DDR4@ 2133Mhz

 

[Bjorn Smith]

But then again - my disks have a MAX theorerical bandwidth of 2x3.2GB/s - so I should be covered so my disks are not bottlenecked by the network.

 

[Rand__]

You will not get that from FreeNas anyway, no worries

Kidding aside, I dont think the network will be a limitation here.

Autotune is not really recommended any more I think, but I have seen no other guide on tuning this than calomel and maybe some info on the MLX page

 

[Bjorn Smith]

Yeah - just did a fio randomwrite and I could get at most 1.7GB/s - which is enough for my needs - and anyway - iops is what matters really when you need it for VM's. I just like to see big numbers and like that vmotions goes fast if I move storage.

 

[Rand__]

so whats your setup now,
FN box on one end and the other?

 

[Laugh|nGMan]

OK
netflix blog talks about pbufs

Serving 100 Gbps from an Open Connect Appliance

By Drew Gallatin

netflixtechblog.com

 

[Bjorn Smith]

so whats your setup now,
FN box on one end and the other?

2xXeon e5-26763v3 + RAM running my main esxi
1xE3-1220v6 64GM ram running backup vms for when the big guy has maintenance
1xXeon D-1528/128 GB ram for isos, movies, backup
1xAtom C3338 running my pfsense

So suddenly after having downsized my homelab I suddenly have a heap of servers again Its a plague.

 

[Rand__]

Lol, its the swing of the pendulum
Q is whether you're on the up or down movement

What kind of storage does your ESXi box have?

 

[Bjorn Smith]

Q is whether you're on the up or down movement

I am planning to have peaked at the moment - I dont want more servers - although I might want faster servers at some point and then I will eventually end up with more servers again.

What kind of storage does your ESXi box have?

Just a SATA ssd for boot and a SATA ssd for booting "important" VM's that cannot wait for then NAS to come up, i.e. vcenter, dns, dhcp. The rest runs from the NAS

 

[Rand__]

Just wondered where you dump the VMs to with the 1.7GB/s
I move between vSan and my TNC box and vSan can't keep up on the writes to it (as soon as cache is full)

 

[i386]

It is working as it should - but doing a iperf from my esxi box to the TrueNAS server I can "only" get around 77-82 Gbps

77-82 GBit/s out of the box is pretty impressive. Are you using the freebsd or mellanox drivers?

Single socket v2 Ivy hitting 54 GB/s, so v3 maybe max. 65 GB/s. With 2400 memory ~76 GB/s ?!

With memory it's usually not Gigabit per Second, but Gigabytes per scond. 76GByte/s are about 760 GBit/s, enough for a 100GBE link

 

[Bjorn Smith]

Just wondered where you dump the VMs to with the 1.7GB/s
I move between vSan and my TNC box and vSan can't keep up on the writes to it (as soon as cache is full)

Actually I havent done speed tests of migrating from my "virtual" Freenas with NVME drives and my new FreeNAS - I will update with how "fast" it will be. But I am guessing slower than MAX of that I can do locally.

 

[Bjorn Smith]

77-82 GBit/s out of the box is pretty impressive. Are you using the freebsd or mellanox drivers?

Just the built in drivers in freebsd . 3.5.2 which is the same version on mellanox's homepage.

With memory it's usually not Gigabit per Second, but Gigabytes per scond. 76GByte/s are about 760 GBit/s, enough for a 100GBE link

Well according to wikipedia Peak transfer rrate for DDR4-2133 is 17066.67 MB/s so 76GB/s is 2400Mhz ram in quad channel setup. But still 17GB/s is plenty for 100GBE.

And apparently my motherboard only supports dual channel setup, so at most I would get would be 2x17GB.

So if I need more speed than that I would need to exchange the motherboard.

 

[Rand__]

ah, thats to the TNC box, yeah thats fast for me too. Not even close to 100G, but looking impressive

 

[jpmomo]

Hi,

I am just setting up my new TrueNAS server - and I have a ConnectX-5 100Gbps card that I have installed.

Machine have 128GB RAM, a Xeon E5-1650v3, SuperMicro X10SRi-F.

Network is set to 9k jumbo frames.

It is working as it should - but doing a iperf from my esxi box to the TrueNAS server I can "only" get around 77-82 Gbps

Is this as expected or is there any tuning I can do in TrueNas to make it run faster.

Client connecting to 10.100.10.202, TCP port 5001
TCP window size:  516 KByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  8] local 10.100.10.184 port 13844 connected with 10.100.10.202 port 5001
[  4] local 10.100.10.184 port 59671 connected with 10.100.10.202 port 5001
[  5] local 10.100.10.184 port 50657 connected with 10.100.10.202 port 5001
[  3] local 10.100.10.184 port 16646 connected with 10.100.10.202 port 5001
[  7] local 10.100.10.184 port 10172 connected with 10.100.10.202 port 5001
[  6] local 10.100.10.184 port 24179 connected with 10.100.10.202 port 5001
[ ID] Interval       Transfer     Bandwidth
[  8]  0.0-10.0 sec  15.7 GBytes  13.5 Gbits/sec
[  4]  0.0-10.0 sec  12.7 GBytes  10.9 Gbits/sec
[  5]  0.0-10.0 sec  15.7 GBytes  13.5 Gbits/sec
[  3]  0.0-10.0 sec  15.7 GBytes  13.5 Gbits/sec
[  7]  0.0-10.0 sec  14.1 GBytes  12.1 Gbits/sec
[  6]  0.0-10.0 sec  15.9 GBytes  13.7 Gbits/sec
[SUM]  0.0-10.0 sec  89.8 GBytes  77.2 Gbits/sec

I have turned on autotune and rebooted the server, which gave me the following settings:

View attachment 17658
View attachment 17659

I know there are tuning for 10Gbps, but I wonder if my machine is simply too slow for 100GBps? My CPU usage goes to around 75% when running the iperf test.

Thanks in advance for any responses.

I have tuned a little according to: FreeBSD Network Performance Tuning @ Calomel.org - and my CPU dropped to around 50% and I gained on average 5 Gbps - so not a lot increase, but better CPU usage.

I wonder if I would gain the little extra I "need" to get to 100Gbps if I had a 8 core CPU? Like the E5-1680 v4

Hello Bjorn,
Can you let me know the details of your iperf command? I have a similar setup with mellanox connectx-5 nics that I am trying to optimize with TN as well. I have a w10 workstation with 100G nics connected to an arista 100G sw. The TN server also has the same nics and is connected to the same switch. How do you setup the iperf test in your setup? I shouldn't have any hw issues with regards to the cpus or memory bandwidth. I have an epyc rome 7h12 on the w10 client side and a 7502 on the TN side. Plenty of 3200mhz ram on both sides as well.
thanks for any details.

 

[jpmomo]

I figured out that TN comes with iperf (not iperf3) built in. I was able to get around 85Gbps without any tweaking. I just wanted to spot check while I wait on my hw based traffic generators. Even with the amd rome cpus, the numa topology can make a difference at the high end of testing. I didn't know how to check any of the details of the mellanox drivers (mft, etc) on the TN side for further tuning. I am not sure if that is exposed (ex. installing ofed sw on the TN server.)

 

[Bjorn Smith]

Hi,
I cannot remember precisely what iperf command arguments I used - but use multiple threads for sure - up to the number of cpu's that is available on both ends, i.e. if one have 10 cores and the other 8 - it probably does not make sense to go above 8.

And also tunables are important.

kern.ipc.maxsockbuf=61440000 
net.inet.tcp.recvbuf_inc=524288  
net.inet.tcp.recvbuf_max=16777216 
net.inet.tcp.recvspace=4194304  
net.inet.tcp.sendbuf_inc=2097152 
net.inet.tcp.sendbuf_max=16777216 
net.inet.tcp.sendspace=4194304 
net.isr.bindthreads=1
net.isr.maxthreads=-1

i.e. the ones with big numbers are just increasing buffer sizes etc, so the network can handle the higher throughput - you should not copy mine - but find out what works for you - it depends on how much memory you want to allocate to the network stack.

The net.isr.* is about how many cores that is available for packet processing - per default freebsd only allocates 1 cores - and setting net.isr.maxthreads=-1 - then it uses all cores in the system if needed. The net.isr.bindthreads=1 forces connections to stay on the same core and not flip around, which is good for throughput, otherwise you will have cache misses etc.

My number above is for my truenas with 256GB of RAM - doing nothing else than serve data via NSF to eSXI.

And I am not sure if my tunables has anything to do with it, or its the upgrade to truenas, but this is what my memory layout looks like on the dashboard. Notice the "Services" usage.


So either there is a memory leak somewhere in truenas - which is not apparent in "top" - or ixsystems calculate weirdly - and strangely enough my ZFS cache usage has dropped at the same time as Services has gone up - even though there is still Free memory.