FreeNAS 9.3 (CIFS poor 10GBe performance) Looking for tweaks / hints / whatever

[Kristian]

Trying to do some tests.... FreeNAS is booting up SINCE 9 minutes...
I am about to throw everything out of the window

 

[Kristian]

Results:
FreeNAS was a b***h.
When I finally managed to establish a direct connection the results were:

180 MB/s sustained in iPerf...

so less than with all the switches and different cables...
I tried 2 different DACs and 2 brands of transceivers with OM3 in the middle... no change... 180 MB/s dead on...
Forgot about the top command to look for cpu utilization :-(

When I noticed it was already 10PM I accecpted this result I switched to windows on both machines
and iPerf'ed the direct connection with those 3 cables:

The results looked like this:

erver listening on TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
OpenSCManager failed - Zugriff verweigert (0x5)
[128] local 192.168.178.30 port 5001 connected with 192.168.178.96 port 59192
[ ID] Interval  Transfer  Bandwidth
[128]  0.0- 1.0 sec  0.41 GBytes  0.41 GBytes/sec
[128]  1.0- 2.0 sec  0.40 GBytes  0.40 GBytes/sec
[128]  2.0- 3.0 sec  0.38 GBytes  0.38 GBytes/sec
[128]  3.0- 4.0 sec  0.45 GBytes  0.45 GBytes/sec
[128]  4.0- 5.0 sec  0.41 GBytes  0.41 GBytes/sec
[128]  5.0- 6.0 sec  0.41 GBytes  0.41 GBytes/sec
[128]  6.0- 7.0 sec  0.39 GBytes  0.39 GBytes/sec
[128]  7.0- 8.0 sec  0.38 GBytes  0.38 GBytes/sec
[128]  8.0- 9.0 sec  0.39 GBytes  0.39 GBytes/sec
[128]  9.0-10.0 sec  0.38 GBytes  0.38 GBytes/sec
[128]  0.0-10.0 sec  3.97 GBytes  0.40 GBytes/sec

I opened the resource monitor and I noticed this: pleas see the picture.
So it seems it is the cpu that is the bottleneck...

Anyways: If I could archive those speeds in FreeNAS instead of the 180 MB/s everything would be good...

Edit 01 from this morning:

@Kristian Yeah, I'd cut as much out your network path as you can and see if things improve. I suppose its possible you could have a faulty switch or something, but it would indeed be a bit unexpected. You're sure you've got up-to-date Intel drivers installed on your workstation, correct? Just because Windows recognizes the card doesn't mean you'll get great performance straight out of the gate. (Windows 10 sees ConnectX-2 cards just fine, but without drivers they max @ about 6Gb/s, for example)
You could also try booting into a Linux LiveCD (Ubuntu would be fine), install Samba4, and create a share to /tmp to test as well; that would narrow it down to whether BSD is what needs tweaking or something else in your network.

Drivers are Version 20.2 released 08/20/2015
Booting to a Linux LiveCD was not possible today because of time issues.
I will try this tonight.

 

[Kristian]

By the way: This is a screenshot copying a file that is 8GBs sized from a Xpenology box with an Asus P9A-I/C2550/SAS/4L in it to a RAM Drive in my Workstation (over the complete topology)
I would really like to know if it would be possible to run iPerf somehow on Xpenology just to cross reference the results.

 

[canta]

By the way: This is a screenshot copying a file that is 8GBs sized from a Xpenology box with an Asus P9A-I/C2550/SAS/4L in it to a RAM Drive in my Workstation (over the complete topology)
I would really like to know if it would be possible to run iPerf somehow on Xpenology just to cross reference the results.

this is a single file copy or multiple file copies?

 

[Kristian]

Single File

 

[Marsh]

I would really like to know if it would be possible to run iPerf somehow on Xpenology just to cross reference the results.

I installed and ran iPerf on Xpenology before ( May this year ).
I did not wrote down the steps of installation.
I remember I used google to find the iPerf installation steps.

 

[Kristian]

I installed and ran iPerf on Xpenology before ( May this year ).
I did not wrote down the steps of installation.
I remember I used google to find the iPerf installation steps.

Thanks Marsh, I was able to install iPerf on my Xpenology box...

But the result is not really what I expected:

Workstation -- all my network topology - P9A-I (Avoton C2550) mtu 9000

bin/iperf.exe -c 192.168.178.66 -P 1 -i 1 -p 5001 -f G -t 20 -T 1
------------------------------------------------------------
Client connecting to 192.168.178.66, TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
[236] local 192.168.178.30 port 53146 connected with 192.168.178.66 port 5001
[ ID] Interval  Transfer  Bandwidth
[236]  0.0- 1.0 sec  1.16 GBytes  1.16 GBytes/sec
[236]  1.0- 2.0 sec  1.15 GBytes  1.15 GBytes/sec
[236]  2.0- 3.0 sec  1.15 GBytes  1.15 GBytes/sec
[236]  3.0- 4.0 sec  1.15 GBytes  1.15 GBytes/sec
[236]  4.0- 5.0 sec  1.15 GBytes  1.15 GBytes/sec
[236]  5.0- 6.0 sec  1.15 GBytes  1.15 GBytes/sec
[236]  6.0- 7.0 sec  1.15 GBytes  1.15 GBytes/sec
[236]  7.0- 8.0 sec  1.15 GBytes  1.15 GBytes/sec
[236]  8.0- 9.0 sec  1.15 GBytes  1.15 GBytes/sec
[236]  9.0-10.0 sec  1.15 GBytes  1.15 GBytes/sec
[236] 10.0-11.0 sec  1.15 GBytes  1.15 GBytes/sec
[236] 11.0-12.0 sec  1.15 GBytes  1.15 GBytes/sec
[236] 12.0-13.0 sec  1.15 GBytes  1.15 GBytes/sec
[236] 13.0-14.0 sec  1.15 GBytes  1.15 GBytes/sec
[236] 14.0-15.0 sec  1.15 GBytes  1.15 GBytes/sec
[236] 15.0-16.0 sec  1.15 GBytes  1.15 GBytes/sec
[236] 16.0-17.0 sec  1.15 GBytes  1.15 GBytes/sec
[236] 17.0-18.0 sec  1.15 GBytes  1.15 GBytes/sec
[236] 18.0-19.0 sec  1.15 GBytes  1.15 GBytes/sec
[236] 19.0-20.0 sec  1.14 GBytes  1.14 GBytes/sec
[ ID] Interval  Transfer  Bandwidth
[236]  0.0-20.0 sec  23.1 GBytes  1.15 GBytes/sec
Done.

I would have expected a result worse than the C2750 ... ?!?


direction reversed mtu 9000

bin/iperf.exe -s -P 0 -i 1 -p 5001 -f G
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
OpenSCManager failed - Zugriff verweigert (0x5)
[ 56] local 192.168.178.30 port 5001 connected with 192.168.178.66 port 35233
[ ID] Interval  Transfer  Bandwidth
[ 56]  0.0- 1.0 sec  0.64 GBytes  0.64 GBytes/sec
[ 56]  1.0- 2.0 sec  0.64 GBytes  0.64 GBytes/sec
[ 56]  2.0- 3.0 sec  0.61 GBytes  0.61 GBytes/sec
[ 56]  3.0- 4.0 sec  0.59 GBytes  0.59 GBytes/sec
[ 56]  4.0- 5.0 sec  0.62 GBytes  0.62 GBytes/sec
[ 56]  5.0- 6.0 sec  0.62 GBytes  0.62 GBytes/sec
[ 56]  6.0- 7.0 sec  0.62 GBytes  0.62 GBytes/sec
[ 56]  7.0- 8.0 sec  0.64 GBytes  0.64 GBytes/sec
[ 56]  8.0- 9.0 sec  0.62 GBytes  0.62 GBytes/sec
[ 56]  9.0-10.0 sec  0.56 GBytes  0.56 GBytes/sec
[ 56] 10.0-11.0 sec  0.63 GBytes  0.63 GBytes/sec
[ 56] 11.0-12.0 sec  0.65 GBytes  0.65 GBytes/sec
[ 56] 12.0-13.0 sec  0.62 GBytes  0.62 GBytes/sec
[ 56] 13.0-14.0 sec  0.59 GBytes  0.59 GBytes/sec
[ 56] 14.0-15.0 sec  0.59 GBytes  0.59 GBytes/sec
[ 56] 15.0-16.0 sec  0.62 GBytes  0.62 GBytes/sec
[ 56] 16.0-17.0 sec  0.65 GBytes  0.65 GBytes/sec
[ 56] 17.0-18.0 sec  0.65 GBytes  0.65 GBytes/sec
[ 56] 18.0-19.0 sec  0.65 GBytes  0.65 GBytes/sec
[ 56]  0.0-20.0 sec  12.4 GBytes  0.62 GBytes/sec

next is the ubuntu live cd on the 2750

Update: I am too stupid to install samba 4 with the live cd. Same applies to iperf and the live cd...
Doing a install on a spare ssd

 

[Kristian]

iPerf result with Workstation -- complete topology -- Supermicro A1SAM 2750 (with Ubuntu 14.02 LTS) (mtu 9000)

bin/iperf.exe -c 192.168.178.96 -P 1 -i 1 -p 5001 -f G -t 20 -T 1
------------------------------------------------------------
Client connecting to 192.168.178.96, TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
[260] local 192.168.178.30 port 62441 connected with 192.168.178.96 port 5001
[ ID] Interval  Transfer  Bandwidth
[260]  0.0- 1.0 sec  0.96 GBytes  0.96 GBytes/sec
[260]  1.0- 2.0 sec  1.01 GBytes  1.01 GBytes/sec
[260]  2.0- 3.0 sec  0.70 GBytes  0.70 GBytes/sec
[260]  3.0- 4.0 sec  0.71 GBytes  0.71 GBytes/sec
[260]  4.0- 5.0 sec  1.00 GBytes  1.00 GBytes/sec
[260]  5.0- 6.0 sec  0.69 GBytes  0.69 GBytes/sec
[260]  6.0- 7.0 sec  1.00 GBytes  1.00 GBytes/sec
[260]  7.0- 8.0 sec  0.68 GBytes  0.68 GBytes/sec
[260]  8.0- 9.0 sec  0.63 GBytes  0.63 GBytes/sec
[260]  9.0-10.0 sec  0.76 GBytes  0.76 GBytes/sec
[260] 10.0-11.0 sec  0.69 GBytes  0.69 GBytes/sec
[260] 11.0-12.0 sec  1.00 GBytes  1.00 GBytes/sec
[260] 12.0-13.0 sec  0.97 GBytes  0.97 GBytes/sec
[260] 13.0-14.0 sec  0.68 GBytes  0.68 GBytes/sec
[260] 14.0-15.0 sec  0.97 GBytes  0.97 GBytes/sec
[260] 15.0-16.0 sec  0.99 GBytes  0.99 GBytes/sec
[260] 16.0-17.0 sec  0.99 GBytes  0.99 GBytes/sec
[260] 17.0-18.0 sec  0.61 GBytes  0.61 GBytes/sec
[260] 18.0-19.0 sec  0.72 GBytes  0.72 GBytes/sec
[260] 19.0-20.0 sec  0.99 GBytes  0.99 GBytes/sec
[ ID] Interval  Transfer  Bandwidth
[260]  0.0-20.0 sec  16.8 GBytes  0.84 GBytes/sec
Done.

vice versa from c2750 to the workstation

bin/iperf.exe -s -P 0 -i 1 -p 5001 -f G
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
OpenSCManager failed - Zugriff verweigert (0x5)
[300] local 192.168.178.30 port 5001 connected with 192.168.178.96 port 36150
[ ID] Interval  Transfer  Bandwidth
[300]  0.0- 1.0 sec  0.42 GBytes  0.42 GBytes/sec
[300]  1.0- 2.0 sec  0.44 GBytes  0.44 GBytes/sec
[300]  2.0- 3.0 sec  0.46 GBytes  0.46 GBytes/sec
[300]  3.0- 4.0 sec  0.44 GBytes  0.44 GBytes/sec
[300]  4.0- 5.0 sec  0.45 GBytes  0.45 GBytes/sec
[300]  5.0- 6.0 sec  0.44 GBytes  0.44 GBytes/sec
[300]  6.0- 7.0 sec  0.45 GBytes  0.45 GBytes/sec
[300]  7.0- 8.0 sec  0.44 GBytes  0.44 GBytes/sec
[300]  8.0- 9.0 sec  0.43 GBytes  0.43 GBytes/sec
[300]  9.0-10.0 sec  0.44 GBytes  0.44 GBytes/sec
[300] 10.0-11.0 sec  0.44 GBytes  0.44 GBytes/sec
[300] 11.0-12.0 sec  0.43 GBytes  0.43 GBytes/sec
[300] 12.0-13.0 sec  0.44 GBytes  0.44 GBytes/sec
[300] 13.0-14.0 sec  0.43 GBytes  0.43 GBytes/sec
[300] 14.0-15.0 sec  0.44 GBytes  0.44 GBytes/sec
[300] 15.0-16.0 sec  0.42 GBytes  0.42 GBytes/sec
[300] 16.0-17.0 sec  0.44 GBytes  0.44 GBytes/sec
[300] 17.0-18.0 sec  0.45 GBytes  0.45 GBytes/sec
[300] 18.0-19.0 sec  0.41 GBytes  0.41 GBytes/sec
[300] 19.0-20.0 sec  0.41 GBytes  0.41 GBytes/sec
[ ID] Interval  Transfer  Bandwidth
[300]  0.0-20.0 sec  8.73 GBytes  0.44 GBytes/sec

So I think I should test the 2750 with xpenology


installing samba 4 on ubuntu is a bit more complicated than I thought.
But smbstatus --version is returning Version 4.1.6-Ubuntu.
How should I proceed now?

 

[canta]

...

I would have expected a result worse than the C2750 ... ?!?


...

~8Gb network to network .. any processor should achieve that easily with 10Gb connection.


the question is running samba to transfer data with multiple big files how many Gb rate can be achieved

the last time copying image of my corrupted windows 10 ( to be sure before overwritting with my current backup) to my ZoL macihine can sustain ~900Mb on 1Gb ethernet connection via samba 4.
the Zol Machine used very low cpu power, the main obvious is one core got hit by samba. and utilization on that core is very low..10% ( as I remember).
ZoL eat much 11M memory of 16M . got to love ZFS that loves RAM.

 

[Rain]

installing samba 4 on ubuntu is a bit more complicated than I thought.
But smbstatus --version is returning Version 4.1.6-Ubuntu.
How should I proceed now?

Sorry, I should have been a bit more clear. $ sudo apt-get install samba should have taken care of it.

Add this to the very bottom of /etc/samba/smb.conf:

[tmp]
path = /tmp
read only = no
browseable = yes
guest ok = yes
read only = no

And restart Samba: # /etc/init.d/samba restart

Then you should be able to access \\1.1.1.1\tmp on your Windows machine (where 1.1.1.1 is the IP address of the Ubuntu machine). /tmp is (for all intents and purposes) a RAM disk, so benchmark copies to your hearts consent. One thing to note though: Copies to the share might stutter (it can be quite drastic) as more RAM is allocated by tmpfs; copies from the share (after you've copied a large file over to it already) to your workstation should go at "full speed" though.

Also, I'd make sure that the "Receive Buffers" and "Transmit Buffers" values (Right click the NIC > Properties > Advanced (> Performance Options)) are set to the maximum value then can be on your workstation. No tweaking should be necessary on Linux/FreeBSD to get full speeds.

 

[Kristian]

alright... will try that... unfortunatelly not before tommorow evening...

update:

Did it! Thanks. (But did not tweak the buffers yet... that follows tommow... its already midnight)

Result: Copy a 8GB file from /tmp (ubuntu c2750) to ramdisk windows: 427 MB/s sustained.

vice versa

Copy same 8GB file from ramdisk windows to /tmp (ubuntu c2750) --> see jpg (peak speed 324 MB/s)

 

[canta]

if you are running on two linux, copy via samba to /dev/null.

 

[Kristian]

Still have to tweak the buffers tomorrow... I read that just now... and its already past midnight.

 

[Kristian]

The send and receive buffers were already tweaked...

P9A --> C2750

Client connecting to 192.168.178.30, TCP port 5001
TCP window size: 0.00 GByte (default)
> [  3] local 192.168.178.66 port 36993 connected with 192.168.178.30 port 5001
[ ID] Interval  Transfer  Bandwidth
[  3]  0.0- 1.0 sec  0.66 GBytes  0.66 GBytes/sec
[  3]  1.0- 2.0 sec  0.60 GBytes  0.60 GBytes/sec
[  3]  2.0- 3.0 sec  0.68 GBytes  0.68 GBytes/sec
[  3]  3.0- 4.0 sec  0.68 GBytes  0.68 GBytes/sec
[  3]  4.0- 5.0 sec  0.64 GBytes  0.64 GBytes/sec
[  3]  5.0- 6.0 sec  0.58 GBytes  0.58 GBytes/sec
[  3]  6.0- 7.0 sec  0.59 GBytes  0.59 GBytes/sec
[  3]  7.0- 8.0 sec  0.68 GBytes  0.68 GBytes/sec
[  3]  8.0- 9.0 sec  0.58 GBytes  0.58 GBytes/sec
[  3]  9.0-10.0 sec  0.68 GBytes  0.68 GBytes/sec
[  3]  0.0-10.0 sec  6.38 GBytes  0.64 GBytes/sec

C2750 -> P9A

iperf -c 192.168.178.66 -P 1 -i 1 -p 5001 -fG -t10
------------------------------------------------------------
Client connecting to 192.168.178.66, TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
[  3] local 192.168.178.96 port 35881 connected with 192.168.178.66 port 5001
[ ID] Interval  Transfer  Bandwidth
[  3]  0.0- 1.0 sec  0.75 GBytes  0.75 GBytes/sec
[  3]  1.0- 2.0 sec  0.72 GBytes  0.72 GBytes/sec
[  3]  2.0- 3.0 sec  0.74 GBytes  0.74 GBytes/sec
[  3]  3.0- 4.0 sec  0.74 GBytes  0.74 GBytes/sec
[  3]  4.0- 5.0 sec  0.72 GBytes  0.72 GBytes/sec
[  3]  5.0- 6.0 sec  0.75 GBytes  0.75 GBytes/sec
[  3]  6.0- 7.0 sec  0.70 GBytes  0.70 GBytes/sec
[  3]  7.0- 8.0 sec  0.70 GBytes  0.70 GBytes/sec
[  3]  8.0- 9.0 sec  0.75 GBytes  0.75 GBytes/sec
[  3]  9.0-10.0 sec  0.75 GBytes  0.75 GBytes/sec
[  3]  0.0-10.0 sec  7.31 GBytes  0.73 GBytes/sec

Workstation --> P9A

bin/iperf.exe -c 192.168.178.66 -P 1 -i 1 -p 5001 -f G -t 10 -T 1
------------------------------------------------------------
Client connecting to 192.168.178.66, TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
[240] local 192.168.178.30 port 52593 connected with 192.168.178.66 port 5001
[ ID] Interval  Transfer  Bandwidth
[240]  0.0- 1.0 sec  1.16 GBytes  1.16 GBytes/sec
[240]  1.0- 2.0 sec  1.15 GBytes  1.15 GBytes/sec
[240]  2.0- 3.0 sec  1.15 GBytes  1.15 GBytes/sec
[240]  3.0- 4.0 sec  1.15 GBytes  1.15 GBytes/sec
[240]  4.0- 5.0 sec  1.15 GBytes  1.15 GBytes/sec
[240]  5.0- 6.0 sec  1.15 GBytes  1.15 GBytes/sec
[240]  6.0- 7.0 sec  1.15 GBytes  1.15 GBytes/sec
[240]  7.0- 8.0 sec  1.15 GBytes  1.15 GBytes/sec
[240]  8.0- 9.0 sec  1.15 GBytes  1.15 GBytes/sec
[240]  9.0-10.0 sec  1.15 GBytes  1.15 GBytes/sec
[240]  0.0-10.0 sec  11.5 GBytes  1.15 GBytes/sec

Workstation --> C2750

bin/iperf.exe -c 192.168.178.96 -P 1 -i 1 -p 5001 -f G -t 10 -T 1
------------------------------------------------------------
Client connecting to 192.168.178.96, TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
[236] local 192.168.178.30 port 52910 connected with 192.168.178.96 port 5001
[ ID] Interval  Transfer  Bandwidth
[236]  0.0- 1.0 sec  0.38 GBytes  0.38 GBytes/sec
[236]  1.0- 2.0 sec  0.72 GBytes  0.72 GBytes/sec
[236]  2.0- 3.0 sec  0.71 GBytes  0.71 GBytes/sec
[236]  3.0- 4.0 sec  0.72 GBytes  0.72 GBytes/sec
[236]  4.0- 5.0 sec  0.69 GBytes  0.69 GBytes/sec
[236]  5.0- 6.0 sec  0.38 GBytes  0.38 GBytes/sec
[236]  6.0- 7.0 sec  0.98 GBytes  0.98 GBytes/sec
[236]  7.0- 8.0 sec  0.30 GBytes  0.30 GBytes/sec
[236]  8.0- 9.0 sec  0.51 GBytes  0.51 GBytes/sec
[236]  9.0-10.0 sec  0.39 GBytes  0.39 GBytes/sec
[236]  0.0-10.0 sec  5.80 GBytes  0.58 GBytes/sec
Done.

 

[Kristian]

Real World speed from workstation RAM Disk to ubuntu back and forth is still the same values as yesterday...
More suggestions what I could test?

 

[Marsh]

Try running iperf with multiple threads to find the max capped network speed.

iperf -P (1-10)
see Iperf cheat sheet | JamesCoyle.net

 

[Kristian]

3 multiple threads bring up the speed to line speed