10 GbE slow between CentOS and FreeNAS

[fraenki]

Hi,

I have several CentOS and FreeNAS servers in my 10 GbE network, all in the same network. I ran many tests with iperf: on most hosts I get 9.3 Gb/s (which I consider full speed), while the results for some hosts vary between 1.1 Gb/s and 9.3 Gb/s.

The "slow" servers:

root@freenas:~ # iperf -c centos-02
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  5.61 GBytes  4.81 Gbits/sec
[  3]  0.0-10.1 sec  2.05 GBytes  1.75 Gbits/sec
[  3]  0.0-10.0 sec  5.52 GBytes  4.74 Gbits/sec
[  3]  0.0-10.0 sec  1.91 GBytes  1.64 Gbits/sec
[  3]  0.0-10.1 sec  6.19 GBytes  5.27 Gbits/sec
[  3]  0.0-10.0 sec  2.06 GBytes  1.77 Gbits/sec

Now the fun: It's only slow on one direction. The other direction is perfectly fine and reaches full speed! See below...

user@centos-02:~ > iperf -c freenas
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.7 GBytes  9.15 Gbits/sec
[  3]  0.0-10.0 sec  10.9 GBytes  9.33 Gbits/sec
[  3]  0.0-10.0 sec  10.8 GBytes  9.24 Gbits/sec
[  3]  0.0-10.0 sec  10.9 GBytes  9.39 Gbits/sec

Another fun fact: I have two other boxes with the same hardware configuration as centos-02, and iperf between these CentOS hosts is fast in both directions...

user@centos-02:~ > iperf -c centos-03
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  10.8 GBytes  9.31 Gbits/sec
[  4]  0.0-10.0 sec  10.7 GBytes  9.15 Gbits/sec

user@centos-03:~ > iperf -c centos-02
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.8 GBytes  9.30 Gbits/sec
[  3]  0.0-10.0 sec  10.8 GBytes  9.26 Gbits/sec

So, why is the connection between centos-02 (or centos-03) and freenas so slow?

To make things even worse: I have more CentOS boxes running on old Fujitsu hardware that reach (almost) full speed with FreeNAS (in both directions)...

root@freenas:~ # iperf -c centos-10
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.1 GBytes  8.71 Gbits/sec
[  3]  0.0-10.0 sec  9.87 GBytes  8.47 Gbits/sec
[  3]  0.0-10.0 sec  10.6 GBytes  9.10 Gbits/sec
[  3]  0.0-10.0 sec  9.99 GBytes  8.58 Gbits/sec

(I think centos-10 would reach 9.3 Gb/s, but there is currently some network traffic on this host.)

HARDWARE:
freenas hardware: Fujitsu RX200 S6, Xeon E5645, 24 GB RAM, Intel X550-T2 NIC
centos-02 hardware: Supermicro X10DRI, Xeon E5-2630L v3, 128 GB RAM, Intel X550-T2 NIC
centos-10 hardware: Fujitsu RX200 S6, Xeon E5645, 24 GB RAM, Intel X550-T2 NIC

Initially I thought that this low performance was caused by counterfeit Intel NICs, but since then I've replaced the NIC in FreeNAS and centos-02 with a new genuine Intel X550-T2/-T1 NIC – but nothing has changed.

I've also tested a direct connection between FreeNAS and centos-02, but with the same poor results. I've replaced all cables too.

Thanks!

- Frank

 

[Blue)(Fusion]

Some things to check:

At the bottom of iperf, it should show retries or re-transmits or something like that. What is the count there?

• Do the NICs have enough PCIe lanes? I.e. is it in a x8 port and is the x8 port actually giving x8 lanes? It should only need 4x but I've discovered that CX2/CX3 cards will NOT give full bandwidth with anything less than 8x slots (PCIe 2.0 spec).
  
• Are firewalls/iptables disabled? Perhaps filtering packets is slowing things down.

FWIW, I get the following speeds across VLANs on the switch, with ACLs (ICX6610). And come to think of it. I think the only reason it's not wirespeed is because the iperf3 server is on a KVM guest.

~ $ iperf3 -c slb1
Connecting to host slb1, port 5201
[  5] local 10.1.10.10 port 53414 connected to 10.1.6.40 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   877 MBytes  7.36 Gbits/sec    0   3.77 MBytes      
[  5]   1.00-2.00   sec  1022 MBytes  8.58 Gbits/sec    0   3.77 MBytes      
[  5]   2.00-3.00   sec  1022 MBytes  8.58 Gbits/sec    0   3.77 MBytes      
[  5]   3.00-4.00   sec  1.05 GBytes  8.99 Gbits/sec    0   3.77 MBytes      
[  5]   4.00-5.00   sec  1.08 GBytes  9.24 Gbits/sec    0   3.77 MBytes      
[  5]   5.00-6.00   sec  1.00 GBytes  8.61 Gbits/sec    0   3.77 MBytes      
[  5]   6.00-7.00   sec  1014 MBytes  8.50 Gbits/sec   24   1.88 MBytes      
[  5]   7.00-8.00   sec  1.09 GBytes  9.32 Gbits/sec    0   1.88 MBytes      
[  5]   8.00-9.00   sec  1.04 GBytes  8.91 Gbits/sec    0   1.88 MBytes      
[  5]   9.00-10.00  sec  1018 MBytes  8.54 Gbits/sec    0   1.88 MBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.1 GBytes  8.66 Gbits/sec   24             sender
[  5]   0.00-10.04  sec  10.1 GBytes  8.63 Gbits/sec                  receiver

iperf Done.

Here's physical to physical via same VLAN (ICX6610):

~ $ iperf3 -c gluster1
Connecting to host gluster1, port 5201
[  5] local 10.1.4.10 port 33184 connected to 10.1.4.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.15 GBytes  9.91 Gbits/sec    0   3.11 MBytes       
[  5]   1.00-2.00   sec  1.15 GBytes  9.89 Gbits/sec    0   3.11 MBytes       
[  5]   2.00-3.00   sec  1.15 GBytes  9.89 Gbits/sec    0   3.25 MBytes       
[  5]   3.00-4.00   sec  1.15 GBytes  9.89 Gbits/sec    0   3.25 MBytes       
[  5]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec    0   3.25 MBytes       
[  5]   5.00-6.00   sec  1.15 GBytes  9.89 Gbits/sec    0   3.25 MBytes       
[  5]   6.00-7.00   sec  1.15 GBytes  9.89 Gbits/sec    0   3.25 MBytes       
[  5]   7.00-8.00   sec  1.15 GBytes  9.90 Gbits/sec    0   3.25 MBytes       
[  5]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec    0   3.25 MBytes       
[  5]   9.00-10.00  sec  1.15 GBytes  9.89 Gbits/sec    0   3.25 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec    0             sender
[  5]   0.00-10.04  sec  11.5 GBytes  9.85 Gbits/sec                  receiver

iperf Done.