Slow 10Gbe SFP+ on Chelsio T520-SO

[MrRedHat]

I have a Chelsio T520-SO-CR with 10GbE SFP+. I connect it to a Quanta LB6M with the brocade firmware on it along with 10GbE SFP+ from fs.com in it. I put them in my Dell PowerEdge R710’s and transferring between my R710’s I get 1GB/sec. Pretty cool to see a 60GB file transfer in 60 seconds.

The problem I’m having is on my desktop, I put the Chelsio T520 NIC in it and I only get about 50MB/sec transfer rate. Sometimes it will spike up to 160MB/sec, but typically it’ll settle around 45/50MB/sec.

I’m wondering if it’s my desktop that’s the issue, but I don’t have the slightest clue to what would be causing it. I know it’s not a disk issue, Crystal Disk Mark tells me my Reads are 2,548MB/sec and my Writes are 1,520MB/sec, so I don’t think it’s a disk issue. My hardware on desktop is an Asus ROG Maximus VIII Extreme with 32GB RAM with a Samsung 950 NVMe M.2 SSD.

I don’t think it’s the Dell R710’s since they happily saturate the 10Gbe SFP+ cards between each other through the Quanta LB6M. I tried different fiber patch cables, swapped out different Chelsio NIC’s, I even tried running the Chelsio in an another Asus MB (Rampage IV Extreme) and had the same results. I’m wondering I there is some type of motherboard setting or configuration I should be doing or something.

 

[i386]

Do you use windows? If so can you run NTttcp with multithreads and post the results?

 

[MrRedHat]

This is the output from workstation.

Thread  Time(s) Throughput(KB/s) Avg B / Compl
======  ======= ================ =============
     0   10.121        99272.404     65536.000


#####  Totals:  #####


   Bytes(MEG)    realtime(s) Avg Frame Size Throughput(MB/s)
================ =========== ============== ================
      981.187500      10.121       1394.761           96.946


Throughput(Buffers/s) Cycles/Byte       Buffers
===================== =========== =============
             1551.131      16.009     15699.000


DPCs(count/s) Pkts(num/DPC)   Intr(count/s) Pkts(num/intr)
============= ============= =============== ==============
    34942.199         0.764       51048.019          0.523


Packets Sent Packets Received Retransmits Errors Avg. CPU %
============ ================ =========== ====== ==========
      737653           270067        1189      0      5.075

 

[i386]

Hmm... Weird. Are you sure that's the 10gbe nic and not an onboard nic?

 

[MrRedHat]

Yup, it's definitely going through the 10GbE NIC. It's using fiber and I have the onboard completely
disconnected. I was just wondering if there's maybe a Windows thing that's keeping it locked at 1Gbit. I don't know what it would be and it can't be the OS, because I even tried Linux and I got the same results.

I then took the same NIC that was in my desktop and put it in my Dell R710 and did a test between another R710. It can't be the NIC and I used the same fiber/port on the switch as the desktop, so I don't think it's the NIC (look below for the results).

Strange that it would run so slow in both my computers with Asus MB's? One is a Rampage IV Extreme and the other is a Maximus VIII Extreme. So that's why I was thinking maybe there is a MB setting, or maybe slot (I tried them all).

Thread  Time(s) Throughput(KB/s) Avg B / Compl
======  ======= ================ =============
     0    1.058      1155206.049     65536.000


#####  Totals:  #####


   Bytes(MEG)    realtime(s) Avg Frame Size Throughput(MB/s)
================ =========== ============== ================
     1193.562500       1.058       1455.969         1128.131


Throughput(Buffers/s) Cycles/Byte       Buffers
===================== =========== =============
            18050.095       2.893     19097.000


DPCs(count/s) Pkts(num/DPC)   Intr(count/s) Pkts(num/intr)
============= ============= =============== ==============
    29357.278         1.600       50972.590          0.922


Packets Sent Packets Received Retransmits Errors Avg. CPU %
============ ================ =========== ====== ==========
      859593            49708           0      0      4.123

 

[BLinux]

when running the throughput test, do you see what CPU utilization is like? how about interrupts?

 

[i386]

@BLinux
Cpu utilization last line, interrupt count is in the line above cpu utilization.

 

[MrRedHat]

@i386
Thanks for the help. I figured out the issue. Stupid me... I had my desktop NIC and my servers on different subnets, so it was routing the packets through my trusted interface on my firewall. Thought I checked it, but I guess I didn't.