Slow 10Gbe SFP+ on Chelsio T520-SO

Discussion in 'Networking' started by MrRedHat, Jun 30, 2018.

  1. MrRedHat

    MrRedHat New Member

    Joined:
    May 25, 2012
    Messages:
    7
    Likes Received:
    0
    I have a Chelsio T520-SO-CR with 10GbE SFP+. I connect it to a Quanta LB6M with the brocade firmware on it along with 10GbE SFP+ from fs.com in it. I put them in my Dell PowerEdge R710’s and transferring between my R710’s I get 1GB/sec. Pretty cool to see a 60GB file transfer in 60 seconds.

    The problem I’m having is on my desktop, I put the Chelsio T520 NIC in it and I only get about 50MB/sec transfer rate. Sometimes it will spike up to 160MB/sec, but typically it’ll settle around 45/50MB/sec.

    I’m wondering if it’s my desktop that’s the issue, but I don’t have the slightest clue to what would be causing it. I know it’s not a disk issue, Crystal Disk Mark tells me my Reads are 2,548MB/sec and my Writes are 1,520MB/sec, so I don’t think it’s a disk issue. My hardware on desktop is an Asus ROG Maximus VIII Extreme with 32GB RAM with a Samsung 950 NVMe M.2 SSD.

    I don’t think it’s the Dell R710’s since they happily saturate the 10Gbe SFP+ cards between each other through the Quanta LB6M. I tried different fiber patch cables, swapped out different Chelsio NIC’s, I even tried running the Chelsio in an another Asus MB (Rampage IV Extreme) and had the same results. I’m wondering I there is some type of motherboard setting or configuration I should be doing or something.
     
    #1
  2. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,710
    Likes Received:
    423
    Do you use windows? If so can you run NTttcp with multithreads and post the results?
     
    #2
  3. MrRedHat

    MrRedHat New Member

    Joined:
    May 25, 2012
    Messages:
    7
    Likes Received:
    0
    This is the output from workstation.

    Code:
    Thread  Time(s) Throughput(KB/s) Avg B / Compl
    ======  ======= ================ =============
         0   10.121        99272.404     65536.000
    
    
    #####  Totals:  #####
    
    
       Bytes(MEG)    realtime(s) Avg Frame Size Throughput(MB/s)
    ================ =========== ============== ================
          981.187500      10.121       1394.761           96.946
    
    
    Throughput(Buffers/s) Cycles/Byte       Buffers
    ===================== =========== =============
                 1551.131      16.009     15699.000
    
    
    DPCs(count/s) Pkts(num/DPC)   Intr(count/s) Pkts(num/intr)
    ============= ============= =============== ==============
        34942.199         0.764       51048.019          0.523
    
    
    Packets Sent Packets Received Retransmits Errors Avg. CPU %
    ============ ================ =========== ====== ==========
          737653           270067        1189      0      5.075
    
     
    #3
  4. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,710
    Likes Received:
    423
    Hmm... Weird. Are you sure that's the 10gbe nic and not an onboard nic?
     
    #4
  5. MrRedHat

    MrRedHat New Member

    Joined:
    May 25, 2012
    Messages:
    7
    Likes Received:
    0
    Yup, it's definitely going through the 10GbE NIC. It's using fiber and I have the onboard completely
    disconnected. I was just wondering if there's maybe a Windows thing that's keeping it locked at 1Gbit. I don't know what it would be and it can't be the OS, because I even tried Linux and I got the same results.

    I then took the same NIC that was in my desktop and put it in my Dell R710 and did a test between another R710. It can't be the NIC and I used the same fiber/port on the switch as the desktop, so I don't think it's the NIC (look below for the results).

    Strange that it would run so slow in both my computers with Asus MB's? One is a Rampage IV Extreme and the other is a Maximus VIII Extreme. So that's why I was thinking maybe there is a MB setting, or maybe slot (I tried them all).


    Code:
    Thread  Time(s) Throughput(KB/s) Avg B / Compl
    ======  ======= ================ =============
         0    1.058      1155206.049     65536.000
    
    
    #####  Totals:  #####
    
    
       Bytes(MEG)    realtime(s) Avg Frame Size Throughput(MB/s)
    ================ =========== ============== ================
         1193.562500       1.058       1455.969         1128.131
    
    
    Throughput(Buffers/s) Cycles/Byte       Buffers
    ===================== =========== =============
                18050.095       2.893     19097.000
    
    
    DPCs(count/s) Pkts(num/DPC)   Intr(count/s) Pkts(num/intr)
    ============= ============= =============== ==============
        29357.278         1.600       50972.590          0.922
    
    
    Packets Sent Packets Received Retransmits Errors Avg. CPU %
    ============ ================ =========== ====== ==========
          859593            49708           0      0      4.123
     
    #5
  6. BLinux

    BLinux cat lover server enthusiast

    Joined:
    Jul 7, 2016
    Messages:
    2,370
    Likes Received:
    838
    when running the throughput test, do you see what CPU utilization is like? how about interrupts?
     
    #6
  7. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,710
    Likes Received:
    423
    @BLinux
    Cpu utilization last line, interrupt count is in the line above cpu utilization.
     
    #7
  8. MrRedHat

    MrRedHat New Member

    Joined:
    May 25, 2012
    Messages:
    7
    Likes Received:
    0
    @i386
    Thanks for the help. I figured out the issue. Stupid me... I had my desktop NIC and my servers on different subnets, so it was routing the packets through my trusted interface on my firewall. Thought I checked it, but I guess I didn't.
     
    #8
Similar Threads: Slow 10Gbe
Forum Title Date
Networking 10GbE network speed slow on Netgear XS716-T Sep 21, 2019
Networking MNPA19-XTR Connectx-2 / Windows 10 - Slow LAN download speed Sep 10, 2019
Networking Quanta LB6M / TurboIron 24x slow transfers Apr 4, 2019
Networking 10 GbE slow between CentOS and FreeNAS Apr 1, 2019
Networking 10Gig to Wifi slower than 1Gig to Wifi Apr 23, 2018

Share This Page