40GbE card appears capped at 14Gbps

Discussion in 'FreeBSD FreeNAS and TrueNAS Core' started by Prototype, Feb 12, 2020.

  1. Prototype

    Prototype New Member

    Joined:
    Feb 12, 2020
    Messages:
    5
    Likes Received:
    0
    Hey, back down the rabbit hole of "how to use mellanox cards with 0 linux knowledge" d:^)

    I currently have a pair of CX314A (Connect X-3)(40GbE Ethernet Only) connected over SR4 transceivers[Mellanox 40G QSFP+ MC2210411-SR4 Transceiver Module] and OM4 MMF (12 strand) [3m (10ft) 12-Fiber OM4 50/125 MPO Trunk Cable, Polarity B, LSZH] I've made no changes outside of increasing MTU to 9000 on both FreeNas and Windows. Yet transfer speeds seem hard capped at 14 Gb/s. This is especially noticeable when running multiple transfers from FreeNas to the same Windows machine on different drives. Neither the FreeNas array nor Windows drive destinations are causing the bottleneck. One thing I have noticed is the "Media Subtype" in Freenas showing CR4 rather than SR4. Is this a potential cause? Any suggestions on how to edit this property?

    Thank you kindly in advance. unknown.png
     
    #1
  2. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    4,020
    Likes Received:
    692
    How have you measure the 14 gigabit?
    Have you run iperf yet? Is it both ways or one way?
     
    #2
  3. Prototype

    Prototype New Member

    Joined:
    Feb 12, 2020
    Messages:
    5
    Likes Received:
    0
    I've only transferred files back and forth via an SMB share through Windows Explorer. The 14Gb/s cap applies both ways, yes. I'm unfamiliar with running iperf through FreeNAS, though looking into it.
     
    #3
  4. fossxplorer

    fossxplorer Active Member

    Joined:
    Mar 17, 2016
    Messages:
    448
    Likes Received:
    60
    For me your issue doesn't sound like it's related to cabling since 14Gbps isn't a multiple of FDR10 signaling rates (https://en.wikipedia.org/wiki/InfiniBand) which seems to be what the 40GbE cards use. But i might very well be wrong :)
    Is your setup tuned according to Mellanox tuning guide with regard to IRQ affinity, performance profile of the server both in BIOS and in the OS (eg. power states and performance profile within the OS). Not familiar with FreeBSD at all, for instance in RHEL/CentOS you have different performance profile one can set/use.

    Another thing i learned from experimenting with Mellanox ConnectX-3 VPI cards (56Gb IB and 40GbE) in Ethernet mode is that the TCP/IP performance is CPU bound.

    And as @Rand__ indicates, try to measure the pure network performance using iperf and similar software.
     
    #4
  5. Prototype

    Prototype New Member

    Joined:
    Feb 12, 2020
    Messages:
    5
    Likes Received:
    0
    Thanks for the reply. It turns out updating the firmware and running iPerf gives much better results. However, Windows SMB continue to tank. I believe enabling RDMA might be a solution - did this on Windows side already, any ideas how to go about it on FreeNAS side?

    iPerf 1.png iPerf 2.png SMB.png
     
    #5
  6. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    4,020
    Likes Received:
    692
    No RDMA on FreeNas unfortunately ... you will need to run Linux with ZoL for this
     
    #6
  7. Prototype

    Prototype New Member

    Joined:
    Feb 12, 2020
    Messages:
    5
    Likes Received:
    0
    Ah that's quite a bummer. Perhaps NFS can give slightly better numbers though. Running into this issue with any file larger than ~1 MB. Any thoughts? Enabled NFS client on windows and set UID/GID to 0 (root). io error.JPG
     
    #7
  8. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    4,020
    Likes Received:
    692
    Well Nfs on Windows usually is slower

    Have you tuned both sides with increased buffers and threads? Or at least the freenas side with 10GBe optimizations? TCP Offloads?

    Whats your expectation set?
    Looks like your 3 large "backup files" combined to 1,5GB/s which is not a bad write value - whats your target array?
    Mind that network latency always take a big chunk of local performance - so I agree that RDMA is needed but its not there yet unfortunately...
     
    #8
  9. weust

    weust Active Member

    Joined:
    Aug 15, 2014
    Messages:
    306
    Likes Received:
    26
    Just wondering, doesn't the Windows machine need a slightly higher MTU because Windows add some of its own to it?
    I had that at home. Wasn't able to ping using a MTU of 9000.

    Can you ping with 9000?
     
    #9
  10. Prototype

    Prototype New Member

    Joined:
    Feb 12, 2020
    Messages:
    5
    Likes Received:
    0
    If by "ping" you mean simply pinging from Windows and moving data, yep works without an issue. Windows side is giving me the same results with 9000 and 9014. However, a Chelsio T580 LP-CR that I have around only has set options for Jumbo Packets at 4088 or 9014, for what it's worth. Chelsio set and Mellanox set all cap at around the same point of ~14 Gbps transfer speeds via SMB.
     
    #10
  11. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,775
    Likes Received:
    443
    Try robocopy and more threads
     
    #11
    WillowMaori likes this.
  12. thingy2098

    thingy2098 New Member

    Joined:
    Mar 16, 2018
    Messages:
    15
    Likes Received:
    8
    It does seem a little odd that your 3 transfers all seem to bump right up against the theoretical SATA3 6Gb/s limit / SAS 6Gb/s (single port) limit.
    Since you can get 40Gb/s through iperf, you've at least confirmed the network stack on both sides is capable, as is the physical link.

    Are you running FreeNAS (or the Windows) side on bare metal or in a VM? When I was doing performance tuning on my file server VM on a reasonably capable ESXi host, I actually needed to fiddle with processor affinity and some VM settings to even get it to sustain 10Gb reliably; I can only imagine 40Gb is that much harder.

    My situation was complicated by the fact that my ESXi host had dual CPUs and each PCIe slot was only physically connected to one of the two CPUs PCIe lanes, and any time the data had to cross the inter-CPU bus performance fell through the floor.

    I had to make sure to keep my HBA and my 10gig NIC on the same CPU, and make sure that my file server VM could only create threads on that one.

    SMB transfers are murder on the CPU without offload. This is a 10GbE transfer with nothing else happening, using up almost all 16 cores of the one Opteron CPU it was allotted. My hardware is old to be sure, but there's no way it'd be capable of 40gig over SMB.

    Image link

    [​IMG]
    [​IMG]
     
    #12
  13. WillowMaori

    WillowMaori New Member

    Joined:
    Mar 6, 2020
    Messages:
    2
    Likes Received:
    0
    I agree with you, trying to use robocopy or any its alternatives like teracopy, securecopy or gs richcopy360 ....etc that can work with multithreading technology can help.
     
    #13
Similar Threads: 40GbE card
Forum Title Date
FreeBSD FreeNAS and TrueNAS Core Displayport 1.2 (MST) graphics card on FreeBSD that actually works? Nov 8, 2017
FreeBSD FreeNAS and TrueNAS Core most affordable 24pt SAS2 Expander cards? (w/cables, not chassis/backplane) Feb 15, 2017
FreeBSD FreeNAS and TrueNAS Core Mellanox CX4 ConnectX VPI cards and FreeNAS Jul 23, 2016
FreeBSD FreeNAS and TrueNAS Core Do Mellanox ConnectX-2 EN 10GbE Cards work in FreeNAS? May 3, 2015

Share This Page