40GbE card appears capped at 14Gbps

[Prototype]

Hey, back down the rabbit hole of "how to use mellanox cards with 0 linux knowledge" d:^)

I currently have a pair of CX314A (Connect X-3)(40GbE Ethernet Only) connected over SR4 transceivers[Mellanox 40G QSFP+ MC2210411-SR4 Transceiver Module] and OM4 MMF (12 strand) [3m (10ft) 12-Fiber OM4 50/125 MPO Trunk Cable, Polarity B, LSZH] I've made no changes outside of increasing MTU to 9000 on both FreeNas and Windows. Yet transfer speeds seem hard capped at 14 Gb/s. This is especially noticeable when running multiple transfers from FreeNas to the same Windows machine on different drives. Neither the FreeNas array nor Windows drive destinations are causing the bottleneck. One thing I have noticed is the "Media Subtype" in Freenas showing CR4 rather than SR4. Is this a potential cause? Any suggestions on how to edit this property?

Thank you kindly in advance.

 

[Rand__]

How have you measure the 14 gigabit?
Have you run iperf yet? Is it both ways or one way?

 

[Prototype]

How have you measure the 14 gigabit?
Have you run iperf yet? Is it both ways or one way?

I've only transferred files back and forth via an SMB share through Windows Explorer. The 14Gb/s cap applies both ways, yes. I'm unfamiliar with running iperf through FreeNAS, though looking into it.

 

[fossxplorer]

For me your issue doesn't sound like it's related to cabling since 14Gbps isn't a multiple of FDR10 signaling rates (https://en.wikipedia.org/wiki/InfiniBand) which seems to be what the 40GbE cards use. But i might very well be wrong
Is your setup tuned according to Mellanox tuning guide with regard to IRQ affinity, performance profile of the server both in BIOS and in the OS (eg. power states and performance profile within the OS). Not familiar with FreeBSD at all, for instance in RHEL/CentOS you have different performance profile one can set/use.

Another thing i learned from experimenting with Mellanox ConnectX-3 VPI cards (56Gb IB and 40GbE) in Ethernet mode is that the TCP/IP performance is CPU bound.

And as @Rand__ indicates, try to measure the pure network performance using iperf and similar software.

 

[Prototype]

For me your issue doesn't sound like it's related to cabling since 14Gbps isn't a multiple of FDR10 signaling rates (https://en.wikipedia.org/wiki/InfiniBand) which seems to be what the 40GbE cards use. But i might very well be wrong
Is your setup tuned according to Mellanox tuning guide with regard to IRQ affinity, performance profile of the server both in BIOS and in the OS (eg. power states and performance profile within the OS). Not familiar with FreeBSD at all, for instance in RHEL/CentOS you have different performance profile one can set/use.

Another thing i learned from experimenting with Mellanox ConnectX-3 VPI cards (56Gb IB and 40GbE) in Ethernet mode is that the TCP/IP performance is CPU bound.

And as @Rand__ indicates, try to measure the pure network performance using iperf and similar software.

Thanks for the reply. It turns out updating the firmware and running iPerf gives much better results. However, Windows SMB continue to tank. I believe enabling RDMA might be a solution - did this on Windows side already, any ideas how to go about it on FreeNAS side?

 

[Rand__]

No RDMA on FreeNas unfortunately ... you will need to run Linux with ZoL for this

 

[Prototype]

No RDMA on FreeNas unfortunately ... you will need to run Linux with ZoL for this

Ah that's quite a bummer. Perhaps NFS can give slightly better numbers though. Running into this issue with any file larger than ~1 MB. Any thoughts? Enabled NFS client on windows and set UID/GID to 0 (root).

 

[Rand__]

Well Nfs on Windows usually is slower

Have you tuned both sides with increased buffers and threads? Or at least the freenas side with 10GBe optimizations? TCP Offloads?

Whats your expectation set?
Looks like your 3 large "backup files" combined to 1,5GB/s which is not a bad write value - whats your target array?
Mind that network latency always take a big chunk of local performance - so I agree that RDMA is needed but its not there yet unfortunately...

 

[weust]

Just wondering, doesn't the Windows machine need a slightly higher MTU because Windows add some of its own to it?
I had that at home. Wasn't able to ping using a MTU of 9000.

Can you ping with 9000?

 

[Prototype]

Just wondering, doesn't the Windows machine need a slightly higher MTU because Windows add some of its own to it?
I had that at home. Wasn't able to ping using a MTU of 9000.

Can you ping with 9000?

If by "ping" you mean simply pinging from Windows and moving data, yep works without an issue. Windows side is giving me the same results with 9000 and 9014. However, a Chelsio T580 LP-CR that I have around only has set options for Jumbo Packets at 4088 or 9014, for what it's worth. Chelsio set and Mellanox set all cap at around the same point of ~14 Gbps transfer speeds via SMB.

 

[i386]

Try robocopy and more threads

 

[thingy2098]

It does seem a little odd that your 3 transfers all seem to bump right up against the theoretical SATA3 6Gb/s limit / SAS 6Gb/s (single port) limit.
Since you can get 40Gb/s through iperf, you've at least confirmed the network stack on both sides is capable, as is the physical link.

Are you running FreeNAS (or the Windows) side on bare metal or in a VM? When I was doing performance tuning on my file server VM on a reasonably capable ESXi host, I actually needed to fiddle with processor affinity and some VM settings to even get it to sustain 10Gb reliably; I can only imagine 40Gb is that much harder.

My situation was complicated by the fact that my ESXi host had dual CPUs and each PCIe slot was only physically connected to one of the two CPUs PCIe lanes, and any time the data had to cross the inter-CPU bus performance fell through the floor.

I had to make sure to keep my HBA and my 10gig NIC on the same CPU, and make sure that my file server VM could only create threads on that one.

SMB transfers are murder on the CPU without offload. This is a 10GbE transfer with nothing else happening, using up almost all 16 cores of the one Opteron CPU it was allotted. My hardware is old to be sure, but there's no way it'd be capable of 40gig over SMB.

Image link

 

[WillowMaori]

Try robocopy and more threads

I agree with you, trying to use robocopy or any its alternatives like teracopy, securecopy or gs richcopy360 ....etc that can work with multithreading technology can help.