Can't Get Over 400MB/s On One Workstation...

[uldise]

i can't see, how you can achieve more than 1Gbe copy speed when iperf3 clearly states it 1GBe..
some power management profiles active? is it on full performance?

 

[mattr]

Yeah now you see why I've been racking my brain over this for months... lol. None of it makes any sense.

The "turn this device off to save power" was checked. I unchecked it and the iperf3 results actually went up to 2G speeds but the file copy is still stuck at 400MB/s. I should note I've seen the iperf results jump around from 1G to 4G speeds previously but typically they just show 1G speeds.

 

[uldise]

try to load Live Linux, and repeat iperf3 test, if you can. then we will see - it's a Windows or not..

 

[uldise]

and are your power plan in Windows set to High Performance?

 

[mattr]

So I just used an Ubuntu live image and got essentially the same results. The iperf3 results shows between 1G to 4G connection. The file copy in the linux live image was actually stuck around 250MB/s.

 

[tinfoil3d]

dmesg | grep -i limited
and look for your card pci id in lspci. i had this case with limited lanes in a very legacy system.

 

[uldise]

The iperf3 results shows between 1G to 4G connection

this is not ok. speed should be almost steady.. do you have another card to try?

i had this case with limited lanes in a very legacy system.

Yes, very similar for me too - cant get full 10Gbe on SM X8 system. after switching to X9, full 10Gbe saturated without any other changes.

 

[jdnz]

do you have multiple NICs on the truenas system you’re running the other end of iperf3 on - same as the client end iperf3 will by default bind to the first interface it finds, if the machine has multiple NICs you’ll need to use the -B option to bind to the correct (10gbe) i/f

 

[mattr]

dmesg | grep -i limited
and look for your card pci id in lspci. i had this case with limited lanes in a very legacy system.

This returns nothing. It's an X11 system with gen 6 Xeon so it's not what I would call "legacy".

 

[mattr]

this is not ok. speed should be almost steady.. do you have another card to try?


Yes, very similar for me too - cant get full 10Gbe on SM X8 system. after switching to X9, full 10Gbe saturated without any other changes.

I've tried several cards. My X8 system seems to be my fastest....

 

[uldise]

are your BIOS at latest version? are you tried load BIOS defaults? are you configured any PCIe Bifurcation, that would limit PCIe count to card?

 

[mattr]

are your BIOS at latest version? are you tried load BIOS defaults? are you configured any PCIe Bifurcation, that would limit PCIe count to card?

Yes I'm on the latest BIOS. I don't see any bifurcation settings in the BIOS. Not sure its supported on this board.

 

[uldise]

so, your iperf3 tests was with only NIC connected to the system? try disconnect all other stuff, boot drive included.
are you tried swapping cables, and ports on switch side too? i have 16-XG but ES version - far from the latest firmware, and was observed that middle ports have with DACs, but works just fine with SFP+ optics.

 

[RTM]

This returns nothing. It's an X11 system with gen 6 Xeon so it's not what I would call "legacy".

It is not something limited to legacy systems, but if that command resulted in nothing, I doubt it is the reason.
If you want to be sure, you would need to figure out what the actual linkwidth is, here is how I usually do this:

$ lspci
..
3d:00.0 Non-Volatile memory controller: Sandisk Corp Device 5006
..

Then I do something like this:

$ sudo lspci -s 3d:00.0 -vv | grep -i lnk
        LnkCap:    Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <256ns, L1 <8us
        LnkCtl:    ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
        LnkSta:    Speed 8GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+

Note that the "3d:00.0" comes from the first command, you will need to find the value for your NIC, if it not labeled Supermicro etc., you may need to look for keywords like 82599 or X520. The line should be labeled with "Ethernet Controller" after "3d:00.0" or whatever it will be.
Look at the LnkSta, where it specifies "Width x2" in my case, this SSD is limited to x2 in my Thinkpad (thank you Lenovo ).

 

[mattr]

So, I'm chasing some sort of mythical evil here. This isn't worth troubleshooting anymore. I moved my desktop workstation (which hits 800MB/s with ease) down to the rack mount workstation (the one stuck at 400MB/s) and now the hardware from my desktop workstation is stuck at 400MB/s.

Just by putting the desktop hardware in the server chassis and moving it 40ft closer that machine is now stuck at 400MB/s. Yes I tried with the same type fiber/SFP+ transceivers and tried with multiple DAC cables..... I'm just going with the belief that the chassis is cursed. There honestly isn't anything to troubleshoot at this point. The issue makes no sense.

EDIT: oh and I put the hardware from the rack mount workstation up in my office and it now hits 800MB/s.

EDIT2: the hardware that is in the rack mount workstation now is the X99 machine from my office which has tons of bandwidth available.

 

[RTM]

Assuming everything is cabled the same (meaning the desktop workstation hardware is now using the same wiring as the former rack mount workstation hardware), it sure sounds like the issue is with the switch.

To verify, you could try connecting the two devices that you run iperf directly, ie without the switch.

 

[mattr]

Assuming everything is cabled the same (meaning the desktop workstation hardware is now using the same wiring as the former rack mount workstation hardware), it sure sounds like the issue is with the switch.

To verify, you could try connecting the two devices that you run iperf directly, ie without the switch.

Yeah I plugged the "new" hardware directly into the same cables. However, I've tried multiple switch ports.