New 10Gbe install, wildly different speeds on similar hardware

[Terry Kennedy]

What about do you get with "iperf -c <server> -w 256k -P 4" ?

My rule of thumb is "if you have to use options with iperf to get decent performance, you're doing something wrong":

(0:50) host:/sysprog/terry# iperf -c test2
------------------------------------------------------------
Client connecting to test2, TCP port 5001
TCP window size: 32.0 KByte (default)
------------------------------------------------------------
[  3] local 10.20.30.40 port 23245 connected with 10.20.30.42 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  11.3 GBytes  9.73 Gbits/sec

This is between a pair of R710's with X540-T1 cards and a Dell 8024 switch, using FreeBSD.

 

[coolrunnings82]

My rule of thumb is "if you have to use options with iperf to get decent performance, you're doing something wrong":

(0:50) host:/sysprog/terry# iperf -c test2
------------------------------------------------------------
Client connecting to test2, TCP port 5001
TCP window size: 32.0 KByte (default)
------------------------------------------------------------
[  3] local 10.20.30.40 port 23245 connected with 10.20.30.42 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  11.3 GBytes  9.73 Gbits/sec

This is between a pair of R710's with X540-T1 cards and a Dell 8024 switch, using FreeBSD.

I suspect this is a Windowsish thing or a driver issue.

Sent from my SM-G900V using Tapatalk

 

[Terry Kennedy]

I suspect this is a Windowsish thing

That would count as "doing something wrong".

 

[zhoulander]

For some reason Windows needs a larger TCP window for me to get 10g, and 4 threads to get 10G to my Freenas box. Can't really get around using Windows because I need it for Premiere. Been trying to use Ubuntu for other things but got really frustrated when the stupid unlock screen refused to take my valid password (verified with CTRL+ALT+F1).

 

[Gerk]

I understand the challenges and limitations for the most part (more so now that I've jumped in two feet first).

Nothing "windows-ish" in my setup here at all, this was all FreeNAS/Ubuntu/CentOS testing happening, and on the (literally) exact same boot ISO one machine is just sort of 'stuck' on only receiving at half speed. The thing that is puzzling the hell out of me is how the interface can send at full bandwidth but only receive at half bandwidth and only on one particular machine ... I guess at this point I've narrowed it down to being a motherboard/PCIe riser issue(? I think?)

I suppose I can try swapping out the PCIe riser with the other machine (I have two almost identical).

And the moral of the story is that I guess that I should be doing a little server juggling -- the machine that can only do half bandwidth one-way should probably NOT be my main file server :/

 

[djflow195]

Have you done any tuning?

For Linux https://fasterdata.es.net/host-tuning/linux/
For Freebsd/FreeNAS https://fasterdata.es.net/host-tuning/freebsd/

 

[Gerk]

Have you done any tuning?

For Linux https://fasterdata.es.net/host-tuning/linux/
For Freebsd/FreeNAS https://fasterdata.es.net/host-tuning/freebsd/

No I hadn't, but I just tried the options suggested there for the FBSD tuning and it didn't seem to make any real difference. I'm still seeing that wall of ~5Gb/s (that I can push closer to 6 but never quite hit it if I ramp up more connections).

I think I'm just going to take this is a sign from the universe and transplant the machines and see what happens. That's about the only thing I haven't done yet.