Yea, the CPU should be more than capable. See my posts in this thread: https://forums.servethehome.com/ind...mance-looking-for-tweaks-hints-whatever.7031/. Specifically, this post (#12) where I tested Sequential Samba4 performance on a L5520 underclocked to 1.6GHz with only 2 cores enabled.For what its worth:iperf is sending a single stream (meaning a single thread). On a 5520 you will bottleneck a single core of the CPU long before you reach 10Gbe throughput. You've got to get all 4 cores (or even all 8 threads) active before you'll see 10Gbe on the wire.
Hardware is dual E5520, network card is X540-T1. FreeBSD 8.4.Code:CPU: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (2275.82-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x106a5 Family = 6 Model = 1a Stepping = 5 (0:4) test1:/sysprog/terry# iperf -c test2 ------------------------------------------------------------ Client connecting to test2, TCP port 5001 TCP window size: 32.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.20.30.40 port 26252 connected with 10.20.30.41 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.2 sec 11.7 GBytes 9.89 Gbits/sec
Does it look better now?This one of those thing where you need to set max switch device MTU to something like 9216? It's different for various vendors but it's worth looking into, seems like you have some HEAVY fragmentation/re-transmissions going on, bet a wireshark session replayed would look...ummm...interesting :-D
That is probably most likely because you are using Windows. It is sad well known fact that the TCP implementation for Windows has serious performance/latency issues.I have clearly seen on my 10G network w/out jumbo frames I can maybe push roughly 6Gbps until I turn up jumbo then I get line rate. With GigE yeah not much point to use jumbo, on 10G it is fairly dramatic (to the tune of 25-30% throughput increase I have found in my testing).
Can anyone else confirm similar results or am I just making this sh|t up? heh :-D
2cents. YMMV
EDIT: Proof in pudding. Comparison at least on my network w/ non-jumbo and jumbo enabled so you cannot tell me for the life of me that jumbo does not do anything or is even close to 'marginal' speed increases.
This is VM to VM on different hosts, hosts have x520 intel adapter and VM's have vmxnet3 virtual nics, forcing traffic up over ex3300 and the aforementioned infrastructure so this is abt as representative to 'real-world' as you can get. I recently had this debate w/ a colleague of mine who touted that jumbo was mostly worthless or introduced more complexity than benefits gained then he saw these results and quickly retracted.
meanwhile@mw-dev1:~$ iperf -c 10.2.2.10 -t 20 -m -P3 -i 4
------------------------------------------------------------
Client connecting to 10.2.2.10, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 5] local 10.2.2.20 port 53256 connected with 10.2.2.10 port 5001
[ 3] local 10.2.2.20 port 53254 connected with 10.2.2.10 port 5001
[ 4] local 10.2.2.20 port 53255 connected with 10.2.2.10 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 4.0 sec 1.43 GBytes 3.07 Gbits/sec
[ 4] 0.0- 4.0 sec 1.31 GBytes 2.81 Gbits/sec
[ 5] 0.0- 4.0 sec 1.55 GBytes 3.33 Gbits/sec
[SUM] 0.0- 4.0 sec 4.29 GBytes 9.21 Gbits/sec
[ 5] 4.0- 8.0 sec 1.68 GBytes 3.62 Gbits/sec
[ 3] 4.0- 8.0 sec 1.34 GBytes 2.89 Gbits/sec
[ 4] 4.0- 8.0 sec 1.36 GBytes 2.91 Gbits/sec
[SUM] 4.0- 8.0 sec 4.38 GBytes 9.41 Gbits/sec
[ 5] 8.0-12.0 sec 1.60 GBytes 3.44 Gbits/sec
[ 3] 8.0-12.0 sec 1.42 GBytes 3.04 Gbits/sec
[ 4] 8.0-12.0 sec 1.36 GBytes 2.92 Gbits/sec
[SUM] 8.0-12.0 sec 4.38 GBytes 9.40 Gbits/sec
[ 3] 12.0-16.0 sec 1.33 GBytes 2.85 Gbits/sec
[ 4] 12.0-16.0 sec 1.35 GBytes 2.90 Gbits/sec
[ 5] 12.0-16.0 sec 1.71 GBytes 3.67 Gbits/sec
[SUM] 12.0-16.0 sec 4.38 GBytes 9.41 Gbits/sec
[ 5] 16.0-20.0 sec 1.70 GBytes 3.65 Gbits/sec
[ 5] 0.0-20.0 sec 8.25 GBytes 3.54 Gbits/sec
[ 5] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
[ 3] 0.0-20.0 sec 6.85 GBytes 2.94 Gbits/sec
[ 3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
[ 4] 0.0-20.0 sec 6.72 GBytes 2.89 Gbits/sec
[ 4] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
[SUM] 0.0-20.0 sec 21.8 GBytes 9.37 Gbits/sec
meanwhile@mw-dev1:~$ iperf -c 10.2.2.10 -t 20 -m -P3 -i 4
------------------------------------------------------------
Client connecting to 10.2.2.10, TCP port 5001
TCP window size: 325 KByte (default)
------------------------------------------------------------
[ 5] local 10.2.2.20 port 53259 connected with 10.2.2.10 port 5001
[ 4] local 10.2.2.20 port 53258 connected with 10.2.2.10 port 5001
[ 3] local 10.2.2.20 port 53257 connected with 10.2.2.10 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 4.0 sec 1.34 GBytes 2.87 Gbits/sec
[ 3] 0.0- 4.0 sec 1.31 GBytes 2.81 Gbits/sec
[ 5] 0.0- 4.0 sec 1.87 GBytes 4.01 Gbits/sec
[SUM] 0.0- 4.0 sec 4.52 GBytes 9.70 Gbits/sec
[ 5] 4.0- 8.0 sec 1.83 GBytes 3.92 Gbits/sec
[ 4] 4.0- 8.0 sec 1.37 GBytes 2.95 Gbits/sec
[ 3] 4.0- 8.0 sec 1.40 GBytes 3.00 Gbits/sec
[SUM] 4.0- 8.0 sec 4.60 GBytes 9.87 Gbits/sec
[ 5] 8.0-12.0 sec 1.97 GBytes 4.23 Gbits/sec
[ 4] 8.0-12.0 sec 1.33 GBytes 2.85 Gbits/sec
[ 3] 8.0-12.0 sec 1.30 GBytes 2.80 Gbits/sec
[SUM] 8.0-12.0 sec 4.60 GBytes 9.88 Gbits/sec
[ 5] 12.0-16.0 sec 1.93 GBytes 4.15 Gbits/sec
[ 4] 12.0-16.0 sec 1.33 GBytes 2.86 Gbits/sec
[ 3] 12.0-16.0 sec 1.34 GBytes 2.88 Gbits/sec
[SUM] 12.0-16.0 sec 4.60 GBytes 9.89 Gbits/sec
[ 3] 0.0-20.0 sec 6.74 GBytes 2.89 Gbits/sec
[ 3] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)
[ 5] 16.0-20.0 sec 1.82 GBytes 3.90 Gbits/sec
[ 5] 0.0-20.0 sec 9.41 GBytes 4.04 Gbits/sec
[ 5] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)
[ 4] 0.0-20.0 sec 6.77 GBytes 2.91 Gbits/sec
[ 4] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)
[SUM] 0.0-20.0 sec 22.9 GBytes 9.84 Gbits/sec
That's ALMOST laughable, I almost NEVER use windows at all on my LAN unless absolutely necessary (15+ yrs as Unix/Linux Systems Engineer from all three proprietary Unix's as well as just about every Linux flavor you can think of and some BSD here and there). Of the 30+ VM's i run day in and day out maybe a small handful 3-4 are Windows.That is probably most likely because you are using Windows. It is sad well known fact that the TCP implementation for Windows has serious performance/latency issues.
Linux is much better, while FreeBSD is the king(maybe Solaris is even better, I don't know much about it).
... With GigE yeah not much point to use jumbo ...
I tried this. I am starting to think its the cables or the transceivers.I've been trying to troubleshoot my ipsec S2S VPN throughput due to WAN latency and found the SpeedGuide.net tool "SG TCP Optimizer" really amazing to quickly tweak your TCP settings.
Set the slider to 100+Mbps and "Optimal"; and you can see that it will increase your TCP Window Auto-Tuning to "experimental" and other setttings like Receive-Side Scaling (RSS), Congestion, etc...all of these will have impact on TCP throughput in the LAN also.
According to the Dell website these the 8x slot is electrically 8x also, the 16x is also 16x. not sure why this happened?Your 10gbe card is an X8 card so that shouldn't matter - unless the X8 slot is only x4 electrical (which is actually likely on many server MB).
What MB are you using?
NICE! Surprisingly, no one suggested that. Well, the 8x slot on your MB should have worked in the first place. That's bizzare. What's the MB on your boxes?GREAT NEWS!!
I am the biggest IDIOT ever. I switched the card in the Server from the PCI 8X slot to the PCI 16x and see the different. Issue fixed
iperf3.exe -c amaran3
Connecting to host amaran3, port 5201
[ 4] local 192.168.0.85 port 49903 connected to 192.168.0.41 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 1.08 GBytes 9.31 Gbits/sec
[ 4] 1.00-2.00 sec 1.09 GBytes 9.35 Gbits/sec
[ 4] 2.00-3.00 sec 1.08 GBytes 9.32 Gbits/sec
[ 4] 3.00-4.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 4] 4.00-5.00 sec 1.09 GBytes 9.36 Gbits/sec
[ 4] 5.00-6.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 4] 6.00-7.00 sec 1.09 GBytes 9.36 Gbits/sec
[ 4] 7.00-8.00 sec 1.10 GBytes 9.43 Gbits/sec
[ 4] 8.00-9.00 sec 1.10 GBytes 9.43 Gbits/sec
[ 4] 9.00-10.00 sec 1.09 GBytes 9.39 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 10.9 GBytes 9.37 Gbits/sec sender
[ 4] 0.00-10.00 sec 10.9 GBytes 9.37 Gbits/sec receiver
iperf Done.
Thank you
According to Dell when I called them, they confirmed the 8x should be able to work well this the Intel 10GBE cards. I am starting to think the riser card is limited bandwidth. I moved this card to the 16x on the other risercard.NICE! Surprisingly, no one suggested that. Well, the 8x slot on your MB should have worked in the first place. That's bizzare. What's the MB on your boxes?
Do you have any ebay links for a good switch I can upgrade to?Mellanox ConnectX-2 is super cheap. ~ $15 - $20 bucks. But it's a bit dated and doesn't have Remote Direct Memory Access (RDMA). Also, when buying these cards, be sure you get the correct profile for your available slots on your server chassis. Low profile or regular profile. Finding the low profile bracket for them is nearly impossible.
With RDMA you can truly test Windows SMB Direct, SMB Multi-Channel, etc. Check this link. http://www.mellanox.com/page/microsoft_based_solutions
Might try ConnectX-3 but I think those are for 40 GbE. Time to update your "newly obsolete" switch. LOL