10g Finding Bottleneck

[ColPanic]

Im trying to locate the bottleneck on a 10g network between 2 esxi hosts.

Both machines have Mellanox 10g SFPs with a Cisco cable between. Both are running ESXi 6. I created a new vSwitch in each and added a VMNET3 NIC to each Win2012r2 VM, added the 10g Nics to the vSwitches, configured static IPs, set MTU to 9000 - all good.

Then in each VM I created a RAM disk and shared it. Locally, ATTO can read and write at about 2.5 GB/s but when I copy a file between the RAM disks on the 2 VMs it hits about 300 MB/s but thats it.

I'm new to 10g networking. Can anyone tell me where I'm going wrong?

 

[Rand__]

Different driver? Or optimization settings? I know I had to tune my Intel 10GBe adapters to perform better... not sure whether thats needed/possible on those though

 

[BackupProphet]

Have you checked CPU load when doing that file copy? The higher CPU load, the slower the file transfer will be. If you want to benchmark your 10G network, I highly recommend iperf.

 

[mervincm]

Before you muss with jumbo frames, why not simplify and see what you get without it. done wrong it will kill performance, and you should be able to see a lot more that what you are seeing with normal sized frames, so they are not helping yet.

 

[ColPanic]

CPU load is basically zilch - these are the only VMs running. Host 1 is 2x E5-2670 host 2 is i7-4790k.
I turned disabled jumbo frames and it dropped to 110MB/s. Drivers are the same. Its all just esxi 6.0.0.
I'll see what iPerf says. And maybe I'll try passing the NICs through to the VMs. That should tell me if its something on the windows end or not.

 

[BackupProphet]

I also get slow performance on Windows for file copy. In fact I get much better performance using iSCSI.

 

[mervincm]

maybe boot both systems off of a live linux boot and transfer some data between. I like to fall back to repeatable simple configuration when looking into items like this one.

In any case I saw over 9Gbit/sec using iperf between my systems w cheap broadcom 10g cards w/o jumbo frames.

Today I only have a Win10 desktop, and an XPenology Linux based NAS with 10G, and I see peaks about 700MB/sec windows file transfers to NAS (normal frames)

 

[aero]

Standard size (1500byte) frames are getting a bad rap these days. However, the throughput difference between using the default frame size and jumbo is negligible.

Jumbo frames are 99% efficient, while 1500 byte MTU is 95% efficient. This works out to maximum throughput difference of only 51MB/s. That's not much when we're talking about 1.25GB/s.

However, this assumes you have decent CPUs to handle smaller packets (interrupts) and decent NICs. Most NICS handle the actual frame segmentation these days (LSO/LRO).

 

[NetWise]

Windows file sharing isn't multithreaded and takes time to speed up and slow down. You'd want to run multiple parallel copies at once. Don't bother watching the GUI for hoe fast the file is moving, watch the NIC performance to see the aggregate.


Sent from my iPhone using Tapatalk

 

[razvanc.mobile]

did you set jumbo frames on all devices involved, virtual and physical? nics, vswitches etc?
i'm not sure how much your setup would be affected by fragmentation, in my case the impact was nowhere this big (basically had jumbo set on all except a physical switch, and iperf results were just a bit over 9gbps, after setting jumbo on the switch as well, went up to 9.7gbps)



Sent from my SM-G930F using Tapatalk

 

[aero]

In his scenario he indicates the hosts are directly connected with no physical switch in between.

Fyi, layer 2 switches don't fragment packets. They will simply drop them if they are bigger than the mtu. Routers (and layer 3 switches) fragment packets.

I'm guessing netwise hit the nail on the head in his post, although we'll have to wait and see how the OP's latest tests turn out.

 

[mervincm]

In his scenario he indicates the hosts are directly connected with no physical switch in between.

Fyi, layer 2 switches don't fragment packets. They will simply drop them if they are bigger than the mtu. Routers (and layer 3 switches) fragment packets.

I'm guessing netwise hit the nail on the head in his post, although we'll have to wait and see how the OP's latest tests turn out.

True MS Windows SMB (3) is the best I have ever seen when it comes to making the most of a variety of possible links between 2 boxes.

In any case I agree completely with your jumbo frame comment. When things are working correctly there is not much benefit, and a large downside when things are not setup perfectly. My advice is to ignore them till you have everything else working as expected. once everthing is working you integrate then to get their advantages (higher payload/overhead per packet, and less total packets so less CPU /INT etc.