ESXi 6.0 rather slow driver performance

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

weust

Active Member
Aug 15, 2014
353
44
28
44
I run a ESXi 6.0 setup on a Supermicro A1SRi-2758F using a USB key to boot ESXi from, a 240GB SSD as a datastore, and a 480GB SSD as a RDM on the Datastore for a specific VM.

I notice that when I copy something from within the VM with has the RDM drive mounted (FreeBSD installation) to my NAS, I get a maximum of 43MByte/s throughput.
When I copy something from my desktop to the NAS I reach 100 to 105 MByte/s.
Both the ESXi and the desktop are connected to the same switch.

I have a vSwitch with two NICs for VM LAN usage (got a seperate just for management) and I already tried removing one NIC, and setting the physicial switch ports to normal.
Meaning no LACP or Static port channel. Can't get that to work properly anyway.

Since I can't copy directly to the RDM drive, I copied something to the Datastore from my desktop. I get a max of 70MByte/s there using the same file from the same hard drive as I copy to the NAS.

What could I be missing here? Why is my performance so low?
 

Rhinox

Member
May 27, 2013
144
26
18
What drive-controller do you have in your ESXi-server? I hope you do not use sata-ports on your motherboard because if you do, then there is your problem: ESXi does NOT do any disk-caching. That's why disk-performance is low. I'm actually surprised you get that high transfer-speed. Probably thanks to SSD, otherwise it would be much lower...
 

yaxisandxaxis

Member
Jan 26, 2015
46
10
8
51
These days with SSDs no issue using onboard SATA.

Now normal performance on a busy system with vm exits is another story.
 

weust

Active Member
Aug 15, 2014
353
44
28
44
I am using the onboard SATA III ports for the SSD's.
ESXi boots from a USB key.

One VM is my OPNsense firewall/router, the other a FreeBSD with the RDM as the second drive.
Barely any I/O for most of the time.

I will look for a controller. Hope it can still fit in.
 

weust

Active Member
Aug 15, 2014
353
44
28
44
These days with SSDs no issue using onboard SATA.

Now normal performance on a busy system with vm exits is another story.
No issue? 40MByte/s or 550MByte/s is quite an issue imo.

It's just weird there is no setting, that I know of, to enable caching if using a USB key to boot ESXi from.

What I do wonder is why is read performance so low? That's not cached the way writing is.
 

weust

Active Member
Aug 15, 2014
353
44
28
44
I ordered a LSI 9212-4i controller of ebay.
Hope it comes with the four cables, and not just one though.
Might ask the seller that just to be sure.
 

weust

Active Member
Aug 15, 2014
353
44
28
44
Card came in earlier this week and got it installed.
Cables are useless to me as they don't have SATA connectors on the drive side...

RDM failed somehow. Couldn't get it back, so with too much wasted time and pain I got the data off to the NAS, and created a datastore from the drive, and added a full size vmdk for the VM on it.
It's copying data back at only 37.3 MByte/s. Feels too slow.

Will try copying data from the SSD to the NAS later. Hope it's better then the 43 MByte/s.
But VMWare in the Infrastructure Client says "Unknown" for both drives on "Hardware Acceleration".
 

weust

Active Member
Aug 15, 2014
353
44
28
44
And the speed went up to a max of 44.1 MByte/s.
I don't get it. Why such a slow read speed?
Can't find anything usefull online about it as well...
 

NetWise

Active Member
Jun 29, 2012
596
133
43
Edmonton, AB, Canada
So some things to note and questions:

* ONLY The A1SA1 is listed on VMware's HCL, including ALL makes of C2700 CPU, and that's the only SuperMicro - and it's only certified to v5.5 - VMware Compatibility Guide: System Search Remember that vendors certify and test, not VMware
* Onboard SATA has never been 'supported' by VMware.
* Are they in SATA/AHCI/RAID mode or anything?
* Have you installed VMware Tools on the VM?
* What vDisk and vNIC are you using?
* "Hardware Accelleration" is referring to VAAI, and that's going to be an array thing - it will not show anything for local disk. "Unknown" is fully expected.
* What do copies look like coming from the other VM on the other SSD?
* Can you try one of the SATA2 vs SATA3 ports, just for troubleshooting?
* What is your copy speed to and from the same VM to itself? If that ALSO is not getting better than 50MB/sec, then it's not network etc.
 

weust

Active Member
Aug 15, 2014
353
44
28
44
- Not entirely sure on the mode. Can't seem to find that. Controller wise they are two seperate SATA devices. ESX shows them as SAS in esxcfg-mpath.
- VMWare Tools are installed and running inside the VM
- What do you mean "what vDISK and vNIC"?
- Copying inside the VM between SSD's (OS is on the other datastore) is fine. 250 MByte/s. So it's a network thing.
The other VM is an OPNsense installation, btw. Not using that for testing. Will build a seperate VM soon.
- I'm almost inclined on testing the onboard controller for it's speed now, but switching everything over again........sigh

So, I can conclude it's a network thing. Feels like 110+20 USD plus 40 Euro (custom crap) wasted of money, but, oh well..
Now to figure out what is wrong with my network setup.

The switch ports are basic. The VSwitch has two NIC's setup in "Route based on originating virtual port ID" with both adapters as Active Adapters. This is for both the vSwitch and the LAN Virtual Machine Port Group.
I tried setting is to "Route based on IP "hash" and putting one adapter in Standby Adapters, but that made no difference.
 

weust

Active Member
Aug 15, 2014
353
44
28
44
Problem seems to be FreeBSD. Not sure where yet, will try to figure that out later.

I created a VM and installed Windows Server 2012 R2 in it.
Did a simple test of copying a large file from the NAS to the drive, and back to the NAS.
From the NAS at around 110 MByte/s and to at about 105 or so MByte/s.
That is consistent with my physical desktop performance.

Was hoping to stay away from Linux, but will give that a go as well.
Wonder how performance is there now.

That FreeBSD VM was the first after the OPNsense VM.
Stupid of me not to test any further before buying the extra controller :-(
 

weust

Active Member
Aug 15, 2014
353
44
28
44
Performance testing is something I hadn't done while running Hyper-V on the server.
Problem I have since March, and not me alone, is that x.org doesn't want to run anymore on Hyper-V.
This is with a manual installation. The forks like GhostBSD and PC-BSD work fine...

I was really hoping to stay away from Linux, since I utterly hate it, but will test it to see for myself what the performance is.
 

weust

Active Member
Aug 15, 2014
353
44
28
44
Because they all install loads of stuff I don't want.
And they all do not have a customuzation option before installing everything.

I just want a lean and clean desktop. Lumina for example, with just a browser, Double Commander and a torrent client.
So, I do not need several multimedia packages, LibreOffice, etc.

But, I strongly thinking of doing it anyway, and then spend a lot of time removing all that...
 

weust

Active Member
Aug 15, 2014
353
44
28
44
I don't care too much what they stand for, but not against it either.

Today I gave Arch Linux a shot on Hyper-V on Windows 8.1.
That actually went very nice. Even got screen resolution to a useful setting.