Low disk performance on ESXi 6.0/5.5/5.0 with H700 on Dell T7400

[Confusioner]

Hi,
I'm pulling my hair trying to figure out where the problem is. Here is the setup:

- Dell T7400 Workstation (Intel 5400 Chipset)
- 2 physical x Intel Xeon X5482 @ 3.2Ghz
- 64GB DDR2-5300F 667mhz ram (8x8GB)
- [Disabled] Onboard Perc 6/i (LSI 1068e SAS/SATA 3.0Gb)
- Dell Perc H700 Integrated, 6.0 Gb/s, 512MB, Has functional Battery, Forced Write back enabled, FW 12.10.3 (plugged to 16x PCIe 2.0) - Salvaged from a Dell T710 Server
- [Raid-0 on H700] 2 x Seagate Constellation ES 1 TB 7200 RPM SAS 2.0 6 GB/s (ST31000424SS) - Stripe size: 64KB
- 1 x Crucial M550 240GB SSD connected to onboard Sata AHCI @ 3.0Gb/s

The problem: If boot to a disk that has Win7x64 on it, I get disk speeds around 270MB/s from SAS Raid-0 to SSD or other way around. If I boot to ESXi 6.0/5.5/5.0 on the same machine without changing anything, I get disk speeds 100-110MB/s inside the VMs (ALL Thick disks, eager zeroed, Win2008R2 and Win2012R2). Copying between the datastores seems to have same speeds. I have tried various LSI drivers, flashed the H700 multiple times, Installed ESXi to different disks but nothing substantially changed. I played with Disk Share value on the VMs and increased to 4000 max, which resulted in extra 15MB/s increase in speed. I read numerous articles regarding disk queue depth, disk latency but the fact that Win7x64 gets the proper speeds pointing to non-hardware issue. Any help will be highly appreciated.

 

[Confusioner]

Last night I did more test and it might shed some light on the performance issue:

 

[Terry Kennedy]

- Dell Perc H700 Integrated, 6.0 Gb/s, 512MB, Has functional Battery, Forced Write back enabled, FW 12.10.3 (plugged to 16x PCIe 2.0) - Salvaged from a Dell T710 Server

Try upgrading to the latest H700 firmware (12.10.7). 12.10.3 is quite old. 12.10.5 is bad (was pulled due to regressions in previously-fixed issues). 12.10.6 is good. The only documented change in 12.10.7 is to handle insanely large arrays. But it's worth being at the latest revision anyway.

I don't know if this holds true for other operating systems, but on FreeBSD the volumes created by the controller do not have I/O caching enabled and it has to be specifically enabled by a utility:

(0:173) gate:~terry# mfiutil cache 0
mfi0 volume mfid0 cache settings:
  I/O caching: writes and reads
  write caching: write-back
write cache with bad BBU: disabled
  read ahead: adaptive
  drive write cache: default

That first line, "I/O caching", defaults to "disabled" on newly-created volumes.

 

[DavidRa]

I'm going to sound like I'm trolling, but you might have hit one of the configurations where ESX performance is sub-par. It's happened to me before - I get pretty good performance from my 2 node C6100 vSAN, but I've had VMware HCL systems from a tier 1 vendor which performed abysmally (5 minutes to boot an Exchange VM on a 6 disk RAID 10!?) Windows is maligned, but it isn't really known for terrible disk performance.

Have you tried using the Dell-specific ESX images rather than the generic ones? It may have different drivers or configurations. Also, you may want to look into partition alignment (Windows 2008 and later does this OK, though there are still opportunities for tweaks if you use VHD instead of VHDX, for example) - from memory VMFS partitions are not aligned by default? Try to align everything to 1MB boundaries and see if it changes.