Is an SSD not enough for an ESXi datastore?

[TeeJayHoward]

Setup:
ESXi node, connected via ConnectX-2 to Windows Server 2012 R2 NAS through a Mellanox unmanaged switch, which runs Services for NFS to share out a single Samsung 850 Pro as a datastore. NAS also runs OpenSM and vCenter Server.

Problem:
100% disk active time at random times. Queue depth of 1 during full utilization, as measured by Resource Manager on the NAS. Active time hovers between 1% and 2% when not being hit. I've got a dozen VMs running off this one SSD. They all lock up until whichever one is using the disk completes its task. If I'm downloading something via yum at 45KB/s on a 5MB/s connection to this SSD which should be able to write at around 400MB/s, the disk is useless to the other VMs until the download finishes.

Goal:
I want all these VMs to share the disk nicely. No one VM should be monopolizing it.

So... How? I honestly don't even know what troubleshooting steps to take. Where are the logs I should be looking at? On the host? On the NAS? What are they called? Heeeeeeelp!

 

[T_Minus]

Problems:

- ConnectX2: You didn't specify but 10GigE? If so that latency will be noticeable and compounded by # of VMs utilizing it, that's less than 1Gig-port per VM.
- Single SSD & 12 VM
- Non-Enterprise SSD not enough OP for the workload, disk is not rated for steady-state you're giving it and can't catch-up
- Latency will be much more noticeable with that many VMs
- Mixed work-load above what SSD is rated to handle. Absolutely none of the "specs" other than mixed-work load really mean anything to you due to how you're using it and what it was designed/made to handle.

I've seen people use 850 Pros in business settings because 'ratings' as basic file systems for small offices if more than 1 person is moving files or accessing/manipulating images/video things slooooooo and lock up. The people I know are using it the same way you are in a Windows NAS / File Server setting, single drive and when it works it works but they also say it will slow/stutter/freeze and then work again.

 

[TeeJayHoward]

ConnectX2: ESXi shows it as 40Gbps, and it's connected to an infiniband switch... But it's got an IP address, and IIRC ConnectX2 doesn't support 40GbE, just 10Gbe.
Non-Enterprise SSD: Could you elaborate on this? What features do Enterprise SSDs have that non-enterprise SSDs don't?
Mixed-work load: I don't suppose you know of a place I can look up this spec for an 850 Pro, and compare it to a nice high-end SSD, do you?
Single SSD: I could switch my datastore from the single SSD to my 24-disk Storage Spaces setup if it would help. I figured that the SSD would have more IOPS and lower latency than the old spinning disks.

There are actually multiple hosts, I just simplified the setup. Right now, 3 are spun up, with 5 sitting in standby should DPM decide that more resources are needed. The 12VMs are distributed across those 3 hosts. Hopefully the network isn't an issue.

 

[DavidRa]

What happens if you attempt IO on the Windows host at that time? It could well be NFS stuffing up - MS is not known for compatibility in that space

 

[TeeJayHoward]

What happens if you attempt IO on the Windows host at that time? It could well be NFS stuffing up - MS is not known for compatibility in that space

The Windows host doesn't seem to have any issues. (The SSD used for the datastore is only used for the datastore - It's not shared with the host for OS or anything.)

 

[vBuild2BAU]

Setup:
ESXi node, connected via ConnectX-2 to Windows Server 2012 R2 NAS through a Mellanox unmanaged switch, which runs Services for NFS to share out a single Samsung 850 Pro as a datastore. NAS also runs OpenSM and vCenter Server.

Problem:
100% disk active time at random times. Queue depth of 1 during full utilization, as measured by Resource Manager on the NAS. Active time hovers between 1% and 2% when not being hit. I've got a dozen VMs running off this one SSD. They all lock up until whichever one is using the disk completes its task. If I'm downloading something via yum at 45KB/s on a 5MB/s connection to this SSD which should be able to write at around 400MB/s, the disk is useless to the other VMs until the download finishes.

Goal:
I want all these VMs to share the disk nicely. No one VM should be monopolizing it.

So... How? I honestly don't even know what troubleshooting steps to take. Where are the logs I should be looking at? On the host? On the NAS? What are they called? Heeeeeeelp!

So 2 things.

1) Take a look at this article : VAAI and the Unlimited VMs per Datastore Urban Myth » boche.net – VMware vEvangelist

This part stood out to me per your problem:
"Queue Depth. There’s only so much active I/O to go around, per LUN, per host, at any given moment in time. When multiple VMs on the same host reside on the same LUN, they must share the queue depth of that LUN. Queue depth is defined in many places along the path of an I/O and at each point, it specifies how many I/Os per LUN per host can be “active” in terms of being handled and processed (decreases latency) as opposed to being queued or buffered (increases latency). Outside of an environment utilizing SIOC, the queue depth that each virtual machine on a given LUN per host must share is 32 as defined by the default vSphere DSNRO value. What this effectively means is that all virtual machines on a host sharing the same datastore must share a pool of 32 active I/Os for that datastore."

2) You can try enabling SIOC : Storage I/O Fairness

 

[vBuild2BAU]

Setup:
ESXi node, connected via ConnectX-2 to Windows Server 2012 R2 NAS through a Mellanox unmanaged switch, which runs Services for NFS to share out a single Samsung 850 Pro as a datastore. NAS also runs OpenSM and vCenter Server.

Problem:
100% disk active time at random times. Queue depth of 1 during full utilization, as measured by Resource Manager on the NAS. Active time hovers between 1% and 2% when not being hit. I've got a dozen VMs running off this one SSD. They all lock up until whichever one is using the disk completes its task. If I'm downloading something via yum at 45KB/s on a 5MB/s connection to this SSD which should be able to write at around 400MB/s, the disk is useless to the other VMs until the download finishes.

Goal:
I want all these VMs to share the disk nicely. No one VM should be monopolizing it.

So... How? I honestly don't even know what troubleshooting steps to take. Where are the logs I should be looking at? On the host? On the NAS? What are they called? Heeeeeeelp!

Also :
Warning: Windows Services for UNIX is not a supported storage solution for use with ESX, and the information in this article is provided as-is. For a full list of supported storage solutions, refer to our Hardware Compatibility List. VMware provides support only for the devices that are listed in the Hardware Compatibility List. If you are having a technical issue with 3rd party hardware or software and it is not found on this list, see our 3rd Party Hardware and Software support policy.

Reference: https://kb.vmware.com/selfservice/m...nguage=en_US&cmd=displayKC&externalId=1004490

 

[DavidRa]

The Windows host doesn't seem to have any issues. (The SSD used for the datastore is only used for the datastore - It's not shared with the host for OS or anything.)

Yeah I understand that - but is the Windows host able to read/write the SSD during a hang? If so that would point to NFS, network stacks or ESX, but if Windows also hangs I'd be looking at SSD health, Windows drivers or perhaps something weird happening with TRIM.

 

[Deslok]

Why did you chose NFS over ISCSI? What are the dozen vm's doing? I have a dozen at home on a tiered(server 2012 R2 storage spaces) ssd+hdd combination but only one of them has any kind of high disk usage(minecraft when people are playing) the most are fairly idle webservers and a smaller monitoring db(zabbix I use it at work and home) I couldn't run a dozen heavy databases though another dozen smaller webservers wouldnt be an issue since they don't write to disk often(I'd run out of memory first loading up LAMP instances)

 

[TeeJayHoward]

VAAI and the Unlimited VMs per Datastore Urban Myth » boche.net – VMware vEvangelist

Awesome article. I'm going to need to read it a couple times for it to sink in fully. I haven't even met the 10-15-20 rule yet on my datastore, but there's a heck of a lot of good info in there. Thanks!

You can try enabling SIOC : Storage I/O Fairness

I can't believe I didn't know about this. Enabled @ 90%. Thanks!

Yeah I understand that - but is the Windows host able to read/write the SSD during a hang?

I'll try it next time it hangs up. I was trying to avoid touching that SSD during the hangups in a hope that it could get over the hiccup.

Why did you chose NFS over ISCSI? What are the dozen vm's doing?

I had issues with iSCSI disconnecting repeatedly. It was unreliable. I had to install NFS for some VM or another back when, so I used it for the datastore as well. The VMs are as follows:
ark gaming VM (windows, one to four people on depending on day)
private gaming VM (linux based, nobody ever really on)
minecraft (linux based, one or two people on)
nagios (linux, monitoring the hardware only at the moment)
home cinema server (LAMP stack pointing to another NFS share)
reverse proxy (linux, apache)
wiki (LAMP)
workstation VM (Win7 for RDP, almost never used)
workstation VM (Win81 for RDP, almost always in use)
another private gaming VM (linux based, the one that chews up the datastore - one database, four tables. Nothing complex. Just me on it.)
main web site (LAMP)

I also had a Linux workstation VM, but I took that down today. I'll probably build another one soon.

 

[Deslok]

It concerns me that iscsi was disconnecting I've never seen that behavior before unless there were problems with the switching equipment