Help choosing m.2 NVMe enterprise drives for ESXi servers

[SRussell]

I am working on building a 4 node ESXi lab using Supermicro Xeon-D systems. After reviewing the nanoEDGE (Project nanoEDGE aka tiny vSphere/vSAN supported configurations!) guide I wanted to find enterprise endurance drives that are closer to a homelab budget. I have $1,000 allocated for this but I do not mind waiting and saving more for a configuration that will extend the useful life of the lab.

The nanoEDGE guide recommended Intel M.2 NVMe P4511 1TB for the capacity tier and Intel M.2 NVMe P4801 375GB for the cache tier. I have 4x 1.92TB Toshiba HK4 that I was considering using for the capacity tier. I don't have any idea what to use as a replacement for the cache tier that will fit my budget.

 

[Rand__]

That will probably depend on your chassis (space) and whether you want to stay on HCL.
They used the M2 drives to stay in the small form factor, but possibly that's not an issue for you...

 

[Evan]

With those systems and a suitable PCIe card you could get more capacity tier in place also if needed or wanted. Would be interesting to play again with this as the suggested performance surprised me, normally 2-node like that is pretty terrible but granted I have not used PCIe capacity tier in any past testing.

 

[SRussell]

With those systems and a suitable PCIe card you could get more capacity tier in place also if needed or wanted. Would be interesting to play again with this as the suggested performance surprised me, normally 2-node like that is pretty terrible but granted I have not used PCIe capacity tier in any past testing.

I should be good on the capacity tier if I use the 1.92TB drives. I am using the Mini Tower design which will leave me with 4x additional SATA slots if I need to expand the capacity tier.

The caching tier is where I am most uncertain. For a lab do I need more than 500GB? As the end user am I able to perceive the difference between 50,000IOPS and 150,000IOPS? At what point in random r/w can I visibly notice the performance difference?

 

[SRussell]

That will probably depend on your chassis (space) and whether you want to stay on HCL.
They used the M2 drives to stay in the small form factor, but possibly that's not an issue for you...

I would prefer to stay on the HCL. At some point in the future the Supermicro systems will move over to being a "production homelab" and I will use several NUCs for my dev/teardown homelab.

I am using the Mini Tower case. This gives me one PCI-e slot, one M.2, six SATA connections with five usable after accounting for the SATA-DOM.

 

[Evan]

For a small system unless your doing something that’s doing a lot of random IO i doubt at that level you will notice so much difference in performance.

it’s really hard to know the answer your question as it’s based on your workload, read/write ratio etc.

so back in the hybrid days there was a rule of thumb that cache sizing at ~10%
In an all flash environment (SAS3) I have tested large systems where 20% cache capacity was a good place to be, but the number of VM’s was large and the hot data also large.

I would think you should be fine as your thinking.

 

[SRussell]

@Evan @Rand__
Thanks for your insight and suggestions on this.

 

[Evan]

I am really interested to see you results. I have never managed to get that sort of performance from the small config but I have never looked at all NVMe setup. If it works like that it seems to be very viable way to run small setup.

 

[Evan]

Check eBay for Intel 4801x even at 200GB for interest... not cheap ! I think i will let you try first

 

[Rand__]

The caching tier is where I am most uncertain. For a lab do I need more than 500GB? As the end user am I able to perceive the difference between 50,000IOPS and 150,000IOPS? At what point in random r/w can I visibly notice the performance difference?

Background:
I've been running a VSAN cluster at home for years now, switching between 3/ 2+Witness and now 4 nodes.
I started with ssds for both layers, went to P3700(400gb) for cache, then Intel 750's for capacity, now optane (900p) for cache and P3600/P4510's for capacity. I also tried single vs multiple diskgroups and running 6 hosts instead of 3/4.

My personal opinion is that vsan scales like horsesh*t (for my need).
My personal need in this case is
-2-3 Users Max
-20 VMs max
-trying to reach 500 MB/s read/write speed to vsan (for backup purposes)

It fails miserable regardless of what HW I throw at it.

Now don't get me wrong, its working fine in principal, and o/c I am aware that caveats apply in my environment (especially partially non hcl hw and o/c totally uncommon use case (low parallelism));
but it is clearly designed for enterprise environments and will not scale at single user level.

I am fairly sure if I ran a appropriate tests even my environment would show that its capable of x-10k IOPS and whatnot, but in my day to day usage I don't see that. I am still on the quest to get it running to my desire, but at this time I don't think it can. Not even switching to optane 4800x's will boost performance as much as I'd need.

So (after a long rant) that brings us to the point - a faster slog will indeed give you faster speeds on single user but only to a certain extent. Keep in mind that all writes on All Flash Cluster go to the cache disk, so your max write iops is determined by that.
Now also keep in mind that only a small percentage of the theoretical Write capability is actually used for a single VM/User.

If we assume that a single User/VM gets maybe up to 10 or 20% of a drives capability then you can guess how much a 10% better rate of write IOPS will actually bring you - 1-2%.

Now of course it totally depends on your use case (vs mine). Also I could be talking sh*t and just being to plain dumb to run it correctly. Just don't get your hopes up if you see huge numbers being flung around.

 

[Evan]

@Rand__ exactly why I am interested to see...
No worries to run hundreds of VM’s on 10+ nodes with a few disk groups per node if you just want average performance per VM but I was surprised when I see those high iops for this small config, never been my experience that’s been possible but I have never gone to all NVMe and optane solution rather last testing I did was all SAS so figured maybe between NVMe and new VMware versions it’s changed for the better.

 

[Rand__]

The question is always "what do you need/what does your workload look like"...
As soon as you run parallel processes (ideally lots) the aggregated performance is great on NVME based systems (as they basically don't suffer from existing workloads up to a certain point).
The pdf (graph) shows 41k peak read iops/17k peak write IOPS - for what I assume are 25VMs which is o/c a great number, but leaves a meager ~1600 read/~700 write IOPS per VM

Now if vSan were a product that would utilize the same 41k read IOPS for 10 VMs - then we'd be talking - but my experience shows that it would probably not even use half of the total capability for those 10 VMs (might scale a bit beyond 1:1 if the test was cpu/latency bound).

Again my use case - I need 3 or 4 VMs utilizing 80% of the available capability and happy to leave the other 10 VMs with 20 left, and thats not a thing vSan can deliver.

Your mileage will vary