The caching tier is where I am most uncertain. For a lab do I need more than 500GB? As the end user am I able to perceive the difference between 50,000IOPS and 150,000IOPS? At what point in random r/w can I visibly notice the performance difference?
Background:
I've been running a VSAN cluster at home for years now, switching between 3/ 2+Witness and now 4 nodes.
I started with ssds for both layers, went to P3700(400gb) for cache, then Intel 750's for capacity, now optane (900p) for cache and P3600/P4510's for capacity. I also tried single vs multiple diskgroups and running 6 hosts instead of 3/4.
My personal opinion is that vsan scales like horsesh*t (for my need).
My personal need in this case is
-2-3 Users Max
-20 VMs max
-trying to reach 500 MB/s read/write speed to vsan (for backup purposes)
It fails miserable regardless of what HW I throw at it.
Now don't get me wrong, its working fine in principal, and o/c I am aware that caveats apply in my environment (especially partially non hcl hw and o/c totally uncommon use case (low parallelism));
but it is clearly designed for enterprise environments and will not scale at single user level.
I am fairly sure if I ran a appropriate tests even my environment would show that its capable of x-10k IOPS and whatnot, but in my day to day usage I don't see that. I am still on the quest to get it running to my desire, but at this time I don't think it can. Not even switching to optane 4800x's will boost performance as much as I'd need.
So (after a long rant) that brings us to the point - a faster slog will indeed give you faster speeds on single user but only to a certain extent. Keep in mind that all writes on All Flash Cluster go to the cache disk, so your max write iops is determined by that.
Now also keep in mind that only a small percentage of the theoretical Write capability is actually used for a single VM/User.
If we assume that a single User/VM gets maybe up to 10 or 20% of a drives capability then you can guess how much a 10% better rate of write IOPS will actually bring you - 1-2%.
Now of course it totally depends on your use case (vs mine). Also I could be talking sh*t and just being to plain dumb to run it correctly. Just don't get your hopes up if you see huge numbers being flung around.