vsan 6.6? 7.0u3g? etc is so slow it doesnt matter if you use all nvme optane or not. you max out at around 2GBs per host span where the vms are in a "raid1", regardless if you run say a fast SAS SSD as cache/capacity or all p4800x for both and have single 100g or lacp bond. it does however scale, so multiple 2GBs available whenever you double the nodes, which is apparant when you change a vsan policy and see the usage on 100g links, and also in vsan reporting, I actually got it to pass 10GBs across the cluster in a 12 node stretch (6+6) the other day when changeing policy.
stretch vsan also takes all the "fun/good" stuff like rdma away
looking forward to testing v8, but all nvme required for it to run properly, and currently have a lot of enterprice sas since most current gen hw doesnt have enough nvme slots, and u.3 stuff has been delayed, honestly u.3 nvme through a controller has also not been that impressive, but with direct lanes and genoa it will be a game changer.
I have tried to order gen11 hpe no luck.. heck they are still delivering us gen10plus and gen10plus v2 (amd) we orderd last year....
I do have supermicro with deliver on genoa this month.. we will see if thatis true
few days left..
stretch vsan also takes all the "fun/good" stuff like rdma away
when running ZFS on SAS/nvme and iscsi presenting r to vmware, most of the time ZFS is to slow and it eats cpu/ram. there are no winning superfast free/cheap good solutions. ironicly even ZFS will be much faster than most hardware raid solutions on a host directly theese days, as we are only know seeing some sas24/tri mode hwraid kontrollers. even a "cheap" enterprise ssd is 200k iops at 4k "all day" the kontrollers and fs etc... not so much.