Single disk group... How many nodes?well let me say, my (untuned except HW) ZFS filer (no RDMA) has better performance with SAS3 SSDs & SLOG vs my all NVMe vSan (single disk group) for my low user count workload... but to each their own
EXCELLENT!!4 nodes, P4800 as cache x, Intel 4510/P3600
I am using NFS at this time, 56GBit, no RDMA unfortunantly or perf would be significantly better
There is next to no tuning on FreeNas since the pool is fast enough - I loose most on Network (async) and slog (sync low QDs)
Thanks for all the feedback!
I tried vSAN with all SSD's backed with NVMe with the cache layer. Performance was amazing considering it was a 10G setup.
The IOPS were basically local storage speeds and out performed a regular setup with iSCSI and FC using ZFS.
iSER (RoCEv2) is supported in ESXI which was why I was siding towards a linux distro like OSNEXUS. I contacted their APAC contact and they say they no longer support RDMA as the need didn't justify the investment.
[ ... ]