Although this subject is often discussed in favour of NVME across the web, after spending some time doing countless benchmarking (really not fun!) I've come up with a conclusion, would love to hear other opinions/findings on this.
a) NVME drives only perform well with dedicated CPU resources, their performance drops considerably in shared CPU environments, ie Virtualisation.
b) SAS SSD drives with a quality hardware raid controller have a read latency on par with NVME, although write latency is NVMEs' forte. However if we use write-back caching on hardware raid with DDR4 on controller, write latency is on par with NVME.
So, in a non-shared environment, ie. a dedicated NVME storage server/cluster, NVME truly shines.
However, in a shared environment, SAS SSDs on a hardware RAID controller perform just as well.
Now they say IOPs is where its at with NVME, but again, with around 400K IOPS on typical modern SAS SSDs, I really can't see that being a bottleneck in a general 10-20 VM environment.
I'm just putting this out there for debate really.
NVME is more CPU hungry than SAS SSD via RAID hardware, and NVME doesn't crunch out the advertised numbers in a heavily shared server, that is my conclusion thus far. Indeed NVME-oF kicks in here, and thus far with my tests in this arena with RDMA, very impressive performance! (Proxmox to Windows clients, SPDK target to Starwind NVMEoF initiators).
a) NVME drives only perform well with dedicated CPU resources, their performance drops considerably in shared CPU environments, ie Virtualisation.
b) SAS SSD drives with a quality hardware raid controller have a read latency on par with NVME, although write latency is NVMEs' forte. However if we use write-back caching on hardware raid with DDR4 on controller, write latency is on par with NVME.
So, in a non-shared environment, ie. a dedicated NVME storage server/cluster, NVME truly shines.
However, in a shared environment, SAS SSDs on a hardware RAID controller perform just as well.
Now they say IOPs is where its at with NVME, but again, with around 400K IOPS on typical modern SAS SSDs, I really can't see that being a bottleneck in a general 10-20 VM environment.
I'm just putting this out there for debate really.
NVME is more CPU hungry than SAS SSD via RAID hardware, and NVME doesn't crunch out the advertised numbers in a heavily shared server, that is my conclusion thus far. Indeed NVME-oF kicks in here, and thus far with my tests in this arena with RDMA, very impressive performance! (Proxmox to Windows clients, SPDK target to Starwind NVMEoF initiators).