So i know you always want to use mirrors to get the most IOPS out of a pool, so a pool composed of mutliple mirrored vdevs (RAIDZ10 if you will) has always been the norm for any datastore using spinning drives. However with the advent of affordable 2TB SSDs with TEN YEAR warranties i am deploying all SSD datastores for our VDI hosts. I cry a little bit inside knowing half the SSD space is going towards redundancy. Besides resilver times, is there any performance reason not to use raidz2 in SSD datastores?
vdev layout for all SSD datastores (striped mirrors or RAIDZ2?)
- Thread starter chune
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
- 10YR warranty has nothing to do with performance of those SSD
- The reasons behind using a pool of mirrored vdevs with spinners doesn't change when you go to SSD
- The prosumer drives you're considering likely won't perform up to what you think in a multi-use / VM Guest OS environment based on their "rated" specifications from manufacturer. I urge 25% over-provision.
- The reasons behind using a pool of mirrored vdevs with spinners doesn't change when you go to SSD
- The prosumer drives you're considering likely won't perform up to what you think in a multi-use / VM Guest OS environment based on their "rated" specifications from manufacturer. I urge 25% over-provision.
With spindels, this is quite clear.
Iops scale with number of vdevs. While a single disk has about 100 iops, a raid-10 has 200 iops on write and 400 on read. With a massive raid-10 of 10 vdevs you can get 1000 iops on write and 2000 iops on read (as ZFS reads from both disks of a mirror).
With SSD this is different. A cheaper SSD can give you a few thousand iops even under load, enterprise SSDs a few 10000 and NVMe 100000 and more. This changes the game especially as good SSD are expensive. While iops scale as well, even a single raid-z can give you enough iops.
The real problems.
As there is usually no trim support in a raid, you need SSDs with a high overprovisioning and a good firmware or performance will drop after a short time of writing. Even with trim support this will lower your performance under load. This means enterprise SSDs if you want a constant performance
As you always have a background data reordering/ garbage collection, you should use SSDs with a powerloss protection in a production environment.
Your overall experience with SSDs may depend on SSD quality.
Iops scale with number of vdevs. While a single disk has about 100 iops, a raid-10 has 200 iops on write and 400 on read. With a massive raid-10 of 10 vdevs you can get 1000 iops on write and 2000 iops on read (as ZFS reads from both disks of a mirror).
With SSD this is different. A cheaper SSD can give you a few thousand iops even under load, enterprise SSDs a few 10000 and NVMe 100000 and more. This changes the game especially as good SSD are expensive. While iops scale as well, even a single raid-z can give you enough iops.
The real problems.
As there is usually no trim support in a raid, you need SSDs with a high overprovisioning and a good firmware or performance will drop after a short time of writing. Even with trim support this will lower your performance under load. This means enterprise SSDs if you want a constant performance
As you always have a background data reordering/ garbage collection, you should use SSDs with a powerloss protection in a production environment.
Your overall experience with SSDs may depend on SSD quality.
Thanks for the informed reply gea! Is the lack of trim support only an issue on raidz pools? Or will this still be an issue in a mirrored vdev situation? I am currently using 4x 2TB samsung 850 pro SSDs right now in a mirrored vdev layout. If i'm getting the drives for $750 each, do you have any recommendations in a similar price point for enterprise SSDs? I am limited to 8 bays on these servers
The Trim problem is independent from the raid type but the internal garbage collection is quite ok with a larger overprovisiong. Enterprise disks have an internal overprovisioning of up to 40%.
You can manually overprovision new or secure erased SSDs with an HPA or with partitioning to reduce the gap between dekstop SSDs like the Samsung Pro and real enterprise SSDs. There are also some like the Sandisk Pro extreme with a higher overprovisioning per default than the Samsung. I would simply try the behaviour under load. Propably it is already much better than good enough and faster than spindels in any case regarding iops.
For real critical workloads I prefer real enterprise SSDs. While there are also offerings from Samsung my favourite is the Intel S3610 for write heavy workloads and the S3500 with a lower write performance all with powerloss protection but they are more expensive than a Samsung Pro.
I have done some benchmarks with spindle vs SSD vs NVMe. The real future of disks is NVMe.
While the difference is quite small with a single user large file sequential workload with many small files the difference is huge.
see http://napp-it.org/doc/downloads/performance_smb2.pdf
You can manually overprovision new or secure erased SSDs with an HPA or with partitioning to reduce the gap between dekstop SSDs like the Samsung Pro and real enterprise SSDs. There are also some like the Sandisk Pro extreme with a higher overprovisioning per default than the Samsung. I would simply try the behaviour under load. Propably it is already much better than good enough and faster than spindels in any case regarding iops.
For real critical workloads I prefer real enterprise SSDs. While there are also offerings from Samsung my favourite is the Intel S3610 for write heavy workloads and the S3500 with a lower write performance all with powerloss protection but they are more expensive than a Samsung Pro.
I have done some benchmarks with spindle vs SSD vs NVMe. The real future of disks is NVMe.
While the difference is quite small with a single user large file sequential workload with many small files the difference is huge.
see http://napp-it.org/doc/downloads/performance_smb2.pdf