TLC SSD useful in multi-tiered storage?

[billc.cn]

I have been experiementing with multiple ways of setting up a SAN for both VMs and media files. The hardware I have are:

• 8x 2TB HDDs (a mix of 5600 and 7200RPM ones)
• 3x 120GB TLC SSDs
• 2x 250GB TLC SSDs
  
• 2x 32GB Intel Optane memory on order

① My original setup was simple, I had one SSDs each in three ESXi hosts for VM boot disks and the HDDs are mounted on one VM and RAIDed in software.

Performance, as you can expect, is poor. The HDD RAID cannot handle movie playback and a file copy at the same time!! And the SSD write latency sometimes spikes up to several seconds (ESXi always do synchronous writes).

② I then experimented with using 2x250GB TLC SSDs (in RAID 1) as write-back caches for the HDDs (using LVM2/mdcache).

This improved write speed & IOPs a bit but is ultimately limited by the write buffer on the SSDs (slows down to <200MB/s after ~4GB continuous write). Random read IO is also extremely poor. So it seems I am gaining little from the cache SSDs and there will be no hope of using this as a SAN for VMs.

③ I am now looking at using Intel Optane memory as write cache (or maybe ZFS ZIL) instead. It should have better write endurance and IOPs. However, I wonder if it is useful to have the TLC SSDs in this array at all?

④ Would I be better off having an bulk storage SAN with a cheaper (non-Optane) MLC SSD and a separete TLC SSD RAID just for VMs?

 

[i386]

You didn't write what ssds you have used, but from the cpacities I would guess that they are all consumer drives like 850 evos or similar.

2) For write intesnive workloads such as caching I would use enterprise class ssds (like intels dc or samsungs sm/pm series). On specs they are "slower" than the consumer drives, but these values are for sustained workloads (and then they outperform consumer drives!).

3) The 32 gb soptane ssds have poor write performance, get the 900p instead (or the 4800x if you have too much money).

 

[billc.cn]

You didn't write what ssds you have used, but from the cpacities I would guess that they are all consumer drives like 850 evos or similar.

It's a mixture of 840EVO and MX500 drives. I have also tried a new 850Evo. The write performance drop was a bit less but I am not sure if this performance will last as the drive gets old.

3) The 32 gb soptane ssds have poor write performance, get the 900p instead (or the 4800x if you have too much money).

Thanks for the heads up about Optane write performance. I somehow failed to notice that in my research. Unfortuately, as far as caching goes, I cannot see any justification of using a larger SSD. The economics simply breaks down.

I will look into those enterprise SSDs though.

 

[LightKnight]

Instead of using the SSDs as as write-back caches using mdcache (you mean dm-cache?) you should use the 32GB Optane drives as a mirrored SLOG (ZFS ZIL). If this is a home setup they're perfect for the job. STH even published an article on the performance of them not too long ago. Your SSDs could be used for VM storage. You can even partition the 32GB Optane drives to be the SLOGs for both the HDD and SSD pools. And don't worry about the endurance, at writing 50GB/day the Optane will last 10 years.

 

[billc.cn]

I eventually decided to sell the 32GB Optanes I've got due to their limited write speed. (In RAID1, they will be slower than the original RAID array.) I will now wait for Optane 800P to hit the market. The roughly doubled write speed (640MB/s) should be sufficient for me for awhile.