SSD mirrored pool sync write setting

[RyC]

Currently I'm using nappit/omnios with mirrored hard drives to serve NFS storage to ESXi. I'm looking to replace the mirrored hard drives with mirrored Intel DC S3710's. I'm not using any SLOG or SSDs currently, and so I have sync=disabled to make speeds tolerable (I know the risks, this isn't critical storage).

My question is will having sync=default with the Intel SSDs (which is effectively sync=always with NFS on ESXi) noticeably reduce performance? I know S3710/S3700's are used as SLOG devices regularly, but my searching always pulls posts about using them as SLOG devices, not standalone mirrors with no SLOG at all. Any advice appreciated, thank you!

 

[ttabbal]

My guess is that even with sync=always they will outperform HDDs with sync=disabled.

There's always a ZIL. In the configuration you are talking about, it would live on the mirror. If a single drive can be a good SLOG, a mirror shouldn't have trouble without an SLOG.

 

[Rand__]

I'd say it will depend on the numbers - of drives you are replacing and of ssds you are going to use.
For a pair each - yes the ssds will be faster. For 24 drives vs 2 ssds - no

O/c you could just add them as log device, then all those threads would yield great info

 

[RyC]

I'm just going from a mirrored pair of hard drives to a mirrored pair of SSDs. From what I've read, the ZIL resides on the pool drives if you don't have a SLOG, resulting in 4 writes per write to the mirrored pool instead of 2.

My googling fails again, going off this thread on the FreeNAS forum, it looks like a SLOG is recommended even on an all SSD pool if you want higher performance with sync=always (which again is effectively what NFS w/ESXi requests): sync=always is always slow?

 

[Rand__]

Quote from that thread: "While an SLOG does help sync write throughput, a commodity SSD is not sufficient if you want to see the full speed of your disks. Interestingly, having an SSD-based SLOG on an SSD-based zpool does improve performance considerably. Even with a RAM-based SLOG, you still won't reach the speeds of sync=default, but it'll be pretty close."
The SSDs where 850 pros which are not comparable in terms of SLOG capability to the 3710

You will benefit with ssd or hdd pool if the slog is faster in sync writes than your pool, eg
I don't think you'd benefit from a 3710 slog if you have a 3710 pool, but having a 3710 pool you'd benenfit from a p3700 as slog

Edited to show the quote

 

[T_Minus]

Not really "interesting" about why that is.

Most people look at the SLOG as a higher performing device than your pools drives and in turn moving your ZIL off your pool to a higher performing SLOG device increases performance. This is true, but it's only a piece of the performance increase! You will get a boost in performance of the pool without the ZIL present simply by freeing up the IOs utilized by the ZIL, and then coupled with the increased performance of the SLOG device your entire pool now performs better from benefit from both increased write performance and higher performing SLOG device.

With that said, mirrored drives with a SLOG is still not going to perform very well.
It will perform > than mirrored HDD though

 

[dswartz]

Note also that even having an extremely high performance SLOG device, your sync write performance will still be much worse than sync=disabled. I've seen this with a number of SLOG devices, including even a ZEUSRAM. Not bad performance, but not nearly as good as async mode. There seems to be some kind of throttle in operation. And it's not related to the platform (e.g. I can demonstrate this on solaris-type as well as ZoL). I even tried creating an 8GB ram disk and using that as an SLOG and saw this in action.

 

[RyC]

I finally got the pool set up and I tried running benchmarks myself like I should have done

Running ATTO benchmark on Windows 10 with storage backed by the mirrored S3710's running over 10gb through a US-16-XG. Also using napp-it/OmniOS.

sync=disabled


sync=always run 1 and 2



There's definitely a slowdown at the lower chunk sizes, but I'm surprised the higher chunk speeds are close enough. The benchmark with sync=always looks fast enough to me so I'll stick with sync=always, but if I'm misinterpreting the results, someone please let me know

 

[J-san]

Running ATTO benchmark on Windows 10 with storage backed by the mirrored S3710's running over 10gb through a US-16-XG. Also using napp-it/OmniOS.

<snip>

There's definitely a slowdown at the lower chunk sizes, but I'm surprised the higher chunk speeds are close enough.

RyC, Is your storage shared via NFS?
What size S3710 are you running? 400GB?
Did you put the S3710 device in the sd.conf of OmniOS as cache-nonvolatile:true so it doesn't wait for sync write confirmation?

Could you run a Crystal Disk Mark test on it for comparison?
Please select 2 tests @ 1000MB..

Thanks in advance!

 

[gea]

Slog is a solution that was invented in a time (more than 10 years ago) when disks were very slow. Using a separate but much faster extra device for logging was a method to increase secure sync write performance massively (factor 2-10).

To decide if an Slog makes sense, you must compare pool iops=write performance for small random writes and latency with the combined performance of a whole pool with the single Slog. With any disk based setup it is clear that a good Slog like a drambased ZeusRAM always improves sync write performance.

With SSDs this is different. A pool from some SSDs is faster than a single ZeusRAM. If you add a ZeusRAM to an average current SSD only pool, your sync write performance will go down in most cases. You need a ultrafast NVMe like the Intel Optane to improve sync performance as this one is much faster than an average SSD. So effectively an Slog for a fast SSD pool does not make sense unless you want it because your SSDs are not powerloss save.

Without an Slog, all sync writes are logged to the onpool ZIL with the combined performance of all disks. If your SSDs are capable for an Slog for hd disks (low latency, high write iops, powerloss protection) use them without Slog.

If you buy new, use NVMe with Optane as the current champion as you must not care about garbage collection or trim and latency is much lower than on any other. But in general as a rule of thumb expect 4 fast Sata SSD=2 conventional NVMe= 1 Optane perfomancewise (unless low latency is the deciding factor where pure Optane is unbeaten)

 

[J-san]

Thanks Gea, I've got a few 960 GB Intel DC S4600 and one 480 GB on order so I'll do some benchmarks.