ZFS and SSDs

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
So I finally started to move over and play with ZFS. If I have a pool made of SSDs, do I need a ZIL on it? Likewise, I doubt a L2ARC would do much since it would be reading from SSDs anyway?

I have two pools, spinners for bulk and a mirrored pair of SSD for my virtualization stores over NFS. I think the ZIL for the spinners?
 

rubylaser

Active Member
Jan 4, 2013
842
229
43
Michigan, USA
I don't think an L2ARC is much benefit in this scenario unless it's an NVMe drive, but for ZIL, write latency is the other issue.

I have a small RAID10 array for my Proxmox server at home with (6) Intel 730's and I added a 100GB Intel S3700 for the ZIL. The S3700 has MUCH lower latency than the 730's do.
 

gea

Well-Known Member
Dec 31, 2010
2,520
852
113
DE
ZFS collects small random writes and after a few seconds all is written as a large sequential write. Such a write caching improves performance but on a powerloss some seconds of last writes are lost.

While this does not corrupt a CopyOnWrite filesystem like ZFS it may corrupt your ESXi, ext4 or ntfs filesystem on your virtualized environment. If you use databases or other transactional services they may got corrupted as well.

To prevent a dataloss you can use a hardware raid with cache and BBU on older filesystems. With ZFS you can enable sync write what means that you log every single write to a ZIL logdevice. A regular write is done after a few seconds as an async write. On a powerloss, last writes are restored on next boot from the logdevice. Without a dedicated Logdevice all writes are logged to your regular pool what means that every data must be written to the pool twice, once as a slow sync write, once as a fast large async write.

Unless you have ZIL grade enterprise SSDs in your pool, a dedicated ZIL is also helpful with SSD only pools as you can use cheaper SSDs for your pool and one enterprise write optimized SSD (ex ZeusRam or Intel S37x0) for performance and better reliabilty of your pool SSDs (less small random writes).

For your general use spindle pool, you do not need a ZIL as you usually do not use secure sync write there. Skip an L2Arc for more RAM and use it only if you cannot add RAM and an arc check shows a quite low arc hit rate.
 
Last edited:

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
Cool, then my setup is fine for basic use. I found some cheap pricing on Intel S3500 80GBs so used a mirror to put ZIL on front of my 6x6TB spinner pool. The SSD pool is a pair of PM853Ts so it probably doesn't matter one way or the other.

Thanks all!
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,966
4,931
113
Unless you have ZIL grade enterprise SSDs in your pool, a dedicated ZIL is also helpful with SSD only pools as you can use cheaper SSDs for your pool and one enterprise write optimized SSD (ex ZeusRam or Intel S37x0) for performance and better reliabilty of your pool SSDs (less small random writes).
Hey @gea - I also have used the Crucial p320h's and they are AWESOME (SLC PCIe not RAM but still super). Also Fusion-io SLC drives and Intel P3700 could be good options. Even the P3600 and Samsung XS1715's are pretty good. Low write endurance changes with a slight amount of additional OP.

@Chuckleb I think the other question is whether this is going to be a long-term NAS solution, or if you are just learning/ trying stuff out. If you are using it for learning, given your day job, I would actually say plan to implement ZIL and L2ARC to learn about it before you go bigger ZFS.

For folks reading this in the future - that was a bit of person-specific contextual advice. May not be the best idea for someone managing a maximum of 6 drives in a NAS.
 

Chuckleb

Moderator
Mar 5, 2013
1,017
331
83
Minnesota
@Patrick This setup is the home stable setup, migrating off my LSI and LVM setup. Now that I have played with this, we will fire up the big setups at work with my guys. They wanted to try ZFS backing Lustre to see if that would be good for us. This is akin to LLNL (
Lustre with ZFS Example) and see how to modify and make cheaper. Then we start the game of comparing filesystem yet again...