ZFS Questions

[T_Minus]

Please correct / add input to the following. I'm trying to make sure I have all my hosts built-out with parts, and want to make sure 100% I understand my devices needed for L2ARC and SLOG.





ARC Configuration
- Provide as much RAM for ARC as possible for best performance
- You can't designate ZFS ARC max usage per-pool.
- You can designate ZFS ARC max usage but it was deprecated in 2014, you can also now configure % to utilize so some is reserved for other OS duties.
- (If using) The value for vfs.zfs.arc_max needs to be smaller than the value for vm.kmem_size (not only ZFS is using the kmem).

L2ARC Configuration
- L2ARC drive suggestion are MLC SSD, low latency drives.
- Using a L2ARC device will increase the amount of memory ZFS needs to allocate

Can you share L2ARC between pools?
No.

SLOG Configuration (Sometimes called ZIL)
- You can have a SLOG or slog-pool per-drive pool in ZFS.
- Best performance is most likely gained from a SLOG POOL of 2 devices (RAM or SLC SSD).
- Sized 2-8gb seems to be very common
-- This could be important if you wanted to use SSD for Spinning SLOG Pool and RAM for SSD SLOG Pool.
- Larger SSD while not needed often have much higher performance and can be utilized for LARGE over-provisioning, and maximum performance, although they cost more.

 

[wildchild]

sounds about right to me

 

[Entz]

Yeah looks good.
Few notes:

Can you share L2ARC between pools?
You can share the drive, just not the contents. To do this you partition the drive and give each pool a separate partition.

SLOG
You only need a SLOG if you are performing sync writes to your pool. Async writes will bypass the SLOG. Depending on your source it can be advantageous (or required) to set sync=always to ensure writes are properly committed in the event of a power loss or other system issue.

ZIL
There is always a ZIL or rather 2 of them. One on disk and one in memory. All writes to storage come from the in memory one unless a failure occurs , then the transactions are replayed from the on disk one. Because these are sync writes pushing them to your pool is generally extremely slow, so it is advisable to have them written to a SLOG (Fast SSD).

 

[T_Minus]

That's good to know about the L2ARC between pools, I wonder how bad of a performance hit it would take or if it would be dependent upon usage at the time or of it's always 50%/%0 or if there would be SERIOUS wait depending on the jobs going on.

 

[Entz]

I am sure it would depend on both the drive and the usage. SSDs can access random information very fast so I would not expect a massive drop in performance. Worst case would likely be 1/2. Haven't really benchmark it to be honest as I have no where enough ram to even setup a L2ARC

 

[T_Minus]

"The writes of iSCSI targe to disk are syncronous per default. Most write operations of databases are synchronous. Metadata updates are synchronous. So in this cases an SSD can really help you. Or to be more exact: Whenever a seperated ZIL has a lower latency than the primary storage, a ZIL media is a good idea. So for local SAS,FC or SATA storage an SSD is a good idea, for remote storage (like iSCSI over WAN) one or two rotating rust disk can do similar things."

I thought that was interesting from:
Insights into ZFS - today: The nature of writing things - c0t0d0s0.org

 

[T_Minus]

Technically if you had a busy pool used for DB you could add a SSD SLOG POOL to alleviate some of the writes to free up the read IOPs.
But, in most situations it's just easier to use different systems for read / write operations

 

[Entz]

Depends on your iSCSI target. LIO ( and I believe Comstar or whatever FreeNAS is using) write to the underlying disks using whatever sync flag is passed through. For example ESXi datastore writes seem to always be async (may depend on the virtual OS). You can force these as sync at the zfs level though (recommended practice). While on the other hand NFS writes from ESXi are always sync.

I updated my ZIL description as it didn't make as much sense and it should of (end of day )

 

[T_Minus]

Even more great info Entz. Thanks

 

[capn_pineapple]

The only time you really need an L2ARC is if you have a massive spindle array and want to increase reads by having a larger cache than you can provide with RAM alone. From memory, L2ARC won't increase the amount of RAM you require, rather ZFS will use all the RAM it can (minus 1GB for system), with L2ARC providing additional disk based cache. e.g. If you have a system with 64GB RAM, ZFS will access 63GB. If you then have say 500GB of L2ARC on SSD, ZFS will use a total of 563GB as ARC/L2ARC. This means that ZFS can cache more reads, NOT that it needs more RAM.

• http://www.zfsbuild . com/2010/04/15/explanation-of-arc-and-l2arc/ (random bug? remove spaces))
  
• ZFS - Wikipedia, the free encyclopedia
  
• ZFS L2ARC (Brendan Gregg)

As to devices, you want low latency, high endurance, low capacity drives for ZIL/SLOG (mirrored if desperate)
L2ARC, you want high IOPS. Even with a single SSD L2ARC you should saturate a 1GBe link. Adding additional L2ARC just increases capacity, you wouldn't mirror L2ARC drives as the data is already on the larger pools.

You're not wrong in your understanding, but instead of thinking you need more RAM to address a bigger L2ARC, think of the two cache levels as CPUL1 (usually measured in KB) and CPUL2 (1-20MB).

 

[Entz]

The reason why you need more ram is because the l2arc requires memory ( taken from the arc) to maintain the pointers. Numbers I saw show that you need 200-330 bytes per block of l2arc ( depending on version and who you read lol). So it depends on your pool but that would mean you lose 8gb of arc space to have 100gb of l2arc at 4k etc. 500gb would remove up to 40 of your 63gb. Which could be detrimental to overall performance.

 

[T_Minus]

The reason why you need more ram is because the l2arc requires memory ( taken from the arc) to maintain the pointers. Numbers I saw show that you need 200-330 bytes per block of l2arc ( depending on version and who you read lol). So it depends on your pool but that would mean you lose 8gb of arc space to have 100gb of l2arc at 4k etc. 500gb would remove up to 40 of your 63gb. Which could be detrimental to overall performance.

This was my understanding of needing more ARC if you add big L2ARC.

Thanks for confirming with example and #s that's even more useful

 

[capn_pineapple]

I stand corrected, good to know!

From memory though, it's still best to cram as much ARC in as possible, L2ARC is just a cheaper option to get more cache as SSD capacity is cheaper than RAM. That said, having a few 100GB's of L2ARC will still saturate any GbE link, perhaps even a quad trunk so the limiting factor becomes networking more than anything.

 

[T_Minus]

The system will have 64 GB dedicated to the VM, leaving 64gb for other VM/usage, with possibility for another 128gb.

 

[capn_pineapple]

Build log?

 

[Entz]

Yeah. ZFS is a bit odd at times. A L2ARC definitely has its place and helps but you just gotta watch how much you give it. The recommendation from the freenas forums is a good one imo. Never exceed 5x 7/8 of your system ram (ARC size) size as the total L2ARC for your system. So for example 64GB of ram would let you have ~280GB of L2ARC but it is a good idea to test your system heavily to try and find out if this is a good performance balance.

 

[spazoid]

Always monitor your ARC hit ratio with something like arc_summary.pl before adding L2ARC. When I build my first ZFS server I thought adding an L2ARC would be great, but in reality, I very rarely had any hits from L2ARC because I had a 98% hit in ARC. I removed the L2ARC and my ARC hits improved to 99%.

This is a pool used as an ESXi datastore with a few Windows VMs and a bunch of Ubuntu VMs and "only" 16GB of ARC. With 32 or even 64GB of RAM, I doubt you'll see any point in adding L2ARC, but obviously it's dependant on your workload and dataset.

 

[T_Minus]

Very good to know.