TrueNAS 13 U4 Metadata special device size, how to calculate?

[bogi]

Can someone point to information online regarding metadata special device sizing proportions.
ex:
My pool is about ~95TB in size made of 12drives sas3 > 4VDEV zfs1, each contains 3 drives.
I have two m2 NVME 2TB 980pro drives in mirror. ~1.8TB actual size, Is this enough for that pool?

I'd like to know how to figure size of SSDs I need for storing just the metadata.
I wouldn't be using 512k blocks.
Files are mostly .exr open exr or TIFF large files ~20 `50mb each basically long video file sequences.
I don't understand why this information is held in such secrecy.

truenas > SMB share > clients are widnows 10 and 11.
Nas on 40GB network users on 10GB.

 

[DavidWJohnston]

From this post: ZFS Metadata Special Device: Z

How much space do you need for Metadata storage?
The rule of thumb is 0.3% of the pool size for a typical workload. It can vary!

So 95T*0.003 = 285 GB - By that estimate it would appear you have beyond plenty. But have a look at that post for details on how to produce a better estimate specific to your workload.

You also have the option of storing small files in the special class. This is configurable, it could be useful for thumbnails in your case.

 

[bogi]

That link above is helpful, but later on they bring some confusing conclusions, correcting each other; that's' why I started this thread here.
One system admin mentioned to me few months ago that there are some tunables that need to be done for metadata so it fully performs on truenas. What I found online is related to combining L2ARC and metadata together on dedicated device and tuning.
I have collection of some tunables for L2ARC only.
Do you know where to look for metadata tuning, or this is just simple way create special vdev dedicated to the main pool and it's working?

We are testing in our lab few servers and this pure metadata vdev on treunas is bringing great performance on data intensive tasks.
Reading and writing from more than 40+ clients / render nodes, dealing with bunch of files and scenes.

Thank you for your quick response.

 

[DavidWJohnston]

I'll defer to someone else, my ZFS knowledge is a bit lacking in this area, all I can do is more Googling on your behalf.

One thing I did read, is the metadata is critical to the system. If it's lost, all your data is lost. (Unlike L2ARC) So make sure the 980 pro has the write endurance you need, and plan for replacement before failure. In a mirror, they will probably fail quite close together in time because they have identical workloads.

If it was me, I'd probably go with Optane instead of regular SSDs - They have excellent random read/write performance and much higher endurance.

Your strategy of lab testing is extremely important. You can replicate your workload, then measure how much space the metadata is using, etc.

 

[bogi]

This is info I got from iX SE:

It's not that the information is "secret" or "protected" but that the calculation isn't as simple as a "percentage of your data size." Metadata space is directly dependent on the record size of the stored blocks - so a user aiming to use meta acceleration for thousands of small files will have different space needs from your fewer large files.

 

[bogi]

Few more things that might help people test their metadata setup. I will do on my end and get back here.
My storage have a bunch of files, file sequences .exr or .jpg range from 100 to 400 each - video shots, each 20 to 60mb in size, jpg are 800MB to 4MB in size.

Recommendations given to me:
>
Given your described file sizes of 20-60MB, a 1M record size would likely be appropriate here. This would stretch out the metadata-to-data ratio quite a bit. If you've got data on the pool already, you can run zdb -LbbbA -U /data/zfs/zpool.cache poolname against it from a command prompt, and sum up the ASIZE column for everything other than the L0 zvol object and L0 ZFS Directory values - look at the bottom of the output for that.

A couple notes though:
---- only new metadata writes will be put on the special VDEVs - it won't change anything that already exists

---- your metadata VDEV should have at least the same level of redundancy as your pool VDEVs - if it is lost, your pool is unmountable, so consider a 2-way mirror as a minimum, and a 3-way mirror is not unreasonable

---- related to the above, if you have RAIDZ VDEVs for data, and attach a mirrored special VDEV, you will not be able to use zpool remove to back out later on.

---- Having separate SSDs for metadata mitigates the need for a lot of the L2ARC-meta-related tunables (like setting a dataset to `secondarycache=metadata`)

 

[gea]

I prefer thumb rules based on use cases. If you have mainly large files you may not see a performance improvement at all as reads and metadata are cached by Arc/L2Arc and writes to the pool are fast enough. If you want a special vdev "just to be sure its fast", a 3-5% minimum is ok. Often you do not want only metadata but also small io. In such a case it depends more on number of files and recsize settings so maybe 10% is more adequate. If you want to force some performance critical filesyestems onto, you must calculate the needed space additionally eventually with a quota for the filesystems to not go beyond limits.

care about
If the special vdev fails the pool is lost as this is not an optional cache like L2Arc but the real data is there. Also care about ashift. Only if all vdevs have same ashift, you can remove a special vdev.

 

[SnJ9MX]

I went through the exercise of calculating required size in my benchmark post - ZFS SLOG Performance Testing of SSDs including Intel P4800X to Samsung 850 Evo - Austin's Nerdy Things towards the bottom