ZFS Design Principles

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

BackupProphet

Well-Known Member
Jul 2, 2014
1,083
640
113
Stavanger, Norway
olavgg.com
The common pitfall about ZFS information that many fails to tell correctly is memory requirements and requirement of ECC memory. You got the details about ECC memory correct, but memory requirements wrong. ZFS requires almost no memory. You will be just fine on a system with 1GB ram. The recommended is 4GB, as ZFS will then by default enable prefetching which will increase read iops for data on spinning drives. For SSD's prefetching may be unnecessary, you can check the zfs-stats if prefetching has a good effect
Code:
ARC Efficiency:                                 399.70m
        Cache Hit Ratio:                83.25%  332.75m
        Cache Miss Ratio:               16.75%  66.96m
        Actual Hit Ratio:               20.73%  82.87m

        Data Demand Efficiency:         86.35%  74.73m
        Data Prefetch Efficiency:       84.14%  308.71m
 
  • Like
Reactions: Linda Kateley

BackupProphet

Well-Known Member
Jul 2, 2014
1,083
640
113
Stavanger, Norway
olavgg.com
You also get the part about Optane wrong, the only time you want ZIL is when you have spinning drives. For SSD's or Optane there is no need. With the exception of consumer level SSD's though, they will benefit greatly from having an enterprise SSD as ZIL.

The difference between 900p and p4800x is not related to powerloss protection, Optane doesnt need powerloss protection by design. There is also no difference in performance.
There real difference however is that p4800x support the Intel Memory Drive technology which let you use the p4800x as additional system memory.
 

i386

Well-Known Member
Mar 18, 2016
4,217
1,540
113
34
Germany
  • Like
Reactions: gustav

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
The common pitfall about ZFS information that many fails to tell correctly is memory requirements and requirement of ECC memory. You got the details about ECC memory correct, but memory requirements wrong. ZFS requires almost no memory. You will be just fine on a system with 1GB ram. The recommended is 4GB, as ZFS will then by default enable prefetching which will increase read iops for data on spinning drives. For SSD's prefetching may be unnecessary, you can check the zfs-stats if prefetching has a good effect
Code:
ARC Efficiency:                                 399.70m
        Cache Hit Ratio:                83.25%  332.75m
        Cache Miss Ratio:               16.75%  66.96m
        Actual Hit Ratio:               20.73%  82.87m

        Data Demand Efficiency:         86.35%  74.73m
        Data Prefetch Efficiency:       84.14%  308.71m
Irrelevant. Who doesn't have a single 4GB DIMM in 2017.

You also get the part about Optane wrong, the only time you want ZIL is when you have spinning drives. For SSD's or Optane there is no need. With the exception of consumer level SSD's though, they will benefit greatly from having an enterprise SSD as ZIL.

The difference between 900p and p4800x is not related to powerloss protection, Optane doesnt need powerloss protection by design. There is also no difference in performance.
There real difference however is that p4800x support the Intel Memory Drive technology which let you use the p4800x as additional system memory.
A ZIL is on pool until you add a SLOG device such as an Optane which will then house the ZIL
Adding an optane to say a pool of 4x SSD will improve their performance.
 

BackupProphet

Well-Known Member
Jul 2, 2014
1,083
640
113
Stavanger, Norway
olavgg.com
There are plenty of systems with less than 4GB ram. Cloud servers, embedded systems and old systems.

In 999 999 of 1 million cases you do not want to use a whole Optane 900p / p4800x as a standalone SLOG. The simplest and fastest is just using the Optane drive for storage and sync writes. SLOG was implemented to speed up spinning drives, which has awful sync write performance. Enterprise SSD's are easily 1000-10000 times faster and the benefit of having an additional SLOG is really never worth it.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
There are plenty of systems with less than 4GB ram. Cloud servers, embedded systems and old systems.

In 999 999 of 1 million cases you do not want to use a whole Optane 900p / p4800x as a standalone SLOG. The simplest and fastest is just using the Optane drive for storage and sync writes. SLOG was implemented to speed up spinning drives, which has awful sync write performance. Enterprise SSD's are easily 1000-10000 times faster and the benefit of having an additional SLOG is really never worth it.
  • It is not "SIMPLE" to use only optane drive for storage because not many people can load up 4 or 8 optane drives in their chassis at this time. Also, the drives currently have minimal capacity so it is not even a possibility to hit capacities of SSD currently and def. not as simple as a 24x SSD array.

  • An optane will increase performance of enterprise SSD on sync write. You believe what you want you are wrong.








 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
In the airport so detailed feedback later. @gea if you did want to do this as a guest post or series on STH, happy to help with that.
 

gea

Well-Known Member
Dec 31, 2010
3,140
1,182
113
DE
Thanks all for your comments.
I will care about as the pdf is ongoing work

My main intention is not to replace ZFS manuals from Solaris or give a full overview but give some basic ideas why and how things work and give my opinion on controversal things like RAM/ ECC/ Slog/ number of disks per vdev and basic configuration ideas regarding snaps and backup.

about RAM
I thought to make it clear that there is no extra RAM need for ZFS over say ntfs.
Oracle claims 2 GB as minimum for Solaris and that is the case for every 64bit OS

But nobody has fun with a 1 -2 GB ZFS filer on such a 64bit system even when stable.
Unless you use fast disks such a server is simply too slow and who would use expensive disks in a low memory setup as they are more expensive that enough RAM.


about Optane
While Optane can be a RAM replacement the main difference of a 900P to Flash is that you can adress data directly like with RAM and not blockwise like with Flash with the need of Trim or erase prior write. Its clear that a 900P is not RAM. Maybe I must clarify this.

There have been some discussions about Optane and Powerloss Protection. Intel guarantees this only for the 4800. When I bought my first 900P this was also in the specs but removed later by Intel.

Now you can speculate if it is removed due marketing or a technical reason. But PLP is not a guaranteed feature on smaller Optane models.


@Patrick
None of the content is really new and for a main site article propably not detailled enough. Intention is more to give some basic recommendations and collect some answers (and opinions) to all the beginners questions or in my opinion wrong statements like min RAM/ECC vs Scrub of Death, Slog vs write cache, golden number of disks in Raid-Z, performance considerations vs pool layout, first steps and similar.

If you are under the impression that parts can be used for guest posts or a series, I am open.
 

gea

Well-Known Member
Dec 31, 2010
3,140
1,182
113
DE
Yes, the Buyers Guide is not really up to date.
While most ZFS suggestions are the same for BSD, Linux or Solaris,
some are different due different add-ons, driver support or recommendations
where I partly have a different opinion than the FreeNas Guys.

If you like I would be glad to help to improve or add some do and don't for beginners.
This is the essential of my pdf.
 

Linda Kateley

New Member
Apr 25, 2017
21
5
3
62
Minnesota
The common pitfall about ZFS information that many fails to tell correctly is memory requirements and requirement of ECC memory. You got the details about ECC memory correct, but memory requirements wrong. ZFS requires almost no memory. You will be just fine on a system with 1GB ram. The recommended is 4GB, as ZFS will then by default enable prefetching which will increase read iops for data on spinning drives. For SSD's prefetching may be unnecessary, you can check the zfs-stats if prefetching has a good effect
Code:
ARC Efficiency:                                 399.70m
        Cache Hit Ratio:                83.25%  332.75m
        Cache Miss Ratio:               16.75%  66.96m
        Actual Hit Ratio:               20.73%  82.87m

        Data Demand Efficiency:         86.35%  74.73m
        Data Prefetch Efficiency:       84.14%  308.71m
I can't tell you how many times I have heard swags on memory requirements. The only one i know is freenas needs 8GB because the utility needs it, not zfs.
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,529
241
63
Everyone's always saying for 4x8TB you need 32GB RAM, but my FreeNAS never uses more than 12GB
 

gea

Well-Known Member
Dec 31, 2010
3,140
1,182
113
DE
This is also my impression.

For a Solarish based single user ZFS system with not too many active data, 4-8 GB RAM may be enough for a decent performance as you do not need a large readcache.

If this is a multiuser mailserver where you want to read most data from ram, you may want 64-128 GB RAM. It is the use case that demands RAM, not ZFS.