New Storage Server build.

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

modder man

Active Member
Jan 19, 2015
657
84
28
32
I have recently acquired a supermicro 36 Bay chassis. I also have 24 3TB 7200 RPM drives for the start of a new storage server. I have a few different SSD's around as well as a fusion IO. Have used freenas in the past until I found out the fusion IO wasnt supported.

I know there are many different options just thought I would post here and see what suggestions you all had.

If I did ZFS again I think I would likely do 3 x 8disk Z2 not sure one which OS.
I also played with storage spaces a bit but writes with dual parity were really slow.
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
If you do not use the "golden number" of disks per raid-z2 (6,10) disks,
you should increase ZFS blocksize from 128k to 512k or 1M to get full capacity.

Regarding OS with ZFS support you can decide between
- BSD (ex FreeNas)
- Linux (ex OMV)
- OSX (no web-ui)
- Solarish (NexentaStor or Oracle Solaris (ZFS origin)/ a free Solaris fork like OmniOS/ OpenIndiana + my napp-it)

each with its own advantages
 
Last edited:
  • Like
Reactions: pgh5278

modder man

Active Member
Jan 19, 2015
657
84
28
32
I did not realize 6 or 10 were golden numbers, I thought the goal was less than 12 disks more than 6 disks. Could you point me in the direction of some reading that I clearly should have done? Also can a fusion IO be used with omnios? I am using the napp-it appliance for testing.
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
Think of the problem like

You want to write a datablock ex a default ZFS 128k datablock striped even over 4 disks.
This means 32k per disk. If a disk has a physical blocksize of 4k you need 8 blocks without an overhead (8x4=32 blocks).
This is the same with any number of datadisks that are a power of 2 (2,4,8,16).
The redundancy disks came on top (z1:+1disk, z2:+2disks, z3:+3disks)

If you write to 5 disks, you have to write 128k/5= 25,6k per disk, you need 7 physical blocks per disk=7x5=35 blocks, Compared to above you have an overhead of 3 diskblocks = waste.

To reduce the problem you can use disks with a lower physical blocksize like 512B or increase ZFS blocksize

A table that I have seen that shows waste over disks/blocksize
ZFS overhead calc.xlsx
 
  • Like
Reactions: Evan

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
Leaves the question

- is the table wrong
- is this model (datadisks as power of 2 as golden numbers) too simple for all cases

I cannot answer exactly but this is the rule of thumb
and an increase of blocksize helps on other numbers
 

fractal

Active Member
Jun 7, 2016
309
69
28
33
The "golden number" is over rated and less important if you enable compression. Current guidance is you should always enable compression with ZFS. They now specify a "minimum" number of drives for each ZFS level.

10 drives per Z2 vdev (8 + 2) is a very good choice from everything I have heard.
 

modder man

Active Member
Jan 19, 2015
657
84
28
32
So I have noticed these drives are listed as ashift9 in the ZFS info. I assuming that means they are 512b, should I be looking at changing them to 4k. Does it depend on my pool arrangement?
 

modder man

Active Member
Jan 19, 2015
657
84
28
32
Have done a little more Testing. I dont really know why sort of numbers to expect from the array. Looks like since these are 512B disks then 8disk vdevs may be alright?
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
With 512B disks, you can ignore the "golden number rule" or the suggestion to increase blocksize in ZFS.
Main problem with ashift=9 vdevs: If a disk fails you will not be able to replace with a new 4k disk and every new disk that you can buy is 4k

This is why I would force ashift=12 vdevs either with mixing a newer 4k disk among them in every vdev that you add or forcing ashift=12 with modified disk parameters.

On OmniOS you must modify sd.config to force 4k identification of these disks.
ZFS and Advanced Format disks - illumos - illumos wiki
 

modder man

Active Member
Jan 19, 2015
657
84
28
32
Thanks for the advice GEA, I will set the drives to 4k. How do these numbers look for the drives?

upload_2017-1-21_8-29-57.png