Missing space on ZFS pool (Ubuntu)

[ameel]

Hi guys,

I have the following setup:
10 x 3TB in raidz2 on Ubuntu. Using the toshiba DT01ACA300 HDDs

root@RaidServer:/# zpool list
NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
Storage  27.2T  17.1T  10.2T    62%  1.00x  ONLINE  -
root@RaidServer:/# zpool status
  pool: Storage
 state: ONLINE
  scan: none requested
config:
NAME                                  STATE     READ WRITE CKSUM
Storage                               ONLINE       0     0     0
raidz2-0                              ONLINE       0     0     0
ata-TOSHIBA_DT01ACA300_331H9XTGS      ONLINE       0     0     0
ata-TOSHIBA_DT01ACA300_331K8TJGS      ONLINE       0     0     0
ata-TOSHIBA_DT01ACA300_3317U2MGS      ONLINE       0     0     0
ata-TOSHIBA_DT01ACA300_331EJ8NGS      ONLINE       0     0     0
ata-TOSHIBA_DT01ACA300_331JZEUGS      ONLINE       0     0     0
ata-TOSHIBA_DT01ACA300_331K8RJGS      ONLINE       0     0     0
ata-TOSHIBA_DT01ACA300_331HUWWAS      ONLINE       0     0     0
ata-TOSHIBA_DT01ACA300_331446BAS      ONLINE       0     0     0
ata-TOSHIBA_DT01ACA300_331JP4KAS      ONLINE       0     0     0
ata-TOSHIBA_DT01ACA300_3314527AS      ONLINE       0     0     0
errors: No known data errors

root@RaidServer:/# zfs list Storage
NAME      USED  AVAIL  REFER  MOUNTPOINT
Storage  13.2T  7.20T  13.2T  /Storage
root@RaidServer:/# zpool list Storage
NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
Storage  27.2T  17.4T  9.88T    63%  1.00x  ONLINE  -

I thought that raidz2 is equivalent to a raid6, i.e. with 10x3TB HDDs I should be getting capacity 8 x 3000 / 1024 = 23.4375TB

However, I don't understand the following:
First, zpool reports 27.2T instead of 30 (or 29.3T) for total capacity of all the HDDs? Is it missing the capacity for 1 HDD?
zfs also reports available/usable capacity as only 20.4TB instead of 24 (or 23.44TB).

Effectively in the raidz2, I lose slightly more than 1 HDD, i.e. about 3TB of usable/available space (in addition to parity/redundancy).

Is that normal? Can overheads/waste be 3 TB+ ??? If not, where is the missing storage?

At this rate, I'd rather have stayed with mdadm raid6 :/

Thanks!

edit:
Just realised that 3TB is not actual 3000 gigabytes but instead 3,000,000,000,000 bytes -.-
hence formatted = 3TB = 2.72TB formatted x 10 = 27.2TB

However, accounting for 1/64 reserved by zfs:
Taking biggest value: 1/64 * 30TB = 0.46875 TB
21.76TB - 0.46875 = 21.29125TB

zfs only shows 20.4TB usable/available. where did the 0.89TB go?

 

[MiniKnight]

Every time I format drives, then put a fancy FS on, I always end up having way less space than I thought I would. Good to note your mdadm against ZoL usage tho.