Expected performance for ZFS Randz2

[Patrik Dufresne]

Hello,

I need you opinion about the poor performance I've observed on a pretty decent server.
The following stat was observed while creating a GIN Index in a postgresql database.

e.g.:

iostat -x -m 3
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.51    0.00    9.24   86.86    0.00    3.39

Device            r/s     rMB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wMB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dMB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
dm-0             0.33      0.00     0.00   0.00    0.00     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.13
dm-1             0.00      0.00     0.00   0.00    0.00     0.00    5.02      0.05     0.00   0.00    3.73    10.67    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.02   0.40
sda              0.33      0.00     0.00   0.00    1.00     4.00    4.68      0.05     0.33   6.67    0.93    11.43    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.54
sdb           1755.85     23.10     0.00   0.00    0.81    13.47   39.13      0.31     0.00   0.00    7.12     8.24    0.00      0.00     0.00   0.00    0.00     0.00    0.67   34.50    1.72  85.89
sdc           1592.31     20.10     0.00   0.00    0.52    12.93   44.48      0.31     0.67   1.48    4.02     7.22    0.00      0.00     0.00   0.00    0.00     0.00    0.67   14.00    1.01  86.29
sdd           1758.19     23.72     0.00   0.00    0.93    13.82   38.46      0.31     0.33   0.86    4.37     8.21    0.00      0.00     0.00   0.00    0.00     0.00    0.67   21.50    1.81  91.24
sde           1686.62     21.08     0.00   0.00    0.80    12.80   29.43      0.31     0.00   0.00   11.52    10.82    0.00      0.00     0.00   0.00    0.00     0.00    0.67   56.50    1.72  87.49
sdf           1695.99     22.43     0.00   0.00    0.67    13.54   43.81      0.33     0.00   0.00    4.48     7.63    0.00      0.00     0.00   0.00    0.00     0.00    0.67   22.00    1.35  87.49
sdg           1480.27     18.36     0.00   0.00    0.86    12.70   33.44      0.31     0.00   0.00   10.50     9.60    0.00      0.00     0.00   0.00    0.00     0.00    0.67   36.50    1.65  86.69

~22MB/s per disk seams a bit slow for disk throughput.

Here the hardware spec:

• Supermicro X9DRT-HF
• 2x Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
• 128GiB RAM DDR3
• LSI SAS2008 PCI-Express Fusion-MPT SAS-2
• HGST 0F27402 10TB HUH721010AL4200 SAS 12Gb/s 3.5" 7.2K HDD Hard Drive

What do you think ?

 

[mrpasc]

Works as expected. RAIDZ2 (RAID 6 in traditionally speaking) isn’t great with for databases IO. You are limited to the IO of one disk for write.
See Re: choosing the right RAID level for PostgresQL database
Very old but still true.

 

[BackupProphet]

Don't do raid on databases, you will also waste a lot of space. PostgreSQL uses 8K block pages, which means that for every update for one page, you will write 4kb data + 4kb data + 4kb parity + 4kb parity on raidz2, even if you have 10 disks. With same disk space usage you can have 2 mirrors and much better performance.

Even better, use a SSD