How many here run ZFS on Linux and get good fsync performance?

BackupProphet

Well-Known Member
Jul 2, 2014
879
353
63
Stavanger, Norway
kingmakers.no
I just installed an Optane 900p as SLOG on my SSD pool. It made zero difference. I still only get 1/4 of the iops I get from ZFS on FreeBSD with the 32GB Optane. Before I reinstall and test again I would like to know if everyone else also struggle with slow fsync performance with their ZOL installation? I am on Ubuntu 18.04 with ZOL 0.7.9
 
  • Like
Reactions: Monoman

Monoman

Active Member
Oct 16, 2013
376
127
43
Not to be difficult, but if you're testing solutions like this, I'd try proxmox and then also centos for ZOL performance . I've read many times about poor Ubuntu zfs performance. Could be strange vatience.
 
  • Like
Reactions: arglebargle

BackupProphet

Well-Known Member
Jul 2, 2014
879
353
63
Stavanger, Norway
kingmakers.no
I took your suggestion and did some research this weekend. Conclusion ZoL has much better performance on CentOS, but is still slow.

For comparison, same hardware, default settings ZoL 0.7.9, benchmarked with pg_test_fsync

Ubuntu 2200 iops
Debian 2000 iops
CentOS 8000 iops

FreeBSD 16000 iops

Ubuntu + XFS 34000 iops
Ubuntu + EXT4 32000 iops
Ubuntu + BcacheFS 14000 iops
 

Monoman

Active Member
Oct 16, 2013
376
127
43
Would it be possible to redo your test with a Solaris install? Be it Oracle or one of the free versions.
 

amalurk

Active Member
Dec 16, 2016
276
97
28
101
Wow. You should post this to appropriate Debian and Ubuntu lists, maybe someone on the inside will take notice and investigate. Sure seems like there must be some optimizations that could be easily done.
 

MikeWebb

Member
Jan 28, 2018
91
25
18
52
Gosh. Good find. PVE must limp on ZFS by comparison. I can see why RHEL uses XFS. I’ve only just started looking at ZFS tuning for optane and 40g rdma. Looks like there is lots of gains to be made. I got as far as Dataset recordsize and zvol volblocksize, alignment shift then the red wine took over.
 
  • Like
Reactions: Evan and gigatexal

_alex

Active Member
Jan 28, 2016
874
96
28
Bavaria / Germany
Ok, will also use some 500dc or even a pair husmm1680 just to put something reasonable fast behind the slog ... Not sure its only slog that affects this, as it‘s cow...
 

_alex

Active Member
Jan 28, 2016
874
96
28
Bavaria / Germany
Yes, i know the microns can go surprisingly close to 40k sustained 4k iops with fio.

No idea of the pg benchmark numbers, and how they relate - besides it‘s 8k blocks
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,350
1,798
113
CA
Any of ya tested optane in proxmox as storage not SLOG? Was that crippled too?
 

gigatexal

I'm here to learn
Nov 25, 2012
2,876
585
113
Portland, Oregon
alexandarnarayan.com
Is there ZFS on Linux mailing list or something. They should know about this. It shouldn’t be this hard to get super high end drives to perform well.

I did have an idea. To rule out ZFS I think there’s a way to limit if not turn off the CoW nature of it right? I can’t tell where I might have seen that flag. I guess testing the drive as a raw block device and then testing it in say XFS and then in stock or tweaked ZFS is the same idea. I just hate seeing ZFS so slow.
 

_alex

Active Member
Jan 28, 2016
874
96
28
Bavaria / Germany
Haven‘t managed to do anything today, too much day-to-day business :(

I have used the 900p as primary storage with pve, and also via nvmeof directly into vm (sr-iov into vm, initiator on the vf). Performance was always ,on the point‘

Edit: to get the most out of this a recent kernel and blk-mq enabled + the right io-schedulers is absolutely a ,must‘
 
Last edited:
  • Like
Reactions: T_Minus