How many here run ZFS on Linux and get good fsync performance?

[BackupProphet]

I just installed an Optane 900p as SLOG on my SSD pool. It made zero difference. I still only get 1/4 of the iops I get from ZFS on FreeBSD with the 32GB Optane. Before I reinstall and test again I would like to know if everyone else also struggle with slow fsync performance with their ZOL installation? I am on Ubuntu 18.04 with ZOL 0.7.9

 

[Monoman]

Not to be difficult, but if you're testing solutions like this, I'd try proxmox and then also centos for ZOL performance . I've read many times about poor Ubuntu zfs performance. Could be strange vatience.

 

[BackupProphet]

I took your suggestion and did some research this weekend. Conclusion ZoL has much better performance on CentOS, but is still slow.

For comparison, same hardware, default settings ZoL 0.7.9, benchmarked with pg_test_fsync

Ubuntu 2200 iops
Debian 2000 iops
CentOS 8000 iops

FreeBSD 16000 iops

Ubuntu + XFS 34000 iops
Ubuntu + EXT4 32000 iops
Ubuntu + BcacheFS 14000 iops

 

[i386]

Ubuntu 2200 iops
Debian 2000 iops
CentOS 8000 iops

That's a huge difference
Has it something to do with Debian & Centos using "older" kernel versions?

 

[BackupProphet]

Debian was 4.5ish, while CentOS was 3.10ish, Ubuntu is 4.16

 

[Monoman]

Would it be possible to redo your test with a Solaris install? Be it Oracle or one of the free versions.

 

[gigatexal]

wtf - why is there so much performance difference ... this does not bode well. I mean CoW vs. XFS is apples and oranges but still.

 

[amalurk]

Wow. You should post this to appropriate Debian and Ubuntu lists, maybe someone on the inside will take notice and investigate. Sure seems like there must be some optimizations that could be easily done.

 

[MikeWebb]

Gosh. Good find. PVE must limp on ZFS by comparison. I can see why RHEL uses XFS. I’ve only just started looking at ZFS tuning for optane and 40g rdma. Looks like there is lots of gains to be made. I got as far as Dataset recordsize and zvol volblocksize, alignment shift then the red wine took over.

 

[_alex]

Is the difference also present when you compare fio runs?

 

[BackupProphet]

yes

 

[_alex]

I have a Setup with current proxmox and a 900p aic i could benchmark tomorrow - what ssd‘s do you have in the pool and how is the Layout ?

 

[BackupProphet]

Intel S3700 and Micron 500DC (with latest firmware), but only the slog get benched when doing sync writes.

 

[_alex]

Ok, will also use some 500dc or even a pair husmm1680 just to put something reasonable fast behind the slog ... Not sure its only slog that affects this, as it‘s cow...

 

[BackupProphet]

With 2000 iops at 8kb sync writes, even the S3700 or the Micron 500DC is not up to their full potential. I get 7000-8000 iops with S3700 and 9000-10000 iops with Micron 500DC on ext4.

 

[_alex]

Yes, i know the microns can go surprisingly close to 40k sustained 4k iops with fio.

No idea of the pg benchmark numbers, and how they relate - besides it‘s 8k blocks

 

[T_Minus]

Any of ya tested optane in proxmox as storage not SLOG? Was that crippled too?

 

[gigatexal]

Is there ZFS on Linux mailing list or something. They should know about this. It shouldn’t be this hard to get super high end drives to perform well.

I did have an idea. To rule out ZFS I think there’s a way to limit if not turn off the CoW nature of it right? I can’t tell where I might have seen that flag. I guess testing the drive as a raw block device and then testing it in say XFS and then in stock or tweaked ZFS is the same idea. I just hate seeing ZFS so slow.

 

[_alex]

Haven‘t managed to do anything today, too much day-to-day business

I have used the 900p as primary storage with pve, and also via nvmeof directly into vm (sr-iov into vm, initiator on the vf). Performance was always ,on the point‘

Edit: to get the most out of this a recent kernel and blk-mq enabled + the right io-schedulers is absolutely a ,must‘

 

[gigatexal]

Wait you can carve up a 900p with sriov like you might an Ethernet interface?