ZFS on Linux - arc not used

Discussion in 'Solaris, Nexenta, OpenIndiana, and napp-it' started by Stril, Jan 29, 2019.

  1. Stril

    Stril Member

    Joined:
    Sep 26, 2017
    Messages:
    147
    Likes Received:
    6
    Hi!

    I have a strange problem:
    ZoL on Ubuntu 18.04, 128 GB memory of which 96 GB are assigned as ARC. sync=disable

    But:

    ARC Size: 4.22% 4.05 GiB
    Target Size: (Adaptive) 100.00% 96.00 GiB
    Min Size (Hard Limit): 100.00% 96.00 GiB
    Max Size (High Water): 1:1 96.00 GiB

    option zfs_arc_max=103079215104
    option zfs_arc_min=103079215104


    --> ARC is not used.
    There are about 3,500,000 files on the storage and the pool-fill is only 1% (1.2/140TB). Do you have any idea on how to force ARC to be used to speed-up directory listings?

    Thank you for your help!

    Regards,
    Stril
     
    #1
  2. dswartz

    dswartz Active Member

    Joined:
    Jul 14, 2011
    Messages:
    355
    Likes Received:
    27
    what release of ZoL? What are the dataset parameters? (e.g. zfs 'zfs get all XXX | sort')
     
    #2
  3. Stril

    Stril Member

    Joined:
    Sep 26, 2017
    Messages:
    147
    Likes Received:
    6
    Hi!

    Now, ARC is used - 10GB/96GB. My main problem seems to be the "list-performance" on directories with MANY, SMALL files.

    ZoL-version is 0.7.5 on Ubuntu 18.04

    ZFS params are mostly default. The only changes are:

    zpool1 atime off local
    zpool1 compression lz4 local
    zpool1 dedup off default

    zpool1/share acltype posixacl inherited from zpool1
    zpool1/share logbias latency default
    zpool1/share sync disabled local
    zpool1/share xattr sa inherited from zpool1

    The pool is available through SMB.

    Do you have any idea for optimization? Something special for prefetch?

    Stril
     
    #3
  4. dswartz

    dswartz Active Member

    Joined:
    Jul 14, 2011
    Messages:
    355
    Likes Received:
    27
    0.7.5 is pretty old. i'd consider updating. there have been bugs in older releases where ARC collapses to the minimum. i assume primarycache=all?
     
    #4
  5. Stril

    Stril Member

    Joined:
    Sep 26, 2017
    Messages:
    147
    Likes Received:
    6
    Hi!

    Yes, its primarycache=all.
    ZoL 0.7.5 is the latest version, that is provided by the ubuntu repositories. I will try to build a newer one, or do you know any repository with newer packages?
     
    #5
  6. dswartz

    dswartz Active Member

    Joined:
    Jul 14, 2011
    Messages:
    355
    Likes Received:
    27
    I'm running CentOS 7 and have 0.7.12-1. I've found the distro repositories tend to lag badly. In Ubuntu 14 and earlier, there was a PPA you could use, but in 15 and later, you're stuck with the distro's version, it seems. I guess you could download and build from source if you wanted to. Might want to ask on the zfs on linux mailing list - devs hang out there, so someone might be able to speak to this...
     
    #6
  7. Stril

    Stril Member

    Joined:
    Sep 26, 2017
    Messages:
    147
    Likes Received:
    6
    Hi!

    I just upgraded to 0.7.12, but performance is still very bad on scanning large directories. Robocopy from windows to the filer with VERY few or zero changes takes a lot of time.

    Do you have any idea on how to speed this up?
     
    #7
  8. dswartz

    dswartz Active Member

    Joined:
    Jul 14, 2011
    Messages:
    355
    Likes Received:
    27
    Is this the same issue? e.g. small arc?
     
    #8
  9. Stril

    Stril Member

    Joined:
    Sep 26, 2017
    Messages:
    147
    Likes Received:
    6
    Hi!

    Hard so say for me...
    ARC is used, but not too much, but as robocopy does only read timestamp and lists folder, that could be o.k.
    I just see a very bad performance and have no idea on where to take a look at.
     
    #9
  10. m4r1k

    m4r1k Member

    Joined:
    Nov 4, 2016
    Messages:
    42
    Likes Received:
    5
    You assume the issue is ZFS, how about Samba? If you export the same directory content not on ZFS, do you see the same bad performance?

    Edit
    My environment is different but on Solaris-ish I have a couple of folder with more than 100k small files each (I’m a photographer) and exporting those through kernel SMB and NFS listening performance are fine. From my Mac (that has a tarrible smb client) takes maybe 30 second to show the whole content. Connected via WiFi ...
     
    #10
  11. zxv

    zxv The more I C, the less I see.

    Joined:
    Sep 10, 2017
    Messages:
    52
    Likes Received:
    16
    If the issue is robocopy performance, perhaps it would help to separate the samba and zfs aspects.
    Try testing samba with a UFS filesystem or better yet, ramdisk.

    If the issues persist, there are certainly optimizations that can be applied to samba.
    Even if robo copy only reads timestamps, that may be a stat() call regardless, which can be a bottleneck in itself.
    If so, here is an option to optimize samba for "Directories with a Large Number of Files":
    Performance Tuning - SambaWiki
     
    #11
  12. Stril

    Stril Member

    Joined:
    Sep 26, 2017
    Messages:
    147
    Likes Received:
    6
    Hi!

    I just tried to separate the processes. So I added an EXT4-volume on the (unchanged) SMB-server and did a scan of a directory with about 50.000 files.

    SMB with ZFS: 3:16 min
    SMB with EXT4: 0:16 min

    --> Seems to be ZFS/ZoL-related.

    Second scan (scan is time robocopy needs if nothing is changed) takes the same time although everything should be in ARC.

    Do you have any idea on how to solve this?
     
    #12
  13. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    1,997
    Likes Received:
    648
    Try a newer ZoL and/or compare Solaris 11.4
    (or a Solaris fork like OmniOS, not as fast as Solaris with native ZFS but free)
     
    #13
  14. m4r1k

    m4r1k Member

    Joined:
    Nov 4, 2016
    Messages:
    42
    Likes Received:
    5
    Gea is 100% right. This issue is certainly not present in Solaris forks like OmniOSce. What you can try is another ZoL flavor (CentOS, Fedora), FreeBSD, and OmniOSce as well.
     
    #14
    T_Minus likes this.
  15. zxv

    zxv The more I C, the less I see.

    Joined:
    Sep 10, 2017
    Messages:
    52
    Likes Received:
    16
    Yep. It looks like it would take some significant effort to tune ZoL, whereas Omnios or solaris should work well with minimal tuning -- perhaps no tuning for a 1gb network.
     
    #15
  16. Stril

    Stril Member

    Joined:
    Sep 26, 2017
    Messages:
    147
    Likes Received:
    6
    Hi!

    @gea:
    I already did the tests with ZoL 0.7.12

    What I tried now, is FreeNAS (because this was easy):

    SMB with ZoL: 3:16 min
    SMB with EXT4: 0:16 min
    SMB with FreeNAS: 0:19 min

    I would prefer ZoL, but as I do not havy any idea, yet....
     
    #16
  17. Stril

    Stril Member

    Joined:
    Sep 26, 2017
    Messages:
    147
    Likes Received:
    6
    F***, I was wrong...

    Now, I installed FreeNAS on my production hardware and the performance is as bad, as before with ZoL.
    I have NO idea, what happens.

    It's an enterprise-class supermicro server with an LSI 9300 SAS 12G HBA and 32 10 TB Seagate Exos Enterprise SAS Disks.

    My test-FreeNAS-system was a VM with a small single-disk-pool (and MUCH faster).

    Do you have any idea, on how to debug this?
     
    #17
  18. ttabbal

    ttabbal Active Member

    Joined:
    Mar 10, 2016
    Messages:
    663
    Likes Received:
    182
    What style of pool are we talking about here? Perhaps output of "zpool status"?

    I'm surprised that the listing metadata doesn't appear to be cached in any case. That seems like an easy win for the cache. It wouldn't help the first go, but subsequent runs should be great.
     
    #18
  19. Stril

    Stril Member

    Joined:
    Sep 26, 2017
    Messages:
    147
    Likes Received:
    6
    Hi!

    Its a quite str simple pool:

    16 mirrors, each with 2 dry drives. I will insert pool status later...

    Very strange. Arc is bigger than pool data at the moment
     
    #19
  20. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    1,997
    Likes Received:
    648
    Arc generally does noch cache files or sequential data. It caches metadata and random reads based on a read last/read most optimisation. So sequential read is only improved by metadata caching.

    For L2Arc you can enable read ahead that may help a little.

    For the rest, real pool iops and sequential performance is relevant.

    But as said, install Solaris 11.4 (you can download for free, only commercial use is prohibited) and check performance there with native ZFS (do all tests from Windows) and the kernelbased SMB server. Then you have a reference about whats possible with ZFS on your hardware and how slower another setup is. If its as slow, its the hardware, otherwise the OS.
     
    #20
    Last edited: Jan 30, 2019
Similar Threads: Linux used
Forum Title Date
Solaris, Nexenta, OpenIndiana, and napp-it Napp-It Linux - post install sudo problems Aug 28, 2018
Solaris, Nexenta, OpenIndiana, and napp-it Lost and confused... Jan 7, 2019
Solaris, Nexenta, OpenIndiana, and napp-it FreeBSD/FreeNAS vs OmniOS/Napp-it write speeds when used as ESXi NFS VM datastore? Jul 29, 2018
Solaris, Nexenta, OpenIndiana, and napp-it ZFS volumes used by iSCSI showing double space usage in zfs list? Apr 9, 2017
Solaris, Nexenta, OpenIndiana, and napp-it ZFS read caching with SSD: is cache used as well as storage, or instead? Apr 7, 2017

Share This Page