What to expect if FreeNAS loses connection to disk shelf?

Discussion in 'FreeBSD and FreeNAS' started by Cape, Jan 12, 2019.

  1. Cape

    Cape Member

    Joined:
    Oct 28, 2015
    Messages:
    30
    Likes Received:
    2
    I just set up a FreeNAS VM to start experimenting with. VM on ESXi, passthrough of a LSI9200, connected to external disk shelf.
    I figured it would be a good thing to know what would happen if my disk shelf lost power or similar, so I powered it down. This led to two unexpected things:
    1. It took > 1 minute until FreeNAS seemed to notice (pop an alert in the UI, that is)
    2. After it noticed, a lot of things seem to die. The UI no longer works, ssh doesn't work, and even console login stalls after giving the password.

    Is this actually expected behavior, or something likely to be wrong with my install?
     
    #1
  2. NathanM3

    NathanM3 New Member

    Joined:
    Dec 4, 2016
    Messages:
    2
    Likes Received:
    0
    I had this same issue a couple weeks ago. What I found was that freenas was using swap. Freenas uses a 2gb partition on the pool drives for swap. So when the pool drives got disconnected, the swap partition that freenas was using also got disconnected.

    There is a system tunable to adjust the amount of ram the ZFS leaves free. I adjusted this so that more ram is left free and I have not seen it swap since. This system has about 100gb of ram, so it was not caused by a lack of sufficient hardware.

    I did not test whether this completely fixes the crashing problem when disconnecting drives, but it fixed the root cause that I saw. The system is in production so I didn't test it further.
     
    #2
    Last edited: Jan 12, 2019
  3. Cape

    Cape Member

    Joined:
    Oct 28, 2015
    Messages:
    30
    Likes Received:
    2
    Oh, yeah, that would cause problems. This system probably hadn't swapped very much either, but I would expect it to be somewhat confused anyway if swap space goes away without notice.
    Still, seems a bit fragile - I noticed FreeNAS will by default (and with strict warnings about not disabling it) add some swap on all pools, and if it crashes the entire system if it loses any of those? Otherwise I'd have just added another non-external disk as swap space.
     
    #3
  4. Ixian

    Ixian Member

    Joined:
    Oct 26, 2018
    Messages:
    64
    Likes Received:
    8
    Common practice with FreeNAS & ESXi is to move swap to an nvme, the same one you should already have for slog and cache.

    For example I have an Intel DC3700 passed through that I've partitioned for swap, log, and cache. When using ESXi you'll most likely be using NFS to connect to your ZFS datasets and you're going to want something like that anyway for performance reasons. Check out page 3 of this thread for details (actually read the whole thing, it's fantastic and was a major help when I set my own up).
     
    #4
  5. azev

    azev Active Member

    Joined:
    Jan 18, 2013
    Messages:
    550
    Likes Received:
    133
    I think the main issue is that by default zpool provisioning via GUI will always automatically create 2GB swap partition in each drive. I wish there is a way (maybe there is I just dont know how) to customize the swap partition location via gui instead of having to manually create the freebsd-zfs partition & freebsd-swap via gui which will simplify the initial zpool provisioning.
     
    #5
  6. Ixian

    Ixian Member

    Joined:
    Oct 26, 2018
    Messages:
    64
    Likes Received:
    8
    Fair enough. Though it's very simple to do via shell, or just add as a post-init command. This:
    Code:
    swapoff -a ; grep -v -E 'none[[:blank:]]+swap[[:blank:]]' /etc/fstab > /etc/fstab.new && echo "/dev/gptid/24642fd-1467-11e9-ad08-000c296d6423.eli none swap sw 0 0" >> /etc/fstab.new && mv /etc/fstab.new /etc/fstab ; swapon -a
    where you obviously put the gptid of the device you will use for swap instead of the example I provide here, added as an post-init command via the GUI, will do it. You can also run it from the shell if you don't want to reboot right away.

    That will delete the existing pool swaps and establish it on the device of your choosing.
     
    #6
Similar Threads: expect FreeNAS
Forum Title Date
FreeBSD and FreeNAS Question regarding expected performance of my FreeNAS build. Jun 28, 2017
FreeBSD and FreeNAS NFSv4 weirdness in ESXi 6.5 and FreeNAS Saturday at 9:55 PM
FreeBSD and FreeNAS FreeNAS install failed Feb 18, 2019
FreeBSD and FreeNAS Freenas error Feb 15, 2019
FreeBSD and FreeNAS X9SRL-F w/E5-1650v2 or X9DRH-iTF w/dual-E5-2680v2 for FreeNAS? Jan 7, 2019

Share This Page