Docker Swarm + Unifi Switch = Massive Packet Loss?

Discussion in 'Docker and Containers' started by nitrobass24, May 11, 2017.

  1. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    Ran all weekend best I could tell without an issue then I try to stream a TV show and have issues. Shutdown the sonarr container and the issues persisted.

    Moving down the stack. I am basically left with an ESXi or FreeNAS issue.
     
    #21
  2. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    Well it turns out its my FreeNAS storage system. NFSd just stops/starts for no reason, but fails.

    Here is the tail of the /var/log/messages

    Code:
    [root@freenas] ~# cat /var/log/messages | grep nfs
    May 30 00:05:24 freenas notifier: Stopping nfsd.
    May 30 00:05:24 freenas notifier: Stopping nfsuserd.
    May 30 00:05:25 freenas notifier: Starting nfsuserd.
    May 30 00:05:25 freenas notifier: Starting nfsd.
    May 30 00:05:25 freenas nfsd[60563]: Can't read stable storage file: Operation not permitted
    May 30 00:08:22 freenas notifier: nfsd not running? (check /var/run/nfsd.pid).
    May 30 00:08:22 freenas notifier: Stopping nfsuserd.
    May 30 00:08:22 freenas notifier: Starting nfsuserd.
    May 30 00:08:22 freenas notifier: Starting nfsd.
    May 30 00:08:23 freenas nfsd: can't register svc name
    May 30 00:36:34 freenas nfsd: can't register svc name
    [root@freenas] ~#
    
    WTF
     
    #22
  3. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,496
    Likes Received:
    4,440
    What FreeNAS version are you on now?
     
    #23
  4. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    Was on 9.10-U2.

    Last night while it was down, I upgraded it to 9.10-U4. Figured I have nothing to lose.
     
    #24
  5. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    Updated FreeNAS to 9.10 U4 and still have this issue. Definitely, something with FreeNAS, because when it happens I lose all NFS connectivity everywhere simultaneously.
     
    #25
  6. DWSimmons

    DWSimmons Member

    Joined:
    Apr 9, 2017
    Messages:
    44
    Likes Received:
    10
    Just curious if this ever got resolved. The log about permission denied seemed peculiar.
     
    #26
  7. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    No not really any closer to figuring this out, unfortunately. I have completely rebuilt my VMware Cluster and while it does not happen on any regular basis, I have seen it occur once since.

    Considering I lose NFS connectivity on both ESXi host simultaneously I am basically down to some sort of Switch Issue or a FreeNAS issue. I am leaning FreeNAS because I didn't have this issue on my Synology, but I, unfortunately, cant rule out the Switch.

    I might buy two Small SSDs to run the Docker workload and put them back into my Syno and see if I can re-produce the issue.
     
    #27
  8. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    UBNT has agreed to replace my switch after looking at my logs and .supp files for a few weeks. "They would like the switch back for physical inspection".
     
    #28
Similar Threads: Docker Swarm
Forum Title Date
Docker and Containers DOCKER Swarm Advice Oct 30, 2018
Docker and Containers HA Docker Swarm on CentOS 7.3 Apr 30, 2017
Docker and Containers Docker Swarm Management with Rancher - Wow Feb 1, 2017
Docker and Containers Docker Swarm Management with Portainer - Really Cool! Jan 30, 2017
Docker and Containers Docker Swarm management: Rancher, Shipyard, Portainer or other? Jan 28, 2017

Share This Page