HA Docker Swarm on CentOS 7.3

Discussion in 'Docker and Containers' started by nitrobass24, Apr 30, 2017.

  1. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    A Place to document my setup of a Docker Swarm

    Requirements:
    • 2-Node HA (needs to be resilient to network isolation and power failures of host)
      • Turns out you need an odd # of managers, I have added a 1cpu, 1GB CentOS VM to act as a third manager
    • Persistent Data Storage for Containers
    • Inbound DNS resolution for services (e.g. If I have to know what host my service is on that defeats the point)
    Starting point:
    • Minimal Install of CentOS7.3
    • SELinux = Enforcing
    • Firewalld = Enabled

    1. Install Docker
    Code:
    su
    yum makecache fast && yum upgrade
    yum install -y yum-utils
    yum-config-manager \
        --add-repo \
        https://download.docker.com/linux/centos/docker-ce.repo
    yum -y install docker-ce
    systemctl enable docker
    
    2. Add user to docker group (so you dont run as root)
    Code:
    sudo usermod -aG docker $(whoami)
    reboot
    3. Add persistent shared storage
    Code:
    yum install nfs-utils -y
    mkdir -p /mnt/docker/
    nano /etc/fstab
    192.168.10.2:/mnt/Single_845/docker    /mnt/docker    nfs    user,intr,sync    0    0
    reboot
    
    4. Prepare Nodes for Swarm-mode - we need to add the following firewall rules.
    • TCP port 2377 for cluster management communications
    • TCP and UDP port 7946 for communication among nodes
    • UDP port 4789 for overlay network traffic
    Code:
    sudo firewall-cmd --permanent --add-port=2377/tcp
    sudo firewall-cmd --permanent --add-port=7946/tcp
    sudo firewall-cmd --permanent --add-port=7946/udp
    sudo firewall-cmd --permanent --add-port=4789/udp
    sudo firewall-cmd --reload
    5. Initialize Swarm

    On the primary manager
    Code:
    docker swarm init --advertise-addr 192.168.10.221
    To add a secondary manager we first need to run the following on the primary manager
    Code:
    docker swarm join-token manager
    You will receive output similar to the below example
    On the secondary manager
    Code:
    docker swarm join \
        --token SWMTKN-1-2t26wk80ahqmxqd08rlp7ap04ri3s8czljo2h1yi768brcxk9w-2sgsx7hsiuh2n7xozfow32s59 \
        192.168.10.221:2377
    Now our setup looks like this

    6. Install Portainer with a persistent container

    Code:
    mkdir -p /mnt/docker/portainer/data
    docker service create \
         --name portainer \
         --publish 9000:9000 \
         --constraint 'node.role == manager' \
         --mount type=bind,src=/mnt/docker/portainer,dst=/data \
         --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
         portainer/portainer \
        -H unix:///var/run/docker.sock
    
    
     
    #1
    Last edited: May 6, 2017
    msvirtualguy and Patrick like this.
  2. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,524
    Likes Received:
    4,450
    @nitrobass24 if you do this on another machine, do the usermod before rebooting. Otherwise, you have to exit / re-login again.
     
    #2
  3. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    Thanks, moved that step later in the process.
     
    #3
  4. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    Ive got the Swarm up and running. Could use some schooling on Docker+DNS though. All I can find information on is how containers do DNS resolution. But cannot find anything about how a container would do a DDNS update so we know where it is.
     
    #4
  5. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    Even when running both nodes as a manager is doesn't like when you "unplug" the Leader. Will probably add a third Manager node with really low resources to fix this. Basically to serve as a "witness" and settle disputes/elect a new Leader.
     
    #5
    Patrick likes this.
  6. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,524
    Likes Received:
    4,450
    Watch out. I can guess your username now.
     
    #6
  7. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    Updated to make portainer a service in the Swarm cluster
     
    #7
    Biren78 likes this.
  8. Biren78

    Biren78 Active Member

    Joined:
    Jan 16, 2013
    Messages:
    550
    Likes Received:
    94
    Does it work with the docker.sock even if the main manager node goes down? I thought you use the default IP method not the .sock if you're doing it as a service.
     
    #8
  9. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    I am not totally clear on the differences (just figuring this stuff out :) ) , but I am basing this off their documentation.

    Deployment — Portainer 1.12.4 documentation
     
    #9
  10. Biren78

    Biren78 Active Member

    Joined:
    Jan 16, 2013
    Messages:
    550
    Likes Received:
    94
    It's that loading screen after making your password. I don't know how to go back later but you can put a hostname or IP if you select the not local host option.
     
    #10
  11. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    I am just accessing via http://docker:9000

    I have a DNS A record for each of my host that resolves to 'Docker'
     
    #11
  12. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    Well finally making progress on standing up a persistent service on a Docker Swarm using Portainer.

    Lessons Learned
    1. All nodes must have the volumes/mountpoints available
    2. Do not select the ingress network in the Portainer UI (its an internal docker network only) because the service creation will fail
    3. I think I found a bug w/ Portainer. When creating a service with "Bind" mapping it reverts to the "Volumes" type and you have to edit the service after the fact.
     
    #12
    Patrick likes this.
  13. nitrobass24

    nitrobass24 Moderator

    Joined:
    Dec 26, 2010
    Messages:
    1,081
    Likes Received:
    125
    Definitely some bugs with Portainer, but it seems to be better for a pure Swarm Mode setup than Rancher. Have not found a way to run Rancher as a Swarm Service and have it "Adopt" the existing Swarm.
    I have Services for Portainer, Sonarr and NZBGet all running. We will see how this goes, especially given my planned 20TB storage migration for next weekend. If I like how this is behaving I will start to move over other services like NextCloud, DNSMasq, Unifi Controller. Maybe even Plex (but there is WAF impact here that needs to be considered).

    Still need to tune my VMware affinity policies - Seem to have blips in service availability, if VMware is doing a DRS vMotion & Docker Swarm is moving, scaling, etc on a service.
     
    #13
    Patrick likes this.
Similar Threads: Docker Swarm
Forum Title Date
Docker and Containers DOCKER Swarm Advice Oct 30, 2018
Docker and Containers Docker Swarm + Unifi Switch = Massive Packet Loss? May 11, 2017
Docker and Containers Docker Swarm Management with Rancher - Wow Feb 1, 2017
Docker and Containers Docker Swarm Management with Portainer - Really Cool! Jan 30, 2017
Docker and Containers Docker Swarm management: Rancher, Shipyard, Portainer or other? Jan 28, 2017

Share This Page