HA Docker Swarm on CentOS 7.3

nitrobass24

Moderator
Dec 26, 2010
1,088
131
63
TX
A Place to document my setup of a Docker Swarm

Requirements:
  • 2-Node HA (needs to be resilient to network isolation and power failures of host)
    • Turns out you need an odd # of managers, I have added a 1cpu, 1GB CentOS VM to act as a third manager
  • Persistent Data Storage for Containers
  • Inbound DNS resolution for services (e.g. If I have to know what host my service is on that defeats the point)
Starting point:
  • Minimal Install of CentOS7.3
  • SELinux = Enforcing
  • Firewalld = Enabled

1. Install Docker
Code:
su
yum makecache fast && yum upgrade
yum install -y yum-utils
yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
yum -y install docker-ce
systemctl enable docker
2. Add user to docker group (so you dont run as root)
Code:
sudo usermod -aG docker $(whoami)
reboot
3. Add persistent shared storage
Code:
yum install nfs-utils -y
mkdir -p /mnt/docker/
nano /etc/fstab
192.168.10.2:/mnt/Single_845/docker    /mnt/docker    nfs    user,intr,sync    0    0
reboot
4. Prepare Nodes for Swarm-mode - we need to add the following firewall rules.
  • TCP port 2377 for cluster management communications
  • TCP and UDP port 7946 for communication among nodes
  • UDP port 4789 for overlay network traffic
Code:
sudo firewall-cmd --permanent --add-port=2377/tcp
sudo firewall-cmd --permanent --add-port=7946/tcp
sudo firewall-cmd --permanent --add-port=7946/udp
sudo firewall-cmd --permanent --add-port=4789/udp
sudo firewall-cmd --reload
5. Initialize Swarm

On the primary manager
Code:
docker swarm init --advertise-addr 192.168.10.221
To add a secondary manager we first need to run the following on the primary manager
Code:
docker swarm join-token manager
You will receive output similar to the below example
[stephen@CentOS-Docker1 ~]$ docker swarm join-token manager
To add a manager to this swarm, run the following command:

docker swarm join \
--token SWMTKN-1-2t26wk80ahqmxqd08rlp7ap04ri3s8czljo2h1yi768brcxk9w-2sgsx7hsiuh2n7xozfow32s59 \
192.168.10.221:2377
On the secondary manager
Code:
docker swarm join \
    --token SWMTKN-1-2t26wk80ahqmxqd08rlp7ap04ri3s8czljo2h1yi768brcxk9w-2sgsx7hsiuh2n7xozfow32s59 \
    192.168.10.221:2377
Now our setup looks like this
[stephen@CentOS-Docker1 ~]$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
5za096psxgnldirifven3uwd7 CentOS-Docker2 Ready Active Reachable
e80d2ifs6ay2ohpsmi7eln4ds * CentOS-Docker1 Ready Active Leader

6. Install Portainer with a persistent container

Code:
mkdir -p /mnt/docker/portainer/data
docker service create \
     --name portainer \
     --publish 9000:9000 \
     --constraint 'node.role == manager' \
     --mount type=bind,src=/mnt/docker/portainer,dst=/data \
     --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
     portainer/portainer \
    -H unix:///var/run/docker.sock
 
Last edited:

Patrick

Administrator
Staff member
Dec 21, 2010
12,214
5,302
113
@nitrobass24 if you do this on another machine, do the usermod before rebooting. Otherwise, you have to exit / re-login again.
 

nitrobass24

Moderator
Dec 26, 2010
1,088
131
63
TX
Ive got the Swarm up and running. Could use some schooling on Docker+DNS though. All I can find information on is how containers do DNS resolution. But cannot find anything about how a container would do a DDNS update so we know where it is.
 

nitrobass24

Moderator
Dec 26, 2010
1,088
131
63
TX
Even when running both nodes as a manager is doesn't like when you "unplug" the Leader. Will probably add a third Manager node with really low resources to fix this. Basically to serve as a "witness" and settle disputes/elect a new Leader.
 
  • Like
Reactions: Patrick

Biren78

Active Member
Jan 16, 2013
550
94
28
Does it work with the docker.sock even if the main manager node goes down? I thought you use the default IP method not the .sock if you're doing it as a service.
 

Biren78

Active Member
Jan 16, 2013
550
94
28
It's that loading screen after making your password. I don't know how to go back later but you can put a hostname or IP if you select the not local host option.
 

nitrobass24

Moderator
Dec 26, 2010
1,088
131
63
TX
Well finally making progress on standing up a persistent service on a Docker Swarm using Portainer.

Lessons Learned
  1. All nodes must have the volumes/mountpoints available
  2. Do not select the ingress network in the Portainer UI (its an internal docker network only) because the service creation will fail
  3. I think I found a bug w/ Portainer. When creating a service with "Bind" mapping it reverts to the "Volumes" type and you have to edit the service after the fact.
 
  • Like
Reactions: Patrick

nitrobass24

Moderator
Dec 26, 2010
1,088
131
63
TX
Definitely some bugs with Portainer, but it seems to be better for a pure Swarm Mode setup than Rancher. Have not found a way to run Rancher as a Swarm Service and have it "Adopt" the existing Swarm.
I have Services for Portainer, Sonarr and NZBGet all running. We will see how this goes, especially given my planned 20TB storage migration for next weekend. If I like how this is behaving I will start to move over other services like NextCloud, DNSMasq, Unifi Controller. Maybe even Plex (but there is WAF impact here that needs to be considered).

Still need to tune my VMware affinity policies - Seem to have blips in service availability, if VMware is doing a DRS vMotion & Docker Swarm is moving, scaling, etc on a service.
 
  • Like
Reactions: Patrick