Weird networking issue with cloudflared, need help diagnosing

[altano]

I have a pretty basic setup where I have some docker containers hosting various services. They’re all only on my local LAN. The docker containers are on Flatcar Linux, which is in a VM in Proxmox.

In the same Proxmox instance I have a Debian VM with only cloudflared on it. I then expose these services over Cloudflare Tunnel + Access.

Here’s what that all looks like:


Here’s the issue: after not using these services for a long time, Cloudflare Access says “502 Bad Gateway” which means the service is no longer reachable. I can’t diagnose the problem because as soon as I try to ssh into the Flatcar Linux VM that hosts the Docker containers, the services start working again. The act of SSHing into the VM fixes the problem. Then, several days (maybe weeks?) later the services become unavailable again.

Does anyone know what might be the issue here or how I might go about diagnosing it?

 

[altano]

I had the problem again on another system with the same setup. I did some more debugging and I'm really stumped:

* The cloudflared VM can ping the Flatcar VM
* One of my Docker containers is running an http service on port 9000, so I would expect that to be reachable on the Flatcar VM. nmap says port 9000 is filtered:
PORT STATE SERVICE
9000/tcp filtered cslistener
* Additionally, ssh is open on the Flatcar VM. nmap confirms:
PORT STATE SERVICE
22/tcp open ssh

So how can I diagnose why SSHing in on port 22 is making port 9000 reachable?

I'm leaving this machine in a bad state (by not SSHing into the Flatcar VM, which would fix it) so I can continue diagnosing the problem but I'm really stumped. Unfortunately I have Flatcar configured to not have user account passwords, so while the VM is responsive in the Proxmox console, I can't login.

 

[altano]

Anddddd I figured it out:

I figured it out. I think I don't fully understand Flatcar Linux but:

• Flatcar Linux is configured to start Docker through socket activation. When docker.socket detects a connection it starts docker.service and any containers that are set to start.
• On Flatcar Linux, ssh actually starts a Docker container and ssh's into it. You're not ssh'ing into the host.
• When Flatcar would automatically update and reboot, my containers would become inaccessible.
• When I was ssh'ing in, it was starting docker, which would start my http container.
• For some reason, Flatcar's socket activation was working for ssh because port 22 was open, but it wasn't working for the port of my http container (port 9000).

To "fix" this (I don't really understand why these are the defaults and why socket activation doesn't work for my http container), you can set docker.service to always start and not wait for socket activation: Getting started with Docker