What kind of nodes? Generally in Kubernetes, we talk about control plane nodes and worker nodes. You need at least one of each, although you can have both on the same server I believe. The control plane nodes run the main database. So I doubt if you could have one of those offline as a spare. I guess you could have a worker node off line as a spare, but to what end? That defeats the purpose of K3S.
I was not specific in my question because I believed it could distract from my question. So let give some more details.
The basic principal of kubernetes is clear to me. I also understand the difference between control node and worker node. I am planning to run virtualized under Proxmox k3s. Not yet sure if worker role and control role on one node (k3s can do so), of if I split both roles in two nodes. Additionally Rancher on a separate VM. This would be by operative system that gives not mutch more value to me as a single docker instance would do but with less complexity and with less overhead. But as mentioned, I want to get into kubernetes.
My question of for "offline node" is related to an additional worker node or control node to the once running on a different physical system. The purpose would be to get these online only in case of need to shut down the proxmox server for a longer time to be able to continue running some services on these. Spin up B, train A shutdown A and later same in opposite direction.
To your question, I guess you could have a single control plane node and a single worker node on line, and have one or more worker nodes off line but ready to spin up. But Since VMs are essentially free, why bother?
My point was not the number of VM running on one Proxmox system. My point was to have nodes running temporarily on an other system that is not 24/7 connected.
FWIW, the biggest challenge I have overcome with K3S is using centralized shared storage (NFS CSI driver). I integrated NFS into my cluster so that my data could be backed up centrally.
Thank you. I am planning to use local storage provided by Proxmox. Same data I will access via NFS provided by TrueNAS.
Otherwise if your nodes are all VMs, the storage is a bit of a sprawl and gets spread between your worker nodes on the qemu drives that Proxmox sets up (or the equivalent in other hypervisors).
Thank you. I will see if performance vise I can live with this.
I also figured out pretty quickly that K3S is more of a learning experience for me. I really doubt if I would ever use K3S for my "production" environment, which consists of a couple of Wordpress websites, a Nextcloud instance and some other assorted services like Home Assistant, Joplin, Bitwarden, and Photoprism.
Same for me. I want to learn and "play" with overengineered solutions in my home lab.
I am looking with one eye to move away from Proxmox to xcp-ng. To do so I would need to give up on my LXC containers running these kind of services. Docker might be the best way to achieve this. I am going one step further if want to discover kubernetes and maybe later extend the lab depending on the outcome
What is your purpose in moving to k8s? Pods don't exactly live-migrate like VMs under PVE, but are intended to be ephemeral and disposable; you can scale them out and back in response to load. The persistent storage needs to be able to handle this; e.g., by using object storage, RWX filesystems like CephFS, or even just iSCSI volumes, where the high-availability of the backing storage is a separate issue, to be addressed by SAN or clustered storage.
Just to be able to move a part of my operative temporarily from one physical system to another to be able to maintain or update the underlaying virtualization software or the hardware. If doing this on kubernetes layer is not possible or more difficult than doing it on VM layer my question is answerd.
