Rant about stuff that don't do as I would like them to... storage, backup, docker

[SecCon]

I apologize in advance. It may become a very self oriented, narrowminded and illogical thread, assuming someone else wanna participate. I do not feel I am overly incompetent nor inexperienced, after all, I have worked within IT since 2000. Not going to drag a list of what I have done or should be able to do in this.

I just need to vent my frustration and while being a member of most forums pertaining to below mentioned solutions/systems/distros I do not feel I can get any answer from a community that might be oriented towards explaining how to do things in their own system rather that finding something that works for me. I have asked about particular challenges in each and every one of those places, at least tried.

This is not hardware related. I have good hardware. The target machine for this is a Huawei RH2288Hv3 server with almost 30TB LFF on a HBA RAIDZ compatible card, or rather it will be once I start implementing my new solution. 128GB RAM, Dual Xeon E5-2683 v4 and GB networking, with optional 10GB available.

What do I want to accomplish? Basically same as what I have today, but with some hosted apps.

• An SMB Fileshare. (Have)
• Reliable backup of that FS to a network connected NAS, once a day, mirrored. (Have)
• A couple of Virtual machines , Windows. (Have)
• A webapp from Ubiquiti called UISP or UNMS to monitor my UI EdgeMAX devices. (Have)
• A webapp from Matomo to monitor web site traffic (Want - Docker seems the best way)
• A webapp to monitor websites uptime (Want - Docker seems the best way)
• Maybe something else, also via Docker probably
• Easy to manage. I will not be the only one handling this. I can not force a linux learning curve on relatives without my experience or knowledge. It just has to be easy and and run smooth out of the box.

I have tried to accomplish above with:

• TrueNAS (Scale)
• Ubuntu Server
• Fedora Server
• Proxmox
• OpenMediaVault

I have installed and uninstalled all of those at least thrice testing different things on a Supermicro test Server I have available with a minimal config I use for testing out stuff..

All fail for one or another reason. Yes they do. Or I am just not feeling safe with how to do to achieve it, whatever it was lacking.

TrueNAS Scale
Wonderful stuff. Perfect for storage. Made pools, made SMB, but it fails in two important aspects, or rather, I fail.

1. The backup. Rsync. I just don't get it. Tried several methods but it is just like reading a different language. And from what I gather it has to run all the time, syncing all the time. Spent hours reading at the site. So I tried with Syncthing. Never managed to get it properly configured to start with. One backup as mirror daily. Period. How is that not possible?
   
   
2. Docker. TrueNAS Scale comes with K8. It comes with Truecharts. Many apps available at the touch of a button. Yet, while I was testing the guys at Truecharts made some "feature update" and kinda wrecked the system and nothing worked. For just about EVERYONE, not just me. The only answer I could see was "reinstall the apps". I can very seriously live without that. I don't get why K8 is there, but never mind I try to install a docker Portainer standalone, pulling the image and starting it. No go. I tried altering some system wide setting in K8 and at first it worked, in my second installation it did not.

I still believe that TrueNAS Scale should be a serious contender worth figuring out, but I am at a loss right now.

Ubuntu Server / Fedora Server
I list those as one since it boils down to the interface used: Cockpit-Project. Cockpit-Project seems to have intimate ties with 45Drives. Nothing wrong about that, but it also means that most if not all the plugins depends on 45Drives to maintain and develop them. I tried to configure a ZFS storage pool with a SMB share using the Cockpit-Projects plugins and was not able to get it to work despite also using a File-Sharing plugin they also have. There seems to be no backup solution implemented for that either so it kinda fails the prerequisites. Does that mean these servers os's are bad for this? Obviously not, since they seem to be used by others, duh, but I was unable to get almost anything of my prerequisites to run. I tried installing Podman and Portainer to see if that would lead to any working app, but could not get any to work properly, Podman being the worst container management interface I have ever seen. Oh, wait, I have only seen one another. Ah well.

Proxmox
Container management is different and rather full of new terminology like Linux Containers and what not. It's different from any other system I have touched so far and requires additional effort to towards learning new stuff. Again. However, see my post below, I am getting further faster than many others and a lot of functionality you only get with plugins and TrueCharts and stuff like that, comes out of the box so it should be a lot safer, in theory, to use. Some people claim CEPH is awesome.

OpenMediaVault
It seems good enough, but backup woes, rsync and all that. I am very concerned about the future of this solution since it does not seem to have any "mayor players" behind it and I need my stuff to work 10-15 years. Long term reliability?

- - -
So, as a conclusion, you might wonder what kind of miracle system I have today that successfully does everything I want from these others and is simple and reliable enough?

Windows Server 2022 with Hyper-V. I use SyncBack as backup solution that has worked perfectly well for over a decade, no software upgrades broke it ever. I have a virtual machine with a *nux os that can run Docker, and ONLY Docker for any apps I would need to run, see the list above, and do not need that guest OS to do anything else. However, it has it's hickups, Windows acts up sometimes and there are tiny things that are hard to diagnose that occasionally cause a BSOD. Not very cheap either.

 

[opensourcefan]

I have the majority of what you've described running. I'd be happy to help you systematically get it installed and running. I have 4 instances of scale, one using rsync. Proxmox, docker etc.

It'll have to be at a chilled pace, and you may have to ping me to check back in.

 

[opensourcefan]

What helped me was searching for install procedures and not just using one but usually a mix of a few. Same for when problems arise. I should noet that my TrueNAS and Proxmox instances are all installed baremetal.

 

[reasonsandreasons]

For share backup on TrueNAS, it might be worth trying remote ZFS replication instead of something like Rsync or Syncthing. This does require a ZFS target system, but will likely be better than most of your other alternatives. I'm running sort of a reverse of your setup, where my main SMB shares are on a TrueNAS Core box and the backup and containers are on a TrueNAS Scale machine.

I don't have much to add on the K8s stuff, unfortunately. TrueCharts is a bit of a nightmare, though hopefully there will be a better solution soon.

 

[SecCon]

For share backup on TrueNAS, it might be worth trying remote ZFS replication instead of something like Rsync or Syncthing.

Sadly that is not an option since it would require me to buy a new NAS entirely...

@opensourcefan As I type this I am having another go at Proxmox and trying to get pass strange acronyms to actually get to where I want to get done. Editing my original post a bit as well.

 

[DavidWJohnston]

Everything you want is possible, except the last item (ease of management out-of-box for someone who isn't you) might be harder than you think.

The "just work out of box" part might be nearly impossible - it doesn't matter how many things you try, or screenshots of GUIs you look at, IT never works like it's supposed to.

The way I've handled this in the past is to build systems on software that's highly flexible and customizable, then write scripts and a dashboard UI to handle the management, and try to code-in as many edge-cases as possible using my imagination and experience. Of course you'll miss some, and there is a diminishing return the closer you approach 100%.

Making IT products that work "perfect" for others is a highly sought-after ability - The problem is this skill of solving problems before they happen is rarely rewarded because management don't know how good they have it until the person leaves.

 

[zer0sum]

Unraid might be worth a shot?

It is amazingly flexible and has a massive "app" store for docker apps. Matomo and UNMS are both there for instance.
I haven't found anything better for home storage and docker apps. UptimeKuma would be perfect for your web app monitoring

 

[SecCon]

Unraid is on the list. Eventually.

Everything you want is possible, except the last item (ease of management out-of-box for someone who isn't you) might be harder than you think.

The "just work out of box" part might be nearly impossible - it doesn't matter how many things you try, or screenshots of GUIs you look at, IT never works like it's supposed to.

The way I've handled this in the past is to build systems on software that's highly flexible and customizable, then write scripts and a dashboard UI to handle the management, and try to code-in as many edge-cases as possible using my imagination and experience. Of course you'll miss some, and there is a diminishing return the closer you approach 100%.

Making IT products that work "perfect" for others is a highly sought-after ability - The problem is this skill of solving problems before they happen is rarely rewarded because management don't know how good they have it until the person leaves.

Yeah, well, that is just not gonna fly. Say you are not available for any fathomable reason and others need to step up and get something out of your network. Anything you told them to store on your servers, because your servers, as mine, keep track of everything and has a backup and is generally the safest place to keep things. While YOU are there to manage it and fix hickups.

And suddenly you are not. How long will things be accessible? Safely? Backed up?
I don't see many addressing that particular situation. Sure, you can go Cloud, all files and stuff available behind one login, but that is not the current scenario and is probably way more expensive and a few other reasons.

 

[Patrick]

Bryan #2, William, and I were working on a CRAZY home server piece this week... Using Windows 11 Pro as the base.

While not directly applicable, your post reminded me a lot of the reason behind this little project.

 

[SecCon]

@Patrick
That is really interesting, I hope you get to publish an article about that.

I must say I have been pondering that as well, but aside from theoretical issues with server hardware and drivers for Windows Desktop I am now wondering how come I never tested it, as I have tested many other distros. After all, in W11 today you have most of the support you need for Virtual Machines, Linux Kernel, SMB and the very simple, and perhaps rather insecure with default settings, but very practical, RDP.



If you miss anything, just fire up a Linux with the Subsystem.

Even ZFS is on its way. GitHub - openzfsonwindows/ZFSin: OpenZFS on Windows port

EDIT. Hmm, maybe TPM could be an issue on server hardware.

 

[SecCon]

... and add any number of software apps to that... Can't believe I forgot to mention it in my post above.

I need to test this.

 

[SecCon]

Windows 11 Pro as bare metal host.

I will just rant a bit 'bout it here as I do things.

Small HW check is ok. All devices and hdd's and NIC's and stuff identified. ... a few hours later after some conf and windows update ... yeah, everything seems to work.

- Docker Desktop for Windows, which is usable in this environment, is a bit of a challenge. I should love an interface, but I must admit it has several quirks before getting it to run, and having it running. Find nothing about run-as-service, only start as login. Which is not the same as start as service. Windows Service, just to be clear. Still no working container either - obviously went for Portainer to handle Containers with Docker for Windows...
- I think a better approach might be WSL with Ubuntu and since WSL2 is already in place, thanks to some nagging by Docker Desktop for Windows, instead of just throwing in a dependency (!?)... will try that after a bit more fiddling with Docker for Windows. Had to change the container/images storage location among other things.

... to be continued

 

[SecCon]

Sometimes I just dont get things.

WTF is wrong here:



Is Intel blatantly stupid or is Windows finding stuff Intel deprecated?

After the Windows update the Device Manager flags cleared.

 

[mrpasc]

The Intel driver update tool has a huge list of exceptions. All those „chipset“ drivers are excluded for instance. In real it updates GPU, consumer NICs and WiFi/Bluetooth chipsets, that’s it.
See Intel Products That Aren’t Supported by Intel® Driver &... for details.

 

[SecCon]

Worth noting is that running a Linux distro via WSL is not to be confused with running it via Hyper-V.
Yes I tested both.

For some reason the Hyper-V Ubuntu chose to setup its own 172.* network, while it worked otherwise. Attempts to make it cosy inside my 192.* network via manipulation of the HyperV switch manager failed. Been here before, under other circumstances, and once that shitty thing start spitting out random hexadecimal error messages, you are better off just starting over.

On the other hand the WSL Ubuntu distro came, naturally, without DNS confed, so editing the /etc/resolv.conf adding manually, but did not help. Several commands, natural and necessary for Linux and Ubuntu, are also crippled, like systemctl or similar. Despite rebooting the WSL and rebooting the Windows, I was unable to get it to communicate with the Internet.

Remembers how this, and more, was up and running in a couple of hours with Proxmox ...

 

[Sean Ho]

Your described end-user workload is very standard for homelab: NAS, a few Windows VMs, and a couple dockers. It may be helpful to note that sync and backup are distinct tasks; consider restic, borg, veeam, etc.

For a NAS OS, I second the recommendation for Unraid; it has a free trial and a very easy docker management UI. TNC (FreeBSD-based) is also solid; I'd recommend using it just for storage management, while running dockers inside a Linux VM. TNS (Debian-based) is still very new, especially with the k8s/charts part. It'll get there, but give it time. PVE is great for VMs and ok for storage but doesn't provide the management UI that you're probably looking for.

For a container management UI, cockpit is fine; also consider portainer. I don't find management UIs useful; you'll eventually get around to a gitops CI workflow using docker-compose or k8s+argo/flux. But I can empathise with the ease of use of a UI when you're starting out.

Regarding ease of management (e.g., for continuity of service in case you're incapacitated), it's laudable to be thinking ahead. These systems are complex, and a big part of the sysadmin/devops skillset is being able to troubleshoot, self-teach what you don't know, and find solutions. It is impossible to code that into an automated, "self-healing" system; the field changes too quickly. What happens when a drive dies? When an update to nginx/traefik requires migrating ingress definitions to a new syntax? I would not expect a non-tech-savvy friend/relative to be able to step up to administer TNC or even Win11. If that is a requirement, then increase the budget and use MS OneDrive, GDrive, etc. -- outsourcing the admin load.

My disaster-recovery plan in case I get hit by a bus is to document everything for a target audience who can learn enough to at least extract our important documents / photos to be hosted elsewhere. My spouse gets the master password to my password manager. My family would likely sell off my homelab at pitiful prices; so be it.

 

[SecCon]

I always liked the FreeBSD approach, but as I see TrueNAS going away from that with its Debian based Scale distro, I wonder why... (I never - in 20 years -had a working Debian distro on any of my machines, dunno if I am cursed).

Did not want to get in to to much detail about what I would use to host apps I will be running in my landscape, but Portainer is kinda the most relevant candidate. Used it on and off for some time in my experiments.

PVE? Is that Proxmox you refer to or something else, since I consider Proxmox rather solid, but lacking native SMB support. Sure it can be added manually, but why is it not there in the first place? I have a situation where my main backup is on SMB on a NAS that has no support for CEPH or XFS (both available in Proxmox) so my hands are tied there. Unless I spend like 1000€ on new stuff.

Regarding ease of management (e.g., for continuity of service in case you're incapacitated), it's laudable to be thinking ahead. These systems are complex, and a big part of the sysadmin/devops skillset is being able to troubleshoot, self-teach what you don't know, and find solutions. It is impossible to code that into an automated, "self-healing" system; the field changes too quickly.

Are you sure you did not mean to write "laughable"

Yup, you are right about that, however there is no way around it. The way to go for that part, to retain a minimal service for files, is to go to Backblaze Online Backup / Filen – Next Generation End-To-End Encrypted Cloud Storage / End-to-End Encrypted Cloud Storage for Businesses | Tresorit or anything at max 15€ per month for a TB.

 

[Rand__]

Stupid question, but why don't you run Hypervisor + X ? I f you cant find a single OS that does it just use multiple?

Ie Proxmox + a linux vm for docker + a few windows vm's + truenas if u want for nicer storage + something like VeeAM free for backup?
I mean I'd use ESXi (free) for that but only since I know it.

I mean sure, letting a family member run Proxmox/ESXi might take a few days or some good screenshots, but basic operations (restart VM), reboot server, change autostart shouldnt be much of an issue if they are documented?

 

[SecCon]

@Rand__
I did use ESXi. I was also adamant in keeping it updated. Since I do not buy new servers every 3 years, I max them out on CPU as I get them anyways, that became more and more of a challenge due to VMWare's dwindling CPU compatibility. So I dropped it. I also had Veeam running for a little while between two ESXi based HPE servers but sold that off due to power bills. Had access to a lot of free stuff with my WMUG membership, that I also dropped at the same time.

Having said that I think I already use multiple OS's. Or at least two. Counting layers, three seems to be a minimum.

 

[Rand__]

Well keeping the OS updated and jumping on the latest versions are two pair of shoes ...
I mean sure I could throw away a win10 box cause it doesn't run Wn11 but why would I

At this point ESxi 7 has 2 years left and it should run on your box. Now 7 was a bit of an oddball that it was replaced by 8 instead of a subrelease as they did in 5 and 6 so it actually only has the 5 years of regular support and not 5+x as you had on older versions.

Anyhow, it does not need to be ESXi, u can do the same on another hypervisor? And you don't need to run Dual E5's to use a hypervisor either, nor a pair of them.

So, lets do a breakdown - your criteria are:

A - Initial acquisition cost, power bill, noise, heat
B - matching functional requirements - one box, multiple boxes, one os, multiple ones
C - management options - ease of use vs documented activities vs remote management (partially)

Maybe you need to start thinking from the User side down to the hardware and not the other way round?
I.e. What can the relatives(?) on their own, with proper documentation, with phone/remote/vpn support... then move on to what OS's that leaves u and so on.

I ran an ESXi box for years at a relative's place, vpn access for core issues, phone support for weird stuff, self managed (windows) for regular things. If all else failed I'd drive over (an hour by car, so not too big a deal. Also not 100% uptime needed).