Enterprise SSD "small deals"

[BackupProphet]

It doesn't. I'm still looking for a decent Linux Backup program but haven't found one yet. . This is why my 2 backup servers are still Windows.

Backup shouldn't be complicated. Everything you need is covered with ZFS snapshots/send/receive + rsync + cron jobs. Personally I am quite happy with zfs-autobackup.

 

[kapone]

Oh yeah I'd be in a big **** if that happened. Same credential everywhere both for root and normal User . Once again tied to that Issue with Salt and lack of being able to deploy Credentials.

I could in Theory change everywhere except my Workstations, and store the passwords in Bitwarden, but then it's also how to unlock in that event where I got no SSH / Clevis access at boot. Entering 64 Characters via IPMI Web Interface isn't fun .

Immutable backups. That's the only way to protect yourself. They can't be overwritten or deleted (unless rotated).

 

[ca3y6]

But at the risk of stating the obvious, make sure the backup machine has different admin credentials than the rest of the network.

 

[kapone]

But at the risk of stating the obvious, make sure the backup machine has different admin credentials than the rest of the network.

True, but arguably, with immutable backups, it doesn't really matter. Once the backup is written, it is written in stone until you delete/rotate it. Even admin credentials can't modify it.

 

[ca3y6]

But that's the thing I don't get. There is nothing you can setup with root privilege that can't be removed with the same privileges. Assume a semi-competent malware takes control of your backup machine. What will protect those files? He can simply copy an encrypted version of those files locally then delete all backup data.

 

[kapone]

But that's the thing I don't get. There is nothing you can setup with root privilege that can't be removed with the same privileges. Assume a semi-competent malware takes control of your backup machine. What will protect those files? He can simply copy an encrypted version of those files locally then delete all backup data.

The backups would be on a hardened repo. As an e.g. with Veeam:

Hardened Repository - Veeam Backup & Replication User Guide

To protect your backup files from loss as a result of malware activity or unplanned actions, you can add a hardened repository based on a Linux server to your backup infrastructure. A hardened repository...

helpcenter.veeam.com

 

[Fritz]

In my search I found a program called Backups. It's based on Deja something or other. I'm in the process of backing up to a network drive as I type this. This is the first program I have found that actually works. And as a bonus it was stupid to set up. Most backup programs I've seen act as if there's no such thing as a network or just plain don't work as advertised. This one will also do incremental backups after the first which is what I want.

 

[Fritz]

I was looking for Restic when I found Backups. Restic is a CLI only program and I prefer a GUI.

 

[Fritz]

Looks like "Backups" only allows one backup. I can't see where a second or more backups can be set up. Stupidness like this makes Linux look bad.

 

[crlt]

I was looking for Restic when I found Backups. Restic is a CLI only program and I prefer a GUI.

Probably veering too much off topic but there are GUIs for restic such as Backrest.

 

[Mithril]

But that's the thing I don't get. There is nothing you can setup with root privilege that can't be removed with the same privileges. Assume a semi-competent malware takes control of your backup machine. What will protect those files? He can simply copy an encrypted version of those files locally then delete all backup data.

A) Most malware isn't that complicated. The stuff that looks for things on the network is going to be going after common things, off the shelf (both consumer and pro) systems, open SMB shares, etc.
B) Failing a vulnerability a remote system set up with a decent level of login protection, such as a combination of: SSH key only login, good passwords, MFA, no direct root, setting your "backup sync" users to not allow a full shell, not using the default SSH key or location on the client system, not having ANY of the remote login users be SUDOers, having the password to elevate to root/admin be DIFFERENT than any of the remote login passwords. Or the ultimate paranoia, admin login not possible from client system.
C) You are generally NOT going to deal with a direct hack or super sophisticated/targeted malware, the less "like" other people your system is the less things with impact you "fully". ZFS snapshots from a desktop replicated to another machine is going to be super niche for consumers, and somewhat niche for even business.
D) Your NAS is and always should be only PART of your backup solution. 3-2-1 rule or as close as you can get.

Heres a reasonably paranoid system layout that is fairly close to what I actually do:

1) Client systems ideally run ZFS and do snapshots and ZFS send to the NAS, this is managed with syncoid and sanoid where possible to automate snapshots, snapshot retention, and syncing. I have limited permissions delegated to the user that runs sanoid, and limited permissions for the syncoid user and the REMOTE syncoid user (no root user needed for any of it). The syncoid user CANNOT delete snapshots, datasets, etc. I have sanoid on the destination machine running a different retention pattern keeping the snapshor count down (dont need to keep hourlies for months)
2) on the nas root cannot SSH in, users can only SSH in with SSH pubkey pairs. The users that run automated backups via scripts (not the normal user accounts for me/humans) can't even get a normal shell and are not in sudors.
3) this system itself syncs to ANOTHER machine running ZFS that is "warm storage", it is off except once a week where it boots, pulls snapshots vs zfs send-recieve via a user that logs into the middle machine (again limited permissions, no sudo). this machine allows NO remote access, not even from the middle machine.
4) that 2nd "NAS" is not actually network attached. fiber optic direct between the 2, and fiber to the main switch but only outbound traffic to local on a vlan so it can report job status.
5) (not done yet) power the 3rd system exclusively from a DC-DC psu powered by a LFP battery that is only charged when the machine is disconnected (power surge/lightning isolation paranoia level)

 

[luckylinux]

But at the risk of stating the obvious, make sure the backup machine has different admin credentials than the rest of the network.

Yeah you make it sound so easy ...

I'd like to have a working System.

Even Bitwarden Password Manager is a PITA in its own Right ... Even when you are in the correct Folder, when creating an Item, you STILL need to type/select that Folder and after saving the Password, in order to copy it, you need to select ANOTHER Item, then back to the right one, otherwise the password isn't even available for copying.

The easiest Things (1-Step) get over-complicated for nothing into 3-4 Step Processes.

 

[luckylinux]

Looks like "Backups" only allows one backup. I can't see where a second or more backups can be set up. Stupidness like this makes Linux look bad.

I have the opposite Way. I didn't play much with restic yet, but as a general Rule, if you rely on a GUI-only Program, then you cannot easily script it.

If you can script it (CLI), then you can do whatever YOU want to do it, automate it, schedule, etc.

I like GUIs, I'm not a vim User etc (actually I use nano ), but for many Things the CLI is enough and more easy to configure.

Plus you can actually use the Program on a Server where obviously you don't have a GUI .

 

[luckylinux]

The backups would be on a hardened repo. As an e.g. with Veeam:

Hardened Repository - Veeam Backup & Replication User Guide

To protect your backup files from loss as a result of malware activity or unplanned actions, you can add a hardened repository based on a Linux server to your backup infrastructure. A hardened repository...

helpcenter.veeam.com

Not quite sure how that works.

They say Certificate based Authentication ... How is that different from just using SSH with Certificates (yes ... there is actually an Option to add Certificates to OpenSSH on top of using the Private & Public Key Pair).

Also in that Case, if your management Client gets taken over, then you will be able to connect to the Backup Machine and delete the Backup .

 

[mtg]

Not quite sure how that works.

They say Certificate based Authentication ... How is that different from just using SSH with Certificates (yes ... there is actually an Option to add Certificates to OpenSSH on top of using the Private & Public Key Pair).

Also in that Case, if your management Client gets taken over, then you will be able to connect to the Backup Machine and delete the Backup .

The only solution is WORM tape then

 

[alaricljs]

Backup server should be pulling from the clients and refuse inbound connections from everything except a non-client system... management or jumphost style. That box should also never touch or be mentioned on the clients. Different subnet is good too.

I too use zfs for all my backups and windows hosts store anything important on zfs backed samba. If that's not where it is, it doesn't matter enough and won't be backed up.

 

[ca3y6]

A) Most malware isn't that complicated. The stuff that looks for things on the network is going to be going after common things, off the shelf (both consumer and pro) systems, open SMB shares, etc.
B) Failing a vulnerability a remote system set up with a decent level of login protection, such as a combination of: SSH key only login, good passwords, MFA, no direct root, setting your "backup sync" users to not allow a full shell, not using the default SSH key or location on the client system, not having ANY of the remote login users be SUDOers, having the password to elevate to root/admin be DIFFERENT than any of the remote login passwords. Or the ultimate paranoia, admin login not possible from client system.
C) You are generally NOT going to deal with a direct hack or super sophisticated/targeted malware, the less "like" other people your system is the less things with impact you "fully". ZFS snapshots from a desktop replicated to another machine is going to be super niche for consumers, and somewhat niche for even business.
D) Your NAS is and always should be only PART of your backup solution. 3-2-1 rule or as close as you can get.

I am not going to present myself as an expert on malware, I am not, I am just an expert on the malware that hit me. It used a zero day in smartermail, which is a commercial mail service software, so had to be exposed to the internet to be useful, and not much that could have been done to dodge that bullet (I did patch immediately but that was too late). And we have heard recently that notepad++ was compromised. So I would not assume it only happens to others.

The money in ransomware isn't in consumer, it is in enterprise. They have money and insurance. So do not assume these malware will be unsophisticated and unaware of common enterprise storage features. They are designed to circumvent anti-malware protections. By running enterprise hardware in a homelab, you basically makes yourself a collateral damage of an attack not designed for you.

Building barriers between machines that have no reason to talk to each others on the same network is the one way I could have made this less painful.

 

[MSameer]

I'm not a vim User

<OT hat on>
Emacs <3
<OT hat off>

 

[BackupProphet]

I was looking for Restic when I found Backups. Restic is a CLI only program and I prefer a GUI.

This is the primary reason why you are struggling with Backup on Linux. GUI are much more complicated to create than a CLI, and most developers just want to maintain something simple.

Anyway, I am pretty sure you can VIBE code a GUI for Rsync with Claude.

 

[Fritz]

I'm still looking, being limited to one backup is profoundly stupid. as is the fact that it's so difficult with Linux and so stupid easy with windows. CLI is out of the question, tho it made be easy to code it's anything but to use. It's like groping in the dark and never really knowing if you got it right or not. Plus it's much easier to destroy your data with cli than it is with a gui. And typos and syntax errors will drive you insane. And finally, I'm of the opinion that it's a bad idea to put all your eggs in one basket by using backup software that uses an archive. And as for security and encryption, I don't need any of that and would rather not have to deal with it.