Enterprise SSD "small deals"

[Mithril]

But that's the thing I don't get. There is nothing you can setup with root privilege that can't be removed with the same privileges. Assume a semi-competent malware takes control of your backup machine. What will protect those files? He can simply copy an encrypted version of those files locally then delete all backup data.

A) Most malware isn't that complicated. The stuff that looks for things on the network is going to be going after common things, off the shelf (both consumer and pro) systems, open SMB shares, etc.
B) Failing a vulnerability a remote system set up with a decent level of login protection, such as a combination of: SSH key only login, good passwords, MFA, no direct root, setting your "backup sync" users to not allow a full shell, not using the default SSH key or location on the client system, not having ANY of the remote login users be SUDOers, having the password to elevate to root/admin be DIFFERENT than any of the remote login passwords. Or the ultimate paranoia, admin login not possible from client system.
C) You are generally NOT going to deal with a direct hack or super sophisticated/targeted malware, the less "like" other people your system is the less things with impact you "fully". ZFS snapshots from a desktop replicated to another machine is going to be super niche for consumers, and somewhat niche for even business.
D) Your NAS is and always should be only PART of your backup solution. 3-2-1 rule or as close as you can get.

Heres a reasonably paranoid system layout that is fairly close to what I actually do:

1) Client systems ideally run ZFS and do snapshots and ZFS send to the NAS, this is managed with syncoid and sanoid where possible to automate snapshots, snapshot retention, and syncing. I have limited permissions delegated to the user that runs sanoid, and limited permissions for the syncoid user and the REMOTE syncoid user (no root user needed for any of it). The syncoid user CANNOT delete snapshots, datasets, etc. I have sanoid on the destination machine running a different retention pattern keeping the snapshor count down (dont need to keep hourlies for months)
2) on the nas root cannot SSH in, users can only SSH in with SSH pubkey pairs. The users that run automated backups via scripts (not the normal user accounts for me/humans) can't even get a normal shell and are not in sudors.
3) this system itself syncs to ANOTHER machine running ZFS that is "warm storage", it is off except once a week where it boots, pulls snapshots vs zfs send-recieve via a user that logs into the middle machine (again limited permissions, no sudo). this machine allows NO remote access, not even from the middle machine.
4) that 2nd "NAS" is not actually network attached. fiber optic direct between the 2, and fiber to the main switch but only outbound traffic to local on a vlan so it can report job status.
5) (not done yet) power the 3rd system exclusively from a DC-DC psu powered by a LFP battery that is only charged when the machine is disconnected (power surge/lightning isolation paranoia level)

 

[mtg]

Not quite sure how that works.

They say Certificate based Authentication ... How is that different from just using SSH with Certificates (yes ... there is actually an Option to add Certificates to OpenSSH on top of using the Private & Public Key Pair).

Also in that Case, if your management Client gets taken over, then you will be able to connect to the Backup Machine and delete the Backup .

The only solution is WORM tape then

 

[alaricljs]

Backup server should be pulling from the clients and refuse inbound connections from everything except a non-client system... management or jumphost style. That box should also never touch or be mentioned on the clients. Different subnet is good too.

I too use zfs for all my backups and windows hosts store anything important on zfs backed samba. If that's not where it is, it doesn't matter enough and won't be backed up.

 

[ca3y6]

A) Most malware isn't that complicated. The stuff that looks for things on the network is going to be going after common things, off the shelf (both consumer and pro) systems, open SMB shares, etc.
B) Failing a vulnerability a remote system set up with a decent level of login protection, such as a combination of: SSH key only login, good passwords, MFA, no direct root, setting your "backup sync" users to not allow a full shell, not using the default SSH key or location on the client system, not having ANY of the remote login users be SUDOers, having the password to elevate to root/admin be DIFFERENT than any of the remote login passwords. Or the ultimate paranoia, admin login not possible from client system.
C) You are generally NOT going to deal with a direct hack or super sophisticated/targeted malware, the less "like" other people your system is the less things with impact you "fully". ZFS snapshots from a desktop replicated to another machine is going to be super niche for consumers, and somewhat niche for even business.
D) Your NAS is and always should be only PART of your backup solution. 3-2-1 rule or as close as you can get.

I am not going to present myself as an expert on malware, I am not, I am just an expert on the malware that hit me. It used a zero day in smartermail, which is a commercial mail service software, so had to be exposed to the internet to be useful, and not much that could have been done to dodge that bullet (I did patch immediately but that was too late). And we have heard recently that notepad++ was compromised. So I would not assume it only happens to others.

The money in ransomware isn't in consumer, it is in enterprise. They have money and insurance. So do not assume these malware will be unsophisticated and unaware of common enterprise storage features. They are designed to circumvent anti-malware protections. By running enterprise hardware in a homelab, you basically makes yourself a collateral damage of an attack not designed for you.

Building barriers between machines that have no reason to talk to each others on the same network is the one way I could have made this less painful.

 

[MSameer]

I'm not a vim User

<OT hat on>
Emacs <3
<OT hat off>

 

[BackupProphet]

I was looking for Restic when I found Backups. Restic is a CLI only program and I prefer a GUI.

This is the primary reason why you are struggling with Backup on Linux. GUI are much more complicated to create than a CLI, and most developers just want to maintain something simple.

Anyway, I am pretty sure you can VIBE code a GUI for Rsync with Claude.

 

[Fritz]

I'm still looking, being limited to one backup is profoundly stupid. as is the fact that it's so difficult with Linux and so stupid easy with windows. CLI is out of the question, tho it made be easy to code it's anything but to use. It's like groping in the dark and never really knowing if you got it right or not. Plus it's much easier to destroy your data with cli than it is with a gui. And typos and syntax errors will drive you insane. And finally, I'm of the opinion that it's a bad idea to put all your eggs in one basket by using backup software that uses an archive. And as for security and encryption, I don't need any of that and would rather not have to deal with it.

 

[lopgok]

I'm still looking, being limited to one backup is profoundly stupid. as is the fact that it's so difficult with Linux and so stupid easy with windows. CLI is out of the question, tho it made be easy to code it's anything but to use. It's like groping in the dark and never really knowing if you got it right or not. Plus it's much easier to destroy your data with cli than it is with a gui. And typos and syntax errors will drive you insane. And finally, I'm of the opinion that it's a bad idea to put all your eggs in one basket by using backup software that uses an archive. And as for security and encryption, I don't need any of that and would rather not have to deal with it.

Personally, I have 4 backup file servers. They are powered off expect when actively doing backups. If one gets corrupted, I have the other 3. They are all running linux and use mdadm raid 5.

 

[jode]

Personally, I have 4 backup file servers. They are powered off expect when actively doing backups. If one gets corrupted, I have the other 3. They are all running linux and use mdadm raid 5.

Same, except my backup servers rely on ZFS. Thanks to COW there is no question what blocks/files have changed and "incremental" backups are a matter of seconds vs. minutes.
The machines turn on automatically on a schedule and turn off after completed backup. This lowers exposure to compromise, but more importantly (to me) saves $$$ on the power bill.

 

[Fritz]

Same, except my backup servers rely on ZFS. Thanks to COW there is no question what blocks/files have changed and "incremental" backups are a matter of seconds vs. minutes.
The machines turn on automatically on a schedule and turn off after completed backup. This lowers exposure to compromise, but more importantly (to me) saves $$$ on the power bill.

How do you manage the on/off?

 

[mmo]

should we create a backup thread to discuss/share our best practice of backups?

 

[ca3y6]

How do you manage the on/off?

IPMI, to switch on and off a computer remotely and programmatically.

 

[Auggie]

should we create a backup thread to discuss/share our best practice of backups?

There should probably be a couple of backup threads. For redundancy.

 

[Greg_E]

Since the last deal was pages ago, probably time to start a new thread with the next deal.

 

[ca3y6]

CISCO UCS-SD38TBMS4-EV - 3.84TB £124

 

[jode]

How do you manage the on/off?

The cheapest and easiest is in the UEFI/BIOS power management option. i.e. boot computer daily. There are software packages which can be used to set/update the time of day.

You can use programmable IoT outlets and boot the computer from another one.

I have a "switched" PDU which is the lexus option to power on/off computers over a network.

... and yes, IPMI works, too

 

[Sacrilego]

How do you manage the on/off?

like ca4y6 said, ipmi.

I have a script in TrueNAS scheduled to turn on my backup server via IPMI periodically at night to replicate snapshots.
Helps me keep power usage down and provide me with some protection against hardware failures and ransomware if they somehow end up removing or encrypting all snaphosts on my main storage.

 

[jode]

Since the last deal was pages ago, probably time to start a new thread with the next deal.

MICRON DELL 08G76D 7450 PRO 7.68TB NVME E1.S (Best offer of $560 accepted)

 

[seany]

Same, except my backup servers rely on ZFS. Thanks to COW there is no question what blocks/files have changed and "incremental" backups are a matter of seconds vs. minutes.
The machines turn on automatically on a schedule and turn off after completed backup. This lowers exposure to compromise, but more importantly (to me) saves $$$ on the power bill.

This is what I do too. Main nas is 4x rz2 6 wide. Every 18mo the "smallest" vdev gets totally replaced with what ever id the current best $/tb sas on the used market. Extra disks replace whatever is smallest in the 2 backup nas (one local, one remote) that are 3x rz2 12 wide with no consideration for size in vdev matching. Backup machines are powered up and down via ipmi, remote unit has opnsense + tailscale on it... been working well for years.

 

[foureight84]

MICRON DELL 08G76D 7450 PRO 7.68TB NVME E1.S (Best offer of $560 accepted)

You'll have to let us know how hot this runs