Who makes a good Disk Array Enclosure?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Electrified

New Member
Dec 29, 2025
6
2
3
....so essentially I want access to my storage no matter how many nodes are available
If you use a DAS with sas multipath like the Netapp 4246 and hook it up to two servers both can technically access the disks, however depending on what level of redundancy you are looking for you just shifted your single point of failure to the DAS, if it goes out your data is offline, also sas multipath can be a huge pain on the software side / filesystem / write lock and keep in mind you must only use sas disks or if you have sata disks use interposers otherwise multipath wont work and the disks will only show up on one of the servers

Okay, so essentially I want access to my storage no matter how many nodes are available, does proxmox and ceph do that?
"No matter how many" well if all of your nodes go down your storage is offline.... always....
Do your own research on ceph, but the super simple version is, each block of data is automatically replicated (copied) to as many nodes as you specified usually 3 (or it at least tries), there is a second value called "min replicas" which basically tells ceph if the actual number of currently existing replicas for a block of data is bellow min replica go into protection / read only mode.

Lets say you got 5 servers / nodes (each with 1 osd for simplicity in this example), set replicas to 3 and min replicas to 2.
Ceph writes each block of data to 3 of the nodes, lets say our test block is initially written to node 1, 2 and 3.
Oh no node 2 failed, we are down to two replicas now (since our min replica is 2 everything continues to work normally and data remains accessible) but ceph internally tries to restore the desired state (which is 3) so it starts to copy / replicate the data block to another available node (like node 4 or 5), after a few seconds to minutes the replication to the new node is done and we are in a healthy state again (with node 3 still offline)

Ceph is built for larger self healing clusters, if there is a failure but no nodes left to replicate / heal you run into issues,
3 nodes, Replica 3, min replica 2, 10g networking between the nodes is the absolute minimum, but performance wont be great and you can only loose a single node, 2 nodes must be online for data to stay writable.
Replica 3 / min 2 is fine for homelab but i would recommend not to go below 5 nodes to give ceph some room for replication,
also no matter what do not use consumer SSDs without plp with ceph, ever...

My recommendation, if you want HA storage, use ceph, if you set it up correctly your storage will basically never go down, even if multiple nodes fail, connecting a DAS to two servers only shifts the point of failure and is quite a pain in my experience.

Honorable mention if you want to go full enterprise HA, build out a full SAN solution like Netapp Metrocluster / Ontap, a lot of fun,
but rip your power bill (and licensing can be annoying)
 
  • Like
Reactions: mattventura

macrules34

Active Member
Mar 18, 2016
518
38
28
42
Thankyou @Electrified, that defiantly make sense and I think I’ll go with ceph and not try to do an enterprise style array. Can I present the storage as NFS shares with ceph of will I need something like TrueNas?
 

nabsltd

Well-Known Member
Jan 26, 2022
826
624
93
Unless you get a significantly more expensive x16g3 or x8g4 HBA, your effective throughput ceiling, Best-Case (all-SAS3 drive usage [see below]), will be ~7GB/s (real-world#)
First, 70Gbps isn't much lower than 96Gbps. It would mean that the "20 or so SSDs" I "back-of-the-napkin'd" would turn into 16.

Second, my "48 spinning disks" assumed max throughput near the outer edge. The real-world sequential average is much lower, so even 70Gbps is a pipe dream for those disks.
 

Electrified

New Member
Dec 29, 2025
6
2
3
Thankyou @Electrified, that defiantly make sense and I think I’ll go with ceph and not try to do an enterprise style array. Can I present the storage as NFS shares with ceph of will I need something like TrueNas?
yes, one option is to use cephfs (sort of separate of ceph the block storage / underlying part) or create a VM / Container that then hosts the nfs shares and is internally stored on the ceph pool itself