If you use a DAS with sas multipath like the Netapp 4246 and hook it up to two servers both can technically access the disks, however depending on what level of redundancy you are looking for you just shifted your single point of failure to the DAS, if it goes out your data is offline, also sas multipath can be a huge pain on the software side / filesystem / write lock and keep in mind you must only use sas disks or if you have sata disks use interposers otherwise multipath wont work and the disks will only show up on one of the servers....so essentially I want access to my storage no matter how many nodes are available
"No matter how many" well if all of your nodes go down your storage is offline.... always....Okay, so essentially I want access to my storage no matter how many nodes are available, does proxmox and ceph do that?
Do your own research on ceph, but the super simple version is, each block of data is automatically replicated (copied) to as many nodes as you specified usually 3 (or it at least tries), there is a second value called "min replicas" which basically tells ceph if the actual number of currently existing replicas for a block of data is bellow min replica go into protection / read only mode.
Lets say you got 5 servers / nodes (each with 1 osd for simplicity in this example), set replicas to 3 and min replicas to 2.
Ceph writes each block of data to 3 of the nodes, lets say our test block is initially written to node 1, 2 and 3.
Oh no node 2 failed, we are down to two replicas now (since our min replica is 2 everything continues to work normally and data remains accessible) but ceph internally tries to restore the desired state (which is 3) so it starts to copy / replicate the data block to another available node (like node 4 or 5), after a few seconds to minutes the replication to the new node is done and we are in a healthy state again (with node 3 still offline)
Ceph is built for larger self healing clusters, if there is a failure but no nodes left to replicate / heal you run into issues,
3 nodes, Replica 3, min replica 2, 10g networking between the nodes is the absolute minimum, but performance wont be great and you can only loose a single node, 2 nodes must be online for data to stay writable.
Replica 3 / min 2 is fine for homelab but i would recommend not to go below 5 nodes to give ceph some room for replication,
also no matter what do not use consumer SSDs without plp with ceph, ever...
My recommendation, if you want HA storage, use ceph, if you set it up correctly your storage will basically never go down, even if multiple nodes fail, connecting a DAS to two servers only shifts the point of failure and is quite a pain in my experience.
Honorable mention if you want to go full enterprise HA, build out a full SAN solution like Netapp Metrocluster / Ontap, a lot of fun,
but rip your power bill (and licensing can be annoying)