I was thinking of building my own storage array. I was wondering who makes a DAE that wouldn’t be a pain in the butt to access firmware and other software? If you have any questions feel free to ask.
Who makes a good Disk Array Enclosure?
- Thread starter macrules34
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
What are you talking about? Disk enclosure
SAS (expander)? Sata? SATA
Tower? Rackmount? Rackmount
How many bays? 24+
SAS (expander)? Sata? SATA
Tower? Rackmount? Rackmount
How many bays? 24+
Forget about SATA...you do not want to run 6 (or more) cables between your host computer and your disk enclosure to connect 24 drive.
Get an SAS3 HBA and a disk enclosure with an SAS3 backplane, and connect the two boxes with just 2 cables. This will give you 96Gbps total throughput between the two boxes, which will support 48 spinning disks or 20 or so SSDs. You can use SATA drives in any SAS backplane made in the past 10 years.
Get an SAS3 HBA and a disk enclosure with an SAS3 backplane, and connect the two boxes with just 2 cables. This will give you 96Gbps total throughput between the two boxes, which will support 48 spinning disks or 20 or so SSDs. You can use SATA drives in any SAS backplane made in the past 10 years.
Two caveats:
1) Unless you get a significantly more expensive x16g3 or x8g4 HBA, your effective throughput ceiling, Best-Case (all-SAS3 drive usage [see below]), will be ~7GB/s (real-world#)
2) (regardless of HBA spec) Depending on the 12g:6g (ie, SAS3:SATA3/SAS2) drive mix, your effective throughput ceiling will/might be reduced to ~4GB/s (due to the effect of edge-buffering in SAS3 expanders)
Supermicro is decent, they offer jbod conversion kits for most of their normal server chassis like the 846 or 847
Netapp 4246 is also decent and cheap (you dont need to mess with any of the firmware, it is pretty much plug and play with any somewhat recent LSI HBA card and TrueNas, Proxmox etc) just make sure it has the IOM6 modules and not any of the FC / ONTAP ones
Netapp 4246 is also decent and cheap (you dont need to mess with any of the firmware, it is pretty much plug and play with any somewhat recent LSI HBA card and TrueNas, Proxmox etc) just make sure it has the IOM6 modules and not any of the FC / ONTAP ones
Yeah, do what the poster above says. You can take any chassis (Supermicro or otherwise) with an expander backplane and run at least a single SAS cable between your server and the JBOD box (or more for higher bandwidth). If the backplane doesn't have an expander, use an 82885T expander in the chassis between the upstream server and the backplane. You don't even need the JBOD controller card, you can ground the green wire on the 24-pin ATX power connector to turn on the PSU (but you'd only be able to turn it off or reset it by pulling the power on it).
Also, I'm just going to plug TheDanishGuy's OpenJBOD project as a nice alternative to the SuperMicro JBOD board: https://forums.servethehome.com/index.php?threads/custom-jbod-motherboard.45684/
ATX PSU control, PCI-e power slot, fan control, network connection, web interface...I have two and I'm quite happy with them.
ATX PSU control, PCI-e power slot, fan control, network connection, web interface...I have two and I'm quite happy with them.
Pretty much any adaptec or LSI unit supported by your OS of choice. You will need the correct cable; the netapp has QSFP.
This should work for SAS-3:
This is what you need for SAS-2
Buy a DS4246 instead, unless the DS4243 has IOM6 controller installed, no QSFP cables required. A DS2246 for 2.5" SAS2. If you want SAS3 look at HP 8000 for 3.5" and DS224C for 2.5"
@macrules34 - My humble opinion. Forget about an external DAS unit, if your motherboard/components etc will fit in a regular 4U type chassis.
External units are somewhat of a pain in the...on a day to day basis. Small examples.
- You wanna power down your server...so you do. Well, guess what? The disk array is still churning.
- And the reverse. You have to remember to power up the disk array before you power up your server, otherwise...
- And the cabling is well..additional. Some connector comes lose, because you were fixing something else in the rack and...
I think you see my .02.
Just get a regular Supermicro 4U 846/847, they're built like tanks, the parts are cheap-ish (they seem to have become a lot more expensive lately, but that's market cycles for you...) And the 847 can hold 36 LFF HDDs...that should last you a lifetime (unless you're like me and run PBs of storage...
)
I modded my 847s with the Norco fanwall (which is unobtanium now, but there's other choices), and 5x Delta 120mm fans (these are big, thick, heavy and very low noise at normal operating speeds), and these just sit in my rack and churn. Rock solid, with very little noise.
And in the rack (they're sitting above my Chenbro 48 bay chassis'):
With storage, it's almost always that, you'll end up needing more. Right now prices are a lil nuts, but that'll go away, market cycles and all that.
External units are somewhat of a pain in the...on a day to day basis. Small examples.
- You wanna power down your server...so you do. Well, guess what? The disk array is still churning.
- And the reverse. You have to remember to power up the disk array before you power up your server, otherwise...
- And the cabling is well..additional. Some connector comes lose, because you were fixing something else in the rack and...
I think you see my .02.
Just get a regular Supermicro 4U 846/847, they're built like tanks, the parts are cheap-ish (they seem to have become a lot more expensive lately, but that's market cycles for you...) And the 847 can hold 36 LFF HDDs...that should last you a lifetime (unless you're like me and run PBs of storage...
)I modded my 847s with the Norco fanwall (which is unobtanium now, but there's other choices), and 5x Delta 120mm fans (these are big, thick, heavy and very low noise at normal operating speeds), and these just sit in my rack and churn. Rock solid, with very little noise.
And in the rack (they're sitting above my Chenbro 48 bay chassis'):
With storage, it's almost always that, you'll end up needing more. Right now prices are a lil nuts, but that'll go away, market cycles and all that.
- You wanna power down your server...so you do. Well, guess what? The disk array is still churning.
- And the reverse. You have to remember to power up the disk array before you power up your server, otherwise...
- And the cabling is well..additional. Some connector comes lose, because you were fixing something else in the rack and...
Switched PDU fixes that problem. Before I had one I used cheap $15 sonoff smart plugs.
Never had a SAS cable come loose, just push till they click
I started with a 846 back in the dark ages when TAMS was selling them with opteron MB's. It was great until I filled all 24 slots with 14TB SAS drives, even with a fanwall with 120mm fans and a 3D printed cover for the front of the case holding 3 more fans I could not get acceptable temps for the drives. Moved drives to a DS4246 and was happy until I ran out of PCIe slots. I wanted to run 2 P40 GPU's for LLM use and could not get that to work in the 846. switched to a HP DL580 G9 and have all the space I need. So the 846 might work for some people but it depends on what you want your server to do.
- And the reverse. You have to remember to power up the disk array before you power up your server, otherwise...
- And the cabling is well..additional. Some connector comes lose, because you were fixing something else in the rack and...
Switched PDU fixes that problem. Before I had one I used cheap $15 sonoff smart plugs.
Never had a SAS cable come loose, just push till they click
I started with a 846 back in the dark ages when TAMS was selling them with opteron MB's. It was great until I filled all 24 slots with 14TB SAS drives, even with a fanwall with 120mm fans and a 3D printed cover for the front of the case holding 3 more fans I could not get acceptable temps for the drives. Moved drives to a DS4246 and was happy until I ran out of PCIe slots. I wanted to run 2 P40 GPU's for LLM use and could not get that to work in the 846. switched to a HP DL580 G9 and have all the space I need. So the 846 might work for some people but it depends on what you want your server to do.
Ya, so to touch a bit on what I’m trying to accomplish here, have an array with a redundant cluster with proxmox and ceph. Allowing one controller to fail and have the other pickup when that one fails. That’s why I want a DAE. Separate nodes to allow one to fail and I don’t loose storage connectivity (similar to an enterprise array).
It's not quite that simple. AFAIK Ceph supports multipath for a single OSD, but it doesn't have a means to say "this OSD exists on multiple hosts but only one should be active at one time", unless that's a very new feature that I'm not aware of. I'm not sure that Ceph is even what you'd want here, because you realistically want a minimum of 3 hosts, all with at least one OSD, so that you can use 3/2 host-level replication at a minimum. The point of Ceph is that you get seamless failover by way of a replicating the data across hosts, not by trying to ensure that individual disks are never inaccessible.
Okay, so essentially I want access to my storage no matter how many nodes are available, does proxmox and ceph do that? Just in a different manner than I’m thinking?