Expander and Backplane stacking/cascading to facilitate a modular storage expansion plan ?

Allan74

Member
May 15, 2019
99
11
8
Is there a preferred connection strategy for Expander/Backplane Stacking/Cascading from a single RAID Card ?

If forced to chose, which of the following 2 connection strategies would you settle for ?
Parts count and cost are not an issue, as I have most of the required hardware already.

(As with many of my silly posts, it's MSPaint time once again - Click image for larger view)

post_Series-Parallel_small.jpg

This just seems to be the cheapest and easiest way to expand my Plex Library without getting too exotic.

Please take into account that this is simply a Plex Media Library. Everything Live is already backed up on old/cold storage.
Minimal concurrent Users (<4). Not mission critical. 99% Read Only. Single User (Me) Writing during low volume times to add items.
Data removal may happen in rare cases, but once written to the array(s), it will remain as part of the Read Only Library.

The image below represents my current add-on/expansion storage solution. I have rack space and this seemed to be the easiest way to add
more media storage if/when I require it, without breaking the bank in doing so. Far from an ideal solution, but it does what it was meant to.

post_SHELF.jpg

Each (roughly) 4U 'Array' consists of a Shelf, 3 fan-cooled drive cages (5 drives each), Expander, Powered PCIe Riser, Cables, Drives.
Not pictured, is the 16th drive (a 2.5" SATA SSD Cache Drive) and power switching solution (need to work on), as I am currently just
using a ground jumper wire on the PSU's 24pin, with control via the power switch on the back of the power supply, prior to booting the server.

The current running setup consists of the above 2U the Server and a single external 'Expander' shelf unit (pic above).
I will however be adding more storage (another complete shelf) in the near future and am wondering which connection strategy to employ.

It's not sexy, but it works (so far).
The Server itself contains the last remaining SAS drives that I own, the shelf is all SATA NAS drives.
With the Server itself and one 'shelf' up and running, I will likely add up to 2 more complete shelves.

I appreciate the advice, whatever it may be.
thanks.
Allan

ps.
Just to reiterate, this is simply a Plex Media Library. Everything Live is already backed up on old/cold storage.
Minimal concurrent Users (<4). Not mission critical. 99% Read Only. Single User (Me) Writing during low volume times to add items.
Data removal may happen in rare cases, but once written to the array(s), it will remain as part of the Read Only Library.
 

NateS

Active Member
Apr 19, 2021
126
66
28
Sacramento, CA, US
I would think you would want to choose based on which drives need the uplink bandwidth, or if they're all equally likely to be accessed, then I'd just split the bandwidth as evenly as I could. Everything plugged into each one of the HBA's ports is sharing that same bottleneck, but is independent of anything plugged into the other port.

In your first example, the one expander plugged into the external port of the HBA directly has the full bandwidth of that port, whereas everything else is sharing the other port. In your second example, the expander in your backplane gets full bandwidth and everything external shares the same port. Neither of these is really splitting the ports close to half and half, but I think the second option is a bit better. And since the load on these is light, maybe that's good enough.

One caveat though is that I hear sometimes SAS expanders don't like a mix of SAS and SATA downstream of them. I've never tried it myself, so I don't know if that's always a problem or just with a select few combinations of HBAs and expanders, but I'd recommend you try it before selecting a topology that mixes them. If that does end up being a problem, your second method may be your only option.

Of course, if you need more bandwidth than you can get from the available two ports, you could always get a bigger (or a second) HBA.
 

gregsachs

Active Member
Aug 14, 2018
429
126
43
The advantage of the parallel connection is that if an enclosure goes away due to power loss or whatever, you only lose that enclosure. I think 4x4k streams would be like 150MB/s, right; ie series would be unlikely to be get overloaded.
 

Allan74

Member
May 15, 2019
99
11
8
One caveat though is that I hear sometimes SAS expanders don't like a mix of SAS and SATA downstream of them.
While I can't comment on the most popular IBM 6Gb or HP 3Gb, the Lenovo branded 6Gb expander seems to have zero issues on HW RAID, being fed a single 4 lane connection in order to support an 8 drive SAS array and another 8 drive SATA array simultaneously following a couple months of testing.

I haven't had a chance to try an HBA, although I do have a 9217 4i/4e HBA that I may eventually give up my current 9280 4i/4e ROC for. I've just been using Windows and HW RAID for so many years on my home media server(s) that I fear making the switch to Linux/ZFS and an HBA. It's difficult to break a 10+ year bad habit....lol. I still have my 1st Areca 1220, but it's been LSI HW RAID following that card and to this day.
 

kapone

Well-Known Member
May 23, 2015
1,049
621
113
s there a preferred connection strategy for Expander/Backplane Stacking/Cascading from a single RAID Card ?
You're doing yourself a dis-service by doing this. RAID cards are not expensive, just add another one/more. The flexibility you'll gain from point to point connections is well worth it. The only downside is a slightly higher power draw due to the additional cards, but if you're at that point (in terms of disk shelves, HDDs) where you're cascading stuff, a slightly higher power draw is probably not a concern.

I have four disk shelves, each with 24x 3.5" drives. I started with an Adaptec 78165 (4x external SAS ports) and actually tried what you're trying. "Wide" (i.e. 2x SAS ports) SAS links to the first expander, and cascade from there. Long story short...just got another 78165 and now each disk shelf has 2x SAS ports connected to it. Now, I can power on/off any expander/disk shelf without worrying about any chaining, or worry about bandwidth.
 
Last edited:
  • Like
Reactions: dawsonkm and NateS

Allan74

Member
May 15, 2019
99
11
8
You're doing yourself a dis-service by doing this.
Fair enough.
While I have a healthy enough collection of HW SAS2108 based LSI cards to choose from, all are only 8 lanes per (twin 8087's).
I should be able to run a pair under the same Management Console, providing there isn't a driver conflict, as they won't match.

The 9280 4i/4e (shown in the pic above) will stay as my primary card, as it's already running.
My 12 bay Intel backplane only requires a single 8087 and the 9280's sff-8088 is already in use to the first external DIY disk shelf/expander.

Rather than adding an expander internally to the mix off the primary Raid card, I will add a 2nd 2 port card (even though both ports are internal) and run it out the back using the 8087/8088 bulkhead slot that I bought. This way every array will get it's own 4 lanes directly from a Raid card rather than piggybacking expanders. This will give me the ability to run 3 external connections and 1 internal independently.

I guess it will come down to whether the current 9280 4i/4e gets along better with a 9261-8i or a 9260-8i.
The 9260 requires a HW Key to enable advanced cache functions, like the 9280...whereas the 9261's only require an activation code.
I think the 9280 and 9260 may be a closer match, as they even share the same HW Function Key part#.

Unfortunately 4 discrete lanes per backplane is as far as I am going, no new cards, so I hope it works well in the end.
 
  • Like
Reactions: NateS

UhClem

Active Member
Jun 26, 2012
164
63
28
NH, USA
While I can't comment on the most popular IBM 6Gb or HP 3Gb, the Lenovo branded 6Gb expander seems to have zero issues on HW RAID, being fed a single 4 lane connection in order to support an 8 drive SAS array and another 8 drive SATA array simultaneously following a couple months of testing.
Have you checked prices on those Lenovo expanders lately? I bought 2 for $15 each, about 6 months ago. Price now ~$100! Note on the expander: you can run 5 output ports (for 20 drives). Also bought that same HDD cage--a nice budget solution.
 

Allan74

Member
May 15, 2019
99
11
8
Have you checked prices on those Lenovo expanders lately? I bought 2 for $15 each, about 6 months ago. Price now ~$100!
I just noticed that. I found an Amazon.ca listing for $30 CAD shipped (~$25 USD) and ordered a couple of the 'Lenovos' to keep as spares just in case.
I also still have a couple 'IBM' 6Gb originals, one of which still needs a firmware update to recognize my drives, but the Lenovo branded expanders seem to work right out of the box and all have common firmware....and all have worked so far.

Note on the expander: you can run 5 output ports (for 20 drives)
Yah, I tested that. Unfortunately I am only going to use 16 outputs per and keep a single Expander (fed a single 4 lane connection) per shelf. Up to 15 drives for the array (5 drive cage x 3) with the 16th connection going to a cache SSD. I am also a little gun shy of long build/rebuild times, so I like to deploy smaller HDDs, as I only run a RAID5. I have a bunch of SAS 2TB Seagate Constellations as well as 3TB IronWolf SATA drives and have just built the last array using 4TB IronWolf SATAs. While I do have larger drives, I mainly use them as Cold Storage/Backups.
The extra 4 connections will be perfect candidates to connect large single drives to, to back up each array, then unplug to be kept as cold storage, which is what I normally do with all the media that I keep online in the Plex setup. Again, it's convenience, not mission critical.

Also bought that same HDD cage--a nice budget solution
I like how 3 of them fit nicely and can be secured on a standard Rack Shelf. The drive spacing in each cage is a bit tight, even with the fan, but I am using IronWolf drives that barely break room temp, even under heavy loads. I wouldn't want to load them up with 7200RPM SAS drives unless I had a 120mm 'finger-tip-eating' Delta fan in front of each cage...lol
 

Allan74

Member
May 15, 2019
99
11
8
Here they are on Amazon.ca

I was able to purchase them previously for about $15-$20 USD each/shipped, but the price has since gone up for North American listings.

You may want to try Aliexpress, as they are always cheaper over there.

As far as a part#, none really exists. They are however the ONLY 5-drive cages (with 120mm fan mount and drive rails) I found, whereas most of the China OEM units are 4-drive.