After some suggestions for Chassis/Expander upgrade

[Eds89]

Hi guys.

I'm looking to increase my storage server capacity at home, which is on a Supermicro X9SRA motherboard, with an Intel E5-2618L v2, an LSI 9260-8i and 8 WD Red drives.

The desktop case it is in, has another 4 drive bays, but with the 9260 only having two internal ports, I need either another RAID controller or an expander. I tried an HP expander, but got unlucky, with it only negotiating SATA at 1.5Gbps.

I started looking at the LogicCase SC-4324S which I think is basically a Norco 4224, and then the Chenbro CK23601 36 port expander. The SC-4324 will give me plenty of drive bays moving forward, and the Chenbro should allow connection of all of them using just one port on the RAID controller.

Do you think this would be the simplest and cheapest way to get me onto the 24 bay ladder? Or, is there a chassis out there with a built in expander that might work out at a similar price? I'm in the UK, and options from the states are incredibly expensive when you take postage into account.

Thanks for any advice you guys might have for me!

Eds

 

[Terry Kennedy]

Thanks for any advice you guys might have for me!

1) Avoid putting SATA drives behind SAS expanders whenever possible. Bad Things can happen during error recovery.

2) Consider a controller with more ports. External ones seem to be cheaper than internal ones, so if you have the slots for it, get a slot filler with SAS connectors and loop the ports back in. This works because all SAS cables are straight-through (unlike the "forward or reverse breakout" SATA cables).

3) Consider adding a second controller, if you have the slots for it. Unless you need a single hardware RAID volume spanning all of your drives. But that's unlikely, given you have files on your existing volume - you'd probably create a new volume for the new drives.

4) Look around for decommissioned SAS drives - a lot of 4TB drives are getting upgraded to larger capacity drives here in NYC, and I expect the same thing will be happening in other areas with large datacenters. This can often be a case of "who you know" if you want to get things for the lowest cost - by the time they get to eBay they've been marked up a couple times. Example (from a few years ago):

 

[Eds89]

Hi Terry,

Thanks for the advice.

What kind of problems would I expect to encounter with SATA on SAS expanders? I did have a bit of a search on this scenario, but couldn't find much in terms of failures/issues. A lot of people confirmed their setups were working (nothing mentioned about error recovery operations).
Is it only on recovery I would expect issues, or in general operation as well? Is it likely to be dependent on workload, such as bandwidth or IOPS?

I will investigate other controllers, to see if this may be a more economical route to handle this. I already have some SAS SFF external/internal brackets which would fit that need.

I was originally thinking of having multiple large volumes handled by one controller, but I am probably going to rethink my volume structure a bit (storage is mainly for Films, TV, Music, Photo storage etc) so I would just go for multiple volumes and split across multiple controllers if it fits the bill.

Would you say any SAS drives are likely to be as good as/better than the WD reds for basic media server storage in RAID? Performance is less of a priority here, and capacity is the main concern. I suspect even second hand, enterprise SAS drives are going to be much more expensive(?). Seems the second hand market/availability in the UK is really quite poor, and most things would have to be shipped over from the states, and postage costs tend to make that prohibitive.

Many thanks
Eds

 

[Terry Kennedy]

What kind of problems would I expect to encounter with SATA on SAS expanders? I did have a bit of a search on this scenario, but couldn't find much in terms of failures/issues. A lot of people confirmed their setups were working (nothing mentioned about error recovery operations). Is it only on recovery I would expect issues, or in general operation as well? Is it likely to be dependent on workload, such as bandwidth or IOPS?

Here's the main issue - if you have SATA drives cabled directly to a SAS controller, the controller does the SAT (SCSI-ATA Translation). So the controller "knows" what is on its ports and can report appropriate status to your operating system.

When you put a SAS expander in there, the SAS controller is speaking SAS (not SATA) to the SAS expander, which is performing its own SAT. The controller doesn't know about this unless there's some very-tightly coupled stuff going on between them with SES, so the controller reports to your OS that a SAS drive is having some problem, when it is really a SATA drive. The expander chip is running its own, vendor-specific, firmware that you probably can't get (even if there were updates for it) and is thus an opaque "black box". When it does something, it often doesn't tell the controller what it is doing. A common issue with older SAS expanders talking to SATA drives was that when the SATA druve hung or took too long to complete an operation, the expander would go "Free bus resets for everybody!", resulting in a bunch of drives (not just the problematic one) going offline at the controller. And there goes your RAID set.

Would you say any SAS drives are likely to be as good as/better than the WD reds for basic media server storage in RAID? Performance is less of a priority here, and capacity is the main concern. I suspect even second hand, enterprise SAS drives are going to be much more expensive(?). Seems the second hand market/availability in the UK is really quite poor, and most things would have to be shipped over from the states, and postage costs tend to make that prohibitive.

I've been SAS-only on the stuff I build for some time now (aside from things like DVD drives) and have been very happy with them. Currently I'm using 8TB HGST He8 drives.

The thing to do is to try to find somebody that works at a datacenter, who can tell you when equipment is going to be scrapped. Normally the scrappers come in and take it away "as a favor" (WEEE and all that). Then the scrappers sort out the real junk from the resellable stuff, mark up the good stuff and put it on eBay. If you can get the equipment before the scrappers get to it, you may be able to get it for a good price or even free.

 

[raylangivens]

What kind of problems would I expect to encounter with SATA on SAS expanders?

I'd like to read more about that, too.
I have yet to encounter any issues with my expanders (HP), and I did run some stress tests which included multiple rebuilds of RAID5 arrays.
Also, many SAS/SATA backplanes are expanding backplanes, and they do run perfectly fine with SATA drives.

 

[Terry Kennedy]

I'd like to read more about that, too.
I have yet to encounter any issues with my expanders (HP), and I did run some stress tests which included multiple rebuilds of RAID5 arrays. Also, many SAS/SATA backplanes are expanding backplanes, and they do run perfectly fine with SATA drives.

The issue may be related to only a few expander chipsets (but there aren't that many vendors in that market to begin with). The fundamental problem is that you're "hiding" information from both the controller and the host OS driver, and that may be infornation they'd like to have.

Generally the problems happen at the absolutely worst time - when a drive is starting to go bad and is either hanging the expander port it is on or is generating incorrect bus transactions. As a way to try to deal with those, the expander often initiates a bus reset to one or more targets. Meanwhile, the host controller is upstream of all this and blissfully unaware that anything is happening, until it sees a bunch of drives go offline at the same time (and, hopefully, come back after a bit). But that isn't good for either controller- or host-based RAID.

Dell seems to understand this - if you insist on provisioning your PowerVault with SATA drives to save money, each drive caddy has its own single-port SAT adapter. So the backplane / expander / host controller go "we're all SAS here!" and are happy. This also has the useful (to Dell) side effect that the SATA drive + SAT interposer + special tray work out to be not much less than the equivalent SAS drive was in the first place, so they don't sell many of them.

 

[Eds89]

So the Chenbro uses the LSI SAS2X36 expander chip, and reading through the basic user docs: LSISAS2x36 (782 KB) it does mention SATA support, and more specifically, SAS/SATA enclosure applications.

I can't think of a scenario where a SATA controller would be hooked up to a SAS expander, so I would have expected it to have been designed with SAS controllers in mind even for SATA drives.

Would you say the same based on their specs and documentation, and would you imagine using this chip/card with my current LSI controller is more likely to play nice than anything else?

I don't want you to think I'm arguing the point with you, I just want to gauge the risk I would be taking if I went the expander route (As it still looks the cheapest and simplest to implement).

Thanks.

EDIT
Just re-read, you said chipset not chip.
I guess Chenbro's implementation of that chip on their expander will likely come into play as well.

 

[Terry Kennedy]

I can't think of a scenario where a SATA controller would be hooked up to a SAS expander, so I would have expected it to have been designed with SAS controllers in mind even for SATA drives.

No, SAS controllers need SAS expanders (which can have SAS or SATA drives connected to the expander ports). SATA controllers need SATA port multipliers (different thing) and can't use SAS drives anywhere. If I led you to believe otherwise, I was mistaken.

Would you say the same based on their specs and documentation, and would you imagine using this chip/card with my current LSI controller is more likely to play nice than anything else?

As I mentioned, SATA drives behind a SAS expander have limited visibility at the host controller level.

If you never have a drive hang the SATA bus or generate a timeout, you may never run into the problem. The point I'm trying to make is that if you do run into it, it will be at the absolute worst time, when you're trying to hang onto your RAID set.

System manufacturers have a better grip on what's possible, since they get more information from the controller / component manufacturers, can run modified firmware, and can supply their own OS-level drivers for whatever operating system(s) they support.

 

[Eds89]

No, SAS controllers need SAS expanders (which can have SAS or SATA drives connected to the expander ports). SATA controllers need SATA port multipliers (different thing) and can't use SAS drives anywhere. If I led you to believe otherwise, I was mistaken.

Not your mistake at all. I know SATA host devices cannot support SAS devices, just wasn't sure if a basic expander was an exception to that rule.

Ok understood, appreciate all your input.
Given the cost, I think I may go for the Chenbro expander with a cheap 4 port controller, and use it for my less important volumes.
For my more important volumes, I will go with a second controller.

Cheers

 

[raylangivens]

The issue may be related to only a few expander chipsets (but there aren't that many vendors in that market to begin with). The fundamental problem is that you're "hiding" information from both the controller and the host OS driver, and that may be infornation they'd like to have.

Yeah, maybe some details, but not the essential information for reliable operation.
Health monitoring works with every single HBA I threw at my expanders, management software sees all the same details it sees on SAS drives.

Generally the problems happen at the absolutely worst time - when a drive is starting to go bad and is either hanging the expander port it is on or is generating incorrect bus transactions. As a way to try to deal with those, the expander often initiates a bus reset to one or more targets. Meanwhile, the host controller is upstream of all this and blissfully unaware that anything is happening, until it sees a bunch of drives go offline at the same time (and, hopefully, come back after a bit). But that isn't good for either controller- or host-based RAID.

Even pretty old expanders like the HP 24 bay card don't do that anymore.
There may be expanders floating around, that do this, but that's no reason to generalize that to "don't ever use SATA drives on SAS expanders".
Yes, you do need to stress-test every setup with expanders thoroughly, but you should do that without expanders, too.

Dell seems to understand this - if you insist on provisioning your PowerVault with SATA drives to save money, each drive caddy has its own single-port SAT adapter. So the backplane / expander / host controller go "we're all SAS here!" and are happy. This also has the useful (to Dell) side effect that the SATA drive + SAT interposer + special tray work out to be not much less than the equivalent SAS drive was in the first place, so they don't sell many of them.

What Dell understands is that they can charge a premium on parts by suggesting problems / reliability issues.
Other major manufacturers can and do make SATA drives on SAS expanders chooch perfectly and reliably. Dell either can't (no way!) or doesn't want to, because $$$.

 

[SycoPath]

You guys are way overthinking this. In the SAS1 days, SATA did cause issues.

I've never seen anything in the SAS2 generation have issues with SATA. If my brain isn't tricking me, I remember digging deep into this issue, and the way a SAS controller handles SATA is encapsulation. The expander talks to the controller with -only- SAS protocols and just contains a SATA payload it delivers to the drive inside of a SAS command. The old single drive hanging the array problem was just poorly designed firmware and nobody does that anymore.