Something better than Norco?

[Perry]

Our SAN, for a small video post production facility, uses two Norco enclosures. It's homegrown, and works well, except the enclosures are a problem. When we first built this setup a while back, one of the enclosures came from an auction and one for free from a friend of mine clearing out a storage space. Both worked well, until they didn't.

In January, during a scheduled update to the SAN, we swapped out both chassis for brand new ones. This was because one of them was dropping the connection to one drive, causing RAID rebuilds and performance problems due to failed backplane cards. 9 months later, and one of the new ones is doing the same. I've tried contacting Norco and it's been nothing but crickets. I'm done with them.

Looking for recommendations for a good chassis. I'm ok with a used one if it's truly high quality (HP, SuperMicro, etc). It must meet the following specs:

Both should be at least 4U rackmount (for better cooling - don't like 3U cases)

1) Chassis 1 should have room for an eATX motherboard and full-height PCIe cards
2) Chassis 1 should have room for 20 hot-swap 3.5" SATA 6gb drives
3) Chassis 2 should have room for 16 hot-swap 3.5" SATA 6gb drives

Chassis 2 is our expansion box. I would like to move everything over to the new hardware without having to change our RAID controllers, so our current setup uses one card that has 20 internal ports and one that has 16 external ports (that is 5x SAS and 4x SAS, using breakout cables). So we're looking to mirror what we have in terms of the physical layout, only with better hardware.

Suggestions? are there good name-brand servers that fit generic eATX motherboards that we might look for used? There are a ton of old servers on ebay and for sale locally on craigslist, but I really don't know much about them. I'm not afraid to tinker a bit if need be, the main thing is reliability.

Thanks!

 

[pricklypunter]

The ubiquitous Supermicro, or Chenbro, are the only two chassis manufacturers that I would rate. Comparing them is not really a fair comparison. Chenbro easily have the edge on price, but they are built a little less sturdy than the Supermicro's. Nonetheless, they are very, very good chassis for the money, just not particularly easy to find on the used market, though sometimes you can be lucky. I suspect that most folks that do buy them, keep them pretty much forever

The Supermicro chassis are a tad more expensive, but very, very well put together. It's a whole eco system you are buying into, with accessories galore and extremely good inter-changeability between chassis parts and power supplies, if you like a bit of DIY. There's no doubting that if you buy into Supermicro, you won't be disappointed.

There are lots of other chassis on the market, all built to a price etc, but none are as well thought out, or as well built or as reliable as the offerings from these two in my opinion

 

[Perry]

Thanks. I'm very familiar with Supermicro, and have been using their motherboards for years. In fact, this server uses one. My big concern with the JBOD version at least, is that the backplane seems to use a port expander, and that may mean we have to rebuild half of our 200TB SAN.

In our current setup, the second RAID card in the server has 4 mini SAS ports that connect to some pass-through SAS ports on the back of a Norco case. Internally these have SFF-8087 cables that run to each of the four backplanes. So it's basically a straight path from the card to the drives. My worry is that a Supermicro case that uses port expanders, which it seems their JBOD drives do (two ports for all 16 drives), might prevent the RAID card from seeing the existing setup. I'd like to find a solution that's electrically identical to what I have, just better, as that will let us do a quick transfer of all the drives and then we'll be back up and running. If we have to rebuild, we're looking at several days between backing up everything to big drives, building the arrays and then copying it all back up.

I have no familiarity with Chenbro, but I'm open if it's actually good. Do you know if their backplanes work like Norco, where 1 SFF-8087 breaks out to 4x SATA3 ports?

 

[pricklypunter]

You would be wanting the "A" version of the backplane for the Supermicro chassis, which is a non expander, passthrough backplane, which allows you to use mini-SAS cables. You can also get the TQ version of the backplane, which would be another option, and would allow you to continue using your 8087 breakout cables as you do now, again, it's a passthrough type. The Chenbro chassis also has the options of a passthrough mini-SAS backplane, or an expanded version. I use a 2U Chenbro, mini-SAS, passthrough version, at home and it has been absolutely faultless in 2 years of 24/7 use

 

[i386]

Supermicro +1

I want to try chenbro, but I can't find a proper source/retailer in germany :/

 

[gea]

I got my last Chenbro from ico.de

 

[JustinTeim]

Supermicro +1

I want to try chenbro, but I can't find a proper source/retailer in germany :/

Look here:
Chenbro 4U heise online Preisvergleich / Deutschland

 

[Perry]

The Chenbro 31616 arrived yesterday. I wired in an ATX power supply switch and started installing the drives. Noisy as hell, but it's in its own room so I guess that's ok. However, we're having similar issues to we had with the Norco: While the RAID is rebuilding, drives mysteriously disappear and cause the rebuild to fail. It got 39% of the way through today, then locked up. When I refreshed the drives in the Adaptec BIOS, there were some showing up as "inaccessible" - after rebooting, they come back up. Or removing and replacing them.

So either I have the worst luck with backplanes or something else is going on here. Any ideas? these are on an external box, so the stuff in the signal path is as follows:

Adaptec ASR78165
Mini SAS to SSS8088 Cables: https://www.amazon.com/gp/product/B00S6AT34A
SFF8088 to SFF8087 PCI brackets: https://www.amazon.com/gp/product/B00PRXOQFA
SFF8087-SFF8087 Internal cables: https://www.amazon.com/gp/product/B01KFEVQ4E

This stuff is all rack mounted and the cabling, etc is never touched. I've checked that all the external and internal cables are seated properly. Today I ordered new external cables to try to rule that out. I guess I need to order new PCI brackets and new internal cables too, just to be sure.

Is it possible this is the Adaptec card? that seems unlikely, but I guess you never know. This setup has worked flawlessly for 9 months until it didn't.

 

[pricklypunter]

It's not the backplane that's causing your issue. Make sure your power supply can handle the disk load, plus whatever else you are running on it. Don't make the mistake of just adding up the wattage's, look at the rail current required for both 5V & 12V. You really want about 30-50% headroom there to account for spin-up, little power anomalies, efficiency etc. Heat on the Adaptec card could be an issue for you, they do have a rep for running pretty warm, so make sure you have good airflow and the card is not getting too hot. It could also be flaky, but the only way to know that for sure is to catch/ observe it doing something it shouldn't be doing or to swap it out for a known good replacement. Obviously bad connections on wiring is another possible source of trouble, but the connectors themselves are self wiping contacts, so usually just mating and de-mating a couple of times is enough to clean them up. In your case, less likely the issue I think, unless the cables are physically damaged in some way. RAM issues are another common source of trouble, again, poor connections on the slots will give all manner of little issues as will faulty DIMM's. It's not always obvious either, but often becomes a real issue especially in RAID setups. Pretty much just about anything that can affect the normal operation of the computer/ server, is a suspect. Poor CPU contacts, poor thermal coupling on heatsinks, generally high temperatures, poor power supply regulation/ undersized supply and etc

On a side note, the standard fans in the Chenbro chassis I have were Denki (SanAce irc) and they have a certain whining tone to them when running full tilt, that is just unbearable to my ears. I swapped mine out for some Noctua's that I blast at full speed, it's almost inaudible now. So far I haven't observed my disk temps reaching critical levels, although they do run warmer than I would like in the summer months. I put that down to the lower static pressure of the fans. You should fair a bit better here, because you have the 3U variant and have a wider choice of fans and speeds to choose from, whilst mine is only 2U

 

[Perry]

It's not the backplane that's causing your issue. Make sure your power supply can handle the disk load, plus whatever else you are running on it.

The power supply in the Chenbro 31616 is what we're using. The *only* things this PSU is running are the 16 drives and the fans. There is no motherboard in the chassis. All of the drives are coming up when the system is booted, they're dropping while the system is running, after they all come up. Since the most power is used at startup when they all come on simultaneously, I find it hard to imagine that the PSU is underpowered considering the only things in the box are fans and drives.

Heat on the Adaptec card could be an issue for you, they do have a rep for running pretty warm, so make sure you have good airflow and the card is not getting too hot.

I've got a fan that I never installed on that card, which I'll look into doing today. As I mentioned in another thread here, we're not getting any warnings about overheating (that said, the Adaptec web interface is largely broken for us, and Adaptec has no answers as to why - it takes a minute or more sometimes for a button click to respond -- and this happens even when the system is powered on after being off all night. The BIOS doesn't report temps, as far as I can tell, so I have to go by the web interface. The logs don't indicate overheating, and I know that's something they report because when I first built this system and was running it with the top off the enclosure, it would quickly rise and give me a warning. Putting the lid on returned the cooling to where it should be, and temps rapidly returned to normal range.

Obviously bad connections on wiring is another possible source of trouble, but the connectors themselves are self wiping contacts, so usually just mating and de-mating a couple of times is enough to clean them up.

New external cables arrive today, just to rule that out.

RAM issues are another common source of trouble, again, poor connections on the slots will give all manner of little issues as will faulty DIMM's

Are you talking about RAM on the card or on the motherboard? bad RAM on the motherboard would probably also involve general system instability, which we don't have. Also, there's a second Adaptec card with its own 20-drive raid and it's just fine. No errors there. This is only happening on the external drives.

I'm planning on replacing the fans soon - these are 80mm, I think, but I need to look more closely at them because they're double the depth of normal fans, so not a generic drop-in for sure.

 

[pricklypunter]

I recently experienced power supply sag, causing random disk errors/ faulted disks after starting up and running for around 20 minutes. The Power Supply tested perfectly, but when I sat down and added up the current requirements on each rail, I discovered that the 5V rail was very close to it's rating. A slightly larger supply cured the issue. This was on a chassis that had ran for nearly a year trouble free

Yes I was meaning the RAM on the Mainboard. You are correct, if that were the issue, system instability often is the first sign, but in systems not using raid, it often goes unnoticed for a period, before it becomes obvious that there's something going on that is not quite right.

I wouldn't think it's a wiring issue, but new cables won't do any harm

Are the Adaptec cards identical? Can you temp swap them over to see how it fairs? You should be able to rule it out as the cause.

 

[Perry]

So we had the same problem with the old chassis and the new - they have different PSUs, and the Chenbro is designed for server use (and has 16 bays so I have to think it's up to the task - it could be defective, sure, but we also had issues with the previous PSU in the Norco case, and that makes me think it's something else).

The Adaptec cards are different - one is an internal RAID card and one is external, so there's no way to swap them, unfortunately.

 

[pricklypunter]

If you are sure that the power supply is good, and the head end is otherwise performing as it should and not showing other signs of trouble in your internal array, then that really throws the focus back on the Raid Card, running temperature of it or what's controlling it etc. I would say it's very likely the issue is in the head end, not in the Chenbro chassis.

I don't know which Mainboard you have, but if it is multi-CPU, is the external Raid Card on a slot running from another CPU? If so, it could still be a poor CPU contact/ bad RAM slot from that chip etc. Usually, the simple things are what get ya, so I would start fresh at this point. Give the whole chassis a wee clean and service, including fans, re-dress any wiring that may be impeding air flow, pull what you can and re-seat all power connectors, clean all edge connectors, pull the CPUs and clean those and their heatsinks and renew all thermal grease. I would also pull any peripheral heatsinks if possible and clean and renew their thermal grease too. After running it again for a bit, I might also be tempted to swap the RAM around as well, just in case it is a failing DIMM, see if any issues move with it. Standard maintenance stuff, but often catches those little niggley things that cause random issues

 

[Perry]

The motherboard is a SuperMicro X10SLR-F. Single Xeon CPU with stock intel heatsink/fan, tons of cooling in the case, all fans have been checked and are working correctly.

The inside of the case is spotless and there's nothing impeding airflow (all wiring is along the bottom and sidewalls of the chassis, zip-tied down. The chassis has 3x 120mm fans that suck air through the vents on the drive trays, and 2x 80mm fans on the back to push it out. The air coming out of the chassis is lukewarm at best, and there are no indications the CPU is overheating. If anything, I'd be more inclined to think it's the adaptec, since this is so specific to one set of drives, all on that card.

I can't locate the fan I bought for the Adaptec, which seems to have gone missing in our move to a new office last spring, so I've ordered another. I do have a PCI-slot blower fan that I'm going to throw in there. It lines right up with the heatsink for the raid card, just to see if it helps anything. And I suppose now's as good a time as any to install the BBU on the other card that I bought but didn't have time to put in.

 

[pricklypunter]

If you have a spare PCI-e slot, try moving the card, see if that helps

 

[Perry]

I don't think it's heat related, because drives are not appearing at boot even when the machine is starting from room temperature after having been off overnight. Regardless, I put in a PCI slot blower fan next to the card's heatsink. I also got a new set of cables to connect it to the expansion box. I also reseated all cables (power and data) for everything in the chassis from fans to SATA, removed and reseated the RAM and the PCIe cards, and re-applied CPU thermal paste. (Worth noting that CPU temps are totally normal - 36c), and the Adaptec software reports the RAID card temps as in the "Normal" range.

At boot, even cold boot, we have a drive that shows as missing, but when I put that drive in other slots it works fine.

After rebooting the machine Linux wouldn't start, giving me page after page of errors related to the adaptec card, on three attempted reboots. So I shut it all down, turned off the JBOD box (but left the cables connected) and rebooted. Linux booted with no issues.

Either the problem is the Chenbro chassis (unlikely since I had similar problems with the Norco), or the internal SAS cables - possible but again unlikely, since they were new in January and haven't been touched except to move everything to the new chassis, or the PCI slot External to Internal SAS brackets, or the Adaptec card.

I have some old SAS cables I can try today to see if that does anything. The SAS brackets are pretty damned simple passive connectors, but I suppose it's possible there's a bad connection somewhere. My money is on the Adaptec card though. I guess I'm ordering both a new Adaptec card and a new set of SAS brackets to be here Monday.

 

[Perry]

Quick update: replacing the internal cables made no difference. Still getting a drive showing up as "inaccessible" even though that drive is fine in other slots on the enclosure.

New Adaptec card won't be here until Tuesday, unfortunately. In the mean time, the PCI adapters and yet another set of internal cables will be here tomorrow AM. Hoping to test them over the weekend. But this is feeling more and more like an issue with the card.