Supermicro 3u 16bay 3.5" chassis with SAS2 expander

canta

Well-Known Member
Nov 26, 2014
1,033
204
63
39
According to this thread, if you have one of the latest LSI controller firmwares you shouldn't have problems with large hard drives on SAS1 backplanes. Can anyone confirm this?
I would say good luck!!

I am done with SAS1 expander that gave me troubles :D
 

Cipher

Member
Aug 8, 2014
151
11
18
50
Hi Guys,

I just got a rack and while testing out the mounting this chassis with it's rails I noticed that the right top "hooks" on both outer rails at the front are bent in and won't sit in the square hole. The bottom hook in the front and the back two hooks are fine so it's only the top one for some reason.

Is this an easy fix, and if so, how would I bend these back out? I have no problem doing this myself, I just want to make sure I'm not going to break off the hook or warp some other part of the rail.
 

Dajinn

Active Member
Jun 2, 2015
512
78
28
30
Well, some would argue that they would want the SATAs directly connected to each port - no expander. I personally subscribed to this club (e.g. I want the option to do RAID) and got my two IBM M1015s that I flashed to the LSI firmware, ready for this day. That's 2x8 ports per card, 16 total.
I'm confused by the wording here, can you not do controller based hardware RAID with drives that are attached to an expander?
 

canta

Well-Known Member
Nov 26, 2014
1,033
204
63
39
Hi Guys,

I just got a rack and while testing out the mounting this chassis with it's rails I noticed that the right top "hooks" on both outer rails at the front are bent in and won't sit in the square hole. The bottom hook in the front and the back two hooks are fine so it's only the top one for some reason.

Is this an easy fix, and if so, how would I bend these back out? I have no problem doing this myself, I just want to make sure I'm not going to break off the hook or warp some other part of the rail.
post the pic...

last resort is contacting the seller to get a replacement :)
 
  • Like
Reactions: T_Minus

eduncan911

Active Member
Jul 27, 2015
100
44
28
65
eduncan911.com
eduncan911 said: Well, some would argue that they would want the SATAs directly connected to each port - no expander. I personally subscribed to this club (e.g. I want the option to do RAID) and got my two IBM M1015s that I flashed to the LSI firmware, ready for this day. That's 2x8 ports per card, 16 total.

I'm confused by the wording here, can you not do controller based hardware RAID with drives that are attached to an expander?

There are two ways you can "scale up" to 16, 24, 36+ drives:

* Directly attach each HDD/SSD to a single port on a HBA/Raid controller. E.g, a 1-to-1 mapping.
* Or, use an "Expander" which basically "shares" the bandwidth of 4x SAS/SATA ports across as many drives as you got.

When you are looking at these Supermicro chassis, you'll notice numbers like this:

SC836TQ
SC836EL1
SC836EL2
SC836A/B

Notice the last few letters/numbers. TQ, EL1, EL2, A/B, etc. Those denote what kind of backplane is in the chassis, and how you connect the drives.

"TQ" and "A/B" models means they have an 1-to-1 direct port mapping. If you have a 16 bay "TQ" chassis, that means you need 16 SATA/SAS ports somehow to connect to all of them! This is the 1-to-1 direct port mapping I was quoted in saying above. In a 1-to-1 port mapping to, say, a RAID card, you can do Hardware RAID all day long. No problem as you are using 1 port per HDD.

Now, that can get expensive to have 1-to-1 mappings. For a 16bay chassis, if you want to use all 16 bays with 16 HDDs, that means you need a 16-port HBA RAID card. $$$ (maybe 500+?) Another option is to have 2x 8-port cards, for a lot less money. This is what i did, and bought 2x IBM M1015 a few years ago. I flashed them to the infamous LSI "IT" mode as I didn't need raid. But, you do have the option to flash them to "IR" mode if you really want hardware raid.

Another option (sometimes cheaper) is to get a small 4-port SAS/SATA HBA controller card, and an additional 16-port "Expander" card like the Intel one people are most found of for around $100. That's two cards: one to act as the "head" and another to act as the "expander." The 16-port Expander card will have 4x SFF-8087 connectors, each connector drives 4-ports. Note: This is NOT a 1-to-1 mapping any longer. See below.

"SL1, SL2" all mean some type of "Expander" chip is built in. Or, you could go with the Expander card mentioned above. Either way, these use an "Expander" chip.

Here's where things get confusing with Expander chips. The first generations could only "expand" up to 4 times each port. So, a 4-port SFF-8087 connection could scale to 16 expanded ports, 4 multiplied by each port. Then they got better, now able to share the bandwidth and scale to 24, 36, 48, up to 255 devices - off of a single SFF-8087 connection (255 for SAS2 and above).

There are three major downsides to expanders chips:

* The OS does not see each HDD directly. Instead, the host OS sees whatever the Expander shares back to the HBA card. I am fuzzy on this part myself, as I haven't had an "Expander" before myself - until now. My main issue is HDD Spin Down no longer works. Well, it might for some good SAS expanders.

* Since the OS does not see/control each HDD directly any longer, nor does the HBA card itself as the Expander is controlling all the drives, you loose "hardware raid" abilities from the HBA card.

* You total bandwidth is limited to your HBA's and Expander's Gbps design:

- SAS1 or 3Gbps SATA2 = ~1TB/s max bandwidth for ALL DRIVES
- SAS2 or 6Gbps SATA3 = ~2TB/s max bandwidth for ALL DRIVES
- SAS3 or 12Gbps = ~4TB/s max bandwidth for ALL DRIVES

Typically this doesn't matter much with home servers, as you are only streaming from 1 or 2 disks at once. Even under heavy NZB uncompression, moving and file organizing + streaming, you are hitting maybe 4 or 5 disks at once. My Seagate 4TB Blue NAS drives get about 150 to 170MB/s. So even 5x 150MB/s is only 750MB/s total - that's not even hitting the limits of old old SAS1. The story quickly changes though if you move huge volumes (10+ TB) often across drives in some type of database access.

You start to see slow-downs in SAS1 (and even SAS2) when you look at SSD drives though. SSDs will easily peg a SATA2 connection at 280 MB/s (been there). And SSDs can easily peg an SATA3 connection at 560 MB/s (been there too). So, if you have say just 4 SSDs on a SAS1 controller, while the SSDs could see up to 2.2TB/s bandwidth, you are only going to see 1 TB/s. SAS2 controllers will still be pegged by just 4 SSDs - reading/writing at the same time. 12Gbps SAS SSDs do exist, they are for Enterprise and out of everyone's budget here for home servers. Don't even waste your time searching, you'll be a sad panda like me...

Again, most home servers have no where near that load. SAS1 is fine for most.

I plan on connecting a few SSDs for "first-in" copy operating on my StableBit Drivepool like I have now, in the hot swap bays because my LGA1366 Xeon mobo does not have native 6Gbps - but my IBM M1015 (flashed to LSI IT mode) do! In case you don't know this feature of StableBit Drivepool, you should check it out... All data copied to the "drive pool" can be directed to a single "fast" SSD. once it is copied, it will be moved back into the pool of archived 4TB HDDs under a specified time.
 
Last edited:

markarr

Active Member
Oct 31, 2013
410
110
43
One minor addition is that the A/B backplane isn't an expander its a TQ with mini sas connectors rather than individual sata ports.
 

neo

Well-Known Member
Mar 18, 2015
672
362
63
@eduncan911

Regarding TQ backplanes and connectivity, it is important to note that the E5 V3 chipset has 10 native onboard SATA ports. You'll only need 1 additional LSI IT mode card for connectivity. That's how mine is. When using SATA drives I prefer to have 1-to-1 connectivity.
 

eduncan911

Active Member
Jul 27, 2015
100
44
28
65
eduncan911.com
Thanks @neo.

Not only do the E5 V3 C12 chipset have 10 SATA ports, but they also are all 6 Gbps as well. ;)

Pass through looks to be the only way to spindown the drives. Today I hacked my IBM M1015s - flashed to LSI IT mode - INF drivers to allow for spindown in Windows Server 2012. It worked. But now I see that direct-connect seems to be the only way, not through a SAS Expander. Still, I am guessing.

As you pointed out, having a mobo with ample onboard SATA ports will allow the spindowns. Then, just the power usage of a single LSI HBA (even onboard HBAs use a lot of power too, remember) to drive the last 8 ports. That's only 18 total, which is fine for a SC836 chassis.

At this point, I'm trying to guesstimate the maximum power savings:

1 HBA card + SAS Expander backplane + 20 drives running 24x7
vs
3x HBA cards - 20 drives spun down (remember, spin-up is a Watt spike too)

Obviously, 3x HBA cards uses a lot of wattage. There's also the cost factor (I already have two HBAs).

TL;DR

Yeah, I've been considering that heavily myself - building a low-power media server monster, but be powerful enough for 7 Plex video transcoding streams (I've seen up to 7 concurrent streams on my system, I share a lot with my friends and families). I settled on one of the E5-2643s Engineering Samples on eBay right now for about $300 as it has to be low-budget. It has a CPU passmark of around 14,000 - that's insanely high compared to my current 6000 with my Core i7 950. Problem is that is an ES version, which is clocked lower. So you have to guestimate that CPU passmark would be around 12000.

Unfortunately while the Intel C612 chipset is awesome, with Haswell idle power savings and 9.4 Gb/s bus, it's the cost of entry that has barred it from me. So close... But i can't justify the extra $800 in cost.

DDR4 is just way too expensive, and those SM X10D mobos are on up there too.

I ended up with a Supermicro dual LGA1366 mobo with 2x L5640 Hex core CPUs. CPU Passmark of around 11,000 (14,000 with light overclocked). 11,000 CPU passmark is about 120W max usage (I only saw a delta of 92W), on both CPUs combined (each CPU is a max of 60W). While overclocking, i've seen around 122W delta being pulled from the wall for 14,000 CPU passmark. The mobo, 24gb LDRAM (low voltage) and both CPUs all ran me about $230 on eBay for everything shipped to door and just about matches the same power-to-watt ratio I need for transcoding videos.

In contrast, the E5-2643 is a 130W TDP chip and most reviews show a delta of 125 to 130W delta from the wall. But, I would save the power of one of the HBAs with that setup using direct-connect SATA ports, using the onboard SATA ports. But that's a $800 bet I am not willing to take.
 
Last edited:

JayG30

Active Member
Feb 23, 2015
226
46
28
34
Sorry, didn't read all the text above. However the expanders I've used (supermicro) present all disks to the controller and therefore the OS as individual drives. The expanders is also recognized and presented to the OS. Spindown is a function of the controller, not the expanders from what I understand. Spindown has worked fine for me.

I find SAS2 expander backplanes (E1 or E2) are nice due to simplificty of cabling, better airflow, and with SAS disks and an E2 backplane you can get multipathing (increased performance and reliability). Its also nice for connecting JBOD chassis to scale out for performance and fault tolerance.

Sure you can end up with contention with lots of high speed SSD's and heavy loads, but honestly don't see that as the norm. I think most companies that would have such an issue would scale out the storage and keep expanders instead. I see the TQ models more popular in home labs.
 

eduncan911

Active Member
Jul 27, 2015
100
44
28
65
eduncan911.com
Thanks for that JayG30!

Wow, so spindown does work through the Supermicro SAS expanders?

What exactly is your setup? HBA, operating system, drivers, etc?