Supermicro SAS backplate, LSA 2208 and working with ZFS

[zecas]

Hello and Happy New Year to Everyone,

I'm going to put my hands on supermicro hardware to make a virtualization server, and I have a question regarding proper using it for ZFS.

So first the hardware:

motherboard: X9DRH-7TF
Supermicro | Products | Motherboards | Xeon® Boards | X9DRH-7TF
it contains:
SAS2 from Broadcom 2208 (1GB cache) - SAS 0~3 + SAS4~7
SATA 3.0 - I-SATA 0/1
SATA 2.0 - I-SATA 2~5 + S-SATA 0~3


disk backplate: BPN-SAS2-826EL1
https://www.supermicro.com/manuals/other/BPN-SAS2-826EL_1.0.pdf
similar to the picture:
https://c1.neweggimages.com/ProductImage/V19B_1_20190624500338554.jpg
Connectors: 3 power connectors + 2 SAS connectors
Backplate to 12 SAS/SATA disks

chassis: SSG-6027R-E1R12T
Supermicro | Products | SuperServers | 2U | 6027R-E1R12T
Contains 12 front hot plug 3.5" caddies, 3 lines * 4 columns


So my questions are:

1- The backplate only has 2 SAS connectors, the board only have 2 SAS connectors. This is a basic question but how many disks are possible for each SAS cable? 6 SAS disks, or only 4 SAS disks and I will not have the 12 caddies working?

2- If I connect the SAS cables to the backplate, then connect the disks on the caddies, they should be controlled by the LSI 2208 chip. I know that for ZFS it is recommended direct access to the disks, with no RAID or anything, so will it be possible to configure the LSI 2208 so it provides direct access, with no RAID or anything?
For instance I believe that I could create in a raid controller a logical volume for each disk, but that wouldn't be the case here, correct? If I connect a disk it will be by default directly accessible, right? (like when I connect a SATA disk)

3- Backplate power connection distribution is associated with a line of caddies?

The backplane has 3 power connectors, anyone knows if normally it would be 1 connector to power each "line" of caddies? I didn't find any details yet on the manual.
I would have interest in connecting disks on each "line" of caddies in a way I can minimize impact on a ZFS pool of mirrors if one power is lost, i.e., one line of caddies goes down. That's why I'm interested in that detail.

4- What SAS cables do you recommend for the job? I have to buy a pair of them, so I'm interested in knowing what properties to look for on a good cable. Will it depend on SAS speed of the controller?


Thank you.

 

[Rand__]

0. Its backplane, not backplate
1. The backplane has a SAS expander built in (-826EL1) which means it will multiplex all drive slots into the (one or two cables x4 lanes = 4 or 8 ) SAS connections.
2. Yes you should flash the SAS chip to IT - not entirely sure if that works with this board if it has onboard cache memory, sorry. Else a similar HBA should be available for <$50
3. No, there is no mapping, it will work also with 2 molex connectors if need be. Redundancy is provided by the Power distribution unit which will supply all 3 power cables by one of the two power supplies
4. Usually any branded ones will do fine, Supermicro themselves offer some in their online shop for example (which means you could get the matching length from the part list: CBL-0108L-02)

 

[BeTeP]

So let me get it straight, you googled the backplane part number, found the manual but instead of reading it yourself - you linked it in your post and asked us to read it and write an executive summary for you?

 

[Rand__]

lol

 

[BeTeP]

1. The backplane has a SAS expander built in (-826EL1) which means it will multiplex all drive slots into the (one or two cables x4 lanes = 4 or 8 ) SAS connections.

-826EL1 is a "single port" version of the backplane. You only connect 1 SAS port to an HBA. The second port is for daisy-chaining another backplane.

2. Yes you should flash the SAS chip to IT - not entirely sure if that works with this board if it has onboard cache memory, sorry. Else a similar HBA should be available for <$50

Only 2008 and 2308 IoCs have official IT and IR firmwares. Crossflashing back and forth is pretty fail-proof and officially supported by manufacturer. But SAS2208 is a RAID SOC. There is no official IT firmware for it. What people do is flash SAS2308 (it shares core design with SAS2208) IT firmware on to these. The procedure is less straightforward and unsupported. For a beginner I would recommend just getting another HBA for like $20.




As for the cables - I just go to ebay, search "SFF-8087 lot" in the "Computer Cables" category, apply "new" and "US only" filters, sort by ascending price and look for OEM part numbers in the titles.
I have not paid more than $2 (per unit in bulk orders of 10+ cables shipped) for a brand name brand new internal mini SAS cable in a very long time.

 

[Rand__]

-826EL1 is a "single port" version of the backplane. You only connect 1 SAS port to an HBA. The second port is for daisy-chaining another backplane.

True

This also explains why there was only one SAS cable on the parts list

 

[zecas]

0. Its backplane, not backplate

eheh thanks for the correction

So let me get it straight, you googled the backplane part number, found the manual but instead of reading it yourself - you linked it in your post and asked us to read it and write an executive summary for you?

Please don't go that way. I know there are lots of people in the forums that wait for the community to do the hard work, but I don't consider myself as such. I've been looking for info around the subject just to make an educated choice and these questions were poisoning the solution I had in mind. This is my first server build, so I want to make it right.

When I post questions I normally go to the trouble of putting links to official sites with info or specs (and to serve me as a quick reference). For this backplane part, it just happened that the result pointed to a PDF, that's why I also added a picture just so to illustrate the point of my question.

1. The backplane has a SAS expander built in (-826EL1) which means it will multiplex all drive slots into the (one or two cables x4 lanes = 4 or 8 ) SAS connections.
2. Yes you should flash the SAS chip to IT - not entirely sure if that works with this board if it has onboard cache memory, sorry. Else a similar HBA should be available for <$50
3. No, there is no mapping, it will work also with 2 molex connectors if need be. Redundancy is provided by the Power distribution unit which will supply all 3 power cables by one of the two power supplies
4. Usually any branded ones will do fine, Supermicro themselves offer some in their online shop for example (which means you could get the matching length from the part list: CBL-0108L-02)

-826EL1 is a "single port" version of the backplane. You only connect 1 SAS port to an HBA. The second port is for daisy-chaining another backplane.

Only 2008 and 2308 IoCs have official IT and IR firmwares. Crossflashing back and forth is pretty fail-proof and officially supported by manufacturer. But SAS2208 is a RAID SOC. There is no official IT firmware for it. What people do is flash SAS2308 (it shares core design with SAS2208) IT firmware on to these. The procedure is less straightforward and unsupported. For a beginner I would recommend just getting another HBA for like $20.

As for the cables - I just go to ebay, search "SFF-8087 lot" in the "Computer Cables" category, apply "new" and "US only" filters, sort by ascending price and look for OEM part numbers in the titles.
I have not paid more than $2 (per unit in bulk orders of 10+ cables shipped) for a brand name brand new internal mini SAS cable in a very long time.

True

This also explains why there was only one SAS cable on the parts list

I now finally understand the cable management involved, it also helps to clarify some information on the manual and it's nice that a single cable is sufficient to manage the 12 disks.

I would prefer not to mess with the LSI 2208 firmware on the motherboard chip, so I guess I'll try searching for an HBA SAS controller in IT mode or that will allow me to easily flash to IT mode. I'll search for those 2008 and 2308 chips refereed, for a start.


Thanks

 

[svtkobra7]

Hello and Happy New Year to Everyone,
motherboard: X9DRH-7TF
disk backplate: BPN-SAS2-826EL1
chassis: SSG-6027R-E1R12T

have same exact config x 2
2 x 12 x 10TB WDC WD100EMAZ-00WJTA0
2 RaidZ 3x4 pools, but if one server wasn't continuously replicating to the other, I'm not sure I would favor the increased risk of RaidZ
That mobo is my fav X9.

So let me get it straight, you googled the backplane part number, found the manual but instead of reading it yourself - you linked it in your post and asked us to read it and write an executive summary for you?

• LMFAO
• He is helping you help him ... nice guy it seems.
  
• Here is a funny for you - my post #1 - wherein I thought it would be a good idea to blow air off the mobo over the HDDs - https://forums.servethehome.com/ind...g-supermicro-fan-reversal-venting-room.13923/
• @Rand__ Come on and laugh, you know you want to.
• I still can't believe I went from that inquiry to ripping out the sheetrock, reinforcing the metal studs with 2x4s and 2x6s, installing Single Wood Stud Interior Wall Insulation | ROCKWOOL, an active exhaust on the ceiling for heat extraction, mounting the servers vertically @$$ up (to align with airflow through the closet), and the gap under the door pulls in enough air, with everything remaining in balance. Heck the plywood panel is held in place via a french cleat and gravity, given 30 minutes, and a screwdriver the only evidence that it was ever a server closet would be three or four easily patched holes where the wall side of the cleat is fastened to the blocking behind the sheetrock.
  
• (and no, I've never dropped a HDD - but yes, it is a PITA to service the one against the wall)

-826EL1 is a "single port" version of the backplane. You only connect 1 SAS port to an HBA. The second port is for daisy-chaining another backplane.
Oprocedure is less straightforward and unsupported. For a beginner I would recommend just getting another HBA for like $20.

• meh - HBA takes up a PCIe slot - and he already has what he needs on board - I'd take the $20 and buy a nice cable that you can fashion into cool shapes.
  
• (not that I disagree with a conservative advice in this scenario)
  
• I flashed mine to 9207-8i (actually first time I've needed to do so) and have had no problems
• Flashing LSI 2208 with IT firmware to use as an HBA - My Wired House
  
• It is really only when I take this guy's advice @Rand__ that I end up borking ZFS pools

• I like these - https://www.amazon.com/gp/product/B017CO6JRO - for those of us who are OCD - we can fold them perfectly - increasing airflow and ZFS performance (jk)

 

[svtkobra7]

eheh thanks for the correction
Please don't go that way. I know there are lots of people in the forums that wait for the community to do the hard work, but I don't consider myself as such.

• I think he was joking - to me at least - I followed the implied purpose.

I would prefer not to mess with the LSI 2208 firmware on the motherboard chip, so I guess I'll try searching for an HBA SAS controller in IT mode or that will allow me to easily flash to IT mode. I'll search for those 2008 and 2308 chips refereed, for a start.

• It is so easy though. My Schnauzer actually took over and flashed the 2nd one for me
  
• FYI - you'll be up and running before you can screw 12 HDDs into the caddies
• As an analogy, its like not using the 10 Gb NICs on board, but having the infra in place to do so
• If you screw up there is even a lsi2208fixer.zip floating about - that well - fixes your 2208
• And if you somehow managed to bork your controller (you won't), just buy an HBA (I'm a minimalist)

 

[Rand__]

@Rand__ Come on and laugh, you know you want to.

Nah , I think it has finally worn out

It is really only when I take this guy's advice @Rand__ that I end up borking ZFS pools

 

[i386]

1- The backplate only has 2 SAS connectors, the board only have 2 SAS connectors. This is a basic question but how many disks are possible for each SAS cable? 6 SAS disks, or only 4 SAS disks and I will not have the 12 caddies working?

12 devices. They connect to the expander on the backplane and the expander is connected to the hba/raid controller.

4- What SAS cables do you recommend for the job? I have to buy a pair of them, so I'm interested in knowing what properties to look for on a good cable. Will it depend on SAS speed of the controller?

I prefer the supermicro sas cables becuase they are more flexible/not sturdy as some cheaper no-brand name cables.

-826EL1 is a "single port" version of the backplane. You only connect 1 SAS port to an HBA. The second port is for daisy-chaining another backplane.

It could be used for daisy cahinning OR for dual linking to the controller

 

[zecas]

have same exact config x 2
2 x 12 x 10TB WDC WD100EMAZ-00WJTA0
2 RaidZ 3x4 pools, but if one server wasn't continuously replicating to the other, I'm not sure I would favor the increased risk of RaidZ
That mobo is my fav X9.

• meh - HBA takes up a PCIe slot - and he already has what he needs on board - I'd take the $20 and buy a nice cable that you can fashion into cool shapes.
  
• (not that I disagree with a conservative advice in this scenario)
  
• I flashed mine to 9207-8i (actually first time I've needed to do so) and have had no problems
• Flashing LSI 2208 with IT firmware to use as an HBA - My Wired House
  
• It is really only when I take this guy's advice @Rand__ that I end up borking ZFS pools

• I like these - https://www.amazon.com/gp/product/B017CO6JRO - for those of us who are OCD - we can fold them perfectly - increasing airflow and ZFS performance (jk)

• I think he was joking - to me at least - I followed the implied purpose.

• It is so easy though. My Schnauzer actually took over and flashed the 2nd one for me
  
• FYI - you'll be up and running before you can screw 12 HDDs into the caddies
• As an analogy, its like not using the 10 Gb NICs on board, but having the infra in place to do so
• If you screw up there is even a lsi2208fixer.zip floating about - that well - fixes your 2208
• And if you somehow managed to bork your controller (you won't), just buy an HBA (I'm a minimalist)

Yes, I understood the purpose , but I also took the chance to point out that I did my research ... buuuut ... I have to resort to the community to get opinions from people that deal with this everyday. Just to be sure where to spend the cash and not ending up with stuff that doesn't work as I planned. I may have been a bit too serious on my comment, though ...

I'm glad to find someone with this config I'm putting together, means this is proven to work

I don't have high requirements for storage space, so I was thinking of putting proxmox on 2x SAS 10k 75Gb 2.5" disks (zfs mirror), then a zpool with 2x vdev mirrors:
- zdev-1 (mirror) : 2x WD Ultrastar 1Tb (SATA3 7200rpm 3.5")
- zdev-2 (mirror) : 2x WD Ultrastar 1Tb (SATA3 7200rpm 3.5")
(other specs: 2x E5 2630v2, 128Gb DDR3)

And going SAS to take advantage of the backplane/caddies and use those 2x SAS disks I have around (since it would be a SAS connector). The requirements are not as high as most of you guys, that setup will give me 2Tb total space and 1 drive failure per mirror (I would prefer 2 drives failure per mirror, if budget permits I may go with 3 disks per mirror).

Flashing hardware was not what I had in mind initially, since something can go wrong and normally it's not easy to revert to the original status. Looking at the comment Flashing LSI 2208 with IT firmware to use as an HBA - My Wired House I found someone with a problem using this mobo and backplane, but it may be an isolated case. I'll be digging more info regarding flashing to IT mode.

I'm also searching for an alternative HBA SAS card, sure I would sacrifice a PCIex8 slot but it is still a possible alternative. There are many around on ebay, almost all from china, land of tech . This component will directly work with my data, so if there's one component that must be working correctly, this should be it so I must get a trusty unit. Found a LSI 9207-8i that does not have raid, so it would be a simple HBA to connect to SAS backplane, but seems a bit pricey for what it is (80£, around 95€) https://www.amazon.co.uk/LSI-SAS-9207-8i-8-Port-6Gbps/dp/B00AENN2LW (for the price I almost feel it would be best to flash)

12 devices. They connect to the expander on the backplane and the expander is connected to the hba/raid controller.

I prefer the supermicro sas cables becuase they are more flexible/not sturdy as some cheaper no-brand name cables.

It could be used for daisy cahinning OR for dual linking to the controller

Found a CBL-0109L-02 (22cm) for around 35€, which is also listed in the backplane manual, couldn't go wrong with that. There are cheaper 15€ alternatives, but I have to trust the cable to buy, not sure I would trust a cheap one for my data.

I also liked the one posted by svtkobra7 (from UK: https://www.amazon.co.uk/10Gtek-SFF-8087-0-5-Meter-Packing-technology/dp/B01DCZCRXW) for around 30€.

 

[Rand__]

If you are in EU I can send you a cable, I've got plenty to spare - almost not using SAS2 any more.

Edit: Mind its not one of these fancy ones @svtkobra7 was recommending, but should have some Supermicro left over if you're good with those - just need to know which length you need.

 

[zecas]

If you are in EU I can send you a cable, I've got plenty to spare - almost not using SAS2 any more.

Edit: Mind its not one of these fancy ones @svtkobra7 was recommending, but should have some Supermicro left over if you're good with those - just need to know which length you need.

Thank you very much for your offer, I'll keep it in mind . I'll have to put my hands on the hardware to be sure what I need, but from what I learn on this thread, I'll only need one and just be confident on its quality, more than the looks.

But out of curiosity, what are you guys using now if not SAS for disk connections? Moving to PCIe SSDs?

 

[Rand__]

SAS3 and NVME mostly

 

[itronin]

• I like these - https://www.amazon.com/gp/product/B017CO6JRO - for those of us who are OCD - we can fold them perfectly - increasing airflow and ZFS performance (jk)

those 10Gtek flat 8087 cables are AWESOME! You can also periodically find them on the bay for about $5 each shipped. often listed as 3M SAS...

 

[svtkobra7]

those 10Gtek flat 8087 cables are AWESOME! You can also periodically find them on the bay for about $5 each shipped. often listed as 3M SAS...

Finally somebody gets it! You = my man!

 

[SRussell]

True

This also explains why there was only one SAS cable on the parts list

I read the documentation for SC216, SC847, SC846 and the SAS2-EL1 backplane and I never knew the 1 stood for only one connector.

This is why I spend so much damn time on the board... I learn something new everyday!

 

[SRussell]

SAS3 and NVME mostly

How much NVMe are you using in your lab?

I never know if people are speaking about production at work or at home labs.

 

[svtkobra7]

How much NVMe are you using in your lab?

#subscribed !
#askthealternativetoo_howmuchisundeployed

I never know if people are speaking about production at work or at home labs.

• serious question - as you called out learning stuff (which I enjoy too) - why is it that they are always "home labs" and never referred to as a "home prod" environment or at least something that implies some stablitity? Nothing about the term production implies usage by some Fortune 500 company that with a $25M/yr IT CapEx budget, depends on said infra for $X million of revenue per day, etc.
  • rather it simply means the operational environment post-testing, i.e. dev => qa => prod
• Further to my point, there are probably a number of individuals on this board that have more infra firepower than many Russell 3000 companies - and are every bit as knowledgeable if not more so than their CTOs/CIOs etc.
• I hear "home lab" and I think of somebody studying for a Cisco certification of some sort (disclosure = I know nothing about this sort of thing) or just buying toys to have fun. Or someone getting creative with a soldering iron and hw and plumes of smoke rising into the air.
  
  • At one point in time, I only had one server, and its uptime was not so great as I was learning ... but then I started to find needing to restore all the way from the hypervisor back up to be too time consuming, annoying, and inconvenient, so I moved to two[1] and really don't fiddle around too much anymore, but if I do, never with with both at once as to ensure I'm always up.
  • (always wondered on that point btw - I know it is a silly question of sorts)

[1] I would have a friggin full rack if I had space, but a DC ≠ in a condo closet; heck, it is only 35.5" deep if memory serves so I had to choose been horizontal or vertical mounting on the wall otherwise I wouldn't be able to close the door.