Expanding storage, new to SAS and expanders

[dj3982]

I have a Fractal Design Node 804 case today with 8x3TB NAS drives I need to expand, and I'd like to keep everything a single volume. I've been googling around and wondered if this would work for me as expected (full SATA support, 3TB drives and up, mainly for media server and backup, performance not uber important as it all happens over 1Gbit/s (~110MB/s transfers/writes):

Get a motherboard in the primary case with 2x PCI-E x16 slots and put 2x 9211-8i cards in them:
2x LSI SAS2008-8I SATA 9211-8i 6Gbps 8 Ports HBA PCI-E RAID Controller Card | eBay
+
2x 2 port Internal SFF-8087 to External 8088 PCI mini SAS 26P Adapter SAS RAID WB1 | eBay

Then get 2 other cases with lots of HDD space and put a sas expander in each powered by a riser:
2x New HP SAS Expender Card 24-Port SAS PCI-E Expander Board 468405-001 | eBay
+
2x 4pin USB 3.0 Pci PCI-E Express 1x To 16x Extender Riser Card Adapter Power Cable | eBay

Will this work well with SATA drives? Will it work with 3TB+ drives? Everything passthrough to do whatever I want with the drives? SMART available?

I'm considering pure HBA setup without expanders but I feel I waste a lot of capacity if that HP SAS expander works well for >2TB and transparent SATA. With 2x 8i HBA's I'd get 16 drives opposed to 48 max with expanders. For a media server I really want to have everything on a single volume, not create multiple, so I don't have to split up Plex libraries more than I absolutely need to. At the same time I want a reliable and compatible system, as much as I can without going enterprise. There are cost constraints as this is a private project.

Thanks.

 

[Mishka]

Your setup is basically exactly what I will be going for longterm.

Look for LSI 8E cards on ebay this will save you having to use up an extra PCI bracket slot with the adapter and also mean you have less cables inside the case and gives you space to add some cooling under the HBA so they have better airflow.

I have on my HP SAS expander a 8TB Toshiba NAS drive connected so I believe because they "only" do pass through they dont really care about size but I DID Have to update the HBA to a newer firmware to get it to see over 2TB (Dell H200 reflashed)

Performance on the 8TB drive plugged into Mobo (SATA2 as its old), H200 (SATA3) or into the expander (SATA2) is all identical give or take a teeny bit of leeway

In theory you will end up "limited" on the connection from Expander back to the HBA if you end up having say 20 drives on that expander and they are all being used at the same time but.....that is very unlikely to happen in the real world at max speeds and your servers connection is only 1GBPS anyway.

You mention volume not wanting to split volumes, those cards are not raid so in the OS you would see for example... all 48 drives individually that you can then mount as required unless you are using some kind of software raid/clustering software like Flexraid

 

[dj3982]

Thanks for you input, I've used and enjoyed Stablebit DrivePool since '14, pretty happy with it. Can I ask how you flashed it (hp sas expander)? Where you got the firmware and software you used. Seem to be a lot of spread information around this card, hard to be sure of anything.

 

[Mishka]

I did not flash the SAS expander, just left it with the firmware it came with if it aint broke dont touch it.

I flashed the dell HBA to LSI 9211 firmware through a guide I found online, they are all near identical anyway re the guides https://techmattr.wordpress.com/201...shing-to-it-mode-dell-perc-h200-and-h310/that one seems to ring a very strong bell though and STH has its own thread about doing the same.

If you get the HP SAS expander get the Green one not the yellow ones, green ones are the newer revision of hardware, I managed to get mine fairly cheap by just waiting on ebay for them to come up at auction for about £15, I now have 4 of them as idea is use them to gradually grow my fileserver though I will be getting 24 bay Hotswap rackmount cases as I burn through about 500GB a month without er...absorbing the NAS's friends have

 

[dj3982]

Ah OK, thanks. When I e-bay'd "lsi 8e" that Dell H200 card comes up first actually and at a fair price to me. I'll get 2-4 of those and flash them to IT. And a few more of those green expanders - I share your thinking in terms of gradually growing. May run dual-link between the cards, at least to begin with, unless there's any negatives to it.

I hate buying used, so I try to get new OEM when I can on electrical parts. The only real price for buying from asia is the waiting time (2-4 weeks), but compared to what stuff like that retails for here in Norway I'm very happy to wait.

I'm closer to perhaps 50-100GB a month. I can't seem to find a cheap enough 24 bay at all, they're all at $300+ and that just seems overpriced to me for what it is, so I'm actually thinking of just DIY a case for drives and use direct cables.

 

[dj3982]

It says in the docs for the HP SAS Expander that the 8088 port is for tape, but it can be used as a line in from the HBA as well, right? Perhaps a given, but may as well ask someone who's tried it.

 

[Mishka]

Just regarding the HP SAS Expander, this is mine connected to my H200 flashed to 9211-8i firmware IT mode, MegaRaid Manager can see it as an "enclosure" showing firmware version, serial and er some info i've not seen how to get it to report, literally just plugged it in and let it do its magic giving me the extra ports.



Below is what the drive passes through when highlighting it although I get full smart info in CrystalDiskInfo but had to enable one of the advanced features (ATA pass through I believe)

 

[dj3982]

Thanks, nice info. Got a few cards and cables on the way. Mostly planning what kind of rack I want for drives now.

 

[Mishka]

For me I kind of want hotswap despite it costing about £75 or so more just for that feature, also means in future I could rackmount the fileserver without having to pull it out when a drive has problems.

I recently managed to grab a 9201-16E pretty cheap so it will be my shiny new card and means I can more easily setup the extra storage units, just need to make sure they then boot up before the fileserver itself

 

[dj3982]

Got any link(s) to the exact case(s) you have in mind? I'm open for inspiration

 

[Mishka]

I was eyeing up the Case Logic Logic Case SC-4324 | 4U Server Case w/ 24x 3.5" Hot-Swappable SATA/SAS Drive Bays, 6Gb/s MiniSAS or 4324s (its a bit shorter) I believe in other places (US) they are Norco cases, but generically 24bay Hotswap jobby.

I have managed to get my hands on the supermicro CBE JBOD1 board so can get fans running and use the power buttons on it

Sneaky Edit,

Logic Case 4U Server Rackmount Chassis 10x 3.5" HDD + 1x 2.5" HDD/SSD + 3x 5.25" Bays is about £75, the 4324 is £300, so for just perma mounting hdd it could be fair bit cheaper to go for a non hotswap case but......half as much capacity although does mean for growth long term need another PSU, not as cost effective due to needing another SAS Expander, port used on controller etc etc

 

[Stefan75]

Tons of info about the hp expander here: HP SAS Expander Owner's Thread
My expander came with firmware 2.06, want to update to 2.10 soon.

 

[dj3982]

So, I finally got around to testing my HBAs and expanders.

I had purchased 3x HP SAS Expanders from eBay. Funny thing is that I looked at the revision numbers and just assumed that A03 would have newer firmware than A01, so I used that for a few days until I realized it had FW 1.00, while the other two older revisions actually had 2.06. According to MSM (nice software, glad I found it) the 1.00 card (as expected) would only negotiate 1.5 gbps, while the 2 other cards negotiated 3 gbps. I also found that these cards act like switches, any port can be used in any fashion, so the rear #1 tape-port is the same function-wise as the others. Both dual-link and single-link seemed to work as expected with 9211-8i flashed cards (flash process was easy).

I've binned the 1.00 card as I can't be bothered finding hardware to upgrade it or sell it again as-is. I have the 2 others already with OK firmware for future custom DAS builds. Testing with riser cards, the expanders worked perfectly with just s-ata or molex power, so a DAS will be really easy to make with just a case, PSU, riser-card and a switch for on/off.

I've started with 2x 9211-8i cards for the initial 16 drive server (only using 10 so far). I have a spare PCIe slot for a H200E that I'll probably hook up to a case with 8088e-to-8087i adapter with dual-linking inside to an expander and room for at least 16 drives (150 MB/s to each, theoretically). I can't see my need going past 32 drives in total, so I'm good with this setup. It also leaves with me with 1x spare of every card in my possession.

Ended up mounting a small noctua 40mm fan on the cards, as their heatsink was burning hot to touch. With a fan (even with the noise adapter to lower rpm to ~3700) it was barely warm on even my thin-skinned pinky finger. Probably not a big deal, but makes me sleep better at night - I also like to customize things anyway.

I feel comfortable around basic SAS/SATA storage setups now, I'm glad I jumped into it, late to the party but still. An old thread I know, but wanted to add my findings since I'd started it.

 

[Stefan75]

Glad it worked out for you..
These hba and expanders seem to be very reliable.