Looking for input on Hardware Raid Cards, and Drives recomendations

[fnc1]

Greetings,

This is my first post here... and I am looking for some input.
I am finally getting around to building out my Norco 4020 case...
and could use some input from you guys.

What I am looking for is the ability to expand in the future (my current 15 drive array supermicro is out of space and I know i will need to at some point)

Here is what I would like / am considering:
-Hardware RAID 6 (i really dont want to have to rip 1000 plus discs again)
-6gbs (incase drives get cheap in the future )
-Online Capacity Expansion (OCE)
-Online RAID Level Migration (RLM)
-The ability to add multiple SAS expanders in the future (add a case, ps, SAS card, drives and be good to go)

I like cheap... and the card I am looking at is LSI/3ware specifically the 9750-4i which if i buy this weekend i can get for 263.99 (thanks to the up to 75 off code)
http://www.lsi.com/products/storagecomponents/Pages/3wareSAS9750-4i.aspx

I would like to use this with the 6gbs sas expander and the only one ive found that has enough ports for my 20 bay case, and one external is the chenbro CK23601 http://www.chenbro.com/corporatesite/products_detail.php?sku=187
this card will allow me to plug in the 3ware card, and then all 5 backplanes, and already has an external port (for the future expandability) ~$320 (but are back ordered everywhere)

I also still need to pick out a MB/CPU/MEMORY/PS/DRIVES

Which brings me to my second part (which i would post in the drive section, but since it is related to Hardware RAID, and fits nicely Ill speak to it here:

I would really like some 'green' drives! Since this is for my home, and i dont need super speeds - reliability, and size are instead at the top, and I figure low power consumption, and price point on some of the lower end drives would be a good thing.

I know there are a lot of these 'green' drives out there, but it looks like you can no longer do the firmware upgrades and what not on the WD drives. So i am also looking for recommendations along these lines, specifically 2tb or larger and low power/heat..


So:
1) Is this a good Combo for what I am looking for?
2) Should I seriously consider something else (Hardware RAID) wise?
3) Are there other decently priced 6gbs SAS cards that have the number of ports i should look at?
4) As for Drives, I would really like a good price point on 2tb or larger 'green' drives that work specifically with HW Raid - Whats Available out there?
5) What drives do you recommend?
6) Anything else i forgot or should consider?

 

[Patrick]

On the green drives I am a big fan of 5K3000. LSI controllers are generally very good. You may also want to look at the Intel SAS Expander. Big thing to look at is what OS do you want to use, and why not something like RAID-Z2? (sorry on mobile this weekend others may have longer responses)

 

[fnc1]

Ah sorry forgot to say... I will be using Linux - most likely ubuntu.

Is there a bigger Intel SAS Expander, this one only has 6 ports, so one input, then 5 out to the backplane, which would leave no room to add an external one?

Im not that familiar with RAID-Z2, other than knowing its a non standard raid level, and similar to RAID-6 ... what do i need to know about RAID-Z2

 

[fnc1]

So I read a LOT on ZFS, and it sounds really cool...
I was thinking, Hmmm ok ill just go with Raid-Z2, but then i keep on seeing things about it Needing a LOT of memory, I've seen people on here say that, and other places, live over on FreeNAS forums (which is what i was using, it says to use 1gig of memory per 1T of space, well, if my goal is to get to 40, or even 60 Ts, then i would think this is going to count out ZFS.

Can anyone chime in about this memory issue?

 

[PigLover]

The "1gig per Terabyte" requirement only applies if you have deduplication turned on. Without dedup ZFS performance is good with much more modest amounts of RAM. It does perform better with more memory due to its caching strategy, but massive amounts of RAM are not really required.

 

[fnc1]

Thanks for the info, So if I'm looking to run 40T-60T this can all be accomplished with good performance, and at a decent cost? what would a good amount of memory be? Also i dont think i called it out before, but I have to be able to stream full BR iso's (and am already gigabit everywhere).

 

[Patrick]

Streaming BR you will be fine. Also, with that type of usage you don't need to use fancier setups with L2ARC either. With current RAM prices 16GB is probably fine.

 

[fnc1]

So i went ahead and orderd the 9750-4i for 263.99 I posted the code good for the rest of the day (im not sure in what time zone, in the great deals sub forum) Basically newegg has up to 75$ off their raid stuff, unfortunately it looks like the only sas expander newegg has is the intel 24 port card.

I got this since it was on sale, and Ill need a Raid card or HBA either way, I figure this gives me options.. with a great card, and only an extra 100$ than an HBA...

I'm not sure im sold on Raid-Z2 yet.. All the features sound GREAT...
- but ive also heard they are issues with fixing things if it ever does take a dive!
- And even though things seem somewhat OK, all the sales from SUN to Oracle and going from open to not..
(i guess im just not convinced (even though I kind of want to be) maybe in the future ill give it a try)

But if I do go ahead with RAID 6, what File systems do people like? I used XFS on my last build, and it seems fine...
I guess the main things I have questions about, are what to use for expansion, i know you can add to XFS, and ZFS ( i still use the zfs FS, but just not sure)
I went with XFS before after reading some comparisons, and it seemed like XFS was the best at large files...

Also, I am looking for Low power / low heat, but I was thinking a decent MB and CPU, not necessarily a Server Board and CPU, but my main question was on memory, just how necessary is ECC memory in a raid 6 and or RAID-Z2 setup?

 

[apnar]

For the price of that card you could have grabbed 3 "new" IBM M1015 cards on ebay and not worried about an expander at all. As to your ZFS worries, at least if something goes wrong there is a chance of fixing it, what would you do if the RAID6 goes wrong with your raid card?

Anyway, the big thing with ZFS for me was the data integrity checking, everything is check-summed so if there are any unreported read errors they are caught and corrected. The same rationale lead me to use ECC memory.

 

[fnc1]

In my original post, i said something about future expandability... over at sasexpander.com there are some pictures of what i have in mind. Not that I plan on using their hardware.. but the idea is the same.

Those IBM cards do look decent, but they only support luns up to 64T - and only 32 drives, so your right i would need 3, plus an extra 3 cards for each box i add...and internal to external plate x2 for each box I add.. sff 8087 to sff 8088 in each machine, and those are like ~50 each... plus then the main mother board would have to support that many cards...

with this type of setup, I can add another Case / PS / drives / Sas expander, and be good to go, and control, everything from one single place as this card supports up to 127 devices.

 

[mobilenvidia]

I'm running XFS on my storage system at home, been very good and reliable for years.
It's pretty well supported in everything non Windoze now.

If you are thinking of RAID6 you should get ECC RAM, other wise you are wasting your time any memory errors will be written to the Array.
There is a Desktop board I think with the intel 3450 chipset that takes ECC RAM, ASUS has one.
I was looking at one, but it lacked some features I needed as my desktop rig, but as a server should be ideal.

If you do go to a non Server Board, don't get a Gigabyte, they have issues with LSI controllers.

I've got a Asrock P55 Deluxe3 Mobo with a Xeon X3470 this has been great, except the very lousy Marvell 9128 controller and hence me coming here after I bought a Dell Perc H200 (LSI 9211-8i)
Now I can run full SATA3 speeds
Make sure you have PCie 8x or better slot free for the 9750

 

[fnc1]

mobilenvidia: Im was looking at the Asrock P55 Deluxe3 and it says it does NOT supports ECC?

But i have been trying to find a cheap board that will support ECC memory and has a PCIe 2.0 16x (its hard to find 8x slots) and ive read they should all be backward compatible!

But I have found this at INTELs Site:
The following Intel® Desktop Boards support both ECC and non-ECC DIMMs.
Chipset Desktop Board
X58 DX58SO2; DX58OG; DX58SO (ECC memory supported with an Intel® Xeon® Processor and
BIOS version 5390 or later)
Q57 DQ57TM; DQ57TML (ECC memory supported with an Intel Xeon Processor)
X38 DX38BT
975X D975XBX2; D975XBX
955X D955XBK; D955XCS
925X D925XCV (AA# C57587-403 and later); D925XBC (AA# C59590-306 and later)
875P D875PBZ

The following Intel® Desktop Boards support fully buffered (FB) DIMMs.
Chipset Desktop Board
5400 D5400XS

 

[mobilenvidia]

The AS P55 deluxe3 indeed does not support ECC, I have no need for it.
I have a home server that I keep all my movies, MP3, piccies on etc that I can dish out all over the house.

P7F7-E WS Supercomputer
Was the board I was originally looking at, this does support ECC BUT only with a Xeon CPU

PCIe 16x slots works just fine with PCIe 8x cards

 

[mobilenvidia]

I have a Xeon x3470 which is a 1156 Lynnfield CPU, an engineering sample cheaply of Ebay.
4 Core/8 Thread CPU, allows ECC and also 32GB max (but with slower RAM, on a critical server a good thing)
This will go into the ASUS P7F7-E

I bought the CPU first and then build my system around it.

 

[fnc1]

mobilenvidia: I actually have like 6 or 8 - 2gb chips pf pc2-5300 So it would be nice if i could use that...

But really im just looking for a cheap solution for the mb and cpu, as it doesn't need to be very powerful for serving files in the home.

It would be really nice if i could find a board that supported ECC memory, and could use the intel 620t for 83 bux (its cheap enough, and only uses 35w)

 

[mobilenvidia]

Your RAID6 will be more powerful than the Mobo/CPU to run it if you want to use the PC2-5300

Get your self a nice Xeon L3426 45w (X3470 uses 95w) will fit the ASUS board above
This I would have liked, but price was king and the x3470 is now part of the family.
It will throttle right down to 9w in it's lowest power use (9x clock) which is most of the time
The L3426 will probably use 1/2 that again.

Or go Mini ITX and get a cheap CPU/Mobo combo, no ECC though.

For the Home why RAID6 ? RAID 5 should be ample, 1 drive fails you still have your data

 

[fnc1]

My current set up is raid 5, and i figure well raid 6 will only cost me one more drive, i might as well go with it..

Correct me if I'm wrong but wouldn't using any motherboard with out the ECC memory No matter the raid level, potentially cause issues?

 

[Patrick]

With today's large capacity drives that are really cheap, RAID 6 makes a lot of sense. Just thinking about the data of the value stored versus a few dollars more for another drive. Also, multiple drive failures are not as uncommon as one may think so RAID 6 leaves you much better protected.

 

[mobilenvidia]

RAID 10 is another much speedier option if you need quick access to your files.
Ofcourse 1/2 the drives are mirrors.

I had an IBM server with 8x 73GB 15kRPM SCSI drives via a ServeRAID 7e controller I tested RAID 50 this was cool, couldn't test RAID60
RAID 10 was blazingly fast, and another option for you to consider, even the low end controllers can do this easily.

BUT should your controller die you will need to get another setup exactly the same to be able to retrieve the data.
No matter how many redundant HDD's you have in your RAID do you also allow for this, a redundant Controller