Storage server build suggestions?

[mstroud]

Hello server experts!

I am looking for advice to build a long-term, stable, sustainable storage server for my home. My years of individual, retired PC part built servers must come to an end and I am looking for some assistance to build the right solution for my needs.

Currently I have a mix of FreeNAS of Debian (Proxmox VE) storage running either ZFS or hardware RAID-5 (using this box). I've run out of patience with multiple drive failures using this enclosure and the inability to grow storage. I have a mix of 1TB and 2TB SATA disks, about 15 in all, running across three systems. I want to consolidate it all down to a single box that is failure tolerant and redundant.

I originally looked at this DIY article from 2010 for advice, but the CPU and motherboard landscape has changed a lot since then. I'm looking to the community for suggestions on where to go next and I'd like to build something that will last and have performance, rather than several hodge-podge builds that I have done previously.

Your suggestions and comments are welcome.

 

[sotech]

Norco 4224 + socket 1155 server board, a Xeon CPU (E3-1230 or perhaps a new Ivy Bridge Xeon if they come out soon as predicted), some ECC RAM and a couple of M1015 HBAs would be a good start, imho. I'm a big fan of ZFS so with our server we've flashed the M1015s to IT mode so they're just dumb HBAs rather than raid cards and it works quite well.

 

[mobilenvidia]

A supermicro SC836 or SC936 16 Bay 3u chassis.
Any choice really of Motherboard LGA2011, LGA1155 or even a LGA1156, LGA1155 probably best bang for buck BUT check the PCIe slots you will need 1x PCIe (8x) for each SAS controller
As above a couple of M1015's in IT mode to run ZFS

You'll be laughing

 

[mstroud]

OK, thanks for this first set of feedback. I'll need to investigate the whole M1015 card thing as that was the one area I was hoping to avoid. Ebay has been a sketchy provider of parts for me in the past and my Google searches turn up stories of people having some problems after flashing the wrong version of the firmware (except for mobilenvidia, who seems to have tried everything).

What will I be laughing about? The cost of the SuperMicro chassis vs the Norco one? Unlike the chassis listed in the 2010 review, these 836/936 models don't seem to offer any advantages (other than quality?) over the Norco models, or did I miss something?

Thanks again!

 

[mobilenvidia]

The M1015 cards are probably the most widely used controller in this forum for small servers, the problems you see are often Motherboard related.
LSI cards are known to give issues with some Mobo's, Gigabyte being one whole brand
Server boards seem to have no issues, which probably relates to timings on the Mobo to extract every ounce of performance from desktop boards.

I pointed out the Supermicro chassis as another option, never checked price.
Keep in mind that you need to add a PSU(s) to the Norco

 

[dba]

Norco chassis are definitely a better value, if purchased new, than Supermicro. What you get with Supermicro is slightly better build quality, redundant power supplies, and sometimes a chassis that only works with particular Supermicro motherboards.

If you need redundant power, then it's a more difficult choice. Add redundant power to a Norco and the price starts to reach Supermicro territory.

If you want quiet, Supermicro probably isn't your choice. The stock fans are incredibly powerful - which is great for a server - but are non-standard. The Norco fans are equally noisy, but are standard sized and can be swapped with quiet fans very easily.

And, finally, buying used is a different matter. There are always used Supermicro chassis available on eBay, and sometimes the prices are pretty good, especially considering the redundant power supplies.

One tip: Norco drive sleds include screw holes for both 3.5" and 2.5" drives. It's a real pain to try to mount a 2.5" drive in a Supermicro 3.5" sled.

Lastly, I love the M1015 cards. They are every bit as good as other SAS2008-based cards (which is very very good) and are dirt cheap. Buy a spare just in case!

OK, thanks for this first set of feedback. I'll need to investigate the whole M1015 card thing as that was the one area I was hoping to avoid. Ebay has been a sketchy provider of parts for me in the past and my Google searches turn up stories of people having some problems after flashing the wrong version of the firmware (except for mobilenvidia, who seems to have tried everything).

What will I be laughing about? The cost of the SuperMicro chassis vs the Norco one? Unlike the chassis listed in the 2010 review, these 836/936 models don't seem to offer any advantages (other than quality?) over the Norco models, or did I miss something?

Thanks again!

 

[mstroud]

More good info here, thanks all. I will look into all of this information and report back here when I think I have a reasonable build to put together.

 

[Patrick]

One tip: Norco drive sleds include screw holes for both 3.5" and 2.5" drives. It's a real pain to try to mount a 2.5" drive in a Supermicro 3.5" sled.

Awesome point and something I will be giving them feedback on as I really don't like doing that.

One thing on the used Supermicro v. Norco, one other thing to consider is you can fairly easily stick 120mm fans in the Norco 4U's which make them fairly quiet. No redundant PSUs, but redundant PSUs have small and loud fans.

 

[PigLover]

Actually, the Norco cases don't have PSUs at all - while the SMs usually do. That accounts for $100+ of the price difference (much more, really, when you consider the quality of the PSUs SM uses).

For home users the lower starting price and greater opportunity for customization (silencing, etc) often do give the edge to Norco. I couldn't imagine doing the things I've done to my Norco cases to a SM unit.

 

[mstroud]

One additional question from my searches: does a motherboard exist that has 3 actual electrical x8 PCI-E slots?

My reading indicates that the M1015 card requires x8 electrical slots, but maybe that isn't correct? So far every motherboard I've found has at most 1 x16 and 1 x8 or 2 x8 and 1 x4 slots, which means a max of 16 HDDs, since there are only 8 drives per M1015 card. Did I miss something?

 

[PigLover]

One additional question from my searches: does a motherboard exist that has 3 actual electrical x8 PCI-E slots?

My reading indicates that the M1015 card requires x8 electrical slots, but maybe that isn't correct? So far every motherboard I've found has at most 1 x16 and 1 x8 or 2 x8 and 1 x4 slots, which means a max of 16 HDDs, since there are only 8 drives per M1015 card. Did I miss something?

The M1015 works quite well in an x4 electrical slot. It has a theoretical throughput limitation when doing that, but if you are just using spinny disks (no SSDs) and no expander (limited to 8 drives/card) then you are unlikely to have any trouble with this limit.

Actually, I tested mine in an x1 electrical slot and it still worked - though I don't recommend it in production.

Yes, there are MBs with 3 x8 slots. They do have drawbacks. Some of them use PCIe 'switches' (e.g., SM X8SIA-F) which interfere with device passthrough when used with ESXi. Some of them are built on dual-processor Xeon designs (e.g., SM X8DAH-F) which make them expensive to deploy. There are also new boards in the new E5-2600 socket 2011 family (e.g., SM X9DAi). I don't know of any in the Xeon/1155 family but they probably exist (using switches).

Personally, I run 3x M1015s using the X8SIA-F motherboard. I don't have a problem with the PCIe switch because i am running 'bare metal' Solaris (no ESXi). But even though I have 3 x8 slots I have one of the M1015s plugged into the x4 electrical slot because I have something else in the 3rd X8 slot. It works fine.

 

[mstroud]

Thanks for that info on the x4 PCI-e compatibility! That is excellent to know and isn't something I could find throughout my day of Google searches. I will investigate that motherboard and see if that makes my short list of equipment for the build.

The feedback is the reason why I came to this site. Many, many thanks for the assistance.

 

[mstroud]

Thanks to feedback in this forum, I now have or will have the following parts in hand for this server build:
- Supermicro X9SCM-F
- Intel Xeon E3-1230
- 16GB DDR3 ECC RAM
- 2x IBM M1015
- Norco 4224 chassis
- Norco 120mm fan bracket
- 80+ PSU (600 watt)

I'm still tracking down the following parts:
- 8087 SFF cables

I had planned to start off this new server by just moving over and starting up FreeNAS on it. Am I better off trying to figure out OpenIndiana and NAPP-IT instead?

 

[Patrick]

Ever think of ESXi OpenIndiana/ Napp-IT all-in-one? You can use ESXi 5.0 since you are under 32GB with that platform basically for free. Then you can do other things with the underlying hardware.

FreeNAS is awesome, but performance wise I think OI still is better.

 

[mstroud]

I'm unsure if I want to fiddle around with ESXi. I was really hoping this would be a less complicated storage box than what I had before. Maybe I'll play around with it over the weekend and see if its worth the work it takes to learn it.

Perhaps I'll have better luck with OI now that I have stable, supported hardware, so I can look at that again as well.

 

[RimBlock]

Supermicro do a number of chassis with built in SAS expanders. The SC846E16-R1200 is one and they provide either 1xin/1xout SFF-8087 connector or 2xin/2xout. They are pricey though especially as these come with 1200W redundant PSUs.

Another option is to look at SAS expanders like the Intel RES2SV240 which could save a SAS card or two and are likely to make minimal difference to speed if you are not looking at top flight hard drives or SSDs.

Your setup will run ESXi 5 even though Supermicro are not on the HCL (I have two I am currently building with the same hardware and I had a quick test). I didn't try VT-d but have no reason to believe it will not work.

RB

 

[mstroud]

OK, I just got the PSU and M1015 cards in and they are now running. I'm waiting on the Norco case and 120mm fan card to arrive before I order up the cables.

In the meantime I am curious about this:
- I installed the two M1015 cards and now OpenIndiana boots like molasses running uphill in the winter time.
- No drives are attached at this stage (no cables, waiting on enclosure)
- Only a single boot SATA drive is attached to the host Intel SATA controller
- According to the BIOS, the two cards have firmware 20.10.1-0052 and 20.10.1-0069

Do I / Will I need to do any sort of flash update to these two cards before using them? Is the slowness during boot (and, to be honest, while running) due to no drives being attached? Should I start another post to deal with all these issues?

Many thanks for all the help!

 

[Patrick]

Are you sure it is OI and it is not M1015 boot ROM loading times?

I will tell you this, I have one that is so slow to boot due to option ROMs that it literally takes minutes to hit the Linux boot screens.

 

[mstroud]

I'm not sure how to tell which side (OS or M1015) is the problem. Yes, the OS boot time is a bit slower, but once in the OS everything that happens there seems slower. Attempting to open a shell window, Firefox or the Driver Installer is an exercise in extreme patience.

 

[matt_garman]

If you flash your M1015 cards following the procedure posted on this site, you can opt to leave out the BIOS ROM, which will improve your boot times. I flashed three M1015s this way, and they are working fine plus I have faster boots (although I avoid rebooting as much as possible).