help buying new reliable fileserver/sas expander... 24/36/48 bay

[muthur6k]

First: I apologize for the long and poorly written post. I used two days on it, so my mind might have wandered a bit.

tldr-ish:

1. Max drivesize is determined by the backplane, is that correct?
If so - SAS2 doesn't support larger than 8TB? (example listing)

2. Am I limiting myself to much buying hardware supporting max 8TB now?

3. Choosing a 36 bay supermicro - means I just have 2u for cards.
Am I limiting myself for future upgrades too much by choosing this?

4. Is the Norco 4224 really as bad as it seems?



I have a rack with a norco 4020 full with hitachi 2tb/4tb drives running a debian fileserver in my home.
Nothing fancy - just sharing media and as a central general storage. Single drives using samba.

My old drives are dying one after another (they're 10+ years or so), so about time to upgrade some stuff.


Read the guides here about making a sas expander box using a norco case, and decided to go for the 4224.
So far I have ordered 4 x hp sas expanders and 4 x ibm m1015 cards, and was thinking of running zfs and mirroring.
Then I found there were alot of bad experiences with the norco cases, so I'm leaning towards supermicro or chenbro instead. And that came with more required reading. It's probably 20 years since I worked with enterprise servers so there's a lot of new information to gather, but luckily this site is great for just that.

I've spent the last 6 months reading up on sas expanders and servers, and have now come to a point where I start to lose information as more gets stacked on top. The post about backplanes was really helpful when reading ebay listings.

Drivesize: I had thought about using 8TB drives, but I can see that it might not be enough when mirroring.
So I guess I need a server/expander supporting 12TB or more so to not lag to much behind, or do I?


Going for Norco/Chenbro is the easiest route as I don't have a lot of reading to do regarding hardware, but are my drives safe with this solution?

The cost of a used supermicro is not that far from Norco/Chenbro, and I have to admit I get a tingly feeling inside me when I browse supermicro listings.. (or it might be caused by a reflection I caught of myself in the monitor)..
I know that this hunger can be expensive to feed, and I really don't enjoy spending money on myself.
Going for the supermicro solution would be nice, but I'm no snob about storage - I just need it to be reliable for now. A plus for a used supermicro is that I can get it all in one swoop instead of hunting around for parts to get it installed.

Planning on buying a 1u/2u virtualization server after this purchase if I manage to find one quick, and the fileserver could perhaps be run from here too.
current guests:
* mail
* misc testing machines
* misc developing envirnoments
* wiki (gets hard to keep track of it all)
* github
* web
All for internal use.


At this point I feel like I'm just making things harder than they need to be, so I would really appreciate any answers or pointers to steer me in the right direction.

 

[pricklypunter]

Just some quick pointers...

Chenbro or Supermicro, there's little room between them in reliability terms. 3U at a minimum, if you don't plan on making some modifications to quieten things down for home use, but 4U would be better from this viewpoint. Basically the larger that fans you can fit in the chassis, the slower you can spin them, when needed, therefore the less noise they produce. Pay particular attention to free airflow over your drives, heatsinks and exhaust, and especially use dampers in "spare" drive sleds as airflow will take the path of least resistance. While mirrorred disks are great for applications requiring performance, playing media files is not really in that category, nor for that matter is chuntering data back and forth at 1Gbps from your laptop to mapped drives. I would suggest that you look at only using mirrors for stuff that needs them, and as you have already mentioned using ZFS, choose raidz2 for your general storage needs

 

[MiniKnight]

First when you say Norco, HP SAS expanders and the like that you saw here, you don't mean the 2010 articles that are now over 8 years old https://www.servethehome.com/sas-expanders-diy-cheap-low-cost-jbod-enclosures-raid/ right?

Today, I'd just get this for what you're doing Supermicro CSE-847 4U 36-Bay X9DRD-EF 2x LGA2011 LSI-SAS9266-8i Storage Server | eBay

You may be able to swap or sell the SAS RAID controllers in that for simple JBOD controllers if you want our of that box.​

For CPUs get two of these SR1AB INTEL XEON E5-2660V2 10 CORE 2.20GHz 95W PROCESSOR FOR DELL T620 R620 R720 658759132833 | eBay

RAM get whatever DDR3 size you want for that big box.

Instead of having FreeNAS, run Proxmox on it which does ZFS storage and virtualization. FreeNAS has an easier storage UI but virtualization is not as good as Proxmox. You'll only need 1 box and have 36 bays plus the virtualization server taken care of. Fewer things to go wrong.

If you want to add more just get one of these 4U Supermicro 847E16-RJBOD1 45 HDD Bay 3.5" SAS2/SATA3 JBOD Storage Expander | eBay

And you'll be at 71 drives. Done. HP SAS expanders were great in 2010 until maybe 2012 if they're the ones in that article, but technology has moved one. 8 years ago there were none of these great off lease servers around so that guide was necessary.

 

[Evan]

SAS1 effectively had a 2TB limit. After that I was not aware of any limits.

 

[muthur6k]

The server is going to be located in a rack in the basement, so the noise isn't of imidiate concern.
I was thinking mirroring two and two drives and pool them.. I think. I might have forgotten what the plan was from the start.
I need to get back into that once I'm finished with the hardware.
The cost is high, but if a hardware failure is happening I might not be well enough to rebuild a raid at that time (can take months before I'm able to get down to the servers), so if the raid goes down I'd be able to access the drives directly. It might be a silly solution though.
Perhaps it's smarter to just add a couple of spare drives or something like that.


And yes, the sas expander thread I read was the one from 2010..
Many of the threads and articles I've read lately has turned out to be old - I keep forgetting to look at the date until the end.
93 still sounds just like a while ago too me...


I see they can deliver cpu and ram from the listing you provided the link for.
I might send them a mail and ask for a quote.
Then cry a little.
Then pay up.


I have looked at this supermicro expander: SUPERMICRO CSE-847 CHASSIS 4U SAS + 2xPWS-1K28P-SQ | eBay - the shipping fee is about 1500 usd lower than the one you mentioned.
I forgot to say that I'm located in Norway.
The shipping fee for your server suggestion is 1000 USD - (at the higher end of the scale) so the price doubles + import charges..
But this is hopefully a one time charge. If everything works I'd rather pay some up front than keep fixing it.
The shipping fees are crazy though - they can vary from 200 to 2000 usd for the same equipment.


I'm already running Proxmox, so that was the plan.

But that raid controller installed is all I need to access all those 36 bays?
And software raid vs hardware raid.. is the fear of the controller kneeling also outdated knowledge?
I was originally thinking about using my cards in IT mode and use software raid.
I found a thread about the LSI SAS9266-8i, but I'm not able to read it atm.

thank you for the links and info - I'm one step closer to a solution now.

 

[muthur6k]

Thank you, Evan - I had a hard time finding any information on this subject.

 

[i386]

After that I was not aware of any limits.

lba64 => 64 bit addressing
2^64 * blocksize (512b,4k) ~enough space for the next 5 years

 

[nthu9280]

There are active members from Norway & EU here. @Sable @_rand had some items for sale. Hopefully shipping within EU is much cheaper if they have the SM 847 chassis.

 

[muthur6k]

I was just thinking of what if I'd need to return it for some reason - I'm out another 1000 usd..
You are right, I should see if I'm able to find something closer to home.

Found an expander in Norway for less than the shipping cost from USA:
https://forums.servethehome.com/ind...o-45bay-jbod-847e16-rjbod1.17675/#post-169868
SC847E16-RJBOD1 | 4U | Chassis | Products | Super Micro Computer, Inc.

It's not ideal, but neither is the shipping cost of the others.


Thank you for letting me know. It'll save me some money hopefully.

 

[muthur6k]

Had some downtime so lost two opportunities, just about to order a couple of cpus when the first server was sold - but came over this one: Supermicro 4U X9DR3-F 2x Xeon E5-2640 2.5ghz 8x 4gb 10600R 36-Bay SAS2-846EL1

As far as I can tell this looks good for my use - perhaps add some more memory.

Any thoughts on this one?

The shipping cost seems more reasonable if using jetcarrier.

 

[BLinux]

when buying supermicro X9 motherboards, pay attention to the revision number. try to get something that is 1.2 or 1.02 or something like that. older revisions are likely not to support Ivy Bridge CPUs. *some* of the older revisions can be fixed to support Ivy Bridge, like 1.00A, but not 1.00.

 

[BackupProphet]

I am from Norway and use Jetcarrier for shipping, they are awesome. A few times I have ordered pallets of servers that costed less to ship with Jetcarrier over the atlantic ocean than from CA to New Jersey

Anyway, I would just get a barebone Supermicro 847A. Dont go the expander route, expander is even too slow for the hard drives and SSD's are becoming dirt cheap. The SSD prices as gone down dramatically the last few months.

There are few 847A available though, you have to be patient to find one for less than 500 USD
However there are plenty of 846A if you can live with 24 bays.

I would also highly recommend getting v2 cpu, Intel 2650 V2 is available for 100 USD per cpu. As mentioned earlier you need to double check motherboard support for V2 cpus. For ZFS, Intel Optane 32GB is a great slog device. Cheap and fast. It has low TBW, but unless you only do 100% sync writes the Optane should last a few years.

 

[muthur6k]

Thank you - I will try to get a 1.2 board instead.
I now see some of the boards I've looking at has been lower than 1.2, but most does not specify rev.


I was hoping to buy a complete, working system to make it as easy as possible.
I see that buying components could perhaps be easier on me. The details are always there, but buying components allows me to focus on one part at a time. The item is usually sold before I can gather and digest information on all the parts.

If I go with a 24 bay I guess I'd need an expander soon anyway, but I am looking at both options.

Cpus.. I try to look up information on cpus but find the differences seems so minute. I was aiming for a v2, but just because I was told to.
I have not been able to read up on that yet - I'm still trying to learn the difference between the different motherboards.
All these almost-identical-numbers when naming stuff is not good for the head. When I compare two items I forget what the first was called when I shift focus to the second. It takes a lot of effort


Took me a couple of days to read up on slog devices.
I hadn't planned for the use of a slog device since it was originally meant to be a separate file server - but it looks like it could just as well be hosting all my guests.

I've only skimmed through sync writes when reading up on zfs, but I thought sync writes was the way to go.

Some development/general storage area will be assigned, but you see how long I took answering these posts? I guess I'm not gonna wear out the slog with sync-writes either.


Thanks both for all the information!

 

[nthu9280]

when buying supermicro X9 motherboards, pay attention to the revision number. try to get something that is 1.2 or 1.02 or something like that. older revisions are likely not to support Ivy Bridge CPUs. *some* of the older revisions can be fixed to support Ivy Bridge, like 1.00A, but not 1.00.

It appears to be certain batches of X9 boards. I have older revisions of X9DRD-7LN4F-JBOD and a X9DRW-iF. Rev 1.02A or something like that. But they both work fine with E5-26xx V2. I initially had V1 CPUs in them updated the BIOS and replaced the CPUs. I only came to know of the issues after folks here were discussing it.

 

[BLinux]

It appears to be certain batches of X9 boards. I have older revisions of X9DRD-7LN4F-JBOD and a X9DRW-iF. Rev 1.02A or something like that. But they both work fine with E5-26xx V2. I initially had V1 CPUs in them updated the BIOS and replaced the CPUs. I only came to know of the issues after folks here were discussing it.

yeah, there are 3 batches of X9 boards:

1) rev 1.00 : these boards do not accept v2 CPUs even with latest BIOS and can't be fixed to accept v2 CPUs
2) rev 1.00A or similar : these do not accept v2 CPUs, but can be reworked by Supermicro RMA to accept v2 CPUs.
3) rev 1.02 or 1.2 or similar : these accept v2 CPUs out of the box.

Until recently, I thought there was only #2 and #3 boards. Then I came across some rev 1.00 boards and after a lot of back and forth discussions with Supermicro RMA, found out they are not able to fix these boards to accept v2.

 

[pricklypunter]

If I go with a 24 bay I guess I'd need an expander soon anyway, but I am looking at both options.

If you go the SuperMicro route, my advice would be to try and find a chassis with the "A" version backplane, rather than the expander backplane, or get one and swap it over. Unless that is, you have other reasons for needing the expander. The reason I mention it, is that you can use 12Gbps SATA disks in those, as they are simply pass-thru ports but have the convenience of using Mini-SAS connections to keep wiring tidy

 

[muthur6k]

I have no reason for needing anything - except a place for my files to call home.

like this ebay listing?
"Unit has SAS846A and SAS826A direct attach backplanes"

My order of 4 x hp sas expanders and 4 x ibm m1015 cards arrived yesterday - it would be nice to be able to use them for something.

Did I understand you correctly here?

 

[pricklypunter]

I have no reason for needing anything - except a place for my files to call home.

like this ebay listing?
"Unit has SAS846A and SAS826A direct attach backplanes"

My order of 4 x hp sas expanders and 4 x ibm m1015 cards arrived yesterday - it would be nice to be able to use them for something.

Did I understand you correctly here?

Yes, just like that, the Direct Attach Backplane, either in or out of a chassis already. If you can't find one at reasonable money, look for a bare chassis or one with an old SAS1 BackPlane and an "A" version BackPlane separately, SuperMicro made it really easy to swap out. The M1015 cards are great, you will definitely use those for sure, the expander boards not so much, unless you have a burning desire to expand beyond what a single chassis can hold.

It really depends on how large your storage requires to be, plus a bit of overhead for growing pains and growth in the future. I would be looking at either the HGST HE6 or HE8 disks if buying them used, or the 10TB ones if buying new, they are quite possibly the best bang for buck now going forward. If filling multiple chassis, you'll want to find a good IT recycler that will work with you on bulk pulls. The idea here is to reduce the number of disks, thereby reducing heat/ cooling requirements and running cost and give reliability a boost

 

[muthur6k]

I'm not sure I'm following..
(Image from the listing in my previous post)
I can count 9 sas cables on this picture (there's one for each row of disks, right?):

I would think I'd need one m1015 raid controller (2 ports), and one expander card (8 ports) to use this. Is it not so?


Regarding your advice on disks; that was my understanding too.
I first looked at 8TB, but realized 12TB also existed. Have not been paying attention the last few years.
But when you're talking disks as large as 10- or 12TB wouldn't the rebuilds take forever - and perhaps kill more of them?
Even more so than some states happens/can happen with the smaller drives.

Used disks.. I had never even considered that as an viable option.
One problem I have here is that I have not been able to get anyone willing to pick disks from different batches for me, but I do shop at consumer-oriented stores, not enterprise-minded. That is one of the reasons why I also consider btrfs - then I can add a drive when I want to - instead of buying the whole array-set in one go. I have also bought all my equipment in Norway so far, due to good consumer protection laws. But looking at it now... I have perhaps lost one or two drives the last ten years, so it might not be that important.
I will try to read up on the forum about buying used disks.

 

[Dan991199]

I have a 12TB drive in my norco 4020 no issues here.