help buying new reliable fileserver/sas expander... 24/36/48 bay

Discussion in 'Chassis and Enclosures' started by muthur6k, Jun 22, 2018.

  1. muthur6k

    muthur6k New Member

    Joined:
    Jun 21, 2018
    Messages:
    6
    Likes Received:
    0
    First: I apologize for the long and poorly written post. I used two days on it, so my mind might have wandered a bit.

    tldr-ish:

    1. Max drivesize is determined by the backplane, is that correct?
    If so - SAS2 doesn't support larger than 8TB? (example listing)

    2. Am I limiting myself to much buying hardware supporting max 8TB now?

    3. Choosing a 36 bay supermicro - means I just have 2u for cards.
    Am I limiting myself for future upgrades too much by choosing this?

    4. Is the Norco 4224 really as bad as it seems?



    I have a rack with a norco 4020 full with hitachi 2tb/4tb drives running a debian fileserver in my home.
    Nothing fancy - just sharing media and as a central general storage. Single drives using samba.

    My old drives are dying one after another (they're 10+ years or so), so about time to upgrade some stuff.


    Read the guides here about making a sas expander box using a norco case, and decided to go for the 4224.
    So far I have ordered 4 x hp sas expanders and 4 x ibm m1015 cards, and was thinking of running zfs and mirroring.
    Then I found there were alot of bad experiences with the norco cases, so I'm leaning towards supermicro or chenbro instead. And that came with more required reading. It's probably 20 years since I worked with enterprise servers so there's a lot of new information to gather, but luckily this site is great for just that.

    I've spent the last 6 months reading up on sas expanders and servers, and have now come to a point where I start to lose information as more gets stacked on top. The post about backplanes was really helpful when reading ebay listings.

    Drivesize: I had thought about using 8TB drives, but I can see that it might not be enough when mirroring.
    So I guess I need a server/expander supporting 12TB or more so to not lag to much behind, or do I?
    I was shocked when I counted my drives in use now: 16 x 2tb and 6 x 4tb (+10-ish 4tb usb as backup). It really adds up over the years. I'm in the process of upgrading my entertainment center to 4k, so the ripped images will be larger from now on.

    Going for Norco/Chenbro is the easiest route as I don't have a lot of reading to do regarding hardware, but are my drives safe with this solution?

    The cost of a used supermicro is not that far from Norco/Chenbro, and I have to admit I get a tingly feeling inside me when I browse supermicro listings.. (or it might be caused by a reflection I caught of myself in the monitor)..
    I know that this hunger can be expensive to feed, and I really don't enjoy spending money on myself.
    Going for the supermicro solution would be nice, but I'm no snob about storage - I just need it to be reliable for now. A plus for a used supermicro is that I can get it all in one swoop instead of hunting around for parts to get it installed.

    I am stuck to the bed or the couch for most of the year due to an illness, so any downtime due to hardware failure during the hard period could be long. Having my blurays on a central storage is a gift when I'm not able to change discs all the time.
    That is the same reason I ask for help here - usually I figure this stuff out by myself. If I don't get it done now I'll have to wait a year and start over.


    Planning on buying a 1u/2u virtualization server after this purchase if I manage to find one quick, and the fileserver could perhaps be run from here too.
    current guests:
    * mail
    * misc testing machines
    * misc developing envirnoments
    * wiki (gets hard to keep track of it all)
    * github
    * web
    All for internal use.


    At this point I feel like I'm just making things harder than they need to be, so I would really appreciate any answers or pointers to steer me in the right direction.
     
    #1
  2. pricklypunter

    pricklypunter Well-Known Member

    Joined:
    Nov 10, 2015
    Messages:
    1,340
    Likes Received:
    366
    Just some quick pointers...

    Chenbro or Supermicro, there's little room between them in reliability terms. 3U at a minimum, if you don't plan on making some modifications to quieten things down for home use, but 4U would be better from this viewpoint. Basically the larger that fans you can fit in the chassis, the slower you can spin them, when needed, therefore the less noise they produce. Pay particular attention to free airflow over your drives, heatsinks and exhaust, and especially use dampers in "spare" drive sleds as airflow will take the path of least resistance. While mirrorred disks are great for applications requiring performance, playing media files is not really in that category, nor for that matter is chuntering data back and forth at 1Gbps from your laptop to mapped drives. I would suggest that you look at only using mirrors for stuff that needs them, and as you have already mentioned using ZFS, choose raidz2 for your general storage needs :)
     
    #2
    fossxplorer likes this.
  3. MiniKnight

    MiniKnight Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    2,679
    Likes Received:
    745
    First when you say Norco, HP SAS expanders and the like that you saw here, you don't mean the 2010 articles that are now over 8 years old https://www.servethehome.com/sas-expanders-diy-cheap-low-cost-jbod-enclosures-raid/ right?

    Today, I'd just get this for what you're doing Supermicro CSE-847 4U 36-Bay X9DRD-EF 2x LGA2011 LSI-SAS9266-8i Storage Server | eBay

    You may be able to swap or sell the SAS RAID controllers in that for simple JBOD controllers if you want our of that box.

    For CPUs get two of these SR1AB INTEL XEON E5-2660V2 10 CORE 2.20GHz 95W PROCESSOR FOR DELL T620 R620 R720 658759132833 | eBay

    RAM get whatever DDR3 size you want for that big box.

    Instead of having FreeNAS, run Proxmox on it which does ZFS storage and virtualization. FreeNAS has an easier storage UI but virtualization is not as good as Proxmox. You'll only need 1 box and have 36 bays plus the virtualization server taken care of. Fewer things to go wrong.

    If you want to add more just get one of these 4U Supermicro 847E16-RJBOD1 45 HDD Bay 3.5" SAS2/SATA3 JBOD Storage Expander | eBay

    And you'll be at 71 drives. Done. HP SAS expanders were great in 2010 until maybe 2012 if they're the ones in that article, but technology has moved one. 8 years ago there were none of these great off lease servers around so that guide was necessary.
     
    #3
  4. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    2,087
    Likes Received:
    289
    SAS1 effectively had a 2TB limit. After that I was not aware of any limits.
     
    #4
  5. muthur6k

    muthur6k New Member

    Joined:
    Jun 21, 2018
    Messages:
    6
    Likes Received:
    0
    The server is going to be located in a rack in the basement, so the noice isn't of immidiate concern.
    I was thinking mirroring two and two drives and pool them.. I think. I might have forgotten what the plan was from the start.
    I need to get back into that once I'm finished with the hardware.
    The cost is high, but if a hardware failure is happening I might not be well enough to rebuild a raid at that time (can take months before I'm able to get down to the servers), so if the raid goes down I'd be able to access the drives directly. It might be a silly solution though.
    Perhaps it's smarter to just add a couple of spare drives or something like that.


    And yes, the sas expander thread I read was the one from 2010..
    Many of the threads and articles I've read lately has turned out to be old - I keep forgetting to look at the date until the end.
    93 still sounds just like a while ago too me...


    I see they can deliver cpu and ram from the listing you provided the link for.
    I might send them a mail and ask for a quote.
    Then cry a little.
    Then pay up.


    I have looked at this supermicro expander: SUPERMICRO CSE-847 CHASSIS 4U SAS + 2xPWS-1K28P-SQ | eBay - the shipping fee is about 1500 usd lower than the one you mentioned.
    I forgot to say that I'm located in Norway.
    The shipping fee for your server suggestion is 1000 USD - (at the higher end of the scale) so the price doubles + import charges..
    But this is hopefully a one time charge. If everything works I'd rather pay some up front than keep fixing it.
    The shipping fees are crazy though - they can vary from 200 to 2000 usd for the same equipment.


    I'm already running Proxmox, so that was the plan.

    But that raid controller installed is all I need to access all those 36 bays?
    And software raid vs hardware raid.. is the fear of the controller kneeling also outdated knowledge?
    I was originally thinking about using my cards in IT mode and use software raid.
    I found a thread about the LSI SAS9266-8i, but I'm not able to read it atm.

    thank you for the links and info - I'm one step closer to a solution now.
     
    #5
  6. muthur6k

    muthur6k New Member

    Joined:
    Jun 21, 2018
    Messages:
    6
    Likes Received:
    0
    Thank you, Evan - I had a hard time finding any information on this subject.
     
    #6
  7. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,252
    Likes Received:
    283
    lba64 => 64 bit addressing
    2^64 * blocksize (512b,4k) ~enough space for the next 5 years :D
     
    #7
    Evan likes this.
  8. nthu9280

    nthu9280 Active Member

    Joined:
    Feb 3, 2016
    Messages:
    849
    Likes Received:
    198
    There are active members from Norway & EU here. @Sable @_rand had some items for sale. Hopefully shipping within EU is much cheaper if they have the SM 847 chassis.
     
    #8
  9. muthur6k

    muthur6k New Member

    Joined:
    Jun 21, 2018
    Messages:
    6
    Likes Received:
    0
    I was just thinking of what if I'd need to return it for some reason - I'm out another 1000 usd..
    You are right, I should see if I'm able to find something closer to home.

    Found an expander in Norway for less than the shipping cost from USA:
    https://forums.servethehome.com/ind...o-45bay-jbod-847e16-rjbod1.17675/#post-169868
    SC847E16-RJBOD1 | 4U | Chassis | Products | Super Micro Computer, Inc.

    It's not ideal, but neither is the shipping cost of the others.


    Thank you for letting me know. It'll save me some money hopefully.
     
    #9
  10. muthur6k

    muthur6k New Member

    Joined:
    Jun 21, 2018
    Messages:
    6
    Likes Received:
    0
    #10
  11. BLinux

    BLinux Well-Known Member

    Joined:
    Jul 7, 2016
    Messages:
    1,442
    Likes Received:
    329
    when buying supermicro X9 motherboards, pay attention to the revision number. try to get something that is 1.2 or 1.02 or something like that. older revisions are likely not to support Ivy Bridge CPUs. *some* of the older revisions can be fixed to support Ivy Bridge, like 1.00A, but not 1.00.
     
    #11
  12. BackupProphet

    BackupProphet Active Member

    Joined:
    Jul 2, 2014
    Messages:
    639
    Likes Received:
    222
    I am from Norway and use Jetcarrier for shipping, they are awesome. A few times I have ordered pallets of servers that costed less to ship with Jetcarrier over the atlantic ocean than from CA to New Jersey

    Anyway, I would just get a barebone Supermicro 847A. Dont go the expander route, expander is even too slow for the hard drives and SSD's are becoming dirt cheap. The SSD prices as gone down dramatically the last few months.

    There are few 847A available though, you have to be patient to find one for less than 500 USD
    However there are plenty of 846A if you can live with 24 bays.

    I would also highly recommend getting v2 cpu, Intel 2650 V2 is available for 100 USD per cpu. As mentioned earlier you need to double check motherboard support for V2 cpus. For ZFS, Intel Optane 32GB is a great slog device. Cheap and fast. It has low TBW, but unless you only do 100% sync writes the Optane should last a few years.
     
    #12
  13. muthur6k

    muthur6k New Member

    Joined:
    Jun 21, 2018
    Messages:
    6
    Likes Received:
    0
    Thank you - I will try to get a 1.2 board instead.
    I now see some of the boards I've looking at has been lower than 1.2, but most does not specify rev.


    I was hoping to buy a complete, working system to make it as easy as possible.
    I see that buying components could perhaps be easier on me. The details are always there, but buying components allows me to focus on one part at a time. The item is usually sold before I can gather and digest information on all the parts.

    If I go with a 24 bay I guess I'd need an expander soon anyway, but I am looking at both options.

    Cpus.. I try to look up information on cpus but find the differences seems so minute. I was aiming for a v2, but just because I was told to.
    I have not been able to read up on that yet - I'm still trying to learn the difference between the different motherboards.
    All these almost-identical-numbers when naming stuff is not good for the head. When I compare two items I forget what the first was called when I shift focus to the second. It takes a lot of effort :)


    Took me a couple of days to read up on slog devices.
    I hadn't planned for the use of a slog device since it was originally meant to be a separate file server - but it looks like it could just as well be hosting all my guests.

    I've only skimmed through sync writes when reading up on zfs, but I thought sync writes was the way to go.
    In any case the storage will for the most part contain 4k isos - so I guess I'm not gonna wear out the slog-ssd with those.

    Some development/general storage area will be assigned, but you see how long I took answering these posts? I guess I'm not gonna wear out the slog with sync-writes either.


    Thanks both for all the information!
     
    #13
  14. nthu9280

    nthu9280 Active Member

    Joined:
    Feb 3, 2016
    Messages:
    849
    Likes Received:
    198
    It appears to be certain batches of X9 boards. I have older revisions of X9DRD-7LN4F-JBOD and a X9DRW-iF. Rev 1.02A or something like that. But they both work fine with E5-26xx V2. I initially had V1 CPUs in them updated the BIOS and replaced the CPUs. I only came to know of the issues after folks here were discussing it.
     
    #14
  15. BLinux

    BLinux Well-Known Member

    Joined:
    Jul 7, 2016
    Messages:
    1,442
    Likes Received:
    329
    yeah, there are 3 batches of X9 boards:

    1) rev 1.00 : these boards do not accept v2 CPUs even with latest BIOS and can't be fixed to accept v2 CPUs
    2) rev 1.00A or similar : these do not accept v2 CPUs, but can be reworked by Supermicro RMA to accept v2 CPUs.
    3) rev 1.02 or 1.2 or similar : these accept v2 CPUs out of the box.

    Until recently, I thought there was only #2 and #3 boards. Then I came across some rev 1.00 boards and after a lot of back and forth discussions with Supermicro RMA, found out they are not able to fix these boards to accept v2.
     
    #15
  16. pricklypunter

    pricklypunter Well-Known Member

    Joined:
    Nov 10, 2015
    Messages:
    1,340
    Likes Received:
    366
    If you go the SuperMicro route, my advice would be to try and find a chassis with the "A" version backplane, rather than the expander backplane, or get one and swap it over. Unless that is, you have other reasons for needing the expander. The reason I mention it, is that you can use 12Gbps SATA disks in those, as they are simply pass-thru ports but have the convenience of using Mini-SAS connections to keep wiring tidy :)
     
    #16
    BackupProphet likes this.
Similar Threads: help buying
Forum Title Date
Chassis and Enclosures Help with HP Procurve May 10, 2018
Chassis and Enclosures Chassis Help/Recommendation May 2, 2018
Chassis and Enclosures Need help powering individual Intel sled Apr 26, 2018
Chassis and Enclosures APC Smart-UPS 2200VA cable help Mar 12, 2018
Chassis and Enclosures C6100 ATX PSU help Feb 2, 2018

Share This Page