Need Advice Building a 16+ ZFS NAS

Discussion in 'DIY Server and Workstation Builds' started by Thomas H, Oct 17, 2019.

  1. Thomas H

    Thomas H Member

    Joined:
    Dec 2, 2017
    Messages:
    55
    Likes Received:
    20
    I want to build a 16 drive ZFS NAS with 10G NIC. Possibly want to expand in the future. Currently, I have 10 HDDs and 6 SSDs. These will be in two ZFS RAIDZ2 pools respectively. General usage will be slower HDDs for media and faster SDDs for VMs. For 10G networking I have HP 649281-B21 adapter cards. I have several servers, Lenovo ThinkServer TS440 and Dell PowerEdge R330, but they are not ideal for this. Both have limited drive bays and PCIe expansion capabilities. TS440 can do 8X drives with 1xPCIe3 x16, 1xPCIe2 x4 and R330 can do 4X drives with 2xPCIe3.

    I have seen threads like Supermicro 836 chassis, backplane, SAS expanders but do not know how those work. What do you recommend?
     
    #1
  2. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,273
    Likes Received:
    752
    SuperMicro cases are in general professional quality but very loud (serverroom cases).
    As an option, you can use Jbod cases. You connect them via exernal 12G SAS to one of your servers.

    The cases are available with an expander or only miniSAS connectors to connect disks directly to an HBA.
    You need then enough ports on the HBA(s). In general expander solutions are best with SAS disks while work with Sata.

    For fast SSDs, look at 12G SAS HBAs like Broadcom 9300 or OEM. For disks, 6G HBAs like a 9207 or even 2008 chipset based ones are ok (original BroadCom or OEM from Dell, HP, IBM etc). If possible use an HBA with IT firmware, IR firmware is ok, raid-5 firmware is not ok. In any case, use only BroadCom HBA with LSI chipsets like 3008, 2307 or 2008.

    The HP 649281-B21 may limit your OS choices to Linux. For ZFS, I would prefer Unix with Solarish (where ZFS comes from) or Free-BSD solutions. I would more prefer Intel based nics where driver support in general is best.

    Two Pools (one from disks and a faster one from SSD) is a good solution. With the very new Open-ZFS Allocation Class feature it may be possible to use one pool with disks based vdevs and SSD based vdevs where you can use the SSD vdevs for metadata, dedup tables or small io ex for dedicated filesystems for VMs, see https://forums.servethehome.com/ind...ses-performance-benchmarks.26111/#post-241295
     
    #2
  3. csp-guy

    csp-guy Member

    Joined:
    Jun 26, 2019
    Messages:
    88
    Likes Received:
    19
    Hi Thomas,

    I think in home enviroment (if we are talking home usage of this hardware) you can spare some money.

    If you have a good mid/big tower case with a lot of 5.25 space, you can use this 5.25->3.5 hdd cage.

    5-in-3 Device Module Hard Disk Cage SAS/SATA Expander Enclosure

    Few weeks ago I bought 3 of them, it is 15 disk storage capacity, and I will use in a well cooled atx midi tower.

    Cheers.
     
    #3
    Last edited: Oct 18, 2019
  4. BeTeP

    BeTeP Active Member

    Joined:
    Mar 23, 2019
    Messages:
    246
    Likes Received:
    104
    how so? Mellanox ConnectX-3 is one of best supported cards.

    Keep your R330 and add a cheap external JBOD enclosure to it.
     
    #4
    Thomas H likes this.
  5. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,682
    Likes Received:
    411
    He's talking about solarish operating systems :D
     
    #5
  6. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,273
    Likes Received:
    752
    Yes, from the main three OS families Linux, Windows and Unix (Free-BSD. OSX, Solarish), only the first two have support for nearly any hardware while Unix has its own advantages in some areas with storage one of them.
     
    #6
  7. BeTeP

    BeTeP Active Member

    Joined:
    Mar 23, 2019
    Messages:
    246
    Likes Received:
    104
    I am aware of @gea's affinity. The card is fully supported in Solaris and FreeBSD. I am not 100% about macOS. But if they don't it's just because Apple chose to blacklist it for some reason.
     
    #7
  8. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,273
    Likes Received:
    752
    Solaris and Free-BSD: yes
    Illumos (free Solaris fork): no
    OSX: no (I suppose)
     
    #8
  9. Thomas H

    Thomas H Member

    Joined:
    Dec 2, 2017
    Messages:
    55
    Likes Received:
    20
    Are you saying SuperMicro storage cases like SC826, SC836, SC846, SC847, etc. are very loud? Any JBOD cases you recommend?

    I am starting to read up on SAS expanders. The 6X SSDs I have are HGST HUSMR1650ASS204 (aka Ultrastar SSD1600MR) drives. If I have 6 SSDs (12Gb/s SAS) and 10 HDDs (SATA3) in a JBOD chassis on a single 12G SAS expander connection, will performance be severely crippled (by the SSDs)?

    The HBAs I have are Dell Perc H330 from my R330s. I am planning on flashing them to HBA IT mode for ZFS. Will these work if I go JBOD with expander route?
     
    #9
  10. BeTeP

    BeTeP Active Member

    Joined:
    Mar 23, 2019
    Messages:
    246
    Likes Received:
    104
    If your R330 would have been 8xSFF instead your your 4xLFF - the obvious way to go would be keeping your 12G SAS SSDs in the R330 connected to H330 and getting a separate cheap 6G SAS HBA for the external JBOD.

    Personally I prefer HP and I have replaced front drive cages in HP servers with more suitable config (from LFF to SFF and vice versa) many times before. The previous generation parts are easily available and rather inexpensive. I am not sure if that's an option for Dell.
     
    #10
    Last edited: Oct 19, 2019
    Thomas H likes this.
  11. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,682
    Likes Received:
    411
    It depends :D
    There are different versions of each chassis with different fans and psus.

    The older chassis had the lowest rpm system and rear fans (5k rpm) but psus with fans at 13k rpm running even when the server was shut down. Replace the psus and the pdb and you're good to go.

    Revision B chassis have slightly faster system and rear fans 6.3~7k rpm fans (these fans don't have that annoying whine as the chassis with the slower 5k rpm fans) and sq psus that turned off when the system was shut down. I would look for such a chassis. (They also have space on the back for two additional sata/sas storage devices)
    (~41 dba measured from ~10cm behind the rear fans with a nexus 5x @ 18° C ambient and 30-35° C system temperature)

    The newest versions of these chassis are designed for the scalabe/epyc cpus/tesla accelerators and have system fans that can go up to 11k rpm. You don't want these for a media server :D
     
    #11
    Thomas H likes this.
  12. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,273
    Likes Received:
    752
    SM cases are prepared to be filled up with high performance disks under steady load. Then the cooling is adequate. With less or low power disks outside a datacenter you may want different (more silent) fans.

    about SAS and expander
    Expanders are connected by one or two miniSAS cables with an HBA. Such a cable combines 4 SAS ports and allows to connect a full jbod case with 24 disks or more via the expander. Performance wise, you can combine slower and faster disks without problem but as all data must go through miniSAS, this can limit performance.

    ex.
    The combined performance of 4 x 6G SAS is 24G. If you count 3G for a mechanical disk, the SAS cable allows up to 8 disks to perform with full performance. This doubles with 12G SAS HBAs and expanders or when you connect the expander with 2 x miniSAS cables.

    If you connect Sata SSDs with around 6G performance, you reach the limit with half the disks. If you use 12G SAS SSDs like WD Ultrastar SS530 you use 12G per disk or even 24G when using both SAS ports. In such cases the expander can be a bottleneck.

    As SAS offers a more robust protocol and data transmission, I always prefer SAS disks over Sata disks with an expander. With a 16 (or 24) disk case I would always prefer 2 x 8 port or 1 x 16 port HBA and avoid the expander. Faster and more robust with propably less costs.
     
    #12
    ReturnedSword and Thomas H like this.
Similar Threads: Need Advice
Forum Title Date
DIY Server and Workstation Builds First Time Builder Needs Advice Aug 3, 2019
DIY Server and Workstation Builds Need RAM replacement advice Feb 8, 2019
DIY Server and Workstation Builds need advice about quad GPU build for cuDNN Feb 4, 2019
DIY Server and Workstation Builds Need advice on Massive New Plex server Nov 5, 2018
DIY Server and Workstation Builds Need some advice on finalizing my massive NAS upgrade Oct 9, 2018

Share This Page