ITRonin's FreeNAS lab server

Discussion in 'DIY Server and Workstation Builds' started by itronin, Dec 21, 2018.

  1. itronin

    itronin Active Member

    Nov 24, 2018
    Likes Received:
    Build’s Name:
    Operating System/ Storage Platform: Freenas 11.2-RELEASE
    CPU: e3-1230
    Motherboard: Tyan S5510GM3NR
    Chassis: Corsair Carbide 300R 1st generation modified for 6 x 5.25” Bays
    Zpool Disk 3 x 4 Raid Z1 with log
    8 x 600GB HGST Netapp X422 10K SAS HDD
    4 x 600GB Toshiba AL13SEB600 10K SAS HDD
    2 x 80GB intel DC3500 SATA SSD on 6Gbps SATA
    Zpool SSD 6 x Mirror
    12 x 400GB Hitachi HUSML4040ASS601 SAS SSD
    RAM: 4x 8GB ecc udimm
    Add-in Cards:
    MCX354A-QCBT (649281-B21) flashed to a MCX354A-FCBT 40Gbe
    Dell Perc H310 HBA flashed to LSI IT Mode
    HP Sas Expander flashed to latest fw package
    Power Supply: Corsair HX750
    Other Bits:
    6 x Icy Dock MB994SP-4S Hot Swap for 24 2.5” 15mm SAS hot swap bays
    Corsair H60 AIO cooler
    Corsair SP140 Blue Fans
    Corsair SP120 Blue Fans
    fan extension cable
    6 x SFF8087-SATA cables
    2 x SFF-8087 to SFF-8087 cables
    5.25” 4 bay drive cage cut from 1999 ATX (V1) Lab system case
    JB Weld Black to build the lower cross brace fascia
    Incept Date: 11/2018 Updated 3/2019

    Usage Profile: Provides 10Gbe NFS storage to my esxi systems via storage vlan. This system replaces a Synology DS1010+ with DX510 that was providing NFS via 1Gbe vlans.

    Other information:
    This system has non hot swap space for 8 x 2.5” devices in the remaining lower drive bay using dual drive adapter brackets.

    I lifecycle large chunks of my lab every 5-8 years depending on cash flow and try and get the most bang for the buck. Only in the last 10 years was I able to start writing it off as before that I worked directly for corporations. First lab was 1999, then 2007, then 2012 and now 2018. I still have a chassis, motherboard, cpu, memory and some SCSI devices from 1999 though I am about to punt those to the recyclers if I can get some data off some old DAT tapes.

    (Q) Why the old spinners?
    (A) I have customers with 24 bay drive servers/shelves/NAS which are mostly spinners and will mimic scenarios for support or projects. My customers are mostly SMB and we try and get 7-8 years out of their server and network infrastructure.

    (Q) Why did you build this monstrosity instead of buying a 24 Bay P2000 or an 8/9 5.25” Bay Mid-tower?
    (A) Because I like mid-towers and got it into my head I could build this re-using a compute node I had on hand from the 2012 evolution. 4 x 300R’s fit in each bread rack level very well. Mid towers are a bit more manageable for me but I have to admit this thing weighs about what a 2U server weighs but it is a bit less cumbersome.

    (Q) That is an awful lot of pricey Icy Dock kit. Okay not a question.
    (A) Hot Swap lets me reconfigure very easily for testing things like vSAN or the drives themselves. I’ve been buying ICY Dock products for a long time. I cleaned out the SAS Hot Swap inventory from a business that seems to be dumping their ICY Dock Products and I got a few more vi eBay.

    (Q) The fascia work is a bit messy. Again not a question.
    (A) First time trying to mod fascia. First time working with epoxy. I’m not the most artistically/mechanically inclined person but what I build is usually pretty solid just not that pretty. My OCD kicks in when I look at the lower fascia brace and realize its pushed out on one side and makes the drive cages look like they aren’t level across the front.

    (Q) Why the HP SAS expander instead of another HBA?
    (A) Because I wanted to play with it, and the lower drive bay will accommodate 8 2.5” drives so I’m above 24 drives.

    (Q) Why the LED fan lights?
    (A) I like blinky lights. Makes me feel fuzzy. Practical reason is sometimes my neighbor has to be my remote hands when I am on the road and colored systems help them find things quickly. But really cause I like blinky lights. Compute Nodes have a Red fan in the lower fan position. Labels are on the back.

    (Q) What is the power usage?
    (A) No idea, have not measured but my electric bill did not change when I turned off the synology it replaced and turned this on.

    (Q) How quiet is this system?
    (A) its okay. Its about as quiet as a synology DS2411+ and DX1211 combined. It lives in the basement with the bodies and they don’t seem to mind. ;-).

    Front.jpg Front bezel removed.jpg Side.jpg
    Last edited: Apr 6, 2019
    Mjac, TomUK, Lix and 3 others like this.
  2. T_Minus

    T_Minus Moderator

    Feb 15, 2015
    Likes Received:
    How's performance? Ran some tests?
  3. itronin

    itronin Active Member

    Nov 24, 2018
    Likes Received:
    Not really, not yet. Customer asks picked up at the end of this year so I'm booked out. Hoping to start seeing what it will do in January.
    T_Minus likes this.
  4. PGlover

    PGlover Active Member

    Nov 8, 2014
    Likes Received:
    Can you please explain in more detail what services you provide to SMB customers.
  5. itronin

    itronin Active Member

    Nov 24, 2018
    Likes Received:
    hmm not to hijack my own thread or to turn into an ad - IT Ops and infrastructure. My SMB customers are either self supporting for end user applications or out source that support (T1 to some T2). Escalations to T2 & T3, network, infrastructure, cloud management, monitoring, roadmaps, growth, budgeting, and vendor management they hand to me which is either onsite (local or travel) or remote based. I'm also part of a small group of consultants with complimentary skills and we come together for larger enterprises performing evals, tiger team work, and some cases soup to nuts transformations or IT turn-arounds.
  6. itronin

    itronin Active Member

    Nov 24, 2018
    Likes Received:
    I have updated the original post with the current config.
    Performance numbers are Meh - okayish.

    slow progress on doing anything particularly functional (lack of time) but last weekend I did have a chance to do some preliminary testing using ATTO.

    But no tuning or tweaking except for bumping the NFS server count to 6. No playing with iSCSI for now.

    Overall things saturate at about 1.08GBps read/write largish blocks (pics attached) on the SSD pools regardless of Mirror or Raid Z1 vdevs.

    Perf results lead me to believe I'm throttling somewhere. SLOG has no impact on the disk performance .
    Sync is NOT disabled on datasets. Not close enough for wire speed for me to think its the MTU size.

    More specifics:

    Switch ICX6610-48P, vmware hosts 10Gbe tagged VLANs for vMotion and NFS, freenas 40Gbe tagged VLAN for NFS . Using standard 1500 MTU and probably won't change that as its not worth the hassle.

    Freenas server stock config except NFS server count was changed from 4 to 6
    Pools: SSD's tested in 6x mirror and 4x Raid Z1; Disk 4x Raid Z1
    HP 40Gb IB xflash to Mellanox ConnectX3 40Gbe code. x8 PCIe 3.0 slot.
    Single PERC H310 xflash to LSI 9211 P20. x8 3.0 PCIe slot. (2.0 PCIe device)
    single HP expander, latest FW with 2 x 4 channels to the PERC
    all devices show negotiated at 6Gbps

    vmware hosts stock config
    (vmware51) Tyan S5510GM3NR, E3-1230, 32GB 1333 udimm, Mellanox ConnectX3 10Gbe x8 PCIe 3.0 slot.
    (vmware61) SM X9SRL-F, E5-2670v2, 128GB PC3L 1333, Mellanox ConnectX3 10Gbe x8 PCIe 3.0 slot

    Test client:
    Windows 10
    120GB virtual disk thin provisioned, 4 CPU, 16GB

    vm61-SSD41.jpg vm61-SSD41.jpg vm51-SSD41.jpg vm61-Disk41.jpg vm51-Disk41.jpg
Similar Threads: ITRonin's FreeNAS
Forum Title Date
DIY Server and Workstation Builds Freenas build, with a rack chassis Dec 10, 2019
DIY Server and Workstation Builds Help! I’m building a small FreeNAS server Nov 6, 2019
DIY Server and Workstation Builds HP Microserver Gen10 vs Budget MiniITX FreeNAS Build - 500€ - Europe Aug 4, 2019
DIY Server and Workstation Builds New ESXi/Proxmox/FreeNAS server build advice Feb 19, 2019
DIY Server and Workstation Builds NAS build - 32 disk custom cabinet. Freenas. Dec 16, 2018

Share This Page