Home storage server

Discussion in 'DIY Server and Workstation Builds' started by Kei-0070, May 8, 2019.

  1. Kei-0070

    Kei-0070 Member

    Joined:
    May 3, 2019
    Messages:
    33
    Likes Received:
    13
    This project started out as a budget project to reuse old equipment that I had stashed around the house.

    This was the spec I started with:

    Gigabyte MA-790FXT-UD5P
    AMD Phenom II X4 955
    Corsair Dominator 4GB 1600
    128GB Samsung OEM SSD
    Nvidia Quadro FX3500
    LSI MegaRaid SAS 8888ELP
    Belkin USB 3.0 card
    Prolimatech Megahalems
    Coolermaster CM690-II

    [​IMG]

    Had to order a few bits, namely a new power supply, backplane, breakout cables, disks and fans.
    4x 2TB WD SE drives
    2x LSI CBL-SFF8087OCF-06M forward breakout cables
    2x ICYBOX 553SK SAS/SATA backplanes
    1x Noctua NF-F12 IndustrialPPC 3000RPM PWM
    5x Scythe Kama Flow2 1900RPM Fans
    550W Superflower Golden Green HX PSU

    [​IMG]
    [​IMG]

    Went with opensuse (bottle) at the time which worked well for a few years. 6 disks in RAID 5 benchmarked using the gnome disk utility.
    [​IMG]

    I then decided that I needed to increase the potential drive pool so I managed to pick up a brand new intel RES2SV240 expander card for a reasonable sum.
    [​IMG]

    That went in along with another WD 2TB se disk. This is the config the server has run as since October 2014.
    [​IMG]

    A year later, I swapped out the quadro FX card for an NVS310 that I was given in order to save some power. In 2017, the original opensuse bottle install got corrupted by a botched update so I updated it to leap 42.2 and subsequently 42.3 which it ran up until this year. I planned some big network updates this year in order to try and improve the bandwidth available to my main workstations and the server. I swapped out my dinky netgear GS108 for a juniper EX3300 24P. (dirt cheap new open box) The Cisco 1921 also got a VA-DSL-A ehwic added so I could remove the separate ISP modem.
    [​IMG]

    I had planned on upgrading the server mainboard to a spare amd 990fx/FX8320 that I had been given but sadly having done all the work to swap the board over, it only worked for about 24 hours before the board seemed to fail. (This was 2 weeks ago)
    [​IMG]

    The main reason behind upgrading the board was that I needed more pcie lanes in order to run a gpu, raid card and 10gbe NIC.
     
    #1
  2. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    625
    Likes Received:
    249
    I love that "rack"... :) Believe it or not, at one point, I had built something similar for about 20U worth of space. This was probably ~2008 or so, and back then, even basic four post racks were > $400 and 2x4s were cheap... :)
     
    #2
  3. Kei-0070

    Kei-0070 Member

    Joined:
    May 3, 2019
    Messages:
    33
    Likes Received:
    13
    That little wooden rack came from work. I could have had a whole 24U or 47U rack but lacked the space for it.

    The 990fx motherboard failure triggered a rather extreme OTT server upgrade:
    i5 9600k
    Asus Z390 Prime-A
    16GB Team Group Vulcan 3000C16

    [​IMG]

    Only issue I had was that my original megahalems (775/1366 only) cooler needed some DIY work in order to fit on 1151. I decided that the best approach would be to swipe a backplate from one of my other coolers and make it work with a mixture of mounting hardware. One noctua backplate combined with some bolts with the same threads as the original prolimatech parts and the noctua black plastic spacers filed down to match the thickness of the prolimatech metal spacers. The only bolts I had which were the correct thread and sufficient length had countersunk heads so I found some suitable washers for them. The upper mounting plates then needed the 775 holes filing out towards the 1366 mounting holes as 775 is 72mm spacing, 115x is 75mm and 1366 is 80mm.
    [​IMG]
    [​IMG]

    Backplate fitted.
    [​IMG]

    Mounting plates fitted.
    [​IMG]

    Test run with paste to see how the spread looked. IMO, pretty much as good as any stock mounting setup.
    [​IMG]

    Built. (except the mellanox connectx-3 in the top slot is a dud)
    [​IMG]

    I had a hell of job installing a new version of linux. Some kind of issue regarding the integrated intel gpu driver (i915) resulting in complete system freezes once you select an install option from the usb boot menu. After many hours of trying various kernel options via grub, I gave up and put the quadro back in. That immediately fixed the installation issues. Once I had fedora installed and setup using the quadro, I shut it down and took it out and re-enabled the intel igpu and haven't had a single issue since doing so. Had some fun setting up one samba share which turned out to be SELinux. Only one minor annoyance left to solve is that it keeps booting to the emergency console saying login or press control D to boot into default target. (pressing control D does work perfectly each time)

    I also took the opportunity to run the disk benchmark on the array as I've not rerun it since I built it nearly 5 years ago. Performance seems pretty decent considering the age of the raid controller. (2007)
    [​IMG]
     
    #3
    Boris, ari2asem and Marsh like this.
  4. Kei-0070

    Kei-0070 Member

    Joined:
    May 3, 2019
    Messages:
    33
    Likes Received:
    13
    These arrived in work today. The end is in sight now, just need to get another NIC for the server and some fibre.
    [​IMG]

    At present I have only got one OM3 MM fibre patch cable so I can't test it properly but it was extremely pleasing to see this once I connected my machine up.
    [​IMG]
     
    #4
  5. Jenfil82

    Jenfil82 New Member

    Joined:
    May 9, 2019
    Messages:
    18
    Likes Received:
    4
    Maybe a stupid question but did you consider FreeNAS at some point?
    Just wondering as I am evaluating how to build my own NAS for the moment, so only gathering data at the moment :)
     
    #5
  6. Kei-0070

    Kei-0070 Member

    Joined:
    May 3, 2019
    Messages:
    33
    Likes Received:
    13
    I never really considered it at the time, mainly as I didn't realise it could do more than be a NAS. I wanted my server to be able to be used for background tasks overnight when I want my desktop off. I've gotten used to dealing with linux in that time so it felt easier to stick with what I've gotten used to.
     
    #6
  7. acquacow

    acquacow Active Member

    Joined:
    Feb 15, 2017
    Messages:
    476
    Likes Received:
    234
    I went all low-power xeon-d and FreeNAS specifically because I wanted a 24x7 box that would run all my background utility VMs/etc. and wouldn't require my desktop.

    It runs a few linux VMx, runs plex in a jail, rsync's files to a remote location/etc.

    Then you get the benefits of ECC memory and ZFS's bit-rot protection on all of the data, something you don't really get with other systems.

    The onboard 10gige was also a bonus because my remote access speeds are the same as my local access speeds.
     
    #7
    Last edited: Jun 4, 2019
    T_Minus likes this.
  8. Kei-0070

    Kei-0070 Member

    Joined:
    May 3, 2019
    Messages:
    33
    Likes Received:
    13
    Finally finished sorting out the OS. Had a hell of a job dealing with setting up vncserver due to policykit issues making executing gui applications with elevated permissions difficult. Simple solution was to switch over to using x0vncserver instead which made a lot more sense as it means I'm not running a separate desktop. Still suffering from some intel i915 driver issues. Weirdly, the problems returned when I changed screen. Turning off the window compositing in xfce stopped the shed load of errors and it makes little difference to the usability of the OS.

    Memory usage seems a tad high but from what I can tell, most of it is the lsi megaraid storage manager server. (1.7GB being used by java)
    [​IMG]


    All that I need to sort now is finding a replacement SFP+ NIC for 10Gbe and re-cable the house. I've been looking at solarflare NICs as mellanox connect-x 3's are pretty expensive still and having had one DOA makes me less keen on them. Another intel x710 is a potential option albeit the most expensive.
     
    #8
    Last edited: May 13, 2019
  9. Kei-0070

    Kei-0070 Member

    Joined:
    May 3, 2019
    Messages:
    33
    Likes Received:
    13
    Replacement NIC bought. After some lengthy research, I decided to try solarflare and picked up a SFN7122F card nice and cheap to replace the DOA Mellanox CX3.
    [​IMG]

    Things have been running well so far. Ironed out the vivaldi framework ram issues but plex DLNA server seems hell bent on slowly chewing through ram.
    [​IMG]
     
    #9
  10. Kei-0070

    Kei-0070 Member

    Joined:
    May 3, 2019
    Messages:
    33
    Likes Received:
    13
    Finally picked up a second OM3 fibre so I can do a test run at 10Gb. Performance in one direction looks pretty good, but not so good in the other direction.

    Running my threadripper workstation with the intel x710 as the iperf server net 7Gbps with the 9600k/solarflare NAS as the client. The other way around only got 3.5Gbps. Currently I've adjusted the Tx/Rx buffers on the intel card to their maximums but not made any changes to the solarflare as I'm less familiar with linux driver tweaks.
    [​IMG]
     
    #10
  11. Kei-0070

    Kei-0070 Member

    Joined:
    May 3, 2019
    Messages:
    33
    Likes Received:
    13
    I bought some OM3 fibres from FS along with some LC-LC links for the wall boxes I got from RS a few months back. Wall boxes should be pretty resilient as the fibres will enter at the base of the boxes keeping the connection virtually flush with the wall to avoid breaking them.
    [​IMG]

    Since I now have sufficient fibres for a proper test run, it'd be rude not to. Just a quick test to make sure all ports and fibres work ok, I connected the server with two links using the 3 avago transceivers I have. All working great, the juniper doesn't seem to mind the avago transceiver.
    [​IMG]

    Quick file transfer test from my pc to the server. This is off an nvme ssd to avoid sata bottlenecks. Normal transfers will be a fair bit slower sadly but at least now I can have two pc's copying to the server and two tv's streaming from it without interruption.
    [​IMG]
     
    #11
  12. Kei-0070

    Kei-0070 Member

    Joined:
    May 3, 2019
    Messages:
    33
    Likes Received:
    13
    Bought some new fans for the juniper switch to try and quieten it down a bit without triggering a fan failure warning.

    2x 40x40x28mm 12,500rpm San Ace 109P0412J3013 fans instead of 18,000rpm. Idle fan voltage is 4.5V so something like a noctua that maxes out at 5000rpm would run too slow at idle.
    A pack of molex plug bodies and pins for fan headers (as the fans above come with bare ends)
    A pack of molex ATX pins and sockets for making up custom PSU cables in the future
    A molex pin extraction tool for the ATX pins
    Crimping dies for insulated and un-insulated terminals
    [​IMG]

    Fitted and now tested. All working ok, significantly quieter both at idle and full speed and no fan failure warnings. Best of all, it is still idling at 44 degrees. It does make me wonder if I could have gotten away with the 9500rpm Delta FFB0412VHN or the 11000rpm San Ace 109P0412G3013.
    [​IMG]

    Server cpu temps looking rather chilly at this time of year.
    [​IMG]
     
    #12
    Last edited: Nov 30, 2019
    dawsonkm likes this.
Similar Threads: Home storage
Forum Title Date
DIY Server and Workstation Builds Suggestions for home network/storage build out Jan 12, 2017
DIY Server and Workstation Builds Advice needed for first home storage rack server - 2 options! Oct 6, 2016
DIY Server and Workstation Builds storage for home esxi cluster Oct 2, 2016
DIY Server and Workstation Builds DIY Home lab SAN Storage Jan 21, 2016
DIY Server and Workstation Builds Cheap storage server (home use, maybe esxi) Dec 8, 2015

Share This Page