NVMe storage server on a budget

Discussion in 'DIY Server and Workstation Builds' started by altmind, Nov 8, 2019.

  1. altmind

    altmind Member

    Joined:
    Sep 23, 2018
    Messages:
    64
    Likes Received:
    14
    I'm looking to build/buy a high-performance NVMe storage server for virtualization purposes.

    I need a server with at least 8 hot-swappable nvme (or with storage, adaptable to nvme drives, u.2 maybe?) that does not break the bank(prefferably, under $10K without the drives)

    Plain nvme on-motherboard is sparse and not hotswappable.

    Propietary solutions like Supermicro classis+Intel EDSFF are expensive.

    Is there some way to serve 8 nvme on a bugdet?

    --

    Frankly speaking, I'm not sure what I'm looking for. Maybe a server, but maybe there is a low-latency nvme DAS or maybe I can find hot-swappable pci-to-4xm.2 cards?
     
    #1
    Last edited: Nov 8, 2019
  2. Cixelyn

    Cixelyn New Member

    Joined:
    Nov 7, 2018
    Messages:
    8
    Likes Received:
    8
    Does the Gigabyte R272-Z32 work for your purposes?
    STH had a review on it recently: https://www.servethehome.com/gigabyte-r272-z32-24x-nvme-amd-epyc-7002-server-is-a-home-run/

    Holds 24x NVMe drives which may be overkill for your purposes.

    Base chassis seems to be around $2k or so, so the full system should come out well under $10k depending on how you kit out processor + RAM.

    We're also considering building out a similar storage system; would love to hear what you end up choosing!
     
    #2
  3. altmind

    altmind Member

    Joined:
    Sep 23, 2018
    Messages:
    64
    Likes Received:
    14
    Gigabyte R272-Z32 looks like exactly what I need.
    Does it require these u.2 drives? m.2-2280 can easily be adapted to u.2 drives, right?
    [​IMG]
     
    #3
  4. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,825
    Likes Received:
    1,484
    Why not buy the 2.5" U.2 instead of the m.2 drive + converter?

    You can use converters like those (I have a handful for testing), but for purpose-built nVME chassis that's made for 2.5" U.2 that is what I go with :) (2.5" U2)

    You can also go "cheap" and buy on ebay Intel hotswap NVME drive cage and adapt to a 5.25" or DIY a DAS from it, for NVME HBA I'm sure there are correct external options, but again, depending what you're doing with this you could route the inner cables out another PCIE slot too. This isn't a clean, perfect solution but if you wanted to test NVME or have something "in office" this may work to add to an existing system, and cheaper than a $5-10k$ NVME chassis :D
     
    #4
    ehorn and zack$ like this.
  5. zack$

    zack$ Active Member

    Joined:
    Aug 16, 2018
    Messages:
    205
    Likes Received:
    75
    #5
    T_Minus likes this.
  6. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,825
    Likes Received:
    1,484
    #6
  7. zack$

    zack$ Active Member

    Joined:
    Aug 16, 2018
    Messages:
    205
    Likes Received:
    75
    Amazing how often this happens...jinx !!!:D
     
    #7
  8. altmind

    altmind Member

    Joined:
    Sep 23, 2018
    Messages:
    64
    Likes Received:
    14
    I thought that consumer-grade nvme would be cheaper than enterprise-grade, but larger u.2 drives. Seems they are not.

    Nytro 5000 Enterprise SSD 1920GB u.2@275
    Samsung 970 EVO SSD 2TB nvme@400

    Do you have any recommended u.2 2tb models? The work load is <0.4DWPD.
     
    #8
    Last edited: Nov 8, 2019
  9. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,261
    Likes Received:
    750
    For a virtualisation storage you should care about filesystem consistency of virtualized disks. This can mean ZFS but in any case this means a secure write behaviour. Without ZFS this means a raidcontroller with battery/flash protection, with ZFS this is guaranteed by a ZIL/Slog device.

    In any case you need a powerloss save behaviour what means enterprise grade datacenter SSD or NVMe (Intel, Samsung, Micron etc). This is independent from the protection of a BBU cache or an Slog as they rely as well on the fact that a committed write must be on stable storage. They only protect the OS or controller write cache but not against an unsecure SSD behaviour.

    For a cheap solution with 4 NVMe per 5,25" bay, you may also check https://www.icydock.com/goods.php?id=255 (5,25 case for 4 xU.2 NVMe).

    As a suitable mainboard I would prefer a mainboard like the X11SPH-nCTF | Motherboards | Products | Super Micro Computer, Inc. where you can connect quite easily up to 10 NVMe (2x M.2, 2 x U.2 and 2 x PCI-e adapters with bifurcation)

    This board already has 10G and 12G SAS what would also allow to add high performance SAS disks like WD SS 530 that are near to faster NVMe and propably faster than cheaper NVMe. With the new special vdev feature of ZFS it would even allow mixed pools from disks or ssd and nvme where you can decide which VMs land on NVMe and which on the slower disks.
     
    #9
    Last edited: Nov 9, 2019
    psannz, Evan, ehorn and 1 other person like this.
  10. altmind

    altmind Member

    Joined:
    Sep 23, 2018
    Messages:
    64
    Likes Received:
    14
    these icydocks may be fine for a tower, but are a complete miss for a rack server.
     
    #10
  11. ehorn

    ehorn Active Member

    Joined:
    Jun 21, 2012
    Messages:
    342
    Likes Received:
    52
    Why do you say that? There are plenty of 3u, 4u chassis solutions. With your budget, you will need to be creative. :)
     
    #11
  12. zack$

    zack$ Active Member

    Joined:
    Aug 16, 2018
    Messages:
    205
    Likes Received:
    75
    #12
    Last edited: Nov 9, 2019
  13. altmind

    altmind Member

    Joined:
    Sep 23, 2018
    Messages:
    64
    Likes Received:
    14
    All the servers have different swappable disk trays mounting mechanisms and different sets of screwholes.
    How do you expect to mount this in a rackable server? Get a dremel and cut the front panel, then drill the server chassis to scew it down?
     
    #13
  14. zack$

    zack$ Active Member

    Joined:
    Aug 16, 2018
    Messages:
    205
    Likes Received:
    75
    I've used them in various rackmounted chassis from chenbro, norco and istarusa...no issues.

    Are you building this server from scratch or using an existing chassis?
     
    #14
  15. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    2,859
    Likes Received:
    427
    @gea is not wrong about the new fast SAS SSD, they do really well in a storage pool
     
    #15
Similar Threads: NVMe storage
Forum Title Date
DIY Server and Workstation Builds Looking for help in building small NVMe storage based system Apr 9, 2018
DIY Server and Workstation Builds DIY WS2016/Storage Spaces Box (NVMe -> SSD -> Spinners) May 22, 2017
DIY Server and Workstation Builds Epyc ZFS Server - NVMe vs. SATA/SAS Feb 10, 2018
DIY Server and Workstation Builds Help selecting NVMe drives to finish build May 22, 2017
DIY Server and Workstation Builds Old storage second life Nov 28, 2019

Share This Page