NVMe storage server on a budget

altmind

Active Member
Sep 23, 2018
148
50
28
I'm looking to build/buy a high-performance NVMe storage server for virtualization purposes.

I need a server with at least 8 hot-swappable nvme (or with storage, adaptable to nvme drives, u.2 maybe?) that does not break the bank(prefferably, under $10K without the drives)

Plain nvme on-motherboard is sparse and not hotswappable.

Propietary solutions like Supermicro classis+Intel EDSFF are expensive.

Is there some way to serve 8 nvme on a bugdet?

--

Frankly speaking, I'm not sure what I'm looking for. Maybe a server, but maybe there is a low-latency nvme DAS or maybe I can find hot-swappable pci-to-4xm.2 cards?
 
Last edited:

Cixelyn

Researcher
Nov 7, 2018
40
24
8
San Francisco
Does the Gigabyte R272-Z32 work for your purposes?
STH had a review on it recently: https://www.servethehome.com/gigabyte-r272-z32-24x-nvme-amd-epyc-7002-server-is-a-home-run/

Holds 24x NVMe drives which may be overkill for your purposes.

Base chassis seems to be around $2k or so, so the full system should come out well under $10k depending on how you kit out processor + RAM.

We're also considering building out a similar storage system; would love to hear what you end up choosing!
 

altmind

Active Member
Sep 23, 2018
148
50
28
Gigabyte R272-Z32 looks like exactly what I need.
Does it require these u.2 drives? m.2-2280 can easily be adapted to u.2 drives, right?
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,007
1,569
113
CA
Why not buy the 2.5" U.2 instead of the m.2 drive + converter?

You can use converters like those (I have a handful for testing), but for purpose-built nVME chassis that's made for 2.5" U.2 that is what I go with :) (2.5" U2)

You can also go "cheap" and buy on ebay Intel hotswap NVME drive cage and adapt to a 5.25" or DIY a DAS from it, for NVME HBA I'm sure there are correct external options, but again, depending what you're doing with this you could route the inner cables out another PCIE slot too. This isn't a clean, perfect solution but if you wanted to test NVME or have something "in office" this may work to add to an existing system, and cheaper than a $5-10k$ NVME chassis :D
 
  • Like
Reactions: ehorn and zack$

zack$

Active Member
Aug 16, 2018
434
163
43
Why not buy the 2.5" U.2 instead of the m.2 drive + converter?

You can use converters like those (I have a handful for testing), but for purpose-built nVME chassis that's made for 2.5" U.2 that is what I go with :) (2.5" U2)

You can also go "cheap" and buy on ebay Intel hotswap NVME drive cage and adapt to a 5.25" or DIY a DAS from it, for NVME HBA I'm sure there are correct external options, but again, depending what you're doing with this you could route the inner cables out another PCIE slot too. This isn't a clean, perfect solution but if you wanted to test NVME or have something "in office" this may work to add to an existing system, and cheaper than a $5-10k$ NVME chassis :D
Amazing how often this happens...jinx !!!:D
 

altmind

Active Member
Sep 23, 2018
148
50
28
Why not buy the 2.5" U.2 instead of the m.2 drive + converter?
I thought that consumer-grade nvme would be cheaper than enterprise-grade, but larger u.2 drives. Seems they are not.

Nytro 5000 Enterprise SSD 1920GB u.2@275
Samsung 970 EVO SSD 2TB nvme@400

Do you have any recommended u.2 2tb models? The work load is <0.4DWPD.
 
Last edited:

gea

Well-Known Member
Dec 31, 2010
2,486
837
113
DE
For a virtualisation storage you should care about filesystem consistency of virtualized disks. This can mean ZFS but in any case this means a secure write behaviour. Without ZFS this means a raidcontroller with battery/flash protection, with ZFS this is guaranteed by a ZIL/Slog device.

In any case you need a powerloss save behaviour what means enterprise grade datacenter SSD or NVMe (Intel, Samsung, Micron etc). This is independent from the protection of a BBU cache or an Slog as they rely as well on the fact that a committed write must be on stable storage. They only protect the OS or controller write cache but not against an unsecure SSD behaviour.

For a cheap solution with 4 NVMe per 5,25" bay, you may also check https://www.icydock.com/goods.php?id=255 (5,25 case for 4 xU.2 NVMe).

As a suitable mainboard I would prefer a mainboard like the X11SPH-nCTF | Motherboards | Products | Super Micro Computer, Inc. where you can connect quite easily up to 10 NVMe (2x M.2, 2 x U.2 and 2 x PCI-e adapters with bifurcation)

This board already has 10G and 12G SAS what would also allow to add high performance SAS disks like WD SS 530 that are near to faster NVMe and propably faster than cheaper NVMe. With the new special vdev feature of ZFS it would even allow mixed pools from disks or ssd and nvme where you can decide which VMs land on NVMe and which on the slower disks.
 
Last edited:

altmind

Active Member
Sep 23, 2018
148
50
28
these icydocks may be fine for a tower, but are a complete miss for a rack server.
 

ehorn

Active Member
Jun 21, 2012
342
52
28
Why do you say that? There are plenty of 3u, 4u chassis solutions. With your budget, you will need to be creative. :)
 

zack$

Active Member
Aug 16, 2018
434
163
43
Last edited:

altmind

Active Member
Sep 23, 2018
148
50
28
All the servers have different swappable disk trays mounting mechanisms and different sets of screwholes.
How do you expect to mount this in a rackable server? Get a dremel and cut the front panel, then drill the server chassis to scew it down?
 

zack$

Active Member
Aug 16, 2018
434
163
43
I've used them in various rackmounted chassis from chenbro, norco and istarusa...no issues.

Are you building this server from scratch or using an existing chassis?
 

Evan

Well-Known Member
Jan 6, 2016
3,070
512
113
@gea is not wrong about the new fast SAS SSD, they do really well in a storage pool