All NVME Epyc Small Server

[Hefferbub]

I'm looking to build a server to host 6-10 Linux servers on Proxmox for my small ISP. Not a huge workload, nor do I need tons of storage (<4TB after redundancy).

I currently have a X8DTU generation 2U Supermicro I bought used, which is getting flakey. I'm looking for something relatively fast and quiet and reliable, so I want to go all NVME SSD.

Here is what I am thinking so far, and I would appreciate any comments:

$650 ASRock Rack ROMED8-2T Epyc Rome motherboard. This seems like a great fit because it comes with 6 "SLIMLINE NVMe x8" ports on the motherboard which (If I understand things correctly) allows me to hook up 6 U2 2.5" NVME drives with the right cables.

$1160 EPYC 7302P 16C/32T processor.

$720 8@$90 Kingston Technology 16GB 3200MHZ DDR4 ECC Reg CL22 DIMM 2RX8 Micron E Idt (On supported memory list for MB)

$1600 - 4@$400 2.5” Micron 7300 Pro U2 NVME SSDs 1.9 TB

$152 - 4@$38 - Cables for slim NVME to U2: Slim SAS SFF-8654 4i Straight to SFF-8639 U.2 Cable

$110 Antec Performance Series P101 Silent Black 0.8mm SPCC ATX Mid Tower Case with 8 x 3.5" HDD / 2.5" SSD Removable Bays

$199 Seasonic PRIME TX-750, 750W 80+ Titanium, Full Modular Power Supply

$80 Noctura 140mm CPU Fan for SP3 socket

$110 M2 SSD Micron 2200 series system drive 512GB - includes power loss protection

$4860 + tax and shipping

Thoughts? Suggestions?

 

[williedee]

Have you considered a server style case with multiple PSU options?

 

[Magic8Ball]

Given the low total storage, you could easily add a couple of mirrored 12TB spinners as a backup destination for proxmox to keep a multiple version history for each VM/container? The disks could be sleeping most of time until it's time to backup, and that way you only need to backup the "latest" version elsewhere, but still have the history available if necessary.

 

[Hefferbub]

I keep feeling that the config I described above is too expensive for this use case, so I started to look more into Ryzen/AM4 based approaches. I first looked at consumer/gaming motherboards, but the problem there is that they have only one x16 pcie slot and that is needed to have some kind of (minimal) video card. It also makes it hard to work in any 10GBE LAN.

So recently I reviewed the article here on STH about ASRock Rack Ryzen server motherboards. I'm not thrilled with that particular model, as it uses sodimms and I think it will be hard to find decent ECC sodimms. But they have a more recent model shipping now/soon the X570D4U-2L2T.

This is micro-atx, uses 4 full-size dimms and, with the X570 chip, has better PCIE v4 support and immediate support for the new Ryzen 5000 processors. This kind of board has a BMP/IPMI controller for management and a simple VGA interface, plus dual Intel 10GBE, all without using a slot. There are also 2 PCIE V4 M2 x4 slots, which seems intriguing, although I realize that the second one is not quite as useful, as it feeds through the X570 chip which is connected with only an x4 interface itself, so it would be competing with the 10GBE and USB ports for that bandwidth.

Since I want an all-flash server, I'm looking at a different approach for storage with this simpler, cheaper motherboard. What I am hoping to do is purchase one of the cards that fits in a V4/V4 x16 slot and provides 4 M2 slots at x4 along with a heatsink and fan arrangement.

There are two different kinds of these cards:

The cheapest, and perhaps best performing are a mostly-passive passthrough of the V4 PCIE lanes to the 4 M2 slots (and can theoretically accept V4 M2 SSDs like the Sabrent Rocket). These run $100 or less, but they only work if your motherboard allows you to configure the X16 slot for "bifurcation", or division into 4X, 4X, 4x, 4x in BIOS. The previous generation of this board using the older X470 chipset does allow this, but the current manual for this new one does not mention it, so I may need to wait and try it. A good example of this kind of card is ASROCK HYPER QUAD M.2 CARD.

The second kind of card contains a PCIE switch chip which can do the bifurcation by itself and does not need motherboard support. These are currently limited to PCIE V3 on both the slotand the 4 M2s, but that is still plenty fast. An example of this type is documented in this blog post from Stephen Wagner. This board costs around $340, but even at that price it gives a very valuable capacity for lots of super-fast storage in a small space.

So one of these motherboards (~$400) with a new Ryzen 9 5950x ($750) and 4x 32GB 3200 ECC DIMMS ($600) plus another $300-400 for case. PS, etc. gives you a 16c/32t fast CPU with 128GB ECC, 2x 10GBE, IPMI and an open V4 X16 PCIE slot to implement storage for around $2200.

Then it's just a matter of figuring out if one can use the simpler/cheaper 4x M2 card or the more expensive one, then deciding on the characteristics of the SSDs to use.

I plan to run Proxmox on this box, and I have been a fan of using ZFS for a while, so I'm trying to determine my best strategy for SSDs.

Of course, one approach is to use expensive datacenter-type SSDs here. That gives fewer worries over longevity, but M2 versions are somewhat limited, and it seems to me that performance is often less than that of more recent consumer SSDs using high-end controllers like the Phison PS5012-E12 and TLC flash.

So I'm considering 2TB models like the Sabrent Rockets (in either V3 or V4 PCIE versions) or the Adlink S70, which, with at least 2000 TBW endurance should last a long time on a server like this.

So using the Adlink as an example, I could get around 8TB raw capacity in 4 M2 sticks for about $1000 plus whichever 4X M2 card I need for a price of $1100 or $1350. I can either set them up in ZFS as dual mirrored/striped giving 4TB free capacity or RAIDz1 for 6TB. I think I would probably choose the former, as the performance should be better and recovery from a failed drive simpler/safer. I'll probably use a small SATA SSD I already own as the Proxmox boot drive, keeping storage separate from boot/OS.

So that gives me a total cost for this super fast, quiet and likely low-maintenance server of $3300 to $3550.

That seems like a pretty good value, and I like that all parts are easy-to-get commodity items that are not locked to one vendor or technology generation.

Thoughts/comments/questions are welcome, as are any real-world experiences that might inform this approach.

P.S. @Magic8Ball, I like your suggestion of possibly reusing some of my spinning-rust drives as first-line backup for regular snapshots. That makes a lot of sense.

 

[Magic8Ball]

P.S. @Magic8Ball, I like your suggestion of possibly reusing some of my spinning-rust drives as first-line backup for regular snapshots. That makes a lot of sense.

Thanks. You might want to look at something like MergerFS + Snapraid. It's designed for cold storage (write once, read occasionally) and compared to ZFS it allows you to:

• Use any old bunch of drives (mixed capacity, mixed filesystem)
• Add single drives anytime, allowing easy increase in capacity
• Spin down the drives most of the time to save noise/power/heat
• Runs on native linux, and potentially even inside containers in proxmox
• If a failure happens (meaning a data drive + N parity drives fail) you only lose the data on that drive, not the whole pool

You'd basically schedule an overnight job for proxmox snapshots followed by snapraid parity calc and everything would be nice and secure. It's a good way to re-use old disks with no extra costs.

 

[Hefferbub]

Regarding availability of this new motherboard, I asked the folks at GAME | Gaming Specialist For Consoles, Games & Accessories! who said they can special order it with 1-2 week lead time.

They also said it does not yet have BIOS supporting Ryzen 5000, but once that is released they would be willing to update it before shipping using a 3000 series Ryzen they have on hand.

 

[Vesalius]

You could use the 2 on board m2 nvme as the promox install boot drive in a zfs raid1 mirror. Smaller nvme would be fine here. Give your boot more redundancy. Then add the other larger pci nvme to run all the VM and spinners as additional backup drives.

 

[Hefferbub]

I just found out that the X570D4U-2L2T does support bifurcation on the X16 slot!

Therefore I ordered the inexpensive Asus Hyper M2 Gen 4 card for about $75. In theory, it will give me four Gen 4 M2 slots at full speed—we’ll see.

I’m leaning toward the Addlink s90 2TB Gen 4 SSDs for $345 each. These are super fast and claim a TBW of 3600!

It’s frustrating having to wait for availability on the Ryzen 9 5650x and for ASRock to update the MB bios to support them. Not sure how long that will take to sort out.

 

[istqian]

For you case, I am providing following, as they are cheap.
you can find some second hand parts.

MB SuperMicro X10SRi-F, 300$
CPU XEON E5 2678V3 120$, 12Core24Threads
RAM 128GB RAM 400$, 32GB DDR4*4, 100$ each
A PCI16x to 4 8643 split card, 100$ or less
U2 SSD, P3605 1.6TB, 250$ each, 1000$
any case, 100$
seasonic 600W titan PSU, 150$
any sata ssd, 256GB, 50$

totally will under 2300$
for server, IPMI is a must, so SuperMicro MB is the only choice.
I have some servers spec like this. and it works for two years, very stable.

 

[Hefferbub]

Those do look like a great solution. Thanks for sharing (although my Ryzen build is already underway).

I did some searching on eBay for some of these suggested parts, but they have significantly higher prices. That is often my experience reading posts on this site. I guess you have to take your time and learn to hunt the deals on EBay.

BTW, the ASRock board I’m using also has IPMI and doesn’t require any extra license fees for it.

 

[Hefferbub]

I received all the parts for this and have built it. I decided not to wait for the Ryzen 5000 version of the processor and went with the 3950x for a hundred dollars less. Even it is probably overkill.

With a roomy case and Noctua fans and CPU cooler, the system is near silent and p.s. extremely fast. It was an easy build, although I needed the (included) case fan cable extensions to reach the fan headers. Thermal performance has been good under load testing.

The inexpensive Asus Hyper M2 Gen 4 card works great with this motherboard, so my 2 GEN4 NVME 2TB ssds are running at full speed and I have 2 more M2 slots if I need them.

The BMC/IPMI works fine, and I was able to get the 10Gbe working without issue. Proxmox virtualization software runs without a hitch, including its use of ZFS (I am mirroring the 2 big SSDs).

Overall cost was around $3500.

I think this is a viable alternative to traditional servers, and I can’t wait to turn off that howling 2U Supermicro monster it replaces.

P.S. If you are interesting in buying this motherboard, I recommend the folks at http://gamepc.com. They were able to get the board in about a week, and they were willing to flash the BIOS for me to accommodate Ryzen 5000 before shipping it on. It was a pleasure working with them.

 

[Magic8Ball]

The inexpensive Asus Hyper M2 Gen 4 card works great with this motherboard, so my 2 GEN4 NVME 2TB ssds are running at full speed and I have 2 more M2 slots if I need them.

This card looks great so I'm glad you've reported some success. My only concern is about heat dissipation. Have you noticed anything getting hot? What are the drive temps under full load? Does the fan run constantly and is it loud or annoying?

 

[Hefferbub]

I have not yet stressed the SSDs too much, so I don't have useful temp data yet. I'll report back later.

I have no concerns about heat however. The aluminum cover over the SSDs is thick and heavy, and it comes with thermal pads installed over each slot to transfer heat to it. I have not heard the fan at all, and I suspect things would still work fine even if it were unplugged.

 

[mixtecinc]

For you case, I am providing following, as they are cheap.
you can find some second hand parts.

MB SuperMicro X10SRi-F, 300$
CPU XEON E5 2678V3 120$, 12Core24Threads
RAM 128GB RAM 400$, 32GB DDR4*4, 100$ each
A PCI16x to 4 8643 split card, 100$ or less
U2 SSD, P3605 1.6TB, 250$ each, 1000$
any case, 100$
seasonic 600W titan PSU, 150$
any sata ssd, 256GB, 50$

totally will under 2300$
for server, IPMI is a must, so SuperMicro MB is the only choice.
I have some servers spec like this. and it works for two years, very stable.

Do you have part number and brand for the PCI16x to 4 8643 split card,?

Thanks