ZFS + virtualization box - which platform?

[zzattack]

I'm looking for advice on building a new storage + virtualization home server. Some 12 or so years ago I built my 'current' NAS, consisting of a then brand-new G34 platform with a very cheap Opteron 6128 octacore and nice ASUS KGPE-D16 motherboard housed in a Norco 20-bay case. Over time I've upgraded hard disks and replaced a few fans, but it's time to thank this setup for its service.

QNAP --> self-built
Initially I looked at the enterprisey (enterpricey?) offerings by QNAP and in fact, decided on a TS-h2477XU-RP which was marketed as an ideal ZFS platform with splendidly included virtualization + container options. At €6.5k steep but for a hassle-free setup that would be rock solid I figured it might beat the full DIY route. Admittedly I should have done more research before the purchase. I really gave it a shot, but after two days I returned it. The setup is way too limiting for me. A mere 32GB non-ECC RAM covering 480TB of raw storage on ZFS, total lack of video output of any kind, and inability to fine-tune or customize stuff to my liking ultimately convinced me the only way this build was going to satisfy me is if I redid myself. Oh, and then there's the DEADBOLT/Qlocker stuff and non-handling of that.

Requirements:
So, new setup. And about 6.5k of budget. There's some hard requirements that I think are reasonable and certainly not impossible to meet:
- Filesystem will be ZFS, should be accompanied with sufficient ECC RAM (so 256GB, possibly 512GB)
- OS will be either Proxmox with ZFS-on-Linux or FreeBSD
- IPMI or similar
- Filled up 24-bay 4u chassis
- Preferably, CPU is energy efficient when idle
- Needs at least one 10G uplink, preferably SFP+ over copper

Components:
I've tried to do my diligence so far, and already managed to decide on several parts that are independent of the eventual platform. The following are parts which I've already decided on/purchased:
- 24 bay case -- Huge Data Storage Cloud Server Case 24 Hotswap Bays 4u Rack Big Data Computer Chassis - Buy Huge Storage Rack Case,4u 24 Bays Case,4u Hotswap Product on Alibaba.com
- case looks to be a supermicro-inspired one and I had it configured with simple non-expander SAS3 backplanes
- LSI SAS 9305-24i for the HBA
- accompanying hard disks (seagate exos x20)
- 4x Kingston PCI-e 4.0 2TB NVMe's for storage/zil/l2arc


Platform:
This is where I'm looking for advice. Current generation consumer grade platforms like AM5 (or AM4) and socket 1700 are top picks for their very fast yet efficient processors. My Ryzen workstations perform beautifully and would no doubt do well, but seem limiting in the amount of RAM currently supported on them. Since >=256GB ECC is a firm requirement for me, I must also consider the Threadripper series, but that in turn seems to relinquish any sense of energy efficiency. Furthermore, I'm not opposed to Intel in any way, whether it be Xeon or consumer grade. Suffice to say I'm fairly lost and would very much appreciate suggestions that connect with my intended use.

 

[DavidWJohnston]

I run a 4-host ESXi cluster, and the 2 best hosts are Supermicro X11SPM-TF Intel Xeon LGA-3647 platform and that's been good. I can easily live migrate between my other Intel-based hosts. I run 6x64GB ECC LRDIMMs in each of them and they're solid. They include IPMI which is also super useful.

One possible consideration is the Intel's on-chip video acceleration "Quick Sync" might be useful for Plex or Blue Iris, or similar. AMD doesn't have that - But for newer CPUs I'm not sure this matters anymore.

Lots of people run AMD - Perfectly fine option, I just stuck with Intel for live migration across all the hosts.

Good luck, new builds are always fun.

 

[ipkpjersi]

If it were me, I'd want to do AMD Zen 3/4 with some nice ECC memory, but I also have some older X99 server boards running with ECC memory too. Self built is definitely a better option for customization, and more fun IMO. I mean , I even run a 5900X with ECC memory in my desktop since that's my single-source-of-truth for all my data, so might as well not have it get corrupted at the source lol

I'm less familiar with newer Intel stuff, so I don't really have any recommendations there.

My servers all run Proxmox on an OS SSD with 6 drives each with ZFS RAIDz2, it's Debian-based so it's super stable.

 

[mattventura]

The 9305-24i is not worth the $300+ IMO. A 9300-8i ($60) an 82885T ($40) is much less expensive. Or, just get a chassis with expander backplanes in the first place and avoid the issue altogether (don't forget, cables cost money too). A SAS3 8i card will not bottleneck 20 spinning drives.

For the chassis, if you don't need full-height expansion slots, an 847 provides more bays and some other neat features. Unless you're throwing a GPU in there, there's not much reason go full-height since most modern network and storage cards are LP-friendly. Plus, it's a lot easier to get replacement parts and upgrades, even into the future. If something goes wrong with a random AliExpress chassis, are you going to be able to get replacement parts in a timely manner? Is there any real upgrade path in the future? I've bought these kinds of chassis before, but I wouldn't do it for something mission critical, nor would I use it in a build with a $6500 budget.

For the motherboard and CPU, Epycs are a pretty good option right now. A gen 2 has plenty of I/O, memory, and performance even with a single socket.

 

[i386]

- LSI SAS 9305-24i for the HBA

Another point about hbas (and raid controllers) with >8 ports: they are rare compared to the 8 port cards and it could take weeks if not months to get a replacement (for new ones, probably even worse for used items from ebay, amazon etc.)

 

[zzattack]

Thanks for the advice offered so far. I've tried to familiarize myself with the available platforms. I'm halfway looking at the '7B13' (mentioned in another recent thread) which is virtually guaranteed to be overkill, but also insane value for money. From the same lineup the 7443p is potentially more interesting due to its high boost at 4.0GHz. I would assume the 24 core offering to be significantly less power hungry in idle states compared to the 7763, but this is unsubstantiated. They share similar TDP but I'm unsure if this is at all relevant to idle usage.

The other platform that looks interesting is the Threadripper Pro 5000WX series, e.g. the 5965WX. I'm looking for input on how to weigh pro's and con's of both options. Available Intel options look to be significantly less value for money so I'm not focusing on anything Xeon.


Re.: comments in the thread so far:
- The case needed to be full-height to accomodate the 24 drives.
- The 9305-24i was freely available to me and seemed to simplify the setup by eliminating the need for expanders
- This does not run any mission critical infrastructure. Sure I'd like everything to be stable, but ultimately it's for private use with nobody but me depending on its availability.

 

[mattventura]

- The case needed to be full-height to accomodate the 24 drives.

I was referring to low-profile expansion slots. The 847 fits an extra 12 3.5" bays in the back, but the tradeoff is that the motherboard tray has to move up, so the slots are all LP. It can be nice to have the extra bays for future expansion, plus it's a lot easier to upgrade the rear BP than the front BP (it uses the same parts as the 826 series).

 

[zzattack]

After reading up further I've decided on a Milan 7443p with 512GB DDR4, no l2arc, an optane nvme for zil.
Last piece to pick is the motherboard itself. Deals from China all come with a supermicro H12SSL-i. This board seems to be perfectly adequate. Are there gotchas to be aware of, particularly since this board came out before Milan? Are there better offerings from asrock rack or asus that should be considered?

 

[ano]

7443 = powerrrr, you want the 3200mhz memory to max thatbtw.

H12 = 7443 compatible and very nice combo.

no real gotchas

 

[zzattack]

That's what I went with. Will post some build pics once everything's found its way over here.

 

[zzattack]

Hardware arrived, the case looks and feels quite nice. Board came with CPU and RAM already installed. I believe the ipmi default admin password would be printed on a sticker on the CPU protection cap, but with the CPU preinstalled and the original box not being included, I don't think I can get into the ipmi menus.
From what I understand, resetting the password is possible provided you know the original unique password it came with. Hopefully the seller (tugm4470-8) still has it somewhere.

Might replace the cooler too, air flow in this direction probably doesn't help much.

 

[Stephan]

Check all white labels on the Supermicro board for the password.

 

[ano]

well, hey, nice I have thoose exact drives in X20 series come in 20TB here yesterday

you can reset pw i bios, should be fine, worst case its printed under cpu, so just take it off/on

 

[zzattack]

Found the password, it was simply underneath one of the m.2 nvme's. IPMI loads a lot slower than the older generation. No big deal.

For the SSDs, I received the ASUS Hyper M.2 X16 adapter to add 4x nvme slots. This very cheap adapter works perfectly after forcing bifurcation to 4x4x4x4 on the corresponding pci-e slot (so the 'auto' setting didn't bifurcate at all, causing only a single disk to show up). Doesn't appear to come at any loss of speed either. Parts-wise, most has arrived and I'm now slowly waiting for some more fans and another CPU cooler to come in.

Meanwhile I've installed proxmox-ve and set up the storage pool with 2x 12-disk vdevs in raidz2. The sequential write performance of the pool sadly leaves a lot to be desired. While filling up the pool with my old data, I will maintain sync=off. Most data will not be compressible so also compression=off, and recordsize set to 1M.
Reads easily saturate my 10gbps network link, and writes start out at ~1GB/s but quickly (after <5GB) drop to ~90MB/s. This is well before the arc should have filled up given the 512GB RAM available. Looking at zpool iostat, all 24 disks are doing about ~4-5MB/s. When stopping any file transfers in progress, it takes quite a while before the writes are flushed from RAM to disk, still going at this 4-5MB/s per disk.

I don't think I've missed any of the best practices, so I will have to debug the root cause for this before pumping over my data. Any tips for where to look are appreciated.

 

[ano]

did you run some fio? should be WAY faster

 

[zzattack]

I did, fio runs blazing fast right until the cache fills up and then I can see the same 4-5MB/s-ish performance on all disks simultaneously. So same exact problem basically.

 

[ano]

which cache? drive? arc? where? zil/slog?

 

[zzattack]

RAM write cache.
ARC should be irrelevant for writes. Sync=disabled so zil/slog also irrelevant.

 

[ano]

is drrive cache on?

 

[zzattack]

Yes, on all drives:

root:~/ # hdparm -W /dev/sdx                                                                                                                         [11:41:47]

/dev/sdx:
 write-caching =  1 (on)