Knoll / Chipset-less AM5 Mobo for Proxmox Builds?

[movax]

I'm thinking of upgrading my Proxmox build (currently X11SSH/something with a soldered down Xeon 1585L). The ACS / IOMMU groups aren't the greatest, I can solve this with some OPNsense configuration + networking, but thinking if I had to go from scratch.

All I need I/O wise is a PCIe x16 slot to host my X710-AM4 adapter (quad 10GBase-T). On-board 10G is nice to have, but if PCIe ACS works properly, I can pass-thru one of the ports to the host. The other thing I would like is dual M.2 or at least enough expansion to run a basic ZFS mirror of 2 or 4 TB NVMe drives for VMs.

So requirements:

1. Support 2x NVMe for ZFS mirror
2. PCIe slot 3.0 x8 minimum for X710 NIC
3. IPMI
4. Lowest Power / least add-on peripherals
   1. So no chipset (accept IPMI power draw for convenience...)
5. On-board 10G NIC or ACS / IOMMU support on CPU port that allows splitting of devices on mutli-port NIC.
6. Undervolting / advanced power management features expoed in BIOS
7. Target nominal power draw: <100W

What the box does?

1. Run OPNsense as my main router via pass-thru 10G NIC.
2. Run 'core' VMs (Home Assistant, Homebridge) -- stuff I don't **** with that often.

That's really it... other VMs run on a M90q Gen2 I have. Doing some HA could be nice!

Focus is on low power, so I would put in a low TDP Ryzen part and undervolt it. Probably get the cheapest set of 2x 48 or 32G ECC sticks I can.

ASRock rack seems the best so far, with these as the down-selected ones (some of which were reviewed here):



The first two have 10G LAN, the second two don't. I want 10G for the Proxmox host just 1) homelab desires, 2) speedier backups. I run an OPNsense VM currently where I PCIe pass-through the entire X710 controller, so I don't need it per se on the mobo if the IOMMU groups work as they should.

Questions

• Do these BIOSes expose undervolt settings / AGESA settings like you find on mainstream boards?
• I have read Prom21 / chipsets don't support ACS, but Ryzen PCIe RCs do -- the opposite of the Skylake situation. Is that correct?
• I'd like to rackmount, but it's not strictly necessary -- assuming I can get a 1U barebone chassis to support any of these and a PCIe riser.
• Do these boards properly support bifurcation?
• Are there other X300/KNOLL based mobos I should look at?

I think I'm SOL on getting a separate 10G controller and running more than one drive (WTF?) -- I'd need to get an OCUlink to M.2 adapter, and therefore one of the drives gets half-bandwidth too. Or -- do I find some kind of combo of riser / cables if the PCIe RC can do bifurcation? I only need x8 for the NIC, the other x8 could go to NVMe.

Feels like if PCIe IOMMU works the way it should, AM5D4ID2 is the way to go? I get my 10G to host, I get IPMI, I get two M.2s for NVMe redundancy... no disadvantage.

Basically:

• AM5D4ID-2T/BCM
  • Not preferred due to needing OCUlink to run second drive + limited bandwidth.
• AM5D4ID3-2T/X710
  • Not preferred due to needing OCUlink to run second drive + limited bandwidth.
• EPYC4000D4U
  • Don't need extra SATA controller.
  • Don't need MCIO ports.
  • Doesn't have 10G.
  • Does have 2x M.2.
• AM5D4ID2
  • Doesn't have 10G.
  • Does have 2x M.2.

 

[mattventura]

EPYC4000D4U

• Don't need extra SATA controller.
• Don't need MCIO ports.
• Doesn't have 10G.
• Does have 2x M.2.

The MCIO ports are an alternative to the M.2 ports and the latter 8 lanes on the x16 slot. So if you're using the 2x M.2 and a x16 card, the MCIO ports don't do anything anyway. i.e. you don't actually *lose* anything you would normally have by having those ports.

But it still sounds like the AM5D4ID2 is the best option given your requirements.

 

[movax]

The MCIO ports are an alternative to the M.2 ports and the latter 8 lanes on the x16 slot. So if you're using the 2x M.2 and a x16 card, the MCIO ports don't do anything anyway. i.e. you don't actually *lose* anything you would normally have by having those ports.

But it still sounds like the AM5D4ID2 is the best option given your requirements.

Ooh -- good point.

Now next step is to see if this is used anywhere or on eBay somewhere...