Advice for an AM5 board/build with 40 G fibre?

[Pakna]

I am in the process of transitioning from the venerable X99 (and E5-2697 v4) in my workstation to an AM5 system. The CPU choice is easy (9950X) but I am curious about the motherboards as we know AM5 is a bit scant on PCIe lanes.

What I am looking for is a (workstation) board that will serve:
- a GPU (something in the RTX 4080 range)
- a 40 Gb NIC (Mellanox MCX354A-FCBT - PCIe 3.0 x8)
- a couple of Optane 905p (with adapters to NVMe)
- a single NVMe drive (Samsung 980 Pro)
- 64-128 GB RAM

So far I am looking at Asus ProArt X870E Creator and if my counting is right, the configuration above should be possible with the only compromise that the GPU bandwidth will be cut from x16 to x8 width (which is something I am fine with).

Three questions:
- is my understanding above right or will there be additional compromises (with the above config) I should live with?
- given the Asus' support track record lately, I'd be more than glad to consider any other boards if they can support the config above with the least amount of connectivity compromises. Are there any I should be aware of?
- what RAM should I be looking at? I remember Wendell extolled the trouble-free world of Threadripper (albeit with expensive RAM kits), and the kind of wild west on the X670E (that was then). Is the RAM compatibility situation better these days and do I need to expect issues trying to reach for 128 GB? I see the ProArt X870e boasting 192 GB RAM and DDR5-8000 speeds, but I am kind of skeptical of these claims.

Any thoughts/advice are greatly appreciated.

 

[Tech Junky]

X670E

I use a PG Lightning / ASRock because of the whole lane splitting thing others do. Also, the only significant change with the 870's is adding 2 TB ports as mandatory.

The biggest hurdle with AMD boards is anything that's not CPU attached gets bottlenecked off the chipset to x4 where Intel is x8 at this point.

Asus is a lost cause in most cases if something goes wrong they tend to be a PITA to deal with from what I've seen around different forums. Then again any co's CS is like 911 you hope you never need to use it and if you do you want them to respond promptly.

I don't see the need for 8000 DDR5 but, sure. It's funny how the MOBO co's / amd can just enable more through a UEFI update with some things. I think the real question is what are you doing that needs this? Workstations don't need it 192GB/8000 unless you're doing something that's cutting edge and latency sensitive. Most things adapt to what's available from the system and work just fine.

 

[Pakna]

I use a PG Lightning / ASRock because of the whole lane splitting thing others do.

Not familiar with this - can you explain a bit?

Asus is a lost cause in most cases if something goes wrong they tend to be a PITA to deal with from what I've seen around different forums. Then again any co's CS is like 911 you hope you never need to use it and if you do you want them to respond promptly.

Asus is my last resort, but I just don't know of better options. I am wary of MSI, with Gigabyte I don't have a lot of experience. The only other option is something like AsRock Taichi Lite.

To clarify my question - if I plugged in all of the above to the board, what kind of bandwidth compromise would I be making? I understand that the two PCIe ports would basically work in x8/x8 mode (and I am not concerned with a 1-3% loss of GPU performance here) but I can't grok whether there would be any NVMe drops in bandwidth as well. Taichi Lite spec doesn't seem to indicate it but it looks like some boards do drop the NVMe bandwidth if both PCIe ports are populated. Maybe I am wrong?

I don't see the need for 8000 DDR5 but, sure. It's funny how the MOBO co's / amd can just enable more through a UEFI update with some things. I think the real question is what are you doing that needs this? Workstations don't need it 192GB/8000 unless you're doing something that's cutting edge and latency sensitive. Most things adapt to what's available from the system and work just fine.

No pressing need as well - this will be mostly a coding machine (Java/Rust, containers, IDE and maybe a VM here or there) with an occasional game. 128 GB will be plenty but realistically 64 GB should be good enough. I assume I should be buying a 2x64 GB kit if shooting for the 128 GB RAM?

 

[joerambo]

- what RAM should I be looking at? I remember Wendell extolled the trouble-free world of Threadripper (albeit with expensive RAM kits), and the kind of wild west on the X670E (that was then). Is the RAM compatibility situation better these days and do I need to expect issues trying to reach for 128 GB? I see the ProArt X870e boasting 192 GB RAM and DDR5-8000 speeds, but I am kind of skeptical of these claims.

The problem is, to reach 128GB You pretty much need to run 4x32GB DIMMs and that in turn means running 2DPC with dual ranked memory, putting immense load on memory controller.

For 2DPC with 2R mem I'd expect to run 5200 or maybe 5600 in best case.
Threadrippers exist for a reason, they provide more memory channels, ECC support and most importantly more PCIE lanes for IO.

Optanes perform the best when they are connected directly to the CPU, so that already puts a lot of strain on desktop platform, I don't have that much experience with AMD I/O but consider the following on Intel Z690-Z790 gen:

(If You need high perf graphics card connected, that means primary PCIE16x is reserved for it, even if operating at 8x)

1) CPU is connected directly to PCIE16x primary slot on most motherboards and additional 4x PCIE is provided to M_2 slot.
2) SOME more expensive motherboards allow splitting primary PCIE slot into 8x prim / 8x secondary PCIE from CPU
3) Some even more expensive motherboards allow splitting primary PCIE slot into 8x prim and 8x secondary AND 4x additional M_2 slot that shares those lines with secondary. ( so not possible to have 3x M_2 drives that are connected directly to the cpu, due to sharing ).
4) And most importantly - Intel does not allow PCIE bifurcation -> no splitting 8x into 4+4 and putting M_2 adapter card, just 1 or 0 drives will work.

The new Z890 has more PCIE lanes, but the platform is performance disaster with some strange decisions of where I/O happens on package.

On AMD it is also quite expensive to find a mobo with 3 direct to CPU PCIE lanes , but at least they allow bifurcation of lanes, enabling flexibility.

All of above does not even start to consider the requirements to have 8x physical connector for Mellanox card, does it has to run 8x PCIE as well?

 

[Tech Junky]

Not familiar with this - can you explain a bit?

With the PG all of the slots remain at their set allocation of lanes... x16 stays x16 unless you want to bifurcate top slot into x8/x8 or x4/x4/x4/x4 which is handy when you want to put in a storage card that gives you up to 4 NVME drives off the slot. The devil is in the details in the block diagram for each board.



So, this is the PG layout for lanes. You get slot1/slot3 and primary M2 off the CPU. Then you have the remaining slots / M2 off the chipset which is x4 whether on the primary or secondary chipset for less critical devices. CPU devices have Gen5 bandwidth and most of the others are Gen4 w/ the exception of M2/2.

For the Taichi there's 2 versions the 650 and the 870 and I'll focus on the 870 since it's going to be the best option in regard to lanes.

https://pcpartpicker.com/products/compare/42hFf7,R226Mp,pcdG3C/

So, you still get about the same results with the 870 board except for the splitting of lanes on the top two slots to get x8/x8 where I can bifurcate the top slot and keep x4 on the next one for a high priority device that needs CPU lanes for speed. It's a bit of a wash until you look at the $$$$ $200 vs $500+. For me I can use that $300 on something else like a GPU for $100 to transcode videos quicker and more efficiently than just relting on the CPU to do it 8x longer.

NVMe bandwidth if both PCIe ports are populated. Maybe I am wrong?

Not really. The BW won't be affected until you get beyond M2/1 but even then you're only dropping to Gen4 speeds which is still ~6.5GB/s. The issue would become more apparent if you were doing M2 raid as the drives on M2/2/3/4 would be sharing the chipset X4 total BW uplink.

Another thing to think about is i have 4 slots albeit not all x16 where the Taichi only has 2. They're kind of funky in terms of layout but, still nice to have options even if you chop the BW a bit to other cards. As to the Taichi with the switch between the 2 slots it could be an issue depending on what you do with them.

What I am looking for is a (workstation) board that will serve:
- a GPU (something in the RTX 4080 range)
- a 40 Gb NIC (Mellanox MCX354A-FCBT - PCIe 3.0 x8)
- a couple of Optane 905p (with adapters to NVMe)
- a single NVMe drive (Samsung 980 Pro)

For the optane you could do 4X on the 670 or 2 on the 870 using a slot or put them on M2 on the chipset and potentially bottleneck their performance. I suppose if you wanted to since you would sacrifice a slot potentially doing x4/x4/x4/x4 could add other stuff depending on the adapter unless you're thinking about using the additional M2 sockets instead.

NIC - 670 would be easier to find a home

It's all about being strategic though when planning things out. Top slot is nice to be able split the lanes yourself with the PG 670 depending on the adapter type you decide on the biggest issue though would be the 4080 finding a home with x8. For me it's not an issue though as my priority isn't gaming as it's a headless setup and the GPU is only in there for Plex media conversions.

It gets ugly though when trying to pick the right board if you have more than simple design ideas on how you want things to run. For a simple consumer it's easy as they're not trying to cram a ton of options into a simple setup. Nerds though have to take a deeper look or shoot themselves in the foot with the wrong buy. It takes some planning for sure but, if you want more modularity AMD is the best option unless you have a bigger budget and step up into the server class options where you can have all 7 slots be x16 Gen4/5 with a CPU that supports 100+ lanes. The things I could come up with if I wanted to spend $5K.

 

[Pakna]

Thank you both for your thorough and thoughtful answers - much appreciated and I learned a LOT! I wish I could have liked your posts more than once, but alas.

I pulled the trigger on the X870E Taichi Lite and the 9950X that have miraculously been in stock (both have been unobtainuim in Canada for the past couple of weeks). Took notes from @joerambo on memory controller pressure and landed on 2x32 GB of DDR-6000 which I expect should be more than adequate. I had a long look at Asus' X870E ProArt Creator, but ultimately could not warrant the extra expense (~100 USD) for basically nothing + Asus' very questionable support as of late.

I also considered Threadripper 7960X with Asus TRX50 board but ultimately, I could not stomach the additional Threadripper tax - with the AM5 board, 9950X and 2x32 GB RAM, I am still well below just the cost of the Threadripper CPU, let alone the TRX50 and the RAM....and all that, for what? Less hassle, GPU basically running at half PCIe width (which is what, 1-3% performance penalty, last I read) and a bunch of unused (but paid-for lanes).

So given that, here is what I am planning to do, please correct me if you think there is a better layout:

• GPU -> PCIE1 (x8)
• NIC -> PCIE2 (x8)
• Optane 1 -> M2_1 (CPU)
• Optane 2 -> M2_2 (Chipset)
• 980 Pro SSD -> M2_3 (Chipset)

PCIE1/2 will bifurcate, and NVMe drives will operate at full widths. The M2_2/3 will have a slight latency penalty, but unless the difference is on the order of 10-20% higher latencies, I think I'll be able to live with this. I can still plug in something to M2_4 and that thing will also operate at x4 width.

Does that sound reasonable?