Supermicro recently published the Manual of their H11DSi-NT Motherboard. One of the reasons why I love Supermicro is because they put detailed Block Diagrams so you know what goes connected where (Sorta important if you're into virtualization with Passthrough using QEMU, due the way that IOMMU Groups works). Sadly, with Epyc, the latest Block Diagram seems to be a bit underwhelming, because they treat each Socket as a whole.
Since AMD MCM designs use multiple dies on the same package, you effectively have multiple NUMA Nodes in every Processor, each with its own local CPU and RAM. However, now NUMA Nodes also have an associated local PCIe Controller, as some PCIe Lanes/Slots will be wired to a specific die. If I were to do something like passthroughing a SAS Controller to a VM that is intended to be a File Server, it makes sense that all the CPU, RAM and PCIe resources comes from the same die instead of thrasing the Infinity Fabric.
Currently, Supermicro doesn't really help you to do that, and it is worrying me, cause I think that making accesible the Block Diagrams of every Motherboard made them easy to plan with. Other manufacturers of mainly consumer Motherboards absolutely suck in this respect, and you have to carefully check the specifications to figure out if a given slot comes from the Chipset, is behind a PLX Switch, or something else. But with Epyc, even Supermicro isn't detailed enough...
Since AMD MCM designs use multiple dies on the same package, you effectively have multiple NUMA Nodes in every Processor, each with its own local CPU and RAM. However, now NUMA Nodes also have an associated local PCIe Controller, as some PCIe Lanes/Slots will be wired to a specific die. If I were to do something like passthroughing a SAS Controller to a VM that is intended to be a File Server, it makes sense that all the CPU, RAM and PCIe resources comes from the same die instead of thrasing the Infinity Fabric.
Currently, Supermicro doesn't really help you to do that, and it is worrying me, cause I think that making accesible the Block Diagrams of every Motherboard made them easy to plan with. Other manufacturers of mainly consumer Motherboards absolutely suck in this respect, and you have to carefully check the specifications to figure out if a given slot comes from the Chipset, is behind a PLX Switch, or something else. But with Epyc, even Supermicro isn't detailed enough...