Hey everyone,
has anyone ran some benchmarks with various NUMA configurations on Rome?
The issue with 64 core CPUs is that Windows is only able to address 64 logical cores at once, and thus has an internal "group" subdivision of 64 logical cores.
Even with NPS1 you are only able to create a 64 core VM (which can still access the whole processor's RAM).
So I was wondering which configuration works best if you need large VMs (32/64 cores, 250+GB RAM)
- CCX as NUMA domain
NPS1 for 8-channel memory interleaving
Large VMs run on many NUMA domains (with NUMA spanning disabled still). This would only be ok with NUMA-aware workloads, but does it work well in practice? With SQL server (which is NUMA aware), for example? Actually better than a smaller VM on a single NUMA node?
- NPS 1
VMs can run on a single NUMA domain up to 64 cores, due to windows's limitation. This might make most sense with SMT disabled and would probably provide the most performance per-VM, at the price of less overall resource utilization.
- NPS 2
VMs can run on a single NUMA domain up to 32 physical/64 logical cores. This would probably work OK, but you'd have to live with a maximum of 1/4 or total RAM, although with 64 logical CPUs
- NPS4
Large VMs run on 2 or 4 domains (with NUMA spanning disabled still). This would only be ok with NUMA-aware workloads for large VMs and also allow for decently sized (32 logical core) VMs that run on a single NUMA node.
Experiences/suggestions/recommendations?
has anyone ran some benchmarks with various NUMA configurations on Rome?
The issue with 64 core CPUs is that Windows is only able to address 64 logical cores at once, and thus has an internal "group" subdivision of 64 logical cores.
Even with NPS1 you are only able to create a 64 core VM (which can still access the whole processor's RAM).
So I was wondering which configuration works best if you need large VMs (32/64 cores, 250+GB RAM)
- CCX as NUMA domain
NPS1 for 8-channel memory interleaving
Large VMs run on many NUMA domains (with NUMA spanning disabled still). This would only be ok with NUMA-aware workloads, but does it work well in practice? With SQL server (which is NUMA aware), for example? Actually better than a smaller VM on a single NUMA node?
- NPS 1
VMs can run on a single NUMA domain up to 64 cores, due to windows's limitation. This might make most sense with SMT disabled and would probably provide the most performance per-VM, at the price of less overall resource utilization.
- NPS 2
VMs can run on a single NUMA domain up to 32 physical/64 logical cores. This would probably work OK, but you'd have to live with a maximum of 1/4 or total RAM, although with 64 logical CPUs
- NPS4
Large VMs run on 2 or 4 domains (with NUMA spanning disabled still). This would only be ok with NUMA-aware workloads for large VMs and also allow for decently sized (32 logical core) VMs that run on a single NUMA node.
Experiences/suggestions/recommendations?
Last edited: