Got some new BIOS code running on our R7515's that have 7352's slotted these servers are running ESXi 6.7update3 and I have found that VMware is not talking to UMA correctly limiting VM performance severely.
Enabling 4way vNUMA in the BIOS then adding the vmx config line numa.consolidate=false and then tuning out the numa related values so that your VM masks correctly to the hosts NUMA allocation, we are seeing a massive increase in memory throughput on these VMs and a massive drop in memory latency. We saw the same behavior on 7001 Naples but were not expecting it to be as effective due to the UMA design with the IOD. Boy were we wrong!
Here are some memory throughput/latency numbers.
Single VM, 16vCPUs with 16vSockets (1 per due to cache domains), 256GB of ram fully allocated at powered on
4way NUMA 160GB/s @ 97ns, 2way NUMA 78GB/s @ 119ns, UMA 44GB/s @ 167ns
Thats a server with 8 quad rank DIMMs, when tested with single rank we got the latency down to sub 80ns.
I have tickets opened with AMD, VMware, and Dell on this. But I am posting this information as I am sick and tired of fighting with the OEMs/ODMs to get shit done. So please, feel free to self validate and open tickets if you care too
Enabling 4way vNUMA in the BIOS then adding the vmx config line numa.consolidate=false and then tuning out the numa related values so that your VM masks correctly to the hosts NUMA allocation, we are seeing a massive increase in memory throughput on these VMs and a massive drop in memory latency. We saw the same behavior on 7001 Naples but were not expecting it to be as effective due to the UMA design with the IOD. Boy were we wrong!
Here are some memory throughput/latency numbers.
Single VM, 16vCPUs with 16vSockets (1 per due to cache domains), 256GB of ram fully allocated at powered on
4way NUMA 160GB/s @ 97ns, 2way NUMA 78GB/s @ 119ns, UMA 44GB/s @ 167ns
Thats a server with 8 quad rank DIMMs, when tested with single rank we got the latency down to sub 80ns.
I have tickets opened with AMD, VMware, and Dell on this. But I am posting this information as I am sick and tired of fighting with the OEMs/ODMs to get shit done. So please, feel free to self validate and open tickets if you care too