Only way to learn about this stuff is by asking. In this case, I'd never thought that NUMA could cause any problems nor would I ever turn it off so that's why it sounded odd. I rebooted one of my C6100 test nodes to see what it does. This is dual socket with separate 3 sticks of memory per controller. This is also under CentOS, which is basically what ESX is running underneath, a Redhat variant. The OS itself is NUMA-aware. What you see is that the BIOS setting encapsulates each socket into a NUMA node so that it can properly map the memory and distances. Without this, it flattens it as one level deep so the OS cannot sort out what hardware you have.
So overall, I would recommend turning on NUMA if you've got it.
----
numactl --hardware:
NUMA Off in BIOS:
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 24566 MB
node 0 free: 23361 MB
node distances:
node 0
0: 10
NUMA on in BIOS:
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 8 9 10 11
node 0 size: 12278 MB
node 0 free: 11406 MB
node 1 cpus: 4 5 6 7 12 13 14 15
node 1 size: 12288 MB
node 1 free: 11599 MB
node distances:
node 0 1
0: 10 20
1: 20 10
Link to PDF versions
NUMA Disabled: http://postimg.org/image/838cbezvt/
NUMA Enabled http://postimg.org/image/5n6ixkht5/
So overall, I would recommend turning on NUMA if you've got it.
----
numactl --hardware:
NUMA Off in BIOS:
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 24566 MB
node 0 free: 23361 MB
node distances:
node 0
0: 10
NUMA on in BIOS:
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 8 9 10 11
node 0 size: 12278 MB
node 0 free: 11406 MB
node 1 cpus: 4 5 6 7 12 13 14 15
node 1 size: 12288 MB
node 1 free: 11599 MB
node distances:
node 0 1
0: 10 20
1: 20 10
Link to PDF versions
NUMA Disabled: http://postimg.org/image/838cbezvt/
NUMA Enabled http://postimg.org/image/5n6ixkht5/
I'm really just jumping into server virtualization and modern server know-how altogether. I was thinking that the NUMA option enabled would lock down an evenly distributed set of resources for each CPU node whereas the ESXI engine automatically distributes resources when and where needed and this ability is obstructed if NUMA is enabled. I certainly could have misinterpreted the information.