Proxmox: E-cores for host/zfs, P-Cores for VMs/containers

[WhiteNoise]

Hello,

I have an 13700 with 8 p cores and 8 e cores.
I was wondering
1. If it is a good idea to reserve/pin the e-cores for running all the host related workload including ZFS, and reserve/pin the P-cores for the VMs & containers. The idea is to limit context switch and L2 cache trashing. Also I would like to avoid cores with different architectures in the VMs
2. How would I go about doing it? I have an idea on how to pin a VM to only using P-cores. However, I am not sure how I can limit the host to only use E-cores.

Thanks for help!

 

[efschu3]

I'm doing something similar (using second chiplet of my 5950x for kvm vms and first chiplet for host/lxc stuff). I'm using cgroups v1 for that.
Add something like that to your kernel cmdline (those cores are not used by kernel by default):

systemd.unified_cgroup_hierarchy=0 rcu_nocbs=8-15,24-31 isolcpus=8-15,24-31 nohz_full=8-15,24-31

then using this for setting vm cpus to isolated cores:

GitHub - ayufan/pve-helpers: A set of Proxmox VE scripts that aids with suspend/resume and cpu pinning

A set of Proxmox VE scripts that aids with suspend/resume and cpu pinning - ayufan/pve-helpers

github.com

but it gives the vm cpu cores a host cpu core pool and not a 1to1 pinning, which may increase dpc latency due cache misses. this is why i use cgroups v1 (didnt figure out how to do that with cgroups v2). i wanted to pin host cpu core to guest cpu core to not get cache misses. for that i use following script:

VMID="100"
VIRT_CORES="8-15,24-31"
HOST_CORES="0-7,16-23"
TOTAL_CORES='0-31'
HOST_CORES_MASK=C3             # 0,4, bitmask 0b00010001

taskset --cpu-list  --all-tasks --pid $HOST_CORES "$(< /run/qemu-server/$VMID.pid)"

cset -m set -c $TOTAL_CORES -s machine.slice
cset -m set -c $VIRT_CORES -s user.slice

echo $HOST_CORES_MASK > /sys/bus/workqueue/devices/writeback/cpumask
echo 0 > /sys/bus/workqueue/devices/writeback/numa

cpu_tasks() {
    expect <<EOF | sed -n 's/^.* CPU .*thread_id=\(.*\)$/\1/p' | tr -d '\r' || true
spawn qm monitor $VMID
expect ">"
send "info cpus\r"
expect ">"
EOF
}


    VCPUS=($(cpu_tasks))
    VCPU_COUNT="${#VCPUS[@]}"

    if [[ $VCPU_COUNT -eq 0 ]]; then
        echo "* No VCPUS for VM$VMID"
        exit 1
    fi

    echo "* Detected ${#VCPUS[@]} assigned to VM$VMID..."
    echo "* Resetting cpu shield..."

    # Set qemu task affinity
    core_match=(8 24 9 25 10 26 11 27 12 28 13 29 14 30 15 31)
    for CPU_INDEX in "${!VCPUS[@]}"
    do
        CPU_TASK="${VCPUS[$CPU_INDEX]}"
        echo "* Assigning ${core_match[$CPU_INDEX]} to $CPU_TASK..."
        cset proc --move --pid "$CPU_TASK" --toset=user.slice --force
        taskset -pc "${core_match[$CPU_INDEX]}" "$CPU_TASK"
    done

i run the script after vm start

 

[WhiteNoise]

Yes, I was looking for something like this. Thank you very much for taking the time to respond!