H12SSL-NT - Work around missing PCIe Force Link Speed option

[NablaSquaredG]

Problem:

Gen 4 capable CPU + board (H12SSL-NT), Gen 4 capable SSD (Samsung PM1733) connected to onboard SlimSAS (JNVMe0 / JNVMe1)
Gen 3 capable cable (CBL-SAST-0826) + backplane (BPN-SAS3-826A-N4)

CPU and SSD will negotiate to Gen 4. Results in lots of PCIe errors because cable + backplane are only Gen3 capable

The board H12SSL-NT does not have any option to force PCIe link speed (thanks for nothing, Supermicro :facepalm


Solution (for Proxmox 8.1-1)
Copy pcie_set_speed.sh and PM1733_force_gen3.sh to /usr/local/sbin

Copy PM1733-force-gen3.service to /etc/systemd/system/PM1733-force-gen3.service

systemctl daemon-reload

systemctl start PM1733-force-gen3.service

journalctl -u PM1733-force-gen3.service

If everything looks good:

systemctl enable PM1733-force-gen3.service

Possible Improvements:
Execute even earlier, somewhere in initramfs

License:

pcie_set_speed.sh by https://alexforencich.com/wiki/en/pcie/set-speed

Distributed under CC Attribution-Share Alike 4.0 International (https://creativecommons.org/licenses/by-sa/4.0/)

pcie_set_speed.sh

#!/bin/bash

dev=$1
speed=$2

if [ -z "$dev" ]; then
    echo "Error: no device specified"
    exit 1
fi

if [ ! -e "/sys/bus/pci/devices/$dev" ]; then
    dev="0000:$dev"
fi

if [ ! -e "/sys/bus/pci/devices/$dev" ]; then
    echo "Error: device $dev not found"
    exit 1
fi

pciec=$(setpci -s $dev CAP_EXP+02.W)
pt=$((("0x$pciec" & 0xF0) >> 4))

port=$(basename $(dirname $(readlink "/sys/bus/pci/devices/$dev")))

if (($pt == 0)) || (($pt == 1)) || (($pt == 5)); then
    dev=$port
fi

lc=$(setpci -s $dev CAP_EXP+0c.L)
ls=$(setpci -s $dev CAP_EXP+12.W)

max_speed=$(("0x$lc" & 0xF))

echo "Link capabilities:" $lc
echo "Max link speed:" $max_speed
echo "Link status:" $ls
echo "Current link speed:" $(("0x$ls" & 0xF))

if [ -z "$speed" ]; then
    speed=$max_speed
fi

if (($speed > $max_speed)); then
    speed=$max_speed
fi

echo "Configuring $dev..."

lc2=$(setpci -s $dev CAP_EXP+30.L)

echo "Original link control 2:" $lc2
echo "Original link target speed:" $(("0x$lc2" & 0xF))

lc2n=$(printf "%08x" $((("0x$lc2" & 0xFFFFFFF0) | $speed)))

echo "New target link speed:" $speed
echo "New link control 2:" $lc2n

setpci -s $dev CAP_EXP+30.L=$lc2n

echo "Triggering link retraining..."

lc=$(setpci -s $dev CAP_EXP+10.L)

echo "Original link control:" $lc

lcn=$(printf "%08x" $(("0x$lc" | 0x20)))

echo "New link control:" $lcn

setpci -s $dev CAP_EXP+10.L=$lcn

sleep 0.1

ls=$(setpci -s $dev CAP_EXP+12.W)

echo "Link status:" $ls
echo "Current link speed:" $(("0x$ls" & 0xF))

PM1733_force_gen3.sh

#!/bin/sh

set -o errexit   # abort on nonzero exitstatus
set -o nounset   # abort on unbound variable

lspci | grep "PM173X" | cut -d ' ' -f1 | while read line; do
    echo "Forcing PM173X at $line to Gen3 Link Speed"
    ./pcie_set_speed.sh "$line" 3
    echo ""
done

PM1733_force_gen3.sh

[Unit]
Description=Force PM1733 SSDs to Gen3 Speed
# Run as early as possible
Before=basic.target
After=local-fs.target sysinit.target
DefaultDependencies=no

[Service]
Type=oneshot
ExecStart=PM1733_force_gen3.sh
WorkingDirectory=/usr/local/sbin

[Install]
WantedBy=basic.target

 

[ano]

interesting workaround

in general pm9a3 work far better than pm1733, even in gen3/4 mix backplan settings

usually supermicro will sell pm9a3, not pm1733 just because of many issues with 1733

adjustability for nvme is dissapointing on the h12 / h13