Wrap up...
- Initial hardware preparation was to replace the original thermal paste with Arctic MX4.
- Installation of opnsense bare metal failed due to issues with the hard drive. FreeBSD constantly caused errors. Opnsense on Proxmox runs stable.
BIOS:
- All unneeded hardware off and unplugged: SATA disabled, USB HSII on xHCI disabled, HD audio disabled, Serial I2C5 controller disabled, eMMC 5.1 Controller disabled, Sensor Hub type None
- PECI on (no idea if it has an influence, it just runs)
- Platform PL1&2 disabled (no time yet to play with)
- power limit 4 override disabled
- Package PL1 = 8000, PL2 = 25000, Energy eff-turbo enabled (-> is in the CPU/Turbo sub tab)
- C-States enabled, Package limit C10
Chipset -> PCH-IO Config -> PCI Express Config:
- every unneeded PCIe slot disabled. I am using slot #1, 7,9,10 and 12. Slot 9,10,11 and 12 are the NICs. On the enabled ones, I have ASPM = auto, but L1 low and L1 Substates disabled - had no time to test the influence on Proxmox and Opnsense.
Proxmox:
The installation of Proxmox 7.4 will halt with an x-server error. Identify the VGA port with lspci ( 02:00.0 for me), then create /usr/share/X11/xorg.conf.d/n100.conf
Code:
Section "Device"
Identifier "Card0"
Driver "fbdev"
BusID "pci0:02:0:0:"
EndSection
and then run xinit again.
- CPU governor: powersave
- install PVE-Kernel 6.2, it results in a lower power consumption
Opnsense VM settings in Proxmox
- 4 CPUs, type host, all meltdown etc. mitigations disabled
- hard drive: type lvm
- NICs: both NICs: pass through and not virtual. NIC1 = WAN, NIC2 = LAN
- a third NIC of the box is connected internally to the switch and set to 100 mbit only. It is only used to access the Proxmox host. Important if you run the other NICs on pass through, as you can't share the LAN port with the Proxmox host.
Opnsense tunings:
Tunables:
dev.igc.0.fc =0, dev.igc.1.fc = 0 (Flow control disabled)
hw.acpi.cpu.cx_lowest = C3 (can be set to C8, didn't see any differences in power consumption)
hw.ibrs_disable = 1 disable Spectre V2 mitigation
hw.igc.rx_process_limit = -1 (set unlimited packets per interrupt)
hw.igc.eee_setting = 0 - Enable Energy Efficient Ethernet
vm.pmap.pti = 0 (Meltdown mitigation off)
- Interfaces -> settings: all hardware offloads enabled.
- CPU governor: Hiadaptive, Powerd disabled (had no influence)
Add an additional line in /etc/rc.conf
This excludes IRQ and NICs from being uses as source for random seeds and adds some speed
Stress tests etc.:
- install lm_sensors on proxmox, run "watch sensors" in one shell
- install stress-ng and run "stress-ng --matrix 0 -t 1m" ina second shell to see, how hot the box gets after running for 1 minute.
Package limits can be read out with
Code:
powercap-info -p intel-rapl
To set PL1 = 8W and PL2 to 26 W, run:
Code:
powercap-set -z 0 -p intel-rapl -c 1 -l 26000000
powercap-set -z 0 -p intel-rapl -c 0 -l 8000000
Then run a stress test to see how well the box behaves under load.
With the above settings, my N100 draws between 6 and 12 W (measured on the 12V line, so you might have to add 15-20% if you measure at the wall, depending on the power supply details). I know that some boxes crash after running for a week, but I still consider the system to be stable. I have tried both, from long time idle (to trigger unwanted energy saving mechanisms) to 200 mbit in both directions with many connections at the same time. System is stable, no crash or reboot so far.
