Secure Boot was developed to protect the UEFI from firmware rootkits, not any malware that runs within the OS. It is a security feature that is transperent to the end user, but can be an inconvenience to admins. In this case, reinstalling the OS requires temporarily disabling Secure Boot.
Minisforum ms-01 i9-13900H 96GB with Proxmox
- Thread starter homelabenthusiast
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Not sure what you mean @Arjestin - I've installed ALL my ms01 proxmox nodes with UEFI - no issues there.
Can you please let us know, how the stability worked out over a LONG time for you ?Getting towards 2 months of uptime now, so I think we can comfortably say the problem is resolved, and I'm still on the 1.25 beta BIOS (so as to not upset my uptime!).Code:root@pve1:~# uname -a Linux pve1 6.8.12-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-1 (2024-08-05T16:17Z) x86_64 GNU/Linux root@pve1:~# uptime 13:22:16 up 53 days, 19:52, 1 user, load average: 4.10, 4.05, 3.64
Wanted to get a 2nd one of these for my Proxmox cluster to replace an old HPE DL20-gen9 but held off when it seemed unstable. Thinking I might have the confidence to spend more on these now!
Was hoping for a Black Friday discount on Amazon, but there's no joy there yet.
I have "a few" ms01 in cluster and "patched" the reboot/crash issues with a rather complex watchdog and a PDU that power cycles the node if there is something fishy going on with it. All the nodes were on 1.22, I've upgraded 3 today to 1.27 without changing the RAM speed. Only thing I've changed from default is that I've enabled RTH (Race To Halt - not sure if it makes any difference, I will report on it if I find a difference). originally my nodes were crashing between 2 days and 18 days. Some are more prone to reboots than others, but reshuffling VM's and CT's seems to made no difference - fun fact the most problematic node still kick the bucket with nothing on it (well - it hosts 3 ceph OSD's so it's not completely idle)
Just out of curiosity, if you have/had so many problems, why didn't you ever try BIOS v1.26?
I watched the minisforum website like a hawk for half a year with no upgrades to bios - I've even reflashed all the units with 1.22 early on. Unfortunately I had to get on with things and I've implemented somehow fool proof watchdog and it keept me running on that cluster. I've stumbled on this thread on Friday and started reading. mid way through going through pages I re-checked minisforum suport page and found 1.26 and 1.27. I finished reading whenever anybody had any bad experience with either and pulled the trigger on 1.27 to see how it's going.
I shall add that first 3 units I've got on very early preorder in Dec 2023, last unit arrived circa May 2024.
This is my brand new minisforum ms-01 on BIOS v 1.27 and installed with proxmox. I am disappointed in the instability and crashing of this device. I have the i5 2600H with 64GB Crucial 4800MHz RAM set to 4200 right now to see if that helps with any stability.
Update#1: So I passthrough my iGPU into a Windows 11 VM inside proxmox by injecting a specific ROM file for 12th gen alder lake CPUs. (GitHub - gangqizai/igd: Intel 核显直通 rom / Intel Integrated GPU passrough rom file for PVE). When I did this I was getting consistent crashes on first boot of the VM which looked similar to the pictures above. If I shutdown the VM and rebooted it again then I would not experience the crash and could successfully use windows 11. I was looking at the github issues and saw this (12400 安装驱动后黑屏 · Issue #23 · gangqizai/igd) forum and decided to play around with the DVMT setting within the BIOS. My DVMT was defaulted to 60MB. However, I have found success in the stability of booting my windows 11 VM first time when changing the DVMT value to 12MB. Now I feel like this is pretty low? But it works….
Wondering if anyone experiencing crashes in proxmox would try changing the settings DVMT in BIOS to 12MB and report your findings here.
@JaxJiang Could you look into this as possibly being related to the crashes in proxmox?
Update#2: Well, I was pretty much wrong about everything above.... It was just a not-so-great gen12_igd.rom file and also Proxmox QEMU server causing the issues on version 9.2! So, the fix was to follow: lixiaoliu666/intel6-14rom: build ROM files for Intel 6th-14th Gen iGPU passthrough in PVE VMs（intel6-14代pve虚拟机核显直通使用rom） and use the 6-14.rom in the Windows 11 VM with machine set to pc-i440fx-9.1 and reinstall qemu-server-kvm-9.1.2-3 ( apt reinstall pve-qemu-kvm=9.1.2-3 ). I had set the DVMT in BIOS to 64M.
Last edited:
I've got three MS-01s that have been doing the lockup/death thing since early on (close to a year). I've got 96gb of RAM (crucial). I went through this thread and thought that had done everything to help get it fixed but two of the nodes (one with no VMs even running) still locked up after a couple of hours.
They're all on the 1.27 BIOS, they've all got the 0x00004128 microcode, and I've turned the memory speed down to 4400 on all three nodes. they're all on 8.3.0.
When they lock up I don't see anything interesting (I had one attached to a pi KVM)
Any ideas what else I should try to resolve this?
They're all on the 1.27 BIOS, they've all got the 0x00004128 microcode, and I've turned the memory speed down to 4400 on all three nodes. they're all on 8.3.0.
When they lock up I don't see anything interesting (I had one attached to a pi KVM)
Any ideas what else I should try to resolve this?
Last edited:
So, if anybody reads this thread:
most of my crashes ended when I:
- upgraded bios to newest version
- replaced cpu paste on all the units.
- replaced one unit where CPU was faulty (or something else, but since you can only replace the ram then the unit went for RMA
- replaced 96gb crucial ram with 96GB kingstone ram on one unit.
Units are pretty stable for far, can get as much at 15 days operation out of them with no major issues.
Anyway. This was a painful lesson, and I'm pondering replacing this cluster with something that just works (ie, enterprise grade hardware ... maybe epyc 4005 or go all out for 9005, but It's hard to justify 9005)
most of my crashes ended when I:
- upgraded bios to newest version
- replaced cpu paste on all the units.
- replaced one unit where CPU was faulty (or something else, but since you can only replace the ram then the unit went for RMA
- replaced 96gb crucial ram with 96GB kingstone ram on one unit.
Units are pretty stable for far, can get as much at 15 days operation out of them with no major issues.
Anyway. This was a painful lesson, and I'm pondering replacing this cluster with something that just works (ie, enterprise grade hardware ... maybe epyc 4005 or go all out for 9005, but It's hard to justify 9005)
hey, can you please describe the lockup/death thing? if possible could you please share a screenshot or logs prior to it?
I have a cluster with 4x MS-01 (all three CPU variants) on BIOS v1.26 with 96GB of the same Crucial RAM, they work rock solid, never had a crash, even now in the summer heat (32C today) they run surprisingly well.
I did not edit my memory speed, all kept on default values (whatever they are).
Two of my MS-01 with 13th Gen Intel (i9-13900H) are indeed on microcode 0x4128 as you mentioned, but the other two with 12th Gen Intel CPU are on 0x437 in case that helps.
All 4 NICs are in use, all have a HDMI and USB (piKVM + switch) attached, nothing else.
Here is how I setup my BIOS:
Code:
Note: UEFI Shell (in BIOS main menu) to work needs Shell.efi in the root of attached USB stick.
BIOS Settings:
--------------
Update v1.26 if needed
Main > Set Date/Time
Advanced > CPU > Boot Performance: Max Non Turbo (during boot, no need to rush)
Advanced > CPU > Turbo Mode: Disabled (make sure no power/heat spikes)
Advanced > Onboard Device Setting > Audio: disabled
Advanced > Onboard Device Setting > PCH PCIE PORT (end of list) > Wifi: disabled
Advanced > ACPI > Power: Always on
Security > Secure Boot > Disable
Boot > Quite Boot > Disable (uncheck)
Boot Order:
#1: UEFI NVME (:proxmox)
#2: UEFI CD/DVD
#3: UEFI USB Device
#4: Network
#5: UEFI Hard Disk
BOOT NVME BBS: #1 proxmox
#2 HDD
***************************************************
*** ASPM was already all disabled in BIOS v1.26 ***
*** otherwise manual disable <pre 1.26) ***
***************************************************The second thing I did was to update the 10GB NICs using the 700Series_NVMUpdatePackage_v9_52_EFI.zip from Intel.
Here are my notes:
Code:
- Copy files to EFI disk.
- Boot to EFI USB Stick
fs0:
cd intel710/EFI2x64
nvmupdate64e.efi
choose card, wait.I do not use vPro/Intel AMT/MBEx. It was working, but very unstable/unpredictable (sometimes it worked, sometimes it didn't).
I don't think it's related, but early on I switched the original power supplies with SlimQ 240W GaN chargers.
The only issue I had was a compatibility problem with some Micron Enterprise NVMe SSD drives (7400 PRO, they would sometimes hang on boot), once I replaced with Samsung all was good. I believe this compatibility issue may even be solved with BIOS v1.27 but I don't bother since I got rid of these drives.
Last edited:
I did all of these exact things back in Feb.. BIOS, re-pasted the CPU, replaced the RAM (but with 64GB TeamGroup). I even strapped a giant fan on the side...
I am running an LSI SAS 9300-8e in the PCI slot with 120TB of disk, all on top of bog-standard ubuntu server. Its core workload is largely relegated purely as a NAS at this point due to reliability issues.
Almost like clockwork.. once every 7ish to 10 days or so, the entire system just halts and requires a hard, physical reboot. Logs dont point to any smoking guns. It's so consistent, my solution so far has been to connect the thing to a smart plug, and use Home Assistant to check the server for heartbeats, and when they stop, it hard powers down the server and powers it back up via that smart plug.
So I've just been on the hunt for a small server replacement option with 10gb ever since and have just written off MinisForum for the future.
Last edited:
Hi. There should be a read for how to down reduce the maximum CPU frequency in V1.27 BIOS.
This may be Intel's common unstable problem.
2. Disable Overclocking Lock: Advaned -> CPU configuation ->Overclocking Lock:[ Disable]
Reduce the maximum CPU frequency:
Advaned -> CPU configuation ->Turbo Ratio Limit Options.
P-Core Trubo Ratio Limit Ratio0: 51
P-Core Turbo Ratio Limit Ratio1: 51
E-Core Trubo Ratio Limit Ratio0: 40
E-Core Turbo Ratio Limit Ratio1: 40
E-Core Turbo Ratio Limit Ratio2: 40
E-Core Turbo Ratio Limit Ratio3: 40
Save and reboot enter into BIOS. Advaned -> CPU configuation ->Turbo Ratio Limit Options. Check is it max frequency has been reduced.
Dude, my original problems with ms-01 were so bad that I've had a PDU that can ping specific nodes and if those fail to respond - those will get a hard reboot. Then on HASS I there was a proxmox extension that would monitor the nodes and if something was indicating that there was something wrong - it would tell PDU to kick the power on given node. It was a blast.
TBH I'm now pondering to replace this small cluster with epyc 4005, and delegate cluster to purely testing for work purposes. I'm starting to think that cluster hype was .... a hype
I have updated to v1.27 of the bios and my microcode still says 0x411c - is this the right version for the MS-01 i9-13900h for Proxmox install? If not what version is and how do I obtain it as it is not on the support forum. Thanks
This is unrelated to the BIOS version as far as I can tell.
Mine shows 0x4129 on the MS-01 with i9-13900h
Code:
grep microcode /proc/cpuinfo | uniq
microcode : 0x4129
Code:
dmesg | grep microcode
[ 1.509482] microcode: Current revision: 0x00004129
[ 1.509485] microcode: Updated early from: 0x0000411capt install intel-microcode.If you can't install, you may need to add
non-free-firmware to your repositories first.Google tutorials on adding package intel-microcode to Debian or Proxmox if you can't get it to work.
There have two way upgrade microcode, BIOS and System. If your system already lastest enough, have microcode newer than BIOS, then will replace it. If you want more stablility. You can check there two new menu in CPU Performace setting. -0.2Ghz max Turbo Freq will more stable.