AMD RADEON PRO V620 32GB GDDR6 GPUs - $565

[luckylinux]

On Supermicro X10SRL-F, after installing the 2nd GPU and enabling SR-IOV, Machine is Stuck at POST Code 91 (DXE-BIOS PCI Bus Enumeration).

If SR-IOV is disabled, this results in a Kernel Call Stack / Driver not loading correctly and GPU failing to initialize.

Did anybody face something similar ?

I tried playing with the MMIO High Base / MMIO Size Settings in BIOS but whenever I set something and enable SR-IOV, then I basically get locked out.

To unlock oneself one can do:

/opt/Supermicro/sum/default/sum --journal_path "${SUM_JOURNAL_PATH}" -c LoadDefaultBiosCfg -i "${BMC_IP}" -u "${BMC_USERNAME}" -p "${BMC_PASSWORD}" --reboot

or

ipmitool -I lanplus -H "${BMC_IP}" -U "${BMC_USERNAME}" -P "${BMC_PASSWORD}"  chassis bootdev bios clear-cmos=yes

Needless to say this does NOT solve the Issue .

EDIT 1: after resetting a new Tab appared in BIOS



Great, so the only Thing this Motherboard should do (get these GPUs to work correctly) it isn't doing it .

EDIT 2: so I managed to get SR-IOV Enabled, though I needed to change a few of the Settings previously mentioned, in order for it to work with Supermicro X10SRL-F.

root@pve055:~# for file in $(ls /etc/{modules-load.d,modprobe.d,default/grub.d}/*); do echo -e "# ${file}"; echo -e "=================================================================================================================================="; cat "${file}"; echo -e "==================================================================================================================================\n\n"; done
# /etc/default/grub.d/amdgpu.cfg
# /etc/default/grub.d/amdgpu.cfg
==================================================================================================================================
# Does NOT work
# GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} amdgpu.ppfeaturemask=0xffffbfff amdgpu.ignore_min_pcap=1 amdgpu.runpm=0 pcie_aspm=off amdgpu.aspm=0 pci=realloc,norid"
# GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} amdgpu.ppfeaturemask=0xffffbfff amdgpu.ignore_min_pcap=1 amdgpu.runpm=0 pcie_aspm=off amdgpu.aspm=0 pci=realloc,norid"

# Works, but it's NOT possible to set Power Limits
# GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} amdgpu.ppfeaturemask=0xffffbfff amdgpu.ignore_min_pcap=1 amdgpu.runpm=0 pcie_aspm=off amdgpu.aspm=0 pci=realloc,realloc=on,nocrs"
# GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} amdgpu.ppfeaturemask=0xffffbfff amdgpu.ignore_min_pcap=1 amdgpu.runpm=0 pcie_aspm=off amdgpu.aspm=0 pci=realloc,realloc=on,nocrs"

# Testing
GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} amdgpu.ppfeaturemask=0xffffbfff amdgpu.runpm=1 pcie_aspm=on amdgpu.aspm=1 pci=realloc,realloc=on,nocrs"
GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} amdgpu.ppfeaturemask=0xffffbfff amdgpu.runpm=1 pcie_aspm=on amdgpu.aspm=1 pci=realloc,realloc=on,nocrs"

# Testing
# GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} amdgpu.ppfeaturemask=0xffffbfff amdgpu.runpm=0 pcie_aspm=off amdgpu.aspm=0 pci=realloc,realloc=on,nocrs amdgpu.exp_hw_support=1 amdgpu.ip_block_mask=0xfffffbbf"
# GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} amdgpu.ppfeaturemask=0xffffbfff amdgpu.runpm=0 pcie_aspm=off amdgpu.aspm=0 pci=realloc,realloc=on,nocrs amdgpu.exp_hw_support=1 amdgpu.ip_block_mask=0xfffffbbf"
==================================================================================================================================


# /etc/default/grub.d/cgroups.cfg
==================================================================================================================================
GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} systemd.unified_cgroup_hierarchy=1"
GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} systemd.unified_cgroup_hierarchy=1"
==================================================================================================================================


# /etc/default/grub.d/framebuffer.cfg
==================================================================================================================================
###GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} iommu=pt amd_iommu=on pcie_acs_override=downstream"
###GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} iommu=pt amd_iommu=on pcie_acs_override=downstream"

#GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} initcall_blacklist=sysfb_init"
#GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} initcall_blacklist=sysfb_init"

==================================================================================================================================


# /etc/default/grub.d/iommu.cfg
==================================================================================================================================
#GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} iommu=pt amd_iommu=on pcie_acs_override=downstream"
#GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} iommu=pt amd_iommu=on pcie_acs_override=downstream"

GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} iommu=pt intel_iommu=on"
GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} iommu=pt intel_iommu=on"
==================================================================================================================================


# /etc/default/grub.d/pcie-power-management.cfg
==================================================================================================================================
#GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} pcie_port_pm=off pcie_aspm.policy=performance"
#GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} pcie_port_pm=off pcie_aspm.policy=performance"
#GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} pcie_port_pm=force pcie_aspm.policy=powersave"
#GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} pcie_port_pm=force pcie_aspm.policy=powersave"

GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} pcie_port_pm=off"
GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} pcie_port_pm=off"
==================================================================================================================================


# /etc/default/grub.d/powersave.cfg
==================================================================================================================================
#GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} cpufreq.default_governor=powersave initcall_blacklist=acpi_cpufreq_init amd_pstate.shared_mem=1 amd_pstate=guided"
#GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} cpufreq.default_governor=powersave initcall_blacklist=acpi_cpufreq_init amd_pstate.shared_mem=1 amd_pstate=guided"
#GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} cpufreq.default_governor=powersave initcall_blacklist=acpi_cpufreq_init amd_pstate.shared_mem=1 amd_pstate=guided"
#GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} cpufreq.default_governor=powersave initcall_blacklist=acpi_cpufreq_init amd_pstate.shared_mem=1 amd_pstate=guided"
==================================================================================================================================


# /etc/default/grub.d/proxmox-ve.cfg
==================================================================================================================================
GRUB_DISTRIBUTOR="Proxmox VE"
GRUB_DISABLE_OS_PROBER=true
==================================================================================================================================


# /etc/default/grub.d/zfs.cfg
==================================================================================================================================
GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} root=ZFS=\"rpool/ROOT/debian\" zfs_force=1 boot=zfs"
GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} root=ZFS=\"rpool/ROOT/debian\" zfs_force=1 boot=zfs"
==================================================================================================================================


# /etc/modprobe.d/amd64-microcode-blacklist.conf
==================================================================================================================================
# The microcode module attempts to apply a microcode update when
# it autoloads.  This is not always safe, so we block it by default.
blacklist microcode
==================================================================================================================================


# /etc/modprobe.d/amdgpu.conf
==================================================================================================================================
# Forces the driver to ignore enterprise hardware profile validation and pushes past the strict PSP firmware gating sequence that caused your previous crash
# options amdgpu exp_hw_support=1
# options amdgpu runpm=0
# options amdgpu ip_block_mask=0xfffffbbf
==================================================================================================================================


# /etc/modprobe.d/blacklist-amdgpu.conf
==================================================================================================================================
# Blacklist AMD GPU Driver (causes Kernel Panic during Proxmox VE Host Boot)

blacklist amdgpu
blacklist radeon
blacklist snd_hda_intel
==================================================================================================================================


# /etc/modprobe.d/blacklist-iwlwifi.conf
==================================================================================================================================
# Blacklist WiFi Driver

blacklist iwlwifi
==================================================================================================================================


# /etc/modprobe.d/blacklist-nvidia.conf
==================================================================================================================================
# Blacklist NVIDIA Drivers
blacklist nouveau
blacklist nvidia
==================================================================================================================================


# /etc/modprobe.d/dirtyfrag.conf
==================================================================================================================================
# Linux Kernel Dirty Frag Mitigation
install esp4 /bin/false
install esp6 /bin/false
install rxrpc /bin/false
==================================================================================================================================


# /etc/modprobe.d/disable-algif.conf
==================================================================================================================================
install algif_aead /bin/false
==================================================================================================================================


# /etc/modprobe.d/dkms.conf
==================================================================================================================================
# modprobe information used for DKMS modules
#
# This is a stub file, should be edited when needed,
# used by default by DKMS.
==================================================================================================================================


# /etc/modprobe.d/mdadm.conf
==================================================================================================================================
# mdadm module configuration file
# set start_ro=1 to make newly assembled arrays read-only initially,
# to prevent metadata writes.  This is needed in order to allow
# resume-from-disk to work - new boot should not perform writes
# because it will be done behind the back of the system being
# resumed.  See http://bugs.debian.org/415441 for details.

options md_mod start_ro=1
==================================================================================================================================


# /etc/modprobe.d/pve-blacklist.conf
==================================================================================================================================
# This file contains a list of modules which are not supported by Proxmox VE

# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
==================================================================================================================================


# /etc/modules-load.d/beep.conf
==================================================================================================================================
# Enable Beep / Speaker Module
pcspkr
==================================================================================================================================


# /etc/modules-load.d/modules.conf
==================================================================================================================================
# /etc/modules is obsolete and has been replaced by /etc/modules-load.d/.
# Please see modules-load.d(5) and modprobe.d(5) for details.
#
# Updating this file still works, but it is undocumented and unsupported.
==================================================================================================================================


# /etc/modules-load.d/pcie-passthrough.conf
==================================================================================================================================
# Enable PCIe Passthrough
vfio
vfio_iommu_type1
vfio_pci
==================================================================================================================================


# /etc/modules-load.d/qemu-server.conf
==================================================================================================================================
# Enable vhost_net
vhost_net
==================================================================================================================================


# /etc/modules-load.d/zram.conf
==================================================================================================================================
# Load the zram kernel Module
zram
==================================================================================================================================

All BIOS Settings obtained via

# Save Current Configuration
/opt/Supermicro/sum/default/sum --journal_path "${SUM_JOURNAL_PATH}" -c GetCurrentBiosCfg -i "${BMC_IP}" -u "${BMC_USERNAME}" -p "${BMC_PASSWORD}" --file "${SUM_BIOS_CFG_BACKUP_FILE}" --overwrite

IIRC I had to set all PCIe Slots to x8 or x8x8, disable ASPM, enable SRIOV, set some "average" 24TB for MMIOHBase, set MMIO High Size to the max (1024GB) and Maximum Payload to 256 Bytes.

Maybe there is something else that I forgot.

#Supermicro Update Manager 2.14.0 (2024/02/15)
#File generated at 2026-07-20_16:15:19
#Please refer to SUM User's guide 'Format of BIOS Settings Text File' for usage.

[Advanced|Boot Feature]
Quiet Boot=01                           // Please enter the value in 2 hexadecimal digits. Default value is <<<01>>>       
AddOn ROM Display Mode=01               // *01 (Force BIOS), 00 (Keep Current)       
Bootup NumLock State=01                 // *01 (On), 00 (Off)       
Wait For "F1" If Error=01               // 00 (Disabled), *01 (Enabled)       
INT19 Trap Response=01                  // *01 (Immediate), 00 (Postponed)       
Re-try Boot=00                          // *00 (Disabled), 01 (Legacy Boot), 02 (EFI Boot)       
Watch Dog Function=00                   // *00 (Disabled), 01 (Enabled)       
Power Button Function=01                // *01 (Instant Off), 00 (4 Seconds Override)       
Restore on AC Power Loss=02             // 00 (Stay Off), 01 (Power On), *02 (Last State)       

[Advanced|CPU Configuration]
Clock Spread Spectrum=00                // Please enter the value in 2 hexadecimal digits. Default value is <<<00>>>       
Hyper-Threading (ALL)=00                // 01 (Disable), *00 (Enable)       
Cores Enabled=00                        // Please enter the value in 2 hexadecimal digits. Default value is <<<00>>>       
Monitor/Mwait=01                        // 00 (Disable), *01 (Enable)       
Execute Disable Bit=01                  // 00 (Disable), *01 (Enable)       
PPIN Control=01                         // 00 (Unlock/Disable), *01 (Unlock/Enable)       
Hardware Prefetcher=01                  // *01 (Enable), 00 (Disable)       
Adjacent Cache Prefetch=01              // *01 (Enable), 00 (Disable)       
DCU Streamer Prefetcher=01              // *01 (Enable), 00 (Disable)       
DCU IP Prefetcher=01                    // *01 (Enable), 00 (Disable)       
Direct Cache Access (DCA)=02            // 00 (Disable), 01 (Enable), *02 (Auto)       
X2APIC=00                               // *00 (Disable), 01 (Enable)       
AES-NI=01                               // 00 (Disable), *01 (Enable)       
Intel Virtualization Technology=01      // 00 (Disable), *01 (Enable)       

[Advanced|CPU Configuration|Advanced Power Management Configuration]
Power Technology=01                     // 00 (Disable), *01 (Energy Efficient), 02 (Custom)       
Energy Performance Tuning=01            // 00 (Enable), *01 (Disable)       
Energy Performance BIAS setting.=07     // 00 (Performance), *07 (Balanced Performance), 08 (Balanced Power), 0F (Power)       
Energy Efficient Turbo=00               // *00 (Enable), 01 (Disable)       

[Advanced|CPU Configuration|Advanced Power Management Configuration|CPU P State Control]
EIST (P-States)=01                      // 00 (Disable), *01 (Enable)       
Turbo Mode=01                           // 00 (Disable), *01 (Enable)       
P-State Coordination=00                 // *00 (HW_ALL), 01 (SW_ALL), 02 (SW_ANY)       

[Advanced|CPU Configuration|Advanced Power Management Configuration|CPU HWPM State Control]
Enable CPU HWPM=00                      // *00 (Disable), 01 (HWPM NATIVE MODE), 02 (HWPM OOB MODE)       
Enable CPU Autonomous Cstate=00         // *00 (Disable), 01 (Enable)       

[Advanced|CPU Configuration|Advanced Power Management Configuration|CPU C State Control]
Package C State Limit=03                // 00 (C0/C1 state), 01 (C2 state), 02 (C6(non Retention) state), *03 (C6(Retention) state)       
CPU C3 Report=00                        // *00 (Disable), 01 (Enable)       
CPU C6 Report=01                        // 00 (Disable), *01 (Enable)       
Enhanced Halt State (C1E)=01            // 00 (Disable), *01 (Enable)       

[Advanced|CPU Configuration|Advanced Power Management Configuration|CPU T State Control]
ACPI T-States=01                        // 00 (Disable), *01 (Enable)       

[Advanced|Chipset Configuration|North Bridge|IIO Configuration]
EV DFX Features=00                      // *00 (Disable), 01 (Enable)       
Snoop Response Hold Off=09              // Please enter the value in 2 hexadecimal digits. Default value is <<<09>>>       

[Advanced|Chipset Configuration|North Bridge|IIO Configuration|IIO1 Configuration]
IOU2 (IIO1 PCIe Port 1)=01              // 00 (x4x4), 01 (x8), *FF (Auto)       
CPU SLOT3 PCI-E 3.0 X8=03               // 01 (Gen 1 (2.5 GT/s)), 02 (Gen 2 (5 GT/s)), *03 (Gen 3 (8 GT/s))       
CPU SLOT2 PCI-E 3.0 X4(IN X8)=03        // 01 (Gen 1 (2.5 GT/s)), 02 (Gen 2 (5 GT/s)), *03 (Gen 3 (8 GT/s))       
IOU0 (IIO1 PCIe Port 2)=03              // 00 (x4x4x4x4), 01 (x4x4x8), 02 (x8x4x4), 03 (x8x8), 04 (x16), *FF (Auto)       
CPU SLOT4 PCI-E 3.0 X8(IN X16)=03       // 01 (Gen 1 (2.5 GT/s)), 02 (Gen 2 (5 GT/s)), *03 (Gen 3 (8 GT/s))       
CPU SLOT5 PCI-E 3.0 X8=03               // 01 (Gen 1 (2.5 GT/s)), 02 (Gen 2 (5 GT/s)), *03 (Gen 3 (8 GT/s))       
IOU1 (IIO1 PCIe Port 3)=03              // 00 (x4x4x4x4), 01 (x4x4x8), 02 (x8x4x4), 03 (x8x8), 04 (x16), *FF (Auto)       
CPU SLOT6 PCI-E 3.0 X8(IN X16)=03       // 01 (Gen 1 (2.5 GT/s)), 02 (Gen 2 (5 GT/s)), *03 (Gen 3 (8 GT/s))       
CPU SLOT7 PCI-E 3.0 X8=03               // 01 (Gen 1 (2.5 GT/s)), 02 (Gen 2 (5 GT/s)), *03 (Gen 3 (8 GT/s))       
IIO1 IOU0 Non-Posted Prefetch=00        // 00 (Enable), *01 (Disable)       
IIO1 IOU1 Non-Posted Prefetch=00        // 00 (Enable), *01 (Disable)       
IIO1 IOU2 Non-Posted Prefetch=00        // 00 (Enable), *01 (Disable)       

[Advanced|Chipset Configuration|North Bridge|IIO Configuration|IOAT Configuration]
Enable IOAT=01                          // 00 (Disable), *01 (Enable)       
No Snoop=00                             // *00 (Disable), 01 (Enable)       
Relaxed Ordering=00                     // *00 (Disable), 01 (Enable)       

[Advanced|Chipset Configuration|North Bridge|IIO Configuration|Intel VT for Directed I/O (VT-d)]
Intel VT for Directed I/O (VT-d)=01     // *01 (Enable), 00 (Disable)       
ACS Control=00                          // *01 (Enable), 00 (Disable)       
Interrupt Remapping=00                  // *01 (Enable), 00 (Disable)       

[Advanced|Chipset Configuration|North Bridge|QPI Configuration|QPI General Configuration]
COD Enable=00                           // *00 (Disable), 01 (Enable)       
Early Snoop=02                          // 00 (Disable), 01 (Enable), *02 (Auto)       
Home Dir Snoop with IVT- Style OSB=02   // 00 (Disable), 01 (Enable), *02 (Auto)       
Isoc Mode=00                            // *00 (Disable), 01 (Enable)       

[Advanced|Chipset Configuration|North Bridge|Memory Configuration]
Enforce POR=00                          // *00 (Enabled), 01 (Disabled)       
Memory Frequency=00                     // *00 (Auto), 05 (1333), 07 (1600), 09 (1867), 0B (2133), 0D (2400)       
Data Scrambling=02                      // *02 (Auto), 00 (Disabled), 01 (Enabled)       
tCCD_L Relaxation=01                    // 00 (Disabled), *01 (Auto)       
Enable ADR=01                           // 00 (Disabled), *01 (Hardware Triggered ADR)       
Erase-Arm NVDIMMs=01                    // 00 (Disabled), *01 (Enabled)        Enable ADR = "ADR + NVDIMMs"
Restore NVDIMMs=01                      // 00 (Disabled), *01 (Enabled)        Enable ADR = "ADR + NVDIMMs"
Reset Trigger ADR=00                    // *00 (Disabled), 01 (Enabled)       
S5 Trigger ADR=00                       // *00 (Disabled), 01 (Enabled)       
DRAM RAPL Baseline=02                   // 00 (Disable), 01 (DRAM RAPL Mode 0), *02 (DRAM RAPL Mode 1)       
Set Throttling Mode=02                  // 00 (Disabled), *02 (CLTT)       
A7 Mode=01                              // 00 (Disable), *01 (Enable)       

[Advanced|Chipset Configuration|North Bridge|Memory RAS Configuration]
RAS Mode=00                             // *00 (Independent), 01 (Mirror), 02 (Lockstep Mode)       
Memory Rank Sparing=00                  // *00 (Disabled), 01 (Enabled)       
Multi Rank Sparing=04                   // 01 (One Rank), 02 (Two Rank), 03 (Three Rank), *04 (Auto)        Memory Rank Sparing = "Enabled"
Patrol Scrub=01                         // 00 (Disable), *01 (Enable)       
Patrol Scrub Interval=18                // Please enter the value in 2 hexadecimal digits. Default value is <<<18>>>        Patrol Scrub = "Enable"
Demand Scrub=01                         // 00 (Disable), *01 (Enable)       
Device Tagging=00                       // *00 (Disable), 01 (Enable)       

[Advanced|Chipset Configuration|South Bridge]
Legacy USB Support=00                   // *00 (Enabled), 01 (Disabled), 02 (Auto)       
XHCI Hand-off=01                        // *01 (Enabled), 00 (Disabled)       
EHCI Hand-off=00                        // *00 (Disabled), 01 (Enabled)       
Port 60/64 Emulation=01                 // 00 (Disabled), *01 (Enabled)       
USB 3.0 Support=02                      // 03 (Smart Auto), *02 (Auto), 01 (Enabled), 00 (Disabled)       
EHCI1=01                                // 00 (Disabled), *01 (Enabled)       
EHCI2=01                                // 00 (Disabled), *01 (Enabled)       
XHCI Pre-Boot Driver=01                 // *01 (Enabled), 00 (Disabled)       
Azalia=02                               // 00 (Disabled), 01 (Enabled), *02 (Auto)       
Azalia PME Enable=00                    // *00 (Disabled), 01 (Enabled)        Azalia = "Enabled" or Azalia = "Auto"
SATA Frozen=01                          // 00 (Disabled), *01 (Enabled)       

[Advanced|SATA Configuration]
SATA Controller=01                      // 00 (Disabled), *01 (Enabled)       
Configure SATA as=01                    // 00 (IDE), *01 (AHCI), 02 (RAID)        SATA Controller = "Enabled"
SATA Support Aggressive Link Power Mgmt=00// *00 (Disabled), 01 (Enabled)        Configure SATA as = "AHCI" or Configure SATA as = "RAID"
SATA RAID Option ROM/UEFI Driver=02     // 00 (Disabled), 01 (EFI), *02 (Legacy)        Configure SATA as = "RAID"
Port 0 Hot Plug=01                      // 00 (Disabled), *01 (Enabled)        Configure SATA as = "AHCI" or Configure SATA as = "RAID"
Port 0 Spin Up Device=00                // *00 (Disabled), 01 (Enabled)        Configure SATA as = "AHCI" or Configure SATA as = "RAID"
Port 0 SATA Device Type=00              // *00 (Hard Disk Drive), 01 (Solid State Drive)       
Port 1 Hot Plug=01                      // 00 (Disabled), *01 (Enabled)        Configure SATA as = "AHCI" or Configure SATA as = "RAID"
Port 1 Spin Up Device=00                // *00 (Disabled), 01 (Enabled)        Configure SATA as = "AHCI" or Configure SATA as = "RAID"
Port 1 SATA Device Type=00              // *00 (Hard Disk Drive), 01 (Solid State Drive)       
Port 2 Hot Plug=01                      // 00 (Disabled), *01 (Enabled)        Configure SATA as = "AHCI" or Configure SATA as = "RAID"
Port 2 Spin Up Device=00                // *00 (Disabled), 01 (Enabled)        Configure SATA as = "AHCI" or Configure SATA as = "RAID"
Port 2 SATA Device Type=00              // *00 (Hard Disk Drive), 01 (Solid State Drive)       
Port 3 Hot Plug=01                      // 00 (Disabled), *01 (Enabled)        Configure SATA as = "AHCI" or Configure SATA as = "RAID"
Port 3 Spin Up Device=00                // *00 (Disabled), 01 (Enabled)        Configure SATA as = "AHCI" or Configure SATA as = "RAID"
Port 3 SATA Device Type=00              // *00 (Hard Disk Drive), 01 (Solid State Drive)       
Port 4 Hot Plug=01                      // 00 (Disabled), *01 (Enabled)        Configure SATA as = "AHCI" or Configure SATA as = "RAID"
Port 4 Spin Up Device=00                // *00 (Disabled), 01 (Enabled)        Configure SATA as = "AHCI" or Configure SATA as = "RAID"
Port 4 SATA Device Type=00              // *00 (Hard Disk Drive), 01 (Solid State Drive)       
Port 5 Hot Plug=01                      // 00 (Disabled), *01 (Enabled)        Configure SATA as = "AHCI" or Configure SATA as = "RAID"
Port 5 Spin Up Device=00                // *00 (Disabled), 01 (Enabled)        Configure SATA as = "AHCI" or Configure SATA as = "RAID"
Port 5 SATA Device Type=00              // *00 (Hard Disk Drive), 01 (Solid State Drive)       

[Advanced|sSATA Configuration]
sSATA Controller=01                     // *01 (Enabled), 00 (Disabled)       
Configure sSATA as=01                   // 00 (IDE), *01 (AHCI), 02 (RAID)        sSATA Controller = "Enabled"
sSATA Support Aggressive Link Power Mgmt=00// *00 (Disabled), 01 (Enabled)        Configure sSATA as = "AHCI" or Configure sSATA as = "RAID"
sSATA RAID Option ROM/UEFI Driver=02    // 00 (Disabled), 01 (EFI), *02 (Legacy)        Configure sSATA as = "RAID"
SATA/sSATA RAID Boot Select=01          // 00 (SATA Controller), *01 (sSATA Controller), 02 (Both)       
sSATA Port 0 Hot Plug=01                // 00 (Disabled), *01 (Enabled)        Configure sSATA as = "AHCI" or Configure sSATA as = "RAID"
sSATA Port 0 Spin Up Device=00          // *00 (Disabled), 01 (Enabled)        Configure sSATA as = "AHCI" or Configure sSATA as = "RAID"
Port 0 sSATA Device Type=00             // *00 (Hard Disk Drive), 01 (Solid State Drive)       
sSATA Port 1 Hot Plug=01                // 00 (Disabled), *01 (Enabled)        Configure sSATA as = "AHCI" or Configure sSATA as = "RAID"
sSATA Port 1 Spin Up Device=00          // *00 (Disabled), 01 (Enabled)        Configure sSATA as = "AHCI" or Configure sSATA as = "RAID"
Port 1 sSATA Device Type=00             // *00 (Hard Disk Drive), 01 (Solid State Drive)       
sSATA Port 2 Hot Plug=01                // 00 (Disabled), *01 (Enabled)        Configure sSATA as = "AHCI" or Configure sSATA as = "RAID"
sSATA Port 2 Spin Up Device=00          // *00 (Disabled), 01 (Enabled)        Configure sSATA as = "AHCI" or Configure sSATA as = "RAID"
Port 2 sSATA Device Type=00             // *00 (Hard Disk Drive), 01 (Solid State Drive)       
sSATA Port 3 Hot Plug=01                // 00 (Disabled), *01 (Enabled)        Configure sSATA as = "AHCI" or Configure sSATA as = "RAID"
sSATA Port 3 Spin Up Device=00          // *00 (Disabled), 01 (Enabled)        Configure sSATA as = "AHCI" or Configure sSATA as = "RAID"
Port 3 sSATA Device Type=00             // *00 (Hard Disk Drive), 01 (Solid State Drive)       

[Advanced|PCIe/PCI/PnP Configuration]
PCI PERR/SERR Support=00                // *00 (Disabled), 01 (Enabled)       
Above 4G Decoding=01                    // *00 (Disabled), 01 (Enabled)       
SR-IOV Support=01                       // *00 (Disabled), 01 (Enabled)       
Maximum Payload=01                      // *37 (Auto), 00 (128 Bytes), 01 (256 Bytes)       
Maximum Read Request=37                 // *37 (Auto), 00 (128 Bytes), 01 (256 Bytes), 02 (512 Bytes), 03 (1024 Bytes), 04 (2048 Bytes), 05 (4096 Bytes)       
ASPM Support=00                         // *00 (Disabled), 37 (Auto)       
MMIOHBase=00000002                      // *00000000 (56 TB), 00000001 (40 TB), 00000002 (24 TB), 00000003 (3 TB), 00000004 (2 TB), 00000005 (1 TB)       
MMIO High Size=00000003                 // *00000000 (256 GB), 00000001 (128 GB), 00000002 (512 GB), 00000003 (1024 GB)       
MMCFG BASE=00000000                     // *00000000 (Auto), 00000003 (0x30000000), 00000004 (0x40000000), 00000006 (0x60000000), 00000008 (0x80000000), 00000009 (0x90000000), 0000000A (0xA0000000), 0000000B (0xB0000000), 0000000C (0xC0000000), 0000000D (0xD0000000), 0000000E (0xE0000000)       
PCH SLOT1 PCI-E 2.0 X4(IN X8) OPROM=00  // 00 (Disabled), *01 (Legacy), 02 (EFI)       
CPU SLOT2 PCI-E 3.0 X4(IN X8) OPROM=00  // 00 (Disabled), *01 (Legacy), 02 (EFI)       
CPU SLOT3 PCI-E 3.0 X8 OPROM=00         // 00 (Disabled), *01 (Legacy), 02 (EFI)       
CPU SLOT4 PCI-E 3.0 X8(IN X16) OPROM=00 // 00 (Disabled), *01 (Legacy), 02 (EFI)       
CPU SLOT5 PCI-E 3.0 X8 OPROM=00         // 00 (Disabled), *01 (Legacy), 02 (EFI)       
CPU SLOT6 PCI-E 3.0 X16 OPROM=00        // 00 (Disabled), *01 (Legacy), 02 (EFI)       
CPU SLOT7 PCI-E 3.0 X8 OPROM=00         // 00 (Disabled), *01 (Legacy), 02 (EFI)       
PCI / PCIX / PCIe Slot 8 OPROM=01       // 00 (Disabled), *01 (Legacy), 02 (EFI)       
Onboard LAN OPROM Type=00               // *00 (Legacy), 01 (EFI)       
Onboard LAN1 OPROM=01                   // 00 (Disabled), *01 (PXE), 02 (iSCSI)       
Onboard LAN2 OPROM=00                   // *00 (Disabled), 01 (PXE)       
Onboard Video OPROM=01                  // 00 (Disabled), *01 (Legacy), 02 (EFI)       
VGA Priority=01                         // *01 (Onboard), 02 (Offboard)       
Network Stack=01                        // 00 (Disabled), *01 (Enabled)       
Ipv4 PXE Support=01                     // 00 (Disabled), *01 (Enabled)        Network Stack = "Enabled"
Ipv6 PXE Support=00                     // *00 (Disabled), 01 (Enabled)        Network Stack = "Enabled"

[Advanced|Super IO Configuration|Serial Port 1 Configuration]
Serial Port 1=01                        // Please enter the value in 2 hexadecimal digits. Default value is <<<01>>>       
Change Port 1 Settings=00               // *00 (Auto), 01 (IO=3F8h; IRQ=4;), 02 (IO=3F8h; IRQ=3,4,5,6,7,9,10,11,12;), 03 (IO=2F8h; IRQ=3,4,5,6,7,9,10,11,12;), 04 (IO=3E8h; IRQ=3,4,5,6,7,9,10,11,12;), 05 (IO=2E8h; IRQ=3,4,5,6,7,9,10,11,12;)       

[Advanced|Super IO Configuration|Serial Port 2 Configuration]
Serial Port 2=01                        // Please enter the value in 2 hexadecimal digits. Default value is <<<01>>>       
Change Port 2 Settings=00               // *00 (Auto), 01 (IO=2F8h; IRQ=3;), 02 (IO=3F8h; IRQ=3,4,5,6,7,9,10,11,12;), 03 (IO=2F8h; IRQ=3,4,5,6,7,9,10,11,12;), 04 (IO=3E8h; IRQ=3,4,5,6,7,9,10,11,12;), 05 (IO=2E8h; IRQ=3,4,5,6,7,9,10,11,12;)       
Serial Port 2 Attribute=03              // *03 (SOL), 01 (COM)       

[Advanced|Serial Port Console Redirection]
COM1 Console Redirection=00             // Please enter the value in 2 hexadecimal digits. Default value is <<<00>>>       
COM2/SOL Console Redirection=01         // Please enter the value in 2 hexadecimal digits. Default value is <<<01>>>       
EMS Console Redirection=00              // Please enter the value in 2 hexadecimal digits. Default value is <<<00>>>       

[Advanced|Serial Port Console Redirection|COM1 Console Redirection Settings]
COM1 Terminal Type=01                   // 00 (VT100), *01 (VT100+), 02 (VT-UTF8), 03 (ANSI)       
COM1 Bits per second=07                 // 03 (9600), 04 (19200), 05 (38400), 06 (57600), *07 (115200)       
COM1 Data Bits=08                       // 07 (7), *08 (8)       
COM1 Parity=01                          // *01 (None), 02 (Even), 03 (Odd), 04 (Mark), 05 (Space)       
COM1 Stop Bits=01                       // *01 (1), 03 (2)       
COM1 Flow Control=00                    // *00 (None), 01 (Hardware RTS/CTS)       
COM1 VT-UTF8 Combo Key Support=01       // Please enter the value in 2 hexadecimal digits. Default value is <<<01>>>       
COM1 Recorder Mode=00                   // Please enter the value in 2 hexadecimal digits. Default value is <<<00>>>       
COM1 Resolution 100x31=01               // Please enter the value in 2 hexadecimal digits. Default value is <<<01>>>       
COM1 Legacy OS Redirection Resolution=01// 00 (80x24), *01 (80x25)       
COM1 Putty KeyPad=01                    // *01 (VT100), 02 (LINUX), 04 (XTERMR6), 08 (SCO), 10 (ESCN), 20 (VT400)       
COM1 Redirection After BIOS POST=00     // *00 (Always Enable), 01 (BootLoader)       

[Advanced|Serial Port Console Redirection|COM2/SOL Console Redirection Settings]
SOL Terminal Type=01                    // 00 (VT100), *01 (VT100+), 02 (VT-UTF8), 03 (ANSI)       
SOL Bits per second=07                  // 03 (9600), 04 (19200), 05 (38400), 06 (57600), *07 (115200)       
SOL Data Bits=08                        // 07 (7), *08 (8)       
SOL Parity=01                           // *01 (None), 02 (Even), 03 (Odd), 04 (Mark), 05 (Space)       
SOL Stop Bits=01                        // *01 (1), 03 (2)       
SOL Flow Control=00                     // *00 (None), 01 (Hardware RTS/CTS)       
SOL VT-UTF8 Combo Key Support=01        // Please enter the value in 2 hexadecimal digits. Default value is <<<01>>>       
SOL Recorder Mode=00                    // Please enter the value in 2 hexadecimal digits. Default value is <<<00>>>       
SOL Resolution 100x31=01                // Please enter the value in 2 hexadecimal digits. Default value is <<<01>>>       
SOL Legacy OS Redirection Resolution=01 // 00 (80x24), *01 (80x25)       
SOL Putty KeyPad=01                     // *01 (VT100), 02 (LINUX), 04 (XTERMR6), 08 (SCO), 10 (ESCN), 20 (VT400)       
SOL Redirection After BIOS POST=00      // *00 (Always Enable), 01 (BootLoader)       

[Advanced|Serial Port Console Redirection|EMS Console Redirection Settings]
Out-of-Band Mgmt Port=00                // *00 (COM1), 01 (SOL/COM2)       
EMS Terminal Type=02                    // 00 (VT100), 01 (VT100+), *02 (VT-UTF8), 03 (ANSI)       
EMS Bits per second=07                  // 03 (9600), 04 (19200), 06 (57600), *07 (115200)       
EMS Flow Control=00                     // *00 (None), 01 (Hardware RTS/CTS), 02 (Software Xon/Xoff)       

[Advanced|ACPI Settings]
WHEA Support=01                         // 00 (Disabled), *01 (Enabled)       
High Precision Event Timer=01           // 00 (Disabled), *01 (Enabled)       
PCI AER Support=00                      // *00 (Disabled), 01 (Enabled)       

[Advanced|Trusted Computing]
Security Device Support=01              // 00 (Disabled), *01 (Enabled)       
SHA-1 PCR Bank=01                       // 00 (Disabled), *01 (Enabled)       
SHA256 PCR Bank=02                      // 00 (Disabled), *02 (Enabled)       
TPM State=00                            // *00 (Disabled), 01 (Enabled)       
Pending Operation=00                    // *00 (None), 05 (TPM Clear)       
Platform Hierarchy=01                   // 00 (Disabled), *01 (Enabled)       
Storage Hierarchy=01                    // 00 (Disabled), *01 (Enabled)       
Endorsement Hierarchy=01                // 00 (Disabled), *01 (Enabled)       
TPM2.0 UEFI Spec Version=02             // 01 (TCG_1_2), *02 (TCG_2)       
PH Randomization=00                     // *00 (Disabled), 01 (Enabled)       
Device Select=02                        // 00 (TPM 1.2), 01 (TPM 2.0), *02 (Auto)       
TXT Support=00                          // *00 (Disabled), 01 (Enabled)       

[Event Logs|Change SMBIOS Event Log Settings]
SMBIOS Event Log=01                     // 00 (Disabled), *01 (Enabled)       
Runtime Error Logging Support=01        // 00 (Disable), *01 (Enable)       
Memory Corrected Error Enabling=00      // *00 (Disable), 01 (Enable)       
Memory Corr. Error Threshold=0064       // Please enter the value in 4 hexadecimal digits. Default value is <<<0064>>>       
PCI-Ex Error Enable=00                  // *00 (No), 01 (Yes)       
Erase Event Log=00                      // *00 (No), 01 (Yes, Next reset), 02 (Yes, Every reset)       
When Log is Full=00                     // *00 (Do Nothing), 01 (Erase Immediately)       
Log System Boot Event=00                // 01 (Enabled), *00 (Disabled)       
MECI=01                                 // Please enter the value in 2 hexadecimal digits. Default value is <<<01>>>       
METW=3C                                 // Please enter the value in 2 hexadecimal digits. Default value is <<<3C>>>       

[IPMI|System Event Log]
SEL Components=01                       // 00 (Disabled), *01 (Enabled)       
Erase SEL=00                            // *00 (No), 01 (Yes, On next reset), 02 (Yes, On every reset)       
When SEL is Full=00                     // *00 (Do Nothing), 01 (Erase Immediately)       

[Boot]
Setup Prompt Timeout=0005               // Please enter the value in 4 hexadecimal digits. Default value is <<<0001>>>       
Boot Mode Select=02                     // 00 (LEGACY), 01 (UEFI), *02 (DUAL)       
HDD special boot instance=00            // *00 (Boot Option #1), 01 (Boot Option #2), 02 (Boot Option #3), 03 (Boot Option #4), 04 (Boot Option #5), 05 (Boot Option #6), 06 (Boot Option #7), 07 (Boot Option #8), 08 (Boot Option #9), 09 (Boot Option #10), 0A (Boot Option #11), 0B (Boot Option #12), 0C (Boot Option #13), 0D (Boot Option #14), 0E (Boot Option #15), 0F (Boot Option #16)       
CD/DVD special boot instance=00         // *00 (Boot Option #1), 01 (Boot Option #2), 02 (Boot Option #3), 03 (Boot Option #4), 04 (Boot Option #5), 05 (Boot Option #6), 06 (Boot Option #7), 07 (Boot Option #8), 08 (Boot Option #9), 09 (Boot Option #10), 0A (Boot Option #11), 0B (Boot Option #12), 0C (Boot Option #13), 0D (Boot Option #14), 0E (Boot Option #15), 0F (Boot Option #16)       
USB Hard Disk special boot instance=00  // *00 (Boot Option #1), 01 (Boot Option #2), 02 (Boot Option #3), 03 (Boot Option #4), 04 (Boot Option #5), 05 (Boot Option #6), 06 (Boot Option #7), 07 (Boot Option #8), 08 (Boot Option #9), 09 (Boot Option #10), 0A (Boot Option #11), 0B (Boot Option #12), 0C (Boot Option #13), 0D (Boot Option #14), 0E (Boot Option #15), 0F (Boot Option #16)       
USB CD/DVD special boot instance=00     // *00 (Boot Option #1), 01 (Boot Option #2), 02 (Boot Option #3), 03 (Boot Option #4), 04 (Boot Option #5), 05 (Boot Option #6), 06 (Boot Option #7), 07 (Boot Option #8), 08 (Boot Option #9), 09 (Boot Option #10), 0A (Boot Option #11), 0B (Boot Option #12), 0C (Boot Option #13), 0D (Boot Option #14), 0E (Boot Option #15), 0F (Boot Option #16)       
USB Key special boot instance=00        // *00 (Boot Option #1), 01 (Boot Option #2), 02 (Boot Option #3), 03 (Boot Option #4), 04 (Boot Option #5), 05 (Boot Option #6), 06 (Boot Option #7), 07 (Boot Option #8), 08 (Boot Option #9), 09 (Boot Option #10), 0A (Boot Option #11), 0B (Boot Option #12), 0C (Boot Option #13), 0D (Boot Option #14), 0E (Boot Option #15), 0F (Boot Option #16)       
USB Floppy special boot instance=00     // *00 (Boot Option #1), 01 (Boot Option #2), 02 (Boot Option #3), 03 (Boot Option #4), 04 (Boot Option #5), 05 (Boot Option #6), 06 (Boot Option #7), 07 (Boot Option #8), 08 (Boot Option #9), 09 (Boot Option #10), 0A (Boot Option #11), 0B (Boot Option #12), 0C (Boot Option #13), 0D (Boot Option #14), 0E (Boot Option #15), 0F (Boot Option #16)       
Network special boot instance=00        // *00 (Boot Option #1), 01 (Boot Option #2), 02 (Boot Option #3), 03 (Boot Option #4), 04 (Boot Option #5), 05 (Boot Option #6), 06 (Boot Option #7), 07 (Boot Option #8), 08 (Boot Option #9), 09 (Boot Option #10), 0A (Boot Option #11), 0B (Boot Option #12), 0C (Boot Option #13), 0D (Boot Option #14), 0E (Boot Option #15), 0F (Boot Option #16)       
UEFI Boot Order #1=0000                 // *0000 (Hard Disk), 0001 (CD/DVD), 0002 (USB Hard Disk), 0003 (USB CD/DVD), 0004 (USB Key), 0005 (USB Floppy), 0006 (Network), 0007 (UEFI AP), 0008 (Disabled)        Boot Mode Select = "UEFI"
UEFI Boot Order #2=0001                 // 0000 (Hard Disk), *0001 (CD/DVD), 0002 (USB Hard Disk), 0003 (USB CD/DVD), 0004 (USB Key), 0005 (USB Floppy), 0006 (Network), 0007 (UEFI AP), 0008 (Disabled)        Boot Mode Select = "UEFI"
UEFI Boot Order #3=0002                 // 0000 (Hard Disk), 0001 (CD/DVD), *0002 (USB Hard Disk), 0003 (USB CD/DVD), 0004 (USB Key), 0005 (USB Floppy), 0006 (Network), 0007 (UEFI AP), 0008 (Disabled)        Boot Mode Select = "UEFI"
UEFI Boot Order #4=0003                 // 0000 (Hard Disk), 0001 (CD/DVD), 0002 (USB Hard Disk), *0003 (USB CD/DVD), 0004 (USB Key), 0005 (USB Floppy), 0006 (Network), 0007 (UEFI AP), 0008 (Disabled)        Boot Mode Select = "UEFI"
UEFI Boot Order #5=0004                 // 0000 (Hard Disk), 0001 (CD/DVD), 0002 (USB Hard Disk), 0003 (USB CD/DVD), *0004 (USB Key), 0005 (USB Floppy), 0006 (Network), 0007 (UEFI AP), 0008 (Disabled)        Boot Mode Select = "UEFI"
UEFI Boot Order #6=0005                 // 0000 (Hard Disk), 0001 (CD/DVD), 0002 (USB Hard Disk), 0003 (USB CD/DVD), 0004 (USB Key), *0005 (USB Floppy), 0006 (Network), 0007 (UEFI AP), 0008 (Disabled)        Boot Mode Select = "UEFI"
UEFI Boot Order #7=0006                 // 0000 (Hard Disk), 0001 (CD/DVD), 0002 (USB Hard Disk), 0003 (USB CD/DVD), 0004 (USB Key), 0005 (USB Floppy), *0006 (Network), 0007 (UEFI AP), 0008 (Disabled)        Boot Mode Select = "UEFI"
UEFI Boot Order #8=0007                 // 0000 (Hard Disk), 0001 (CD/DVD), 0002 (USB Hard Disk), 0003 (USB CD/DVD), 0004 (USB Key), 0005 (USB Floppy), 0006 (Network), *0007 (UEFI AP), 0008 (Disabled)        Boot Mode Select = "UEFI"
Legacy Boot Order #1=0000               // *0000 (Hard Disk), 0001 (CD/DVD), 0002 (USB Hard Disk), 0003 (USB CD/DVD), 0004 (USB Key), 0005 (USB Floppy), 0006 (Network), 0007 (Disabled)        Boot Mode Select = "LEGACY"
Legacy Boot Order #2=0001               // 0000 (Hard Disk), *0001 (CD/DVD), 0002 (USB Hard Disk), 0003 (USB CD/DVD), 0004 (USB Key), 0005 (USB Floppy), 0006 (Network), 0007 (Disabled)        Boot Mode Select = "LEGACY"
Legacy Boot Order #3=0002               // 0000 (Hard Disk), 0001 (CD/DVD), *0002 (USB Hard Disk), 0003 (USB CD/DVD), 0004 (USB Key), 0005 (USB Floppy), 0006 (Network), 0007 (Disabled)        Boot Mode Select = "LEGACY"
Legacy Boot Order #4=0003               // 0000 (Hard Disk), 0001 (CD/DVD), 0002 (USB Hard Disk), *0003 (USB CD/DVD), 0004 (USB Key), 0005 (USB Floppy), 0006 (Network), 0007 (Disabled)        Boot Mode Select = "LEGACY"
Legacy Boot Order #5=0004               // 0000 (Hard Disk), 0001 (CD/DVD), 0002 (USB Hard Disk), 0003 (USB CD/DVD), *0004 (USB Key), 0005 (USB Floppy), 0006 (Network), 0007 (Disabled)        Boot Mode Select = "LEGACY"
Legacy Boot Order #6=0005               // 0000 (Hard Disk), 0001 (CD/DVD), 0002 (USB Hard Disk), 0003 (USB CD/DVD), 0004 (USB Key), *0005 (USB Floppy), 0006 (Network), 0007 (Disabled)        Boot Mode Select = "LEGACY"
Legacy Boot Order #7=0006               // 0000 (Hard Disk), 0001 (CD/DVD), 0002 (USB Hard Disk), 0003 (USB CD/DVD), 0004 (USB Key), 0005 (USB Floppy), *0006 (Network), 0007 (Disabled)        Boot Mode Select = "LEGACY"
Dual Boot Order #1=0008                 // 0000 (UEFI Hard Disk), 0001 (UEFI CD/DVD), 0002 (UEFI USB Hard Disk), 0003 (UEFI USB CD/DVD), 0004 (UEFI USB Key), 0005 (UEFI USB Floppy), 0006 (UEFI Network), 0007 (UEFI AP), *0008 (Hard Disk), 0009 (CD/DVD), 000A (USB Hard Disk), 000B (USB CD/DVD), 000C (USB Key), 000D (USB Floppy), 000E (Network), 000F (Disabled)        Boot Mode Select = "DUAL"
Dual Boot Order #2=0009                 // 0000 (UEFI Hard Disk), 0001 (UEFI CD/DVD), 0002 (UEFI USB Hard Disk), 0003 (UEFI USB CD/DVD), 0004 (UEFI USB Key), 0005 (UEFI USB Floppy), 0006 (UEFI Network), 0007 (UEFI AP), 0008 (Hard Disk), *0009 (CD/DVD), 000A (USB Hard Disk), 000B (USB CD/DVD), 000C (USB Key), 000D (USB Floppy), 000E (Network), 000F (Disabled)        Boot Mode Select = "DUAL"
Dual Boot Order #3=000A                 // 0000 (UEFI Hard Disk), 0001 (UEFI CD/DVD), 0002 (UEFI USB Hard Disk), 0003 (UEFI USB CD/DVD), 0004 (UEFI USB Key), 0005 (UEFI USB Floppy), 0006 (UEFI Network), 0007 (UEFI AP), 0008 (Hard Disk), 0009 (CD/DVD), *000A (USB Hard Disk), 000B (USB CD/DVD), 000C (USB Key), 000D (USB Floppy), 000E (Network), 000F (Disabled)        Boot Mode Select = "DUAL"
Dual Boot Order #4=000B                 // 0000 (UEFI Hard Disk), 0001 (UEFI CD/DVD), 0002 (UEFI USB Hard Disk), 0003 (UEFI USB CD/DVD), 0004 (UEFI USB Key), 0005 (UEFI USB Floppy), 0006 (UEFI Network), 0007 (UEFI AP), 0008 (Hard Disk), 0009 (CD/DVD), 000A (USB Hard Disk), *000B (USB CD/DVD), 000C (USB Key), 000D (USB Floppy), 000E (Network), 000F (Disabled)        Boot Mode Select = "DUAL"
Dual Boot Order #5=000C                 // 0000 (UEFI Hard Disk), 0001 (UEFI CD/DVD), 0002 (UEFI USB Hard Disk), 0003 (UEFI USB CD/DVD), 0004 (UEFI USB Key), 0005 (UEFI USB Floppy), 0006 (UEFI Network), 0007 (UEFI AP), 0008 (Hard Disk), 0009 (CD/DVD), 000A (USB Hard Disk), 000B (USB CD/DVD), *000C (USB Key), 000D (USB Floppy), 000E (Network), 000F (Disabled)        Boot Mode Select = "DUAL"
Dual Boot Order #6=000D                 // 0000 (UEFI Hard Disk), 0001 (UEFI CD/DVD), 0002 (UEFI USB Hard Disk), 0003 (UEFI USB CD/DVD), 0004 (UEFI USB Key), 0005 (UEFI USB Floppy), 0006 (UEFI Network), 0007 (UEFI AP), 0008 (Hard Disk), 0009 (CD/DVD), 000A (USB Hard Disk), 000B (USB CD/DVD), 000C (USB Key), *000D (USB Floppy), 000E (Network), 000F (Disabled)        Boot Mode Select = "DUAL"
Dual Boot Order #7=000E                 // 0000 (UEFI Hard Disk), 0001 (UEFI CD/DVD), 0002 (UEFI USB Hard Disk), 0003 (UEFI USB CD/DVD), 0004 (UEFI USB Key), 0005 (UEFI USB Floppy), 0006 (UEFI Network), 0007 (UEFI AP), 0008 (Hard Disk), 0009 (CD/DVD), 000A (USB Hard Disk), 000B (USB CD/DVD), 000C (USB Key), 000D (USB Floppy), *000E (Network), 000F (Disabled)        Boot Mode Select = "DUAL"
Dual Boot Order #8=0000                 // *0000 (UEFI Hard Disk), 0001 (UEFI CD/DVD), 0002 (UEFI USB Hard Disk), 0003 (UEFI USB CD/DVD), 0004 (UEFI USB Key), 0005 (UEFI USB Floppy), 0006 (UEFI Network), 0007 (UEFI AP), 0008 (Hard Disk), 0009 (CD/DVD), 000A (USB Hard Disk), 000B (USB CD/DVD), 000C (USB Key), 000D (USB Floppy), 000E (Network), 000F (Disabled)        Boot Mode Select = "DUAL"
Dual Boot Order #9=0001                 // 0000 (UEFI Hard Disk), *0001 (UEFI CD/DVD), 0002 (UEFI USB Hard Disk), 0003 (UEFI USB CD/DVD), 0004 (UEFI USB Key), 0005 (UEFI USB Floppy), 0006 (UEFI Network), 0007 (UEFI AP), 0008 (Hard Disk), 0009 (CD/DVD), 000A (USB Hard Disk), 000B (USB CD/DVD), 000C (USB Key), 000D (USB Floppy), 000E (Network), 000F (Disabled)        Boot Mode Select = "DUAL"
Dual Boot Order #10=0002                // 0000 (UEFI Hard Disk), 0001 (UEFI CD/DVD), *0002 (UEFI USB Hard Disk), 0003 (UEFI USB CD/DVD), 0004 (UEFI USB Key), 0005 (UEFI USB Floppy), 0006 (UEFI Network), 0007 (UEFI AP), 0008 (Hard Disk), 0009 (CD/DVD), 000A (USB Hard Disk), 000B (USB CD/DVD), 000C (USB Key), 000D (USB Floppy), 000E (Network), 000F (Disabled)        Boot Mode Select = "DUAL"
Dual Boot Order #11=0003                // 0000 (UEFI Hard Disk), 0001 (UEFI CD/DVD), 0002 (UEFI USB Hard Disk), *0003 (UEFI USB CD/DVD), 0004 (UEFI USB Key), 0005 (UEFI USB Floppy), 0006 (UEFI Network), 0007 (UEFI AP), 0008 (Hard Disk), 0009 (CD/DVD), 000A (USB Hard Disk), 000B (USB CD/DVD), 000C (USB Key), 000D (USB Floppy), 000E (Network), 000F (Disabled)        Boot Mode Select = "DUAL"
Dual Boot Order #12=0004                // 0000 (UEFI Hard Disk), 0001 (UEFI CD/DVD), 0002 (UEFI USB Hard Disk), 0003 (UEFI USB CD/DVD), *0004 (UEFI USB Key), 0005 (UEFI USB Floppy), 0006 (UEFI Network), 0007 (UEFI AP), 0008 (Hard Disk), 0009 (CD/DVD), 000A (USB Hard Disk), 000B (USB CD/DVD), 000C (USB Key), 000D (USB Floppy), 000E (Network), 000F (Disabled)        Boot Mode Select = "DUAL"
Dual Boot Order #13=0005                // 0000 (UEFI Hard Disk), 0001 (UEFI CD/DVD), 0002 (UEFI USB Hard Disk), 0003 (UEFI USB CD/DVD), 0004 (UEFI USB Key), *0005 (UEFI USB Floppy), 0006 (UEFI Network), 0007 (UEFI AP), 0008 (Hard Disk), 0009 (CD/DVD), 000A (USB Hard Disk), 000B (USB CD/DVD), 000C (USB Key), 000D (USB Floppy), 000E (Network), 000F (Disabled)        Boot Mode Select = "DUAL"
Dual Boot Order #14=0006                // 0000 (UEFI Hard Disk), 0001 (UEFI CD/DVD), 0002 (UEFI USB Hard Disk), 0003 (UEFI USB CD/DVD), 0004 (UEFI USB Key), 0005 (UEFI USB Floppy), *0006 (UEFI Network), 0007 (UEFI AP), 0008 (Hard Disk), 0009 (CD/DVD), 000A (USB Hard Disk), 000B (USB CD/DVD), 000C (USB Key), 000D (USB Floppy), 000E (Network), 000F (Disabled)        Boot Mode Select = "DUAL"
Dual Boot Order #15=0007                // 0000 (UEFI Hard Disk), 0001 (UEFI CD/DVD), 0002 (UEFI USB Hard Disk), 0003 (UEFI USB CD/DVD), 0004 (UEFI USB Key), 0005 (UEFI USB Floppy), 0006 (UEFI Network), *0007 (UEFI AP), 0008 (Hard Disk), 0009 (CD/DVD), 000A (USB Hard Disk), 000B (USB CD/DVD), 000C (USB Key), 000D (USB Floppy), 000E (Network), 000F (Disabled)        Boot Mode Select = "DUAL"

 

[Filez]

Does it have anything to do with AMD v620 cards requiring PICE 3.0 Atomics support?

 

[luckylinux]

Does it have anything to do with AMD v620 cards requiring PIC3.0 Atomics support?

Was that for me ?

Not sure to be honest. From some sparring with Google Gemini (with lots of False Positives !) it seems more of a PCIe Adressing Issue due to these 384 GB for SR-IOV Virtual Functions.

The Crazy Thing was: if I set 56TB it will POST CODE 91, if I set 1TB it will POST CODE 91, if I choose something in the Middle (24TB) it seems to work.

I'll try to install a 3rd GPU in now. Let's see if that works.

I also just ordered a Supermicro 4028GR-TRT Server for 8xGPUs + 1-2xNICs so it better work ...

 

[Filez]

Was that for me ?

Not sure to be honest. From some sparring with Google Gemini (with lots of False Positives !) it seems more of a PCIe Adressing Issue due to these 384 GB for SR-IOV Virtual Functions.

The Crazy Thing was: if I set 56TB it will POST CODE 91, if I set 1TB it will POST CODE 91, if I choose something in the Middle (24TB) it seems to work.

I'll try to install a 3rd GPU in now. Let's see if that works.

I also just ordered a Supermicro 4028GR-TRT Server for 8xGPUs + 1-2xNICs so it better work ...

Yes, it was for you. I am researching as well, if I can use these cards for inference. I only have x9 and x10 boards, X9 supermicro boards are not useful since ROCm needs PCIE 3.0 atomics for any compute work other than gaming. I was wondering if you are seeing any issues related to that.

Supermicro 4028GR-TRT - great system. Are you going to setup proxmox and then create an AMD side for inferencing and NVIDIA side for reasoning, because I wanted to try that.

 

[luckylinux]

Yes, it was for you. I am researching as well, if I can use these cards for inference. I only have x9 and x10 boards, X9 supermicro boards are not useful since ROCm needs PCIE 3.0 atomics for any compute work other than gaming. I was wondering if you are seeing any issues related to that.

Supermicro 4028GR-TRT - great system. Are you going to setup proxmox and then create an AMD side for inferencing and NVIDIA side for reasoning, because I wanted to try that.

If you have the exact Commands / Test you want to run I can try, but otherwise I really don't know anything about PCIe 3.0 Atomics, sorry .

For the Supermicro 4028GR-TRT I was planning only on LXC to be honest (avoids the whole vendor_reset out of Kernel Driver which might be otherwise needed), and only with either the AMD Radeon Instinct Mi50 16GB/32GB or the AMD Radeon Pro V620 32GB.

Alternatively, if PCIe Reset isn't an Issue, I might split it into 2 x KVM VMs, in order to basically decouple cross-CPU / QPI Traffic, since I heard that dual Socket Boards can have quite a big Performance Penality, if NUMA isn't working properly. That again depends on whether Proxmox VM can bind to a specific CPU Core Range though.

Google Gemini says that (all ?) Supermicro X10 Boards do NOT support more than 3x of these AMD Radeon Pro V620 32GB due to these 384GB Virtual Functions Adressing Issue (and the highest I can set is apparently 1024GB), but given the number of False Reports before, I wouldn't be surprised if I can still get it to work.

 

[luckylinux]

What a Nightmare .

The Supermicro X10SRL-F might not even accept 3 x AMD Radeon Pro V620 GPUs, let alone 4x.

Chassis: CoolerMaster Storm Stryker (9+1 x PCIe Slots)
Motherboard: Supermicro X10SRL-F
PSU: Corsair RM1000x

Guess what the Issues are ? The AMD Radeon Pro V620 is VERY close to the Motherboard !

• USB 2.0 Header Conflict
• SATA Port Conflict
• PCIe Riser Conflict (can't use the 2nd "Slot" hidden by the GPU)
• PCIe Riser is also a bit wider than 2 Slots, thus not an Option for "below" the Motherboard

So it's either use a 10gbe NIC or a HBA/NVMe in the first Slot.

Furthermore this required drilling the end of the SLOT3 PCIe Slot (which is only x4 Electrical)

I'm starting to Think that some Chassis like the Inter-Tech IPC 4W2 / IPC 4W40, a PCIe x16 to 4x M.2 NVMe Adapter and some M.2 to PCIe x16 Adapters. Then I could even use an AMD Ryzen B550 Motherboard there with 4 x PCIe 4.0 Slots bifurcated.

It's really annoying though, as I already had these Motherboards / Chassis, and basically can't use them like this .

 

[luckylinux]

FYI somebody just created a GitHub Repository about cross-flashing (I found it while looking for the Fan Shroud I previously used):

GitHub - Szalacinski/amdv620-to-w6800-flashing-guide: Step-by-step guide to cross-flash the AMD Radeon Pro V620 to a Radeon Pro W6800 for display output, home-lab compatibility, and fan control curves.

Step-by-step guide to cross-flash the AMD Radeon Pro V620 to a Radeon Pro W6800 for display output, home-lab compatibility, and fan control curves. - Szalacinski/amdv620-to-w6800-flashing-guide

github.com

NOT tested, so take it with a huge Grain of Salt.

 

[Filez]

FYI somebody just created a GitHub Repository about cross-flashing (I found it while looking for the Fan Shroud I previously used):

GitHub - Szalacinski/amdv620-to-w6800-flashing-guide: Step-by-step guide to cross-flash the AMD Radeon Pro V620 to a Radeon Pro W6800 for display output, home-lab compatibility, and fan control curves.

Step-by-step guide to cross-flash the AMD Radeon Pro V620 to a Radeon Pro W6800 for display output, home-lab compatibility, and fan control curves. - Szalacinski/amdv620-to-w6800-flashing-guide

github.com

NOT tested, so take it with a huge Grain of Salt.

I saw on reddit people flashed with that bios to get it working but not sure if it still requiring PCIE 3.0 atomics.
For your case issue, can these help?

Would be interesting for dual cpu boards, if one cpu can be dedicated for AMD cards and one for NVIDIA cards.

 

[luckylinux]

For your case issue, can these help?

Not really.

1st Problem: where / how do I install it in a Way that is protected ? I don't really have much free / accessible Space at all

2nd Problem: that is still putting the PCIe Device directly into the PCIe Slot. To solve my Problem it would need to be raised.

A PCIe LP to HP Adapter such as this might, though I'm not sure how to properly mechanically fasten the Device to the Case later (there will be a 40mm Gap):

https://www.aliexpress.com/item/1005009360894287.html

 

[cesarinpillin]

Hey folks question, I bought a second AMD V620 but I noticed this runs CONSIDERABLY hotter ( I bought it from somewhere else). I assume I need to re-paste. Is there any thread or guide to re-paste the V620? I'm more worried about the memory pads for sure.

Update.. I opened it up, re-pasted, etc.. then disaster.

One of the GPU vapor-chamber heat-sink screws blew up.. the head gave up.. the worst part? It was stuck inside.. Also why the heck these screws are hollow?

anyway, anyone knows what is the size of the screws that hold the vapour chamber to the gpu die?

 

[ctrl-alt-del]

I saw on reddit people flashed with that bios to get it working but not sure if it still requiring PCIE 3.0 atomics.

ROCM always requires "PCIe 3.0 atomics", while other SW interfaces like Vulkan work without it (look e.g. at Westmere/Nehalem setups with AMD, those have PCIe 2.0 only).

Atomics should be supported with any PCIe 3.0 or newer implementation, but I have no information whether any unusual PCIe switches might have problems.

 

[cesarinpillin]

Somehow saved my V620 by removing that darn stuck screw.

Anyone can confirm if the screws are M2/5mm phillips reverse flat head?

 

[Whaaat]

reverse flat head

wtf is that?

 

[Cruzader]

wtf is that?

He needs the reverse flathead to remove the screws, the regular one is just for screwing them in.

 

[cesarinpillin]

wtf is that?

Sorry, I took the Aliexpress description when writing. I need to sleep more lol.

Here is what I mean.



normal heads are like rounded from the top.
The one I need is flat with undercut (aka "reverse" in engrish?)

btw, the screws on the gpu vapour chamber V620 are HOLLOW in this one I'm trying to repaste
Not sure if they were aftermarket replaced (mine seems to have had repairs on the power connectors as there is flux residue).

As for the stuck brokenscrew.. luckily it wasn’t tightly screwed in. So I could use an electric screwdriver to slowly drive it out.

 

[luckylinux]

Sorry, I took the Aliexpress description when writing. I need to sleep more lol.

Here is what I mean.

View attachment 49926

normal heads are like rounded from the top.
The one I need is flat with undercut (aka "reverse" in engrish?)

btw, the screws on the gpu vapour chamber V620 are HOLLOW in this one I'm trying to repaste
Not sure if they were aftermarket replaced (mine seems to have had repairs on the power connectors as there is flux residue).

As for the stuck brokenscrew.. luckily it wasn’t tightly screwed in. So I could use an electric screwdriver to slowly drive it out.

With the plastic shroud I was using, I used a M3 x 8mm Screw (Button/Pan head).

But than 2.5mm (?) HEX Drive is not really very good. A Hexagonal would have been better I guess (too late to change now though).

 

[cesarinpillin]

With the plastic shroud I was using, I used a M3 x 8mm Screw (Button/Pan head).

But than 2.5mm (?) HEX Drive is not really very good. A Hexagonal would have been better I guess (too late to change now though).

Are you using the same gpu holder and the same vapour chamber heatsink? or you did also replace it?

I wonder if other video card coolers are compatible.

 

[luckylinux]

Are you using the same gpu holder and the same vapour chamber heatsink? or you did also replace it?

I wonder if other video card coolers are compatible.

I don't know.

I just removed the GPU Holder and installed this GPU Air Shroud for 120mm Fan:

AMD Instinct MI50 120mm fan adapter V1.120 by MagentL

V1.120 modelMI50 fan adapter designed with total length 65mm (without fan) . All screws is M3 10mm+-. Compatible with any 120mm fanIf you buy this ancient artifact for ML see https://github.com/mixa3607/ML-gfx906!!! 3x120 version not tested !!!!!! 2x120 with extra slot version not tested...

www.thingiverse.com

I'm currently in the Process of recreating the Design in FreeCAD and do a 3+2 Slot Configuration so that it can fit on my AMD X570 Motherboards (the Thingiverse Design is for 2+2 or 2+2+2 Slots Spacing).

I didn't touch the Vapour Chamber at all.

 

[luckylinux]

For those interested, I finally managed to do that 3-Slot Spacing 3D Model in FreeCAD which I need especially for X570 Motherboards, with the GPU Adapter (and Inspiration for this Model, though I had to remodel it 100% from scratch) taken from

AMD Instinct MI50 120mm fan adapter V1.120 by MagentL

V1.120 modelMI50 fan adapter designed with total length 65mm (without fan) . All screws is M3 10mm+-. Compatible with any 120mm fanIf you buy this ancient artifact for ML see https://github.com/mixa3607/ML-gfx906!!! 3x120 version not tested !!!!!! 2x120 with extra slot version not tested...

www.thingiverse.com

Model(s) Attached for FreeCAD 1.1.1. Rounded Model had to be splitted into 2 ZIP Files due to 1.0MiB Forum Limit.

It's my first Time really using FreeCAD . It surely doesn't look too good and there are several Things I don't like. But hopefully it's good enough and functional, though it's a bit on the heavy Side in Terms of Material used . I will know in 7-10 Days or so when I get my Order from CraftCloud.

The fact that you have to update Chamfer / Fillet Functions each Time you slightly Change something, then FreeCAD recomputes all subsequent Edge Names, makes it so that you need to manually fix all subsequent Functions by reducing Size and/or selecting the Edge that now got a new Name, thus breaking the Link in FreeCAD. Very annoying .

Furthermore, I couldn't really round the internal Edges in the Corners very well (around the Fan Mounting Holes). 2 are "OK", the other 2 look bad and will need to be sanded off manually. I couldn't really find a good Way to limit the Length of the Edge for doing the Chamfer in a good Way (or, rather, adding a small Pocket and do the Chamfer from there would have created extra Issues related to Alignment with the other Sketches). Basically on 2 Sides I "cheated" and created a 0.1mm Pocket from which I could do the Chamfer on the short-ish Edge. The 3rd Side though would have resulted in misalignment anyway if I did it, so not much difference compared to leaving it full length.

Green is "OK", Red is bad/worse, taken from the rounded Version:



If somebody else is more experienced and wants to share a better Version, that would be appreciated.