Discrete GPU deactivates IPMI iKVM at boot (when cable is plugged)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
E

enJOyIT

New Member
Feb 23, 2020
7
1
1
Hi,

my specs:
motherboard H11SSL-NC Rev. 2 Bios 2.0b
CPU: Epyc 7282
RAM: 32 GB ECC RAM

I want to use a discrete GPU (GTX 1660 Super) for steam in home streaming, which works fine so far!

The problem is when I boot my machine. If I leave the hdmi dummy plug connected to the GPU linux/proxmox is using my GPU for console output instead of the Aspeed 2500 VGA KVM. The result beside a iKVM black screen is, that the VM won't boot (I think it is because the host is using the GPU for itself)

If I unplug the hdmi dummy plug before boot and wait until proxmox has booted all is fine.

I have an AMD Radeon RT 5500 XT here which doesn't have this problem (there are a lot other passthrough problems with this card, so this isn't an option)

Onboard VGA is activated in BIOS of course.

iKVM works until grub is trying to boot linux.

The last booting screen:
last_screen.jpg

After that I get "No Signal".

In BIOS there is an option "AMI Graphic Output Protocol Policy":
bios.jpg

If the hdmi dummy plug is plugged it shows "Output Select". If I disconnect the plug there isn't such an option". Maybe it is connected to that?

I tried to edit grub:
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on nouveau.modeset=0 video=vesafb:off,efifb:off"
with no success

Here are my other configs:

/etc/initramfs-tools/modules:
Code: 
softdep nouveau pre: vfio vfio_pci

vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=10de:21c4,10de:1aeb,10de:1aec,10de:1aed
vfio_pci ids=10de:21c4,10de:1aeb,10de:1aec,10de:1aed
vfio_pci
/etc/modules:
Code: 
vfio
vfio_iommu_type1
vfio_pci ids=10de:21c4,10de:1aeb,10de:1aec,10de:1aed
vfio_virqfd
/etc/modprobe.d/nvidia.conf:
Code: 
softdep nouveau pre: vfio-pci
softdep nvidia pre: vfio-pci
softdep nvidia* pre: vfio-pci
/etc/modprobe.d/blacklist.conf:
Code: 
blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist mpt3sas
blacklist mpt2sas
blacklist amdgpu
blacklist i2c-nvidia-gpu
/etc/modprobe.d/kvm.conf:
Code: 
options kvm ignore_msrs=1
options kvm report_ignored_msrs=0
lspci -nnv:
Code: 
81:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:21c4] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: NVIDIA Corporation Device [10de:21c4]
        Flags: bus master, fast devsel, latency 0, IRQ 70, NUMA node 0
        Memory at f5000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 20030000000 (64-bit, prefetchable) [size=256M]
        Memory at 20040000000 (64-bit, prefetchable) [size=32M]
        I/O ports at 6000 [size=128]
        Expansion ROM at f6000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] #19
        Capabilities: [bb0] #15
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau

81:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aeb] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:21c4]
        Flags: bus master, fast devsel, latency 0, IRQ 71, NUMA node 0
        Memory at f6080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

81:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1aec] (rev a1) (prog-if 30 [XHCI])
        Subsystem: NVIDIA Corporation Device [10de:21c4]
        Flags: bus master, fast devsel, latency 0, IRQ 238, NUMA node 0
        Memory at 20042000000 (64-bit, prefetchable) [size=256K]
        Memory at 20042040000 (64-bit, prefetchable) [size=64K]
        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Power Management version 3
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: vfio-pci

81:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1aed] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:21c4]
        Flags: bus master, fast devsel, latency 0, IRQ 239, NUMA node 0
        Memory at f6084000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Power Management version 3
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: i2c_nvidia_gpu
Any ideas?

If it's the wrong forum maybe someone can move this thread to the correct one?
 
Last edited:
Blinky 42

Blinky 42

Active Member
Aug 6, 2015
615
232
43
48
PA, USA
The Intel supermicro motherboard have a jumper to force the video to Onboard/IPMI or discrete/add-in I would expect the AMD ones have a similar jumper.
 
E

enJOyIT

New Member
Feb 23, 2020
7
1
1
The jumper is set to onboard VGA by default. The strange thing is, that the amd GPU doesn't have this issue?!
 
Z

zir_blazer

Active Member
Dec 5, 2016
357
128
43
Based on my personal experiences in a slighty different case (IGP + discrete Video Card) but also used for Passthrough, your problem is that while the Firmware correctly uses as primary Video output the ASpeed VGA, after Linux boots, usually X.org reconfigures video output and prioritizes the discrete Video Card instead of the IGP, as it does not directly inherits the Firmware video output configuration.
If you're intending to passthrough the GeForce, just blacklist whatever Driver your Linux distribution uses (nouveau or the propietary nVidia ones) and see if X.org ignores it.
 
E

enJOyIT

New Member
Feb 23, 2020
7
1
1
Hi,

forgot to post these configs in my first post:

/etc/modprobe.d/nvidia.conf
Code: 
softdep nouveau pre: vfio-pci
softdep nvidia pre: vfio-pci
softdep nvidia* pre: vfio-pci
/etc/modprobe.d/blacklist.conf
Code: 
blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist mpt3sas
blacklist mpt2sas
blacklist amdgpu
blacklist i2c-nvidia-gpu
/etc/modprobe.d/kvm.conf
Code: 
options kvm ignore_msrs=1
options kvm report_ignored_msrs=0
lspci -nnv:
Code: 
81:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:21c4] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: NVIDIA Corporation Device [10de:21c4]
        Flags: bus master, fast devsel, latency 0, IRQ 70, NUMA node 0
        Memory at f5000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 20030000000 (64-bit, prefetchable) [size=256M]
        Memory at 20040000000 (64-bit, prefetchable) [size=32M]
        I/O ports at 6000 [size=128]
        Expansion ROM at f6000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] #19
        Capabilities: [bb0] #15
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau

81:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aeb] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:21c4]
        Flags: bus master, fast devsel, latency 0, IRQ 71, NUMA node 0
        Memory at f6080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

81:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1aec] (rev a1) (prog-if 30 [XHCI])
        Subsystem: NVIDIA Corporation Device [10de:21c4]
        Flags: bus master, fast devsel, latency 0, IRQ 238, NUMA node 0
        Memory at 20042000000 (64-bit, prefetchable) [size=256K]
        Memory at 20042040000 (64-bit, prefetchable) [size=64K]
        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Power Management version 3
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: vfio-pci

81:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1aed] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:21c4]
        Flags: bus master, fast devsel, latency 0, IRQ 239, NUMA node 0
        Memory at f6084000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Power Management version 3
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: i2c_nvidia_gpu
So I already blacklisted the driver

I see that I maybe forgot the "nvidiafb" to blacklist?

Will try it at the afternoon...
 
Last edited:
Z

zir_blazer

Active Member
Dec 5, 2016
357
128
43
If everything is blacklisted and X.org STILLs doesn't use the ASpeed video output, then you will have to manually create X.org configuration files for it. They look like this.
 
E

enJOyIT

New Member
Feb 23, 2020
7
1
1
Ok, blacklisting of nvidiafb doesn't work too...

So I will try zir_blazer's suggestion.

But I think I'm on the wrong path with that? Xorg is used for GUI? But I don't have/need a GUI?!
 
Last edited:
Z

zir_blazer

Active Member
Dec 5, 2016
357
128
43
enJOyIT said:
Ok, blacklisting of nvidiafb doesn't work too...

So I will try zir_blazer's suggestion.

But I think I'm on the wrong path with that? Xorg is used for GUI? But I don't have/need a GUI?!
Click to expand...
I just checked and Proxmox seems to boot directly to a console, the GUI that I saw were actually from a remote browser accessing it. So yes, it seems that it doesn't use X.org thus what I said previously doesn't apply.

Then it is an issue with KMS (Kernel Mode Setting). So far I have been googling about what to do when the Firmware and Linux Kernel doesn't agree about primary video output and I have been unable to find a useful answer. But you can check these to see if you find more appropiated keywords for googling.

How to get tty to display on another monitor (using nvidia drivers)
How do I change which monitor the TTY shows up on?
 
E

enJOyIT

New Member
Feb 23, 2020
7
1
1
So... I plugged a real monitor to the GPU.

What I see is: the booting is shown on both devices... Onboard VGA and on the seperate monitor until the last booting screen (see first post)

Then the Monitor shows linux grub bootloader and some on the next screen.:
IMG_20200225_185746.jpg

IMG_20200225_185752.jpg

This means to me that linux is using the GPU. And after grub it's going to blacklist the driver? But why? Shouldn't it be before grub?

Code: 
root@pve:~# cat /proc/fb
0 EFI VGA
1 astdrmfb
 
Last edited:
Z

zir_blazer

Active Member
Dec 5, 2016
357
128
43
Given that you have a Supermicro Motherboard, can you try to disable that card Option ROM? Check Advanced, PCIe/PCI/PnP Configuration, check for the CPU Slot 1 PCI-E 3.0 x8 OPROM (Or whatever Slot correspond to your Video Card), and put it in Disabled. If the Firmware doens't initialize a Video Card, maybe GRUB doesn't, either.
 
E

enJOyIT

New Member
Feb 23, 2020
7
1
1
I was able to get the console to my KVM by adding "fbcon=map:1" to the grub boot line...

But linux is still catching the GPU so I can't passthrough the GPU to VM.

@zir_blazer
Will try the BIOS Option

Thank you so far!
 
E

enJOyIT

New Member
Feb 23, 2020
7
1
1
Ok...

I set the GPU-Slot to disabled...

Now I can see the grub bootloader and the "Loading initial ramdisk" on the KVM! But after that the KVM shows black screen again...

But I can start my VM now without problems including passthrough...

So whats going on here? I tried to put "fbcon=map:0" to grub but this doesn't work either.

Edit: It works!!! :)

Reminder for me... just don't forget to update grub (after resetting) :-D

Thank you!!!
 
Last edited:
  • Like
Reactions: Almighty
You must log in or register to reply here.
Top Bottom