list of IOMMU groupings for X11SPH-nCTF

[MikeWebb]

Hi, I had trouble trying to get this information so I thought I would put this here incase someone else would like to know

Supermicro Super Server/X11SPH-nCTF, BIOS 2.0b 02/26/2018

This was after adding "iommu=1 intel_iommu=on" to grub

******
EDIT:
At this point my post goes south please go to post #6 for the IOMMU group list.
https://forums.servethehome.com/ind...-groupings-for-x11sph-nctf.20419/#post-193726

*******

IOMMU Group * 00:00.0 Host bridge [0600]: Intel Corporation Sky Lake-E DMI3 Registers [8086:2020] (rev 04)
00:04.0 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.1 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.2 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.3 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.4 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.5 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.6 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:04.7 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
00:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers [8086:2024] (rev 04)
00:05.2 System peripheral [0880]: Intel Corporation Device [8086:2025] (rev 04)
00:05.4 PIC [0800]: Intel Corporation Device [8086:2026] (rev 04)
00:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2014] (rev 04)
00:08.1 Performance counters [1101]: Intel Corporation Sky Lake-E Ubox Registers [8086:2015] (rev 04)
00:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2016] (rev 04)
00:11.0 Unassigned class [ff00]: Intel Corporation Device [8086:a1ec] (rev 08)
00:11.1 Unassigned class [ff00]: Intel Corporation Device [8086:a1ed] (rev 08)
00:11.5 SATA controller [0106]: Intel Corporation Lewisburg SSATA Controller [AHCI mode] [8086:a1d2] (rev 08)
00:14.0 USB controller [0c03]: Intel Corporation Lewisburg USB 3.0 xHCI Controller [8086:a1af] (rev 08)
00:14.2 Signal processing controller [1180]: Intel Corporation Lewisburg Thermal Subsystem [8086:a1b1] (rev 08)
00:16.0 Communication controller [0780]: Intel Corporation Lewisburg CSME: HECI #1 [8086:a1ba] (rev 08)
00:16.1 Communication controller [0780]: Intel Corporation Lewisburg CSME: HECI #2 [8086:a1bb] (rev 08)
00:16.4 Communication controller [0780]: Intel Corporation Lewisburg CSME: HECI #3 [8086:a1be] (rev 08)
00:17.0 SATA controller [0106]: Intel Corporation Lewisburg SATA Controller [AHCI mode] [8086:a182] (rev 08)
00:1c.0 PCI bridge [0604]: Intel Corporation Lewisburg PCI Express Root Port #1 [8086:a190] (rev f8)
00:1c.5 PCI bridge [0604]: Intel Corporation Lewisburg PCI Express Root Port #6 [8086:a195] (rev f8)
00:1f.0 ISA bridge [0601]: Intel Corporation Lewisburg LPC Controller [8086:a1c2] (rev 08)
00:1f.2 Memory controller [0580]: Intel Corporation Lewisburg PMC [8086:a1a1] (rev 08)
00:1f.4 SMBus [0c05]: Intel Corporation Lewisburg SMBus [8086:a1a3] (rev 08)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Lewisburg SPI Controller [8086:a1a4] (rev 08)
02:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 04)
03:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41)
16:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port 1A [8086:2030] (rev 04)
16:01.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port 1B [8086:2031] (rev 04)
16:02.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port 1C [8086:2032] (rev 04)
16:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
16:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
16:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
16:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:08.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:09.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:09.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
16:0e.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0e.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0f.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:0f.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
16:1d.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2054] (rev 04)
16:1d.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2055] (rev 04)
16:1d.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2056] (rev 04)
16:1d.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2057] (rev 04)
16:1e.0 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2080] (rev 04)
16:1e.1 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2081] (rev 04)
16:1e.2 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2082] (rev 04)
16:1e.3 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2083] (rev 04)
16:1e.4 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2084] (rev 04)
16:1e.5 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2085] (rev 04)
16:1e.6 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2086] (rev 04)
19:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
64:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port 1A [8086:2030] (rev 04)
64:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
64:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
64:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
64:08.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
64:09.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
64:0a.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
64:0a.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
64:0a.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
64:0a.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
64:0a.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
64:0a.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
64:0a.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
64:0a.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
64:0b.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
64:0b.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
64:0b.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
64:0b.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
64:0c.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
64:0c.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
64:0c.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
64:0c.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
64:0c.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
64:0c.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
64:0c.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
64:0c.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
64:0d.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
64:0d.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
64:0d.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
64:0d.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
65:00.0 PCI bridge [0604]: Intel Corporation Device [8086:37c0] (rev 08)
66:03.0 PCI bridge [0604]: Intel Corporation Device [8086:37c5] (rev 08)
67:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X722 for 10GBASE-T [8086:37d2] (rev 08)
67:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Connection X722 for 10GBASE-T [8086:37d2] (rev 08)
b2:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
b2:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
b2:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
b2:0e.0 Performance counters [1101]: Intel Corporation Device [8086:2058] (rev 04)
b2:0e.1 System peripheral [0880]: Intel Corporation Device [8086:2059] (rev 04)
b2:0f.0 Performance counters [1101]: Intel Corporation Device [8086:2058] (rev 04)
b2:0f.1 System peripheral [0880]: Intel Corporation Device [8086:2059] (rev 04)
b2:12.0 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204c] (rev 04)
b2:12.1 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204d] (rev 04)
b2:12.2 System peripheral [0880]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204e] (rev 04)
b2:15.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
b2:16.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
b2:16.4 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)

 

[MikeWebb]

Here is another little tidbit of interest, the DMAR section of the dmesg log. Note the X2APIC bit.

[    0.000000] ACPI: DMAR 0x000000006C9C23B0 000158 (v01 SUPERM SMCI--MB 00000001 INTL 20091013)
[    0.004000] DMAR: Host address width 46
[    0.004000] DMAR: DRHD base: 0x000000c5ffc000 flags: 0x0
[    0.004000] DMAR: dmar0: reg_base_addr c5ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.004000] DMAR: DRHD base: 0x000000e0ffc000 flags: 0x0
[    0.004000] DMAR: dmar1: reg_base_addr e0ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.004000] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    0.004000] DMAR: dmar2: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.004000] DMAR: DRHD base: 0x000000aaffc000 flags: 0x1
[    0.004000] DMAR: dmar3: reg_base_addr aaffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.004000] DMAR: RMRR base: 0x0000006f055000 end: 0x0000006f065fff
[    0.004000] DMAR: ATSR flags: 0x0
[    0.004000] DMAR: RHSA base: 0x000000aaffc000 proximity domain: 0x0
[    0.004000] DMAR: RHSA base: 0x000000c5ffc000 proximity domain: 0x0
[    0.004000] DMAR: RHSA base: 0x000000e0ffc000 proximity domain: 0x0
[    0.004000] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[    0.004000] DMAR-IR: IOAPIC id 12 under DRHD base  0xfbffc000 IOMMU 2
[    0.004000] DMAR-IR: IOAPIC id 11 under DRHD base  0xe0ffc000 IOMMU 1
[    0.004000] DMAR-IR: IOAPIC id 10 under DRHD base  0xc5ffc000 IOMMU 0
[    0.004000] DMAR-IR: IOAPIC id 8 under DRHD base  0xaaffc000 IOMMU 3
[    0.004000] DMAR-IR: IOAPIC id 9 under DRHD base  0xaaffc000 IOMMU 3
[    0.004000] DMAR-IR: HPET id 0 under DRHD base 0xaaffc000
[    0.004000] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.004000] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.004000] DMAR-IR: Enabled IRQ remapping in xapic mode

Once my testing of a newly added batch of HDD's has completed, I'll update my grub and have another look. There is a lot of banter around the net and linux kernel mailing lists regarding x2apic, BIOS and the people that program them.

 

[MikeWebb]

I don't know how I feel that there is no DMAR notice on startup abut IOMMU bing enabled. But from what I've gathered , this is jus t reporting that I have "intel_iommu=on" on my grub.cfg, so it is really no indication that iommu is actually enabled. However, I do have it in my grub config and it's not being reported, so there is that for confusion.

One of my uses cases fro this motherboard is as a napp-t or ESXi AIO server with the onboard SAS3 adaptor and sata ports (the 2 SFF-8087 ports) and the 2 sSATA ports (the two powered data ports) going to mirrored drives for ESXi host.

Unfortunately, it would appear due to the IOMMU grouping that I can only pass the on board LSI3008 SAS3 to the napp-it guest, maybe to sSATA controller as well as I think the two unregistered items in the same group are power controllers for DOM support.

 

[zir_blazer]

Your first post is wrong. It looks like a lspci, there is no grouping information at all. Either wrong copy paste, or wrong script.

intel_iommu=on is enough to get running the part of the Linux Kernel that builds the IOMMU Group info and to do PCI Passthrough with QEMU-KVM-VFIO. I'm not sure if iommu is still used or is legacy since I see iommu=pt often, but didn't looked into it to see what it actually does (Supposedly it "sets the IOMMU for Passthrough mode", but I don't have it and do Passthrough anyways with just intel_iommu=on, so...).
VMWare ESXi is a Type 1 Hypervisor that should need to implement its own low level interfaces, so your findings about what the Linux Kernel sees may not be representative of what ESXi sees. Same with Xen, which had no concept of IOMMU Groups at all back when I used it for PCI Passthrough before migrating to QEMU.

The x2APIC thing has a long history behind it. I wrote about it here. Basically, some Firmwares have a Bit set that tells an OS that checks for it to not enable x2APIC. It was a workaround added since when x2APIC debuted in the Nehalem era, it was bugged, but it was fixed YEARS AGO and there should be no reason why any modern Firmware has it enabled by default. You can either see if you can disable the X2APIC Opt Out Bit in the Firmware, or force the Linux Kernel to use x2APIC anyways with a Kernel Parameter.
You need x2APIC enabled to use APICv (APIC Virtualization) on Intel Ivy Bridge-E+ platforms (Not supported on the consumer platforms. Your Skylake-E should support it).

 

[MikeWebb]

Zir…blazer. You are correct, my first post is wrong. Oh so wrong it’s embarrassing. It would appear that I’m struggling to get IOMMU switched on with this motherboard and the FAQs on the Supermicro site don’t line up with this boards bios, I might send them an email.

I didn’t know that about ESXi, cheers

iommu=pt from my understanding helps a lot with sr-iov performance, according to the Mellanox material I’ve read. I intend to use VF with the nics.

 

[MikeWebb]

OK, found the the BIOS settings to get VT-D and IOMMU and all the magic for passthrough working with this motherboard. Helps when you down download and read the manual (the good old RTFM usually never fails)

BIO settings from page 86 of the manual

Intel® VT for Directed I/O (VT-d)Intel® VT for Directed I/O (VT-d)

Select Enable to use Intel Virtualization Technology for Direct I/O VT-d support by reporting the I/O device assignments to the VMM (Virtual Machine Monitor) through the DMAR ACPI tables. This feature offers fully-protected I/O resource sharing across Intel platforms, providing greater reliability, security and availability in networking and data-sharing. The options are Disable and Enable.

Interrupt Remapping
Use this feature to enable Interrupt Remapping support, which detects and controls external interrupt requests. The options are Enable and Disable.

PassThrough DMA
Use this feature to allow devices such as network cards to access the system memory without using a processor. Select Enable to use the Non-Isoch VT_D Engine Pass Through Direct Memory Access (DMA) support. The options are Enable and Disable.

ATS
Use this feature to enable Non-Isoch VT-d Engine Address Translation Services (ATS) support. ATS translates virtual addresses to physical addresses. The options are En- able and Disable.

Posted Interrupt
Use this feature to enable VT_D Posted Interrupt. The options are Enable and Disable.

Coherency Support (Non-Isoch)
Use this feature to maintain setting coherency between processors or other devices. Select Enable for the Non-Isoch VT-d engine to pass through DMA to enhance system performance. The options are Enable and Disable.

do the usual intel_iommu=on in grub, reboot and admire the groupings

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:2020] (rev 04)
IOMMU Group 10 00:1f.0 ISA bridge [0601]: Intel Corporation Lewisburg LPC Controller [8086:a1c2] (rev 08)
IOMMU Group 10 00:1f.2 Memory controller [0580]: Intel Corporation Lewisburg PMC [8086:a1a1] (rev 08)
IOMMU Group 10 00:1f.4 SMBus [0c05]: Intel Corporation Lewisburg SMBus [8086:a1a3] (rev 08)
IOMMU Group 10 00:1f.5 Serial bus controller [0c80]: Intel Corporation Lewisburg SPI Controller [8086:a1a4] (rev 08)
IOMMU Group 1 00:04.0 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 1 00:04.1 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 1 00:04.2 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 1 00:04.3 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 1 00:04.4 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 1 00:04.5 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 1 00:04.6 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 1 00:04.7 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 11 02:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 04)
IOMMU Group 11 03:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41)
IOMMU Group 12 16:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port 1A [8086:2030] (rev 04)
IOMMU Group 13 16:01.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port 1B [8086:2031] (rev 04)
IOMMU Group 14 16:02.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port 1C [8086:2032] (rev 04)
IOMMU Group 15 16:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
IOMMU Group 15 16:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
IOMMU Group 15 16:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
IOMMU Group 16 16:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 16 16:08.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 16 16:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 16 16:08.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 16 16:08.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 16 16:08.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 16 16:08.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 16 16:08.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 17 16:09.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 17 16:09.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 18 16:0e.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 18 16:0e.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 18 16:0e.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 18 16:0e.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 18 16:0e.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 18 16:0e.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 18 16:0e.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 18 16:0e.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 19 16:0f.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 19 16:0f.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 2 00:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers [8086:2024] (rev 04)
IOMMU Group 2 00:05.2 System peripheral [0880]: Intel Corporation Device [8086:2025] (rev 04)
IOMMU Group 2 00:05.4 PIC [0800]: Intel Corporation Device [8086:2026] (rev 04)
IOMMU Group 20 16:1d.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2054] (rev 04)
IOMMU Group 20 16:1d.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2055] (rev 04)
IOMMU Group 20 16:1d.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2056] (rev 04)
IOMMU Group 20 16:1d.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2057] (rev 04)
IOMMU Group 21 16:1e.0 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2080] (rev 04)
IOMMU Group 21 16:1e.1 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2081] (rev 04)
IOMMU Group 21 16:1e.2 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2082] (rev 04)
IOMMU Group 21 16:1e.3 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2083] (rev 04)
IOMMU Group 21 16:1e.4 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2084] (rev 04)
IOMMU Group 21 16:1e.5 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2085] (rev 04)
IOMMU Group 21 16:1e.6 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2086] (rev 04)
IOMMU Group 22 19:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
IOMMU Group 23 64:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port 1A [8086:2030] (rev 04)
IOMMU Group 24 64:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
IOMMU Group 24 64:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
IOMMU Group 24 64:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
IOMMU Group 25 64:08.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
IOMMU Group 26 64:09.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
IOMMU Group 27 64:0a.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
IOMMU Group 27 64:0a.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
IOMMU Group 27 64:0a.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
IOMMU Group 27 64:0a.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
IOMMU Group 27 64:0a.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
IOMMU Group 27 64:0a.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
IOMMU Group 27 64:0a.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
IOMMU Group 27 64:0a.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
IOMMU Group 28 64:0b.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
IOMMU Group 28 64:0b.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
IOMMU Group 28 64:0b.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
IOMMU Group 28 64:0b.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
IOMMU Group 29 64:0c.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
IOMMU Group 29 64:0c.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
IOMMU Group 29 64:0c.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
IOMMU Group 29 64:0c.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
IOMMU Group 29 64:0c.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
IOMMU Group 29 64:0c.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
IOMMU Group 29 64:0c.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
IOMMU Group 29 64:0c.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
IOMMU Group 3 00:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2014] (rev 04)
IOMMU Group 3 00:08.1 Performance counters [1101]: Intel Corporation Sky Lake-E Ubox Registers [8086:2015] (rev 04)
IOMMU Group 3 00:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2016] (rev 04)
IOMMU Group 30 64:0d.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
IOMMU Group 30 64:0d.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
IOMMU Group 30 64:0d.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
IOMMU Group 30 64:0d.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
IOMMU Group 31 65:00.0 PCI bridge [0604]: Intel Corporation Device [8086:37c0] (rev 08)
IOMMU Group 32 66:03.0 PCI bridge [0604]: Intel Corporation Device [8086:37c5] (rev 08)
IOMMU Group 33 67:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:37d2] (rev 08)
IOMMU Group 34 67:00.1 Ethernet controller [0200]: Intel Corporation Device [8086:37d2] (rev 08)
IOMMU Group 35 b2:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port 1A [8086:2030] (rev 04)
IOMMU Group 36 b2:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
IOMMU Group 36 b2:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
IOMMU Group 36 b2:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
IOMMU Group 37 b2:0e.0 Performance counters [1101]: Intel Corporation Device [8086:2058] (rev 04)
IOMMU Group 37 b2:0e.1 System peripheral [0880]: Intel Corporation Device [8086:2059] (rev 04)
IOMMU Group 38 b2:0f.0 Performance counters [1101]: Intel Corporation Device [8086:2058] (rev 04)
IOMMU Group 38 b2:0f.1 System peripheral [0880]: Intel Corporation Device [8086:2059] (rev 04)
IOMMU Group 39 b2:12.0 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204c] (rev 04)
IOMMU Group 39 b2:12.1 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204d] (rev 04)
IOMMU Group 39 b2:12.2 System peripheral [0880]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204e] (rev 04)
IOMMU Group 4 00:11.0 Unassigned class [ff00]: Intel Corporation Device [8086:a1ec] (rev 08)
IOMMU Group 4 00:11.1 Unassigned class [ff00]: Intel Corporation Device [8086:a1ed] (rev 08)
IOMMU Group 4 00:11.5 SATA controller [0106]: Intel Corporation Lewisburg SSATA Controller [AHCI mode] [8086:a1d2] (rev 08)
IOMMU Group 40 b2:15.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
IOMMU Group 41 b2:16.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
IOMMU Group 41 b2:16.4 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
IOMMU Group 42 b3:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:67df] (rev c7)
IOMMU Group 42 b3:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf0]
IOMMU Group 5 00:14.0 USB controller [0c03]: Intel Corporation Lewisburg USB 3.0 xHCI Controller [8086:a1af] (rev 08)
IOMMU Group 5 00:14.2 Signal processing controller [1180]: Intel Corporation Device [8086:a1b1] (rev 08)
IOMMU Group 6 00:16.0 Communication controller [0780]: Intel Corporation Lewisburg CSME: HECI #1 [8086:a1ba] (rev 08)
IOMMU Group 6 00:16.1 Communication controller [0780]: Intel Corporation Lewisburg CSME: HECI #2 [8086:a1bb] (rev 08)
IOMMU Group 6 00:16.4 Communication controller [0780]: Intel Corporation Lewisburg CSME: HECI #3 [8086:a1be] (rev 08)
IOMMU Group 7 00:17.0 SATA controller [0106]: Intel Corporation Lewisburg SATA Controller [AHCI mode] [8086:a182] (rev 08)
IOMMU Group 8 00:1c.0 PCI bridge [0604]: Intel Corporation Lewisburg PCI Express Root Port #1 [8086:a190] (rev f8)
IOMMU Group 9 00:1c.5 PCI bridge [0604]: Intel Corporation Lewisburg PCI Express Root Port #6 [8086:a195] (rev f8)

 

[zir_blazer]

What about x2APIC and APICv? Did you got them working?

 

[MikeWebb]

When I had ubuntu installed on it I had to add an entry to my grub, I can't remember what exactly but the error tells you what you to add)
I have proxmox 5.2 installed now.

Here is the dmesg output

[    0.000000] x2apic: enabled by BIOS, switching to x2apic ops

[    0.000000] Setting APIC routing to cluster x2apic.

[    0.000000] ACPI: X2APIC_NMI (uid[0xffffffff] high level lint[0x1])

[    0.004000] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.

[    0.004000] DMAR-IR: Enabled IRQ remapping in x2apic mode

 

[zir_blazer]

When I had ubuntu installed on it I had to add an entry to my grub, I can't remember what exactly but the error tells you what you to add)
I have proxmox 5.2 installed now.

Here is the dmesg output

[    0.000000] x2apic: enabled by BIOS, switching to x2apic ops

[    0.000000] Setting APIC routing to cluster x2apic.

[    0.000000] ACPI: X2APIC_NMI (uid[0xffffffff] high level lint[0x1])

[    0.004000] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.

[    0.004000] DMAR-IR: Enabled IRQ remapping in x2apic mode

The x2apic: enabled by BIOS seems better than your previous
DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit. Did you found the X2APIC Opt Out Bit I said early? Since it seems that now the Firmware isn't telling the Linux Kernel to not use x2APIC, that is why it seems that you don't need to override. So it seems that it works now out of the box.

Does...
cat /sys/module/kvm_intel/parameters/enable_apicv
...gives a Y or N?

 

[MikeWebb]

root@pve:~# cat /sys/module/kvm_intel/parameters/enable_apicv
Y

 

[MikeWebb]

When I get a bit of time I'll do a DIY server build post.

I've dropped ESXi as I could only get ver.6.7 as an evaluation and all the tricks to get sata passthrough for the Lewisburg controller no longer work. I've passed both the SATA (not the sSATA) and the LSI controllers through to a FreeNAS guest and have some test shares working. I'm working on GPU passthrough to a win10 guest ATM (for steam streaming as some games aren't available for linux)