Ah, OK, I guess you moved to something more Powerful. I'm also starting setting up some AMD Ryzen (mostly 2nd Hand Parts) Systems that have like 2-4 x more CPU Power

.
NOT all PCIe Slots are populated though
Code:
-[0000:00]-+-00.0 Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers
+-01.0-[01]--+-00.0 Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
| \-00.1 Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
+-14.0 Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller
+-14.2 Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem
+-16.0 Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1
+-16.1 Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #2
+-17.0 Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode]
+-1c.0-[02]--+-00.0 Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1)
| \-00.1 Emulex Corporation ServerView iRMC HTI
+-1c.5-[03]----00.0 Intel Corporation I210 Gigabit Network Connection
+-1c.6-[04]----00.0 Intel Corporation I210 Gigabit Network Connection
+-1d.0-[05]--+-00.0 Intel Corporation I350 Gigabit Network Connection
| +-00.1 Intel Corporation I350 Gigabit Network Connection
| +-00.2 Intel Corporation I350 Gigabit Network Connection
| \-00.3 Intel Corporation I350 Gigabit Network Connection
+-1f.0 Intel Corporation C236 Chipset LPC/eSPI Controller
+-1f.2 Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller
\-1f.4 Intel Corporation 100 Series/C230 Series Chipset Family SMBus
Not exactly what I was expecting.
I guess everything
1X (like onboard NIC, Matrox VGA, etc) are on the Chipset and everything
0X is directly CPU connected. I would have expected an extra Tree Level / Branch for the Chipset though.
But
1c.# corresponds to the PCIe Root Port #, so that is at least an Exception to what I said above.
It does look a bit weird. Maybe depending on the actual "lspci" version.
I would have expected this for the PEG ports (so both x8 ports):
Code:
00:01.0 PCI bridge: Intel Corporation 6th-9th Gen Core Processor PCIe Controller
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller
With the focus on two things:
1.) The PEG ports are listed with lower functions, than the PCH ports! So, "00:01.x"=PEG vs. "00:1x.x"=PCH.
2.) It actually says "Core Processor PCIe Controller" in the description text (if available), which is the PEG.
It is not THAT clear in your example, but you can still see that your Mellanox card is connected to "00:01.0", or in other words, PEG0.
As the PEG can be bifurcated into 3 parts (1x8 and 2x4) at max, you can also have PEG1 (00:01.1) and PEG2 (00:01.2).
The unused ports just don't get displayed.
And like you already realized, the PCH ports have 2 points to identify:
1.) The PCH ports are listed with higher functions, than the PEG ports (00:1x.x)!
2.) It actually says " C230 Series Chipset Family PCI Express Root Port" in the description text, which is a PCH port.
See here for details:
Starting page 830, here is a quote for a better understanding:
Code:
12.1 PCI Express* Port Configuration Registers
Summary
There are twenty sets of the following configuration registers used for PCH PCI
Express* Port Configurations. Each PCH PCI Express* Configuration Register set covers
a single PCI Express* Port and maps out as the following Device/Function:
D28/F0 = Port1
D28/F1 = Port2
D28/F2 = Port3
D28/F3 = Port4
D28/F4 = Port5
D28/F5 = Port6
D28/F6 = Port7
D28/F7 = Port8
D29/F0 = Port9
D29/F1 = Port10
D29/F2 = Port11
D29/F3 = Port12
D29/F4 = Port13
D29/F5 = Port14
D29/F6 = Port15
D29/F7 = Port16
D27/F0 = Port17
D27/F1 = Port18
D27/F2 = Port19
D27/F3 = Port20
In your example lspci output it shows "1c.0", "1c.5", "1c.6" and finally "1d.0".
That means (with reference to the intel document above) you see:
- "1c.0" = 16 + 12, Function 0 = 28, Function 0 = Port 1 : Internal Matrox BMC graphics chip
- "1c.5" = 16 + 12, Function 5 = 28, Function 5 = Port 6 : On-board Intel i210 #1
- "1c.6" = 16 + 12, Function 6 = 28, Function 6 = Port 7 : On-board Intel i210 #2
- "1d.0" = 16 + 13, Function 0 = 29, Function 0 = Port 9 : Intel i350, as connected by you.
Yeah from the US it's quite good at Times, if I can manage to pass the Order via eBay (I usually ALWAYS get denied, maybe it worked because I sent them an Offer instead of putting the Article in my Cart, when it ALWAYS get blocked at Checkout stating that it cannot be delivered to my Country)

.
Otherwise I got 25-40 EUR / Piece Deal on the Mellanox ConnectX-4 LX from UK, which is why it would be good to use them, if I can get ASPM to work correctly

.
I am not sure, as I never owned a ConnectX-4, however from what I read (before getting the X710 in the end), the cards simply don't seem to support it. Why, no idea! But so far I did not find a single story, where someone used ConnectX-4 with active and working (!) ASPM.
The following reads somewhat ambiguous. At first it mentions working ASPM, but then says it doesn't do anything, concerning lower power consumption:
I’ve recently got myself 2 Connectx-4 Lx cards. I wanted to replace very sketchy connectx-3 cards. (OCP2.0 cards on pcie adapters which negotiate only pcie3x2 D:) I’ve found decent cards on aliexpress for 62 EUR without taxes. It’s a steal at that price. Out of the box card didn’t support...
forum.level1techs.com
Weird stuff!