Hello,
I'm using the Gigabyte server G293-Z42 with bifurcation setup.
After inserting the GPUs, I checked the PCIe link status at the device level. It was GEN4 with x4 lane (it's normal with the bifurcation setup)
Here is the lspci tree
However, at the bridge level, I see that the PCIe link is downgraded. The last time it arrived on the same server model is because of the PCIe insertion.
This time I inserted the card very carefully but it still arrives. Is there any way to confirm that it comes from the BIOS driver or hardward setup ?
Is it normal that the PCI bridge support only GEN5 x4 but four devices connected to it support GEN4 x4 each ?
From my PCIe bandwidth measurement, I see a degradation in the bandwidth and the link status at the device should be GEN3 x4 instead of GEN4).
I'm using the Gigabyte server G293-Z42 with bifurcation setup.
After inserting the GPUs, I checked the PCIe link status at the device level. It was GEN4 with x4 lane (it's normal with the bifurcation setup)
Code:
25:00.0 Processing accelerators: Device 0302 (rev 01)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 56
NUMA node: 0
Region 0: Memory at c5380000 (32-bit, non-prefetchable) [size=256K]
Region 1: Memory at c5340000 (32-bit, non-prefetchable) [size=256K]
Region 2: Memory at 6dc70000000 (64-bit, prefetchable) [size=64M]
Region 4: Memory at 6dc60000000 (64-bit, prefetchable) [size=256M]
Capabilities: [40] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 25.000W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 16GT/s (ok), Width x4 (ok)
Code:
+-[0000:40]-+-00.0
| +-00.3
| +-01.0
| +-01.4-[41-4e]----00.0-[42-4e]--+-00.0-[43-48]----00.0-[44-48]--+-10.0-[45]--
| | | +-14.0-[46]--
| | | +-18.0-[47]--
| | | \-1c.0-[48]--
| | \-04.0-[49-4e]----00.0-[4a-4e]--+-00.0-[4b]----00.0
| | +-04.0-[4c]--
| | +-08.0-[4d]--
| | \-0c.0-[4e]--
| +-02.0
| +-03.0
| +-04.0
| +-05.0
| +-07.0
| \-07.1-[4f]--+-00.0
| \-00.1
+-[0000:20]-+-00.0
| +-00.3
| +-01.0
| +-01.4-[21-2e]----00.0-[22-2e]--+-00.0-[23-28]----00.0-[24-28]--+-10.0-[25]--+-00.0
| | | | \-01.0
| | | +-14.0-[26]--+-00.0
| | | | \-01.0
| | | +-18.0-[27]--+-00.0
| | | | \-01.0
| | | \-1c.0-[28]--+-00.0
| | | \-01.0
| | \-04.0-[29-2e]----00.0-[2a-2e]--+-00.0-[2b]--
| | +-04.0-[2c]--
| | +-08.0-[2d]--
| | \-0c.0-[2e]--However, at the bridge level, I see that the PCIe link is downgraded. The last time it arrived on the same server model is because of the PCIe insertion.
This time I inserted the card very carefully but it still arrives. Is there any way to confirm that it comes from the BIOS driver or hardward setup ?
Is it normal that the PCI bridge support only GEN5 x4 but four devices connected to it support GEN4 x4 each ?
From my PCIe bandwidth measurement, I see a degradation in the bandwidth and the link status at the device should be GEN3 x4 instead of GEN4).
Code:
sudo lspci -vvs 21:00.0
21:00.0 PCI bridge: Broadcom / LSI Device c034 (rev a0) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 56
NUMA node: 0
Region 0: Memory at c4000000 (32-bit, non-prefetchable) [size=16M]
Bus: primary=21, secondary=22, subordinate=2e, sec-latency=0
I/O behind bridge: [disabled]
Memory behind bridge: c5000000-c53fffff [size=4M]
Prefetchable memory behind bridge: 000006dc00000000-000006dc73ffffff [size=1856M]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [68] Express (v2) Upstream Port, MSI 00
DevCap: MaxPayload 2048 bytes, PhantFunc 0
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 75.000W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 512 bytes, MaxReadReq 128 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <4us, L1 <32us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 32GT/s (ok), Width x4 (downgraded)
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
$ sudo lspci -vvs 41:00.0
41:00.0 PCI bridge: Broadcom / LSI Device c034 (rev a0) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 37
NUMA node: 0
Region 0: Memory at f0000000 (32-bit, non-prefetchable) [size=16M]
Bus: primary=41, secondary=42, subordinate=4e, sec-latency=0
I/O behind bridge: [disabled]
Memory behind bridge: f1000000-f27fffff [size=24M]
Prefetchable memory behind bridge: 000003c000000000-000003e041ffffff [size=132128M]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [68] Express (v2) Upstream Port, MSI 00
DevCap: MaxPayload 2048 bytes, PhantFunc 0
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 75.000W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <4us, L1 <32us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 32GT/s (ok), Width x4 (downgraded)