Gigabyte MJ11-EC1 EPYC 3151 Mystery

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
S

scorpion1110

New Member
Aug 11, 2024
10
1
3
what if i want to add a GPU something like a quadro p400 or something .

what other cables do i need?

overall whats the power consumption with just 2 hdds?
 
  • Like
Reactions: Reflexiony
Y

Yababakets

New Member
Sep 11, 2024
6
1
3
scorpion1110 said:
what if i want to add a GPU something like a quadro p400 or something .

what other cables do i need?

overall whats the power consumption with just 2 hdds?
Click to expand...

I attempted to connect a Quadro P600 using a CEACENT CNS41CX16W adapter (tried this adapter also) and a SFF-8654 8i PCIe 4.0 85 Ohm cable, but the GPU was not recognized.

However, when using the same adapter and cable with a MSI R5450-MD1GD3H/LP, the GPU was recognized, but there were some issues. The MSI R5450 is supposed to run at 5GT/s (PCIe 2.0), but it is only running at 2.5GT/s (PCIe 1.0).

Additionally, one member in earlier posts mentioned that the Ceacent adapter is limited to 30W, while the Quadro P600 has a 40W TDP, and the MSI R5450 has a 19W TDP.
The Ceacent adapter is powered by a mini 6-pin to 6-pin cable, which is typically rated to output 75W. Manufacturer of Ceacent adapter confirmed max 75W output.

I have reconfigured the Kernel Boot Parameters by adding pcie_aspm=off (to disable energy-saving features for PCIe) and pci=pcie_bus_perf (to prioritize performance for the PCIe subsystem) but no changes.

You can find some output I'm getting below:

Code: 
lspci

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
00:01.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
00:01.5 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7
01:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
02:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
03:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. NV2 NVMe SSD TC2200 (DRAM-less)
04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series]
07:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function
07:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor (PSP) 3.0 Device
07:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 xHCI Compliant Host Controller
08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function
08:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP
08:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
08:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller
Code: 
grep -i pcie
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.1.0-29-amd64 root=UUID=*************** ro quiet splash pcie_aspm=off pci=pcie_bus_perf
[    0.026699] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.1.0-29-amd64 root=************* ro quiet splash pcie_aspm=off pci=pcie_bus_perf
[    0.026771] PCIe ASPM is disabled
[    0.389102] pci 0000:03:00.0: 31.504 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x4 link at 0000:00:01.3 (capable of 63.012 Gb/s with 16.0 GT/s PCIe x4 link)
[    0.390534] pci 0000:06:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x8 link at 0000:00:03.1 (capable of 32.000 Gb/s with 2.5 GT/s PCIe x16 link)
[    1.618158] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x1) d8:5e:d3:14:28:c6
[    1.646525] igb 0000:05:00.0: eth1: (PCIe:2.5Gb/s:Width x1) d8:5e:d3:14:28:c7
[    4.782767] [drm] PCIE GART of 1024M enabled (table at 0x000000000014C000).
Code: 
dmidecode -t slot

# dmidecode 3.4
Getting SMBIOS data from sysfs.
SMBIOS 3.2.1 present.

Handle 0x001C, DMI type 9, 17 bytes
System Slot Information
        Designation: U2_2
        Type: x4 PCI Express
        Current Usage: In Use
        Length: Long
        ID: 1
        Characteristics:
                3.3 V is provided
                Opening is shared
                PME signal is supported
        Bus Address: 0000:03:00.1

Handle 0x001D, DMI type 9, 17 bytes
System Slot Information
        Designation: SL_SAS
        Type: x4 PCI Express
        Current Usage: In Use
        Length: Short
        ID: 2
        Characteristics:
                3.3 V is provided
                Opening is shared
                PME signal is supported
        Bus Address: 0000:01:00.2

Handle 0x001E, DMI type 9, 17 bytes
System Slot Information
        Designation: M2_M
        Type: x4 PCI Express
        Current Usage: In Use
        Length: Short
        ID: 3
        Characteristics:
                3.3 V is provided
                Opening is shared
                PME signal is supported
        Bus Address: 0000:01:00.3

Handle 0x001F, DMI type 9, 17 bytes
System Slot Information
        Designation: U2_2_PortB
        Type: x4 PCI Express
        Current Usage: In Use
        Length: Long
        ID: 4
        Characteristics:
                3.3 V is provided
                Opening is shared
                PME signal is supported
        Bus Address: 0000:03:00.2

Handle 0x0020, DMI type 9, 17 bytes
System Slot Information
        Designation: U2_1_PortA
        Type: x4 PCI Express
        Current Usage: In Use
        Length: Long
        ID: 5
        Characteristics:
                3.3 V is provided
                Opening is shared
                PME signal is supported
        Bus Address: 0000:03:00.3

Handle 0x0021, DMI type 9, 17 bytes
System Slot Information
        Designation: U2_1_PortB
        Type: x4 PCI Express
        Current Usage: In Use
        Length: Long
        ID: 6
        Characteristics:
                3.3 V is provided
                Opening is shared
                PME signal is supported
        Bus Address: 0000:03:00.4
Code: 
lspci -vvv

03:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. NV2 NVMe SSD TC2200 (DRAM-less) (prog-if 02 [NVM Express])
        Subsystem: Kingston Technology Company, Inc. NV2 NVMe SSD [TC2200] (DRAM-less)
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 32
        IOMMU group: 16
        Region 0: Memory at eea00000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 256 bytes, MaxReadReq 256 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (downgraded), Width x4
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq- OBFF Via message/WAKE#, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS+
                LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: Upstream Port
        Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
                Vector table: BAR=0 offset=00002000
                PBA: BAR=0 offset=00003000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn+ ECRCChkCap+ ECRCChkEn+
                        MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [148 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [168 v1] Physical Layer 16.0 GT/s <?>
        Capabilities: [18c v1] Lane Margining at the Receiver <?>
        Capabilities: [1a4 v1] Latency Tolerance Reporting
                Max snoop latency: 5242880ns
                Max no snoop latency: 5242880ns
        Capabilities: [1ac v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=90us PortTPowerOnTime=200us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us LTR1.2_Threshold=0ns
                L1SubCtl2: T_PwrOn=10us
        Capabilities: [1bc v1] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
        Capabilities: [2bc v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
        Capabilities: [2f4 v1] Data Link Feature <?>
        Kernel driver in use: nvme
        Kernel modules: nvme


06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] (prog-if 00 [VGA controller])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 98
        IOMMU group: 19
        Region 0: Memory at 1fff0000000 (64-bit, prefetchable) [size=256M]
        Region 2: Memory at ee720000 (64-bit, non-prefetchable) [size=128K]
        Region 4: I/O ports at c000 [size=256]
        Expansion ROM at ee700000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 256 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #1, Speed 2.5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x8 (downgraded)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee00000  Data: 0000
        Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn+ ECRCChkCap+ ECRCChkEn+
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Kernel driver in use: radeon
        Kernel modules: radeon, amdgpu
Power consumption (idle): 33W and 40W with adapter and gpu.
Build: Jonsbo N3 , 2x 90mm exhaust fans , 1x 120mm cpu fan (450rpm) , 1x Seagate Exos 12TB , 1x Kingston NV2 1TB M.2 NVMe PCIe 4.0 (SNV2S/1000G) , 2x Micron 16GB DDR4 2666Mhz ECC (MTA18ASF2G72PDZ-2G6D1) , CWT 650W Plus Gold PSU.
 
Last edited:
Y

Yababakets

New Member
Sep 11, 2024
6
1
3
scorpion1110 said:
what adapter did u ended up buying?
Click to expand...
CEACENT CNS41CX16W adapter with SFF-8654 8i PCIe 4.0 85 Ohm cable and mini 6-pin to 6-pin cable .

There are couple of things you should now:

The mini 6-pin to 6-pin cable needs to be modified to corespond with the pinout showed by the manufacturer on the product specification page.


The adapter has a resistor (I think) C21 next to the mini 6-pin plug , you will have to cut the mini 6-pin connector's extruded bottom edge off if present to avoid knocking the C21 off the board.


The 30W limit mentioned by another member was for this adapter and not for the Ceacent adapter. So I am not sure where the issue is ,maybe it doesn't support Bus Width above 64 bit but then 3 members reported success for Tesla P4 (256bit) and 1 member for Quadro T1000 (128bit).
 
Last edited:
  • Like
Reactions: etorix
F

forspeed

New Member
Jan 7, 2025
1
0
1
Hi,

I am facing the Same issue under Proxmox. The NIC freezes with the PCIe Lost Link Message after 1-2 days. I use pcie_aspm=Off but also tried the Performance setting, etc.

Did you find a solution? I have ordered parts to use the 8i for an external Card, but the internal ones would be sufficient for me... If they would Work.

Thanks,
Florian

efeschiyan said:
Hey there folks
The board is pretty decent and I've migrated my old ivy-bridge very "temporary" "quick-and-dirty-fix" NAS from an old HP SFF8300 that I built 7 years ago "until I get a proper board for storage some time next year".
The only serious issue I am getting is "enp4s0 / enp5s0: PCIe link lost"
Code: 
Oct 18 04:40:23 srv kernel: igb 0000:05:00.0 enp5s0: PCIe link lost
Oct 18 04:40:23 srv kernel: ------------[ cut here ]------------
Oct 18 04:40:23 srv kernel: igb: Failed to read reg 0xc030!
Oct 18 04:40:23 srv kernel: WARNING: CPU: 3 PID: 719144 at drivers/net/ethernet/intel/igb/igb_main.c:745 igb_rd32+0x93/0xb0 [igb]
Oct 18 04:40:23 srv kernel: Modules linked in: btrfs blake2b_generic xor raid6_pq veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter nf_tables libcrc32c softdog binfmt_misc sunrpc nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd kvm irqbypass crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 sha1_ssse3 ipmi_ssif rapl uas acpi_ipmi pcspkr ahci usb_storage ipmi_si i2c_piix4 ccp libahci ipmi_devintf ipmi_msghandler 8250_dw zfs(PO) spl(O) vhost_net vhost vhost_iotlb tap k10temp efi_pstore dmi_sysfs ip_tables x_tables autofs4 input_leds hid_generic usbkbd dm_crypt bonding tls usbhid hid xhci_pci nvme xhci_pci_renesas igb ast nvme_core xhci_hcd dca i2c_algo_bit nvme_auth mac_hid aesni_intel crypto_simd cryptd
Oct 18 04:40:23 srv kernel: CPU: 3 PID: 719144 Comm: kworker/3:2 Tainted: P           O       6.8.12-2-pve #1
Oct 18 04:40:23 srv kernel: Hardware name: GIGABYTE G431-MM0-OT/MJ11-EC1-OT, BIOS F09 09/14/2021
Oct 18 04:40:23 srv kernel: Workqueue: events igb_watchdog_task [igb]
Oct 18 04:40:23 srv kernel: RIP: 0010:igb_rd32+0x93/0xb0 [igb]
Oct 18 04:40:23 srv kernel: Code: c7 c6 03 34 72 c0 e8 0c 64 50 d2 48 8b bb 28 ff ff ff e8 a0 cf fe d1 84 c0 74 c1 44 89 e6 48 c7 c7 f8 40 72 c0 e8 cd 56 80 d1 <0f> 0b eb ae b8 ff ff ff ff 31 d2 31 f6 31 ff e9 a9 76 87 d2 66 0f
Oct 18 04:40:23 srv kernel: RSP: 0018:ffffa81e19c7bd88 EFLAGS: 00010246
Oct 18 04:40:23 srv kernel: RAX: 0000000000000000 RBX: ffff8fda49994f38 RCX: 0000000000000000
Oct 18 04:40:23 srv kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Oct 18 04:40:23 srv kernel: RBP: ffffa81e19c7bd98 R08: 0000000000000000 R09: 0000000000000000
Oct 18 04:40:23 srv kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000c030
Oct 18 04:40:23 srv kernel: R13: 0000000000000000 R14: 0000000000000000 R15: ffff8fda4b562340
Oct 18 04:40:23 srv kernel: FS:  0000000000000000(0000) GS:ffff8fe13b380000(0000) knlGS:0000000000000000
Oct 18 04:40:23 srv kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 18 04:40:23 srv kernel: CR2: 0000613aca4b31f0 CR3: 0000000195556000 CR4: 00000000003506f0
Oct 18 04:40:23 srv kernel: Call Trace:
Oct 18 04:40:23 srv kernel:  <TASK>
Oct 18 04:40:23 srv kernel:  ? show_regs+0x6d/0x80
Oct 18 04:40:23 srv kernel:  ? __warn+0x89/0x160
Oct 18 04:40:23 srv kernel:  ? igb_rd32+0x93/0xb0 [igb]
Oct 18 04:40:23 srv kernel:  ? report_bug+0x17e/0x1b0
Oct 18 04:40:23 srv kernel:  ? handle_bug+0x46/0x90
Oct 18 04:40:23 srv kernel:  ? exc_invalid_op+0x18/0x80
Oct 18 04:40:23 srv kernel:  ? asm_exc_invalid_op+0x1b/0x20
Oct 18 04:40:23 srv kernel:  ? igb_rd32+0x93/0xb0 [igb]
Oct 18 04:40:23 srv kernel:  ? igb_rd32+0x93/0xb0 [igb]
Oct 18 04:40:23 srv kernel:  igb_update_stats+0x89/0x830 [igb]
Oct 18 04:40:23 srv kernel:  igb_watchdog_task+0x134/0x8a0 [igb]
Oct 18 04:40:23 srv kernel:  process_one_work+0x16d/0x350
Oct 18 04:40:23 srv kernel:  worker_thread+0x306/0x440
Oct 18 04:40:23 srv kernel:  ? __pfx_worker_thread+0x10/0x10
Oct 18 04:40:23 srv kernel:  kthread+0xf2/0x120
Oct 18 04:40:23 srv kernel:  ? __pfx_kthread+0x10/0x10
Oct 18 04:40:23 srv kernel:  ret_from_fork+0x47/0x70
Oct 18 04:40:23 srv kernel:  ? __pfx_kthread+0x10/0x10
Oct 18 04:40:23 srv kernel:  ret_from_fork_asm+0x1b/0x30
Oct 18 04:40:23 srv kernel:  </TASK>
Oct 18 04:40:23 srv kernel: ---[ end trace 0000000000000000 ]---
Oct 18 04:40:34 srv kernel: igb 0000:05:00.0 enp5s0: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 7168 ms
Oct 18 04:40:34 srv kernel: igb 0000:05:00.0 enp5s0: Reset adapter
Oct 18 04:40:35 srv kernel: vmbr0: port 1(enp5s0) entered disabled state
That's a snippet from the latest proxmox 6.8 kernel.
My kernel cmdline is "pci=nommconf ahci.mobile_lpm_policy=0 pcie_port_pm=off pcie_aspm.policy=performance"
Has anyone else encountered dropping pcie link to the network interface? rmmod-ing "igb" and modprobing it again spews another stacktrace and I have to reboot the board to get the link running again. The link gets lost around 24-48 hours after boot and is actually the only issue I have with this board.
I haven't done any bios mods, I've only flashed the BMC firmware with the latest available here - 12.61.21.
Click to expand...
 
M

M10

New Member
Jan 21, 2022
8
1
3
finaly i've put mine in production , so far i have only pfsense on it , four 32GB sticks , one optane 16GB as boot os for proxmox, one port for lan and second for wan on pfsense, idlle consuption 22-26W with 150W pico psu . need to install second one to play with and i need bigger ssd.
 
  • Like
Reactions: etorix
I

insurgensy

New Member
Jan 2, 2024
11
10
3
insurgensy said:
This 10G card here on Ali has regular SAS-8087 connector (disregard M.2 part).
Could we just use SlimSAS-4i to SAS-8087 cable to make this card work via SlimSAS-4i or SlimSAS-8i ?
Has anyone tried this particular card type with regular SAS connector with this board?
Click to expand...
Took some time to test this, but the following results are observed:
Card DOESN'T work from SlimSAS-4i port with SlimSAS-4i to SAS-8087 cable (tested with F02 and F09 BIOS)
Card does work when inserted into M.2 slot as it was designed. It is genuine X520 with ASPM and no weird dmesg errors

Also tested cheap M.2 to PCIe x4 adapter ($5) which works with throughs lots of dmesg errors and no ASPM
 
  • Like
Reactions: Vanetta, UhClem and etorix
D

_Dejan_

Member
Aug 18, 2022
55
18
8
Im read complete thread long time ago and forget a lot of things and now re-read last few(10 pages)...

I have 2 of this boards laying in closet and 16GB sticks from old server and thinking to bring them up for testing and move some VM's to them... I plan use 4x16GB in each of them.

Im not sure if I will use 2x 960GB SATA SSD(Like Micron 5300 Pro/Max) or NVME SSD(Micron 7400 Pro U.3) on each system. NVME can get new at same price as used SATA on ebay... But NVME have a lot of higher power consumption(Idle and Max):
Micron 5300 Pro 960GB
1.5 W (Idle)
2.8 W (Avg)
3.4 W (Max)
Micron 7400 Pro 960GB U.3
4.8 W (Idle)
13.6 W (Read)
22 W (Write)

What is current status of adapters? Is there way(Flash EC0 BIOS or BIOS modded) to use 8i port as 4x4x and use some cable to connect 8i to 2x U2/U3 SSD's? Other option is use 4i To U2 adapter and connect one U3 NVME and M.2 To U2 adapter to connect another U3 NVME. Then 8i port stay if there will be requirements in future for 10G card but I lose option to use M.2 as Proxmox boot drive...

Will flashing MJ11-EC0 F2/F3 BIOS give me something usefull(like bifurcation x4 x4 on SlimSAS-8i port or better power consumption, better FAN control or enable pcie x4 on SlimSAS-4i)? Or is better to use F09 or F9 modded?

Is there some knowledge base for this board with all things on same page/post? Like how to enable additional FAN control, what adapters&cables work and some other usefull things(How decrease power consumption)...

Thanks.
 
D

_Dejan_

Member
Aug 18, 2022
55
18
8
@hmartin Are you talking about board with PLX chip or something else?

Does not EC0 bios allow bifurcation on x16 port? Does not this mean that if I set it to x4 x4 x4 x4 that it will set on 8i slot x4 x4 ?

Is maybe possible use 8i to PCIE slot board and then PCIE to dual U2 board like this:

 
hmartin

hmartin

Well-Known Member
Sep 20, 2017
355
326
63
38
_Dejan_ said:
Are you talking about board with PLX chip or something else?
Click to expand...
No, a retimer (edit: clock buffer) is not the same thing as a PLX switch.

_Dejan_ said:
Does not EC0 bios allow bifurcation on x16 port? Does not this mean that if I set it to x4 x4 x4 x4 that it will set on 8i slot x4 x4 ?
Click to expand...
Yes, the EC0 BIOS does allow bifurcation. If you configure it to x4x4x4x4 then the 8i port will be x4x4.

_Dejan_ said:
Is maybe possible use 8i to PCIE slot board and then PCIE to dual U2 board like this
Click to expand...
Yes, this kind of board should work. The retimer (edit: clock buffer) is the small chip in the middle of the board, just above the PCIe finger connector.
 
Last edited:
  • Like
Reactions: _Dejan_
D

_Dejan_

Member
Aug 18, 2022
55
18
8
Ok thanks for answers because boards with PLX are expensive and then it became much more expensive as SATA option...

What about

SFF-8654 8i to PCIE x8 -> PCIE x8 To 2xSFF-8643 card -> SFF-8643 To SFF-8639 cable -> U2 SSD

So something like this combination will work?

SFF-8654 8i to PCIE x8:

PCIE x8 To 2xSFF-8643 card:

SFF-8643 To SFF-8639 cable:

Delock Products 84823 Delock Cable SFF-8643 male > U.2 SFF-8639 female + SATA power connector 30 cm

www.delock.com www.delock.com
 
U

UhClem

just another Bozo on the bus
Jun 26, 2012
484
294
63
NH, USA
hmartin said:
... Yes, this kind of board should work. The retimer is the small chip in the middle of the board, just above the PCIe finger connector.
Click to expand...
No, that is a clock buffer (likely 1:2), which replicates the (single) REFCLK of the host PCIe port/slot to two REFCLKs, one for each downstream port. [an x16 to 4- x4 board would use a 1:4 clock buffer]

When a SlimSAS-8i connector is found on a "typical" motherboard (for PCIe/NVMe connectivity), the REFCLK replication is included [the 74-pin connector CAN accomodate 2 complete PCIe-x4 connections, OR a single x8 (both, sans power); ref: SFF-9402 Specs]. This (-EC1) board, however, appears to be atypical, lacking that REFCLK replication. This is why, for example, 2x U.2 drives can not be connected (on the 8i) directly with a simple SFF8654_8i-to-2x_SFF8639 cable, like [Link] (despite having x4x4 enabled in BIOS).
No, a retimer is not the same thing as a PLX switch.
Click to expand...
Correct, but both are "active" components, and will (almost always) have a substantial heatsink.
 
  • Like
Reactions: hmartin and etorix
L

liiil

New Member
Feb 5, 2025
1
0
1
toni3131 said:
I'm having the same issue as @ZoMbY on pages 8-9.

I've tested with Hynix 2400Mhz and Samsung 2666Mhz (2x16gb in blue slots). Connected VGA and LAN cable to IPMI.

When pressing the PSU power button the fan starts at max level and settle down after less than a minute. No VGA output on screen, no BMC info or Gigabyte logo. I can access IPMI and it shows only 1 stick of Micron RAM (2400Mhz) even if I have two sticks of different manufacturer. From the BMC console I can't poweroff the server or access the KVM console.

Does anybody have any clue what's happening ?
Click to expand...
I had same problem, and fixed that by clear the CMOS, it is a little bit tricky, you need disconnect the power, then short the Clear CMOS jumper and leave it for a while, then it should boot.
 
D

_Dejan_

Member
Aug 18, 2022
55
18
8
On german forum I see that a lot of users complain that both of integrated LAN cards with Intel I210-AT chipset do not work and stop working after some data is transferred(Some GB of data). Is this true? Im asking this because I don't have plan to use additional network card which will increase power consumption and costs... I need both one for LAN and another for WAN ...
 
You must log in or register to reply here.
Top Bottom