Gigabyte MJ11-EC1 EPYC 3151 Mystery
- Thread starter BMacD
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
That's PCIe 1.0, which I hope is not relevant - 3.0 will do almost 8Gbps and 4.0 will do close to 16Gb with an x1 connection, plenty for dual 2.5G ethernet.
it's how (2.5GT/s) PCIe switch board is wired in Gigabyte server (which is fine for GPU mining/AI config),
SlimSAS 8i port on motherboard itself reports 8GT/s x 8 lanes. (see lspci output attached to my 1st post).
All I see there is a single device that didn't negotiate a 3.0 link, everything else says it should support 3.0, and it makes very little sense for Gigabyte to cap the link version so low that 10 slots can't even use half the bandwidth of the 3.0x8 port.
Also PCIe ASPM is also disabled and I haven't noticed an option in BIOS to enable it (lspci says that it's supported but actual state shows it as disabled).
(I don't see point in spamming forum with extra repeatable descriptions).
It doesn't make sense to me either to limit downstream link speeds/lanes but that's what lspci shows
(however server spec says '10 x PCIe Gen3 x1 GPGPU cards').
Maybe there is a way to re-configure switch (it would be nice to get at least 1x at PCIe 3.0 speeds), but don't know about it.
If you suggest something to try, I can certainly test it.
All downstream slots look the same (maybe modulo 1 of them which seemed to be 2x)
07:0f.0
LnkCap: Port #15, Speed 2.5GT/s, Width x1
^^^^^^^^^^^^^ that's max possible, and it's the same on empty slots
LnkSta: Speed 2.5GT/s, Width x1
^^^^^^^^^^^^^^^^^^^^^ actual state of slot
(unless I grossly misunderstood caps meaning)
And NVME in that port:
08:00.0
LnkCap: Port #0, Speed 8GT/s, Width x4
LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)
wasn't able to convince slot to go with PCIe 3.0 speeds.
@qwerty0987654321 PCIe ASPM isn't available as option in the bios, but it set to on per default on this board. I need to deactivated because it generates a lot of error.
More on a PCIe switch, found this thread
[PATCH v9 00/14] pci: Work around ASMedia ASM2824 PCIe link training failures — Linux RDMA and InfiniBand development
so there is a hope that it will work with 6.5 kernel
[PATCH v9 00/14] pci: Work around ASMedia ASM2824 PCIe link training failures — Linux RDMA and InfiniBand development
so there is a hope that it will work with 6.5 kernel
Can also confirm function of SFF-6543 4i to 4x SATA cable like this one: https://www.amazon.de/dp/B0C712HXCJ
The LnkCap2 lines make me think 3.0 is supported:
Driver bug sounds like a reasonable explanation, you could always build a patched kernel to see how it works.
Code:
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis+still capped at 2.5, with the latest Fedora has to offer (6.7.0-0.rc0.20231106gitd2f51b3516da.9.fc40.x86_64)
Code:
07:00.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #0, Speed 2.5GT/s, Width x2, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x1
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
07:02.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #2, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x1
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
07:03.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #3, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x1
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
07:04.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #4, Speed 2.5GT/s, Width x2, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x1
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
07:06.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #6, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x1
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
07:07.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #7, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x1
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
07:08.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #8, Speed 2.5GT/s, Width x2, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x1
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
07:0a.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #10, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x1
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
07:0b.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #11, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x1
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
07:0c.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #12, Speed 2.5GT/s, Width x2, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x1
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
07:0e.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #14, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x1
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
07:0f.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode])
LnkCap: Port #15, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
LnkSta: Speed 2.5GT/s, Width x1
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
08:00.0 Non-Volatile memory controller: Silicon Motion, Inc. SM2263EN/SM2263XT (DRAM-less) NVMe SSD Controllers (rev 03) (prog-if 02 [NVM Express])
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
I'm not familiar with the connectors used on this machine, but from this thread I think I've understood the following:
My goal is to put together a small NAS with a 10 Gbps NIC. If I can use the 4i for SATA I'd have 4+4 SATA connectors and use the 8i for the NIC.
I'd be content with 2x1 Gbps with the builtin NICs but I have a few 10 Gbps cards so I might as well...
- There are two different SlimSAS/Slimeline connectors, one 4i and one 8i
- The 4i one can be used for SATA expansion with a cable like this: https://www.amazon.de/dp/B0C712HXCJ
- The 8i does not support bifurcation, but with a proper adapter supports PCIe3 x8
My goal is to put together a small NAS with a 10 Gbps NIC. If I can use the 4i for SATA I'd have 4+4 SATA connectors and use the 8i for the NIC.
I'd be content with 2x1 Gbps with the builtin NICs but I have a few 10 Gbps cards so I might as well...
Yeah that should be exactlly how you can use this Board, finding a proper SFF to PCIE 8x Card tho seemed kind of hard for me.
i found one which costs almost as much as the board itself tho i found a different cheaper way, on aliexpress there are SFF to 2x m.2 slot adapter cards, because of the non existing bifurcation the second slot will probably be useless unfortunatly but the first slot should be usable and therefore u could install one of these 10GBe m.2 network cards which is going to be my plan when the adaptor arrives from aliexpress, the onboard m.2 im gonna use as a cache drive and the 8 sata slots for prolly 5-6 harddrives and 2 sata ssds.
Tho m.2 slots are just 4 Lanes which should be fine for even dual 10 GBe NICs with PCIe 3.0, i will report when all the stuff arives
Is it possible to remove the 2.5 inch drives and the separater wall, 3d print some standoffs, cut out a standard io plate and use it with a standard atx mainboard?
If anyone got one of these, can you take a picture from slightly behind that you can see under the power distribution pcb and see the mounted drive cages / separation wall to the mainboard? That would be amazing
Thinking of getting it for a diy project and just using the case + power delivery system, but swapping out the mainboard and maybe even the pcie backplane with some oculink to pcie brakets, to allow for full x8 speeds to all the slots...
If anyone got one of these, can you take a picture from slightly behind that you can see under the power distribution pcb and see the mounted drive cages / separation wall to the mainboard? That would be amazing
Thinking of getting it for a diy project and just using the case + power delivery system, but swapping out the mainboard and maybe even the pcie backplane with some oculink to pcie brakets, to allow for full x8 speeds to all the slots...
You can do SFF-8654 8i to PCIe x8 without breaking the bank (although with shipping it comes to around 40€ which is half the cost of the board):
Breakout PCB, cable.
Not endorsing these particular sellers, just thought they were a good price. I've ordered but haven't received my items yet.
I also suspect that U2_1 can be used if you solder an SFF-8654 8i connector there (Foxconn LDL2743-24E10-9H). Unfortunately, the one seller I've found on AliExpress with that didn't ship before the deadline and the order was cancelled...
Yes it worked for me. An ssd with Server 2019 booted up and recognized the new hardware. Maybe your disk is not GPT formated.
I have instead a problem with NVME m2 not being detected by the BIOS or operating system . I have tried with 2 that are know to work.
BR
Peter
Good to know, thanks.
I was able to boot from a Samsung 980 NVMe drive in the M.2 slot without any issues. Could either be an issue with a BIOS setting or the slot itself imo.
For anyone wanting to atleast use a PCIe card with 4x connection, using the integrated m.2 slot to riser up to a normal PCIe slot does work and ive tested it with a GPU in windows, gets detected fine and also gives full resolution display output with installed drivers. I would assume that a SFF 8654 8i to 2x m.2 adaptor and than risering that up to a full size slot would also work (if at full pcie 3.0 speeds tho we will have to see), that m.2 adaptor is still on its way and i will report back if that works and with what speeds.
Its how i am planning to use this board, as a NAS with 10 GBe SFP+ uplink on that riser card)
I might end up using the integrated m.2 to riser up to a normal port and use the m.2 adaptor card with a nvme drive just so i dont have multiple risers in one chain. not the fine british way of doing smth like that but if its going to work than why not, not gonna host anything critical.
Its how i am planning to use this board, as a NAS with 10 GBe SFP+ uplink on that riser card)
I might end up using the integrated m.2 to riser up to a normal port and use the m.2 adaptor card with a nvme drive just so i dont have multiple risers in one chain. not the fine british way of doing smth like that but if its going to work than why not, not gonna host anything critical.