Gigabyte MJ11-EC1 EPYC 3151 Mystery
- Thread starter BMacD
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
As already said, the EC1 is missing some settings from the EC0 BIOS.
Warning: I advise against flashing your BIOS, it can damage your system. Crossflashing in particular is even more dangerous!
I backed up the BIOS from the EC1 and flashed the current version F02 from the EC0 via AFUWIN.
The system booted up without a problem and the missing settings are now there.
The "SL_SAS Control" option to switch the SFF-8654 4i between PCIe and SATA is there, but I did not test that.
I was interested in setting the PCIe_1, which should be the SFF-8654 8i (or as the silkscreen names it U2_2) to 4x4x4x4.
I connected the SFF-8654 8i port with an Amphenol RSL74 to a generic SFF-8654 8i to 2xM.2 X4 board, equipped with two PCIe 3.0 NVMe drives.
The system was still not able to detect the second drive, so it seems bifurcation still does not work.
In any situation just the one slot worked, and I was not able to get any drive to work in the second slot, even when only one drive was installed.
Maybe the port is really just electrically a X4 port, or the settings from the EC0 BIOS just do not work with the EC1.
Hopefully just my generic SFF-8654 8i to 2xM.2 board does not work, but I doubt that.
Warning: I advise against flashing your BIOS, it can damage your system. Crossflashing in particular is even more dangerous!
I backed up the BIOS from the EC1 and flashed the current version F02 from the EC0 via AFUWIN.
The system booted up without a problem and the missing settings are now there.
The "SL_SAS Control" option to switch the SFF-8654 4i between PCIe and SATA is there, but I did not test that.
I was interested in setting the PCIe_1, which should be the SFF-8654 8i (or as the silkscreen names it U2_2) to 4x4x4x4.
I connected the SFF-8654 8i port with an Amphenol RSL74 to a generic SFF-8654 8i to 2xM.2 X4 board, equipped with two PCIe 3.0 NVMe drives.
The system was still not able to detect the second drive, so it seems bifurcation still does not work.
In any situation just the one slot worked, and I was not able to get any drive to work in the second slot, even when only one drive was installed.
Maybe the port is really just electrically a X4 port, or the settings from the EC0 BIOS just do not work with the EC1.
Hopefully just my generic SFF-8654 8i to 2xM.2 board does not work, but I doubt that.
Last edited:
@user9482389
Do you have a SFF-8654 8i to PCIe x8/x16 Adapter? lspci -vv on Linux would show you, how many lanes are used.
@SadoKitten Is that possibly something you could check? I guess the asm2824 would show up in lspci.
dmidecode -t slot
Note U2_2 and U2_2_PortB
Do you have a SFF-8654 8i to PCIe x8/x16 Adapter? lspci -vv on Linux would show you, how many lanes are used.
@SadoKitten Is that possibly something you could check? I guess the asm2824 would show up in lspci.
dmidecode -t slot
Code:
# dmidecode 3.5
Getting SMBIOS data from sysfs.
SMBIOS 3.2.1 present.
Handle 0x001C, DMI type 9, 17 bytes
System Slot Information
Designation: U2_2
Type: x4 PCI Express
Current Usage: In Use
Length: Long
ID: 1
Characteristics:
3.3 V is provided
Opening is shared
PME signal is supported
Bus Address: 0000:03:00.1
Handle 0x001D, DMI type 9, 17 bytes
System Slot Information
Designation: SL_SAS
Type: x4 PCI Express
Current Usage: In Use
Length: Short
ID: 2
Characteristics:
3.3 V is provided
Opening is shared
PME signal is supported
Bus Address: 0000:01:00.2
Handle 0x001E, DMI type 9, 17 bytes
System Slot Information
Designation: M2_M
Type: x4 PCI Express
Current Usage: In Use
Length: Short
ID: 3
Characteristics:
3.3 V is provided
Opening is shared
PME signal is supported
Bus Address: 0000:01:00.3
Handle 0x001F, DMI type 9, 17 bytes
System Slot Information
Designation: U2_2_PortB
Type: x4 PCI Express
Current Usage: In Use
Length: Long
ID: 4
Characteristics:
3.3 V is provided
Opening is shared
PME signal is supported
Bus Address: 0000:03:00.2
Handle 0x0020, DMI type 9, 17 bytes
System Slot Information
Designation: U2_1_PortA
Type: x4 PCI Express
Current Usage: In Use
Length: Long
ID: 5
Characteristics:
3.3 V is provided
Opening is shared
PME signal is supported
Bus Address: 0000:03:00.3
Handle 0x0021, DMI type 9, 17 bytes
System Slot Information
Designation: U2_1_PortB
Type: x4 PCI Express
Current Usage: In Use
Length: Long
ID: 6
Characteristics:
3.3 V is provided
Opening is shared
PME signal is supported
Bus Address: 0000:03:00.4
The server this is from uses a PCIe switch to get 10x PCIe3x8 links.
But, to answer your question, what about a load of cheap pro-sumer NVMe drives?
PCIe 3.0 is ~1GB/sec per lane.
If I had the choice, I would ABSOLUTELY bifurcate my x16 ports into:
- x2/x2/x2/x2/x2/x2/x2/x2
or even
- x1/x1/x1/x1/x1/x1/x1/x1/x1/x1/x1/x1/x1/x1/x1/x1
There just isn't a low-cost way to do this since PCIe switching bottoms out at like $200.
And that's BEFORE the cost of the rest of the system...
Sadly I received a board without the ATX to 4 pin connector adapter so I need to source this connector on my own. I just sent a question of which type of connector/plug is used since after an hour on Digikey I couldn't find one that matched (none of the typical molex connectors seems to fit).
I really don't want to be soldering wires direct to the board even if that should be easy enough if you attach to the underside of the board.
I really don't want to be soldering wires direct to the board even if that should be easy enough if you attach to the underside of the board.
I dont't think this is possible. The 4-Pin provides standby power, the turn- on signal and something else i don'T remember off the top of my head
Hi, also ordered one of these boards from Ram-Koenig. The 69 Euro offer with ATX adapter was just too good.
Added 2x "SKhynix Huawei 32GB PC4-2400T ECC RDIMM CH121 V3 Compute Node" to the order. According to the MJ11-EC0 specs these should work, right?
@saivert @Kozhuharov
I somewhere found this description in German for the necessary ATX adapter/connectors: https://files.ram-koenig.de/information/649/power-guide_de.pdf
Added 2x "SKhynix Huawei 32GB PC4-2400T ECC RDIMM CH121 V3 Compute Node" to the order. According to the MJ11-EC0 specs these should work, right?
@saivert @Kozhuharov
I somewhere found this description in German for the necessary ATX adapter/connectors: https://files.ram-koenig.de/information/649/power-guide_de.pdf
I recieved my Board today and unfortunatly it seams like my Board is quite dead, it starts up and Management opens i also can boot into Windows Linux and what not but it only lassts for like 30 Seconds before i get either complete kernal crash in Linux or Bluescreen in Windows.
MemTest worked fine however and didnt crash but i cant really do anything in there except test ram which also tested fine.
i contacted Ram-König... lets see how they will handle this.
Edit: i've tried different Ram went from 2x 8gb Ram gaming memory to 1 stick of bog standard oem crucial ram and it seems to work fine now. i have 2x 16gb of ecc 2666 Memory on the way should arrive by tmrw and hopefully that wont give me problems aswell.
MemTest worked fine however and didnt crash but i cant really do anything in there except test ram which also tested fine.
i contacted Ram-König... lets see how they will handle this.
Edit: i've tried different Ram went from 2x 8gb Ram gaming memory to 1 stick of bog standard oem crucial ram and it seems to work fine now. i have 2x 16gb of ecc 2666 Memory on the way should arrive by tmrw and hopefully that wont give me problems aswell.
Last edited:
If anyone have trouble with SATA hotplug/hotswap on proxmox (debian) or ubuntu we have now a fix for this behavior.
Unfortunally SATA hotplug/hotswap don't want to work out of the box on this mainboard on these distros.
The reason for this behavior is the fact that in kernels like 5.15, 5.19 and 6.2 the kernel parameter CONFIG_SATA_MOBILE_LPM_POLICY is default set to 3. This parameter should decrease the energy consumption of sata controller in mobile device. Some of AMDs Epyc CPUs like the 1st and 2nd generations cannot work with this default setting.
I have do some tests with this board and for me the values 0 and 1 worked well.With 2 it also worked, but i had some wierd error messages. So i don't can recommend this value.
The energy consumption increase by 1 or 2 watt. So not so much. To get SATA hotplug/hotswap have to do the following steps.
With ZFs on your "root" storage medium:
Edit the cmdline file with
Without ZFS on "normal" filesystems like ext4 or btrfs:
Edit the grub file with
Nice to know! The parameter can have following sets of values:
Unfortunally SATA hotplug/hotswap don't want to work out of the box on this mainboard on these distros.
The reason for this behavior is the fact that in kernels like 5.15, 5.19 and 6.2 the kernel parameter CONFIG_SATA_MOBILE_LPM_POLICY is default set to 3. This parameter should decrease the energy consumption of sata controller in mobile device. Some of AMDs Epyc CPUs like the 1st and 2nd generations cannot work with this default setting.
I have do some tests with this board and for me the values 0 and 1 worked well.
The energy consumption increase by 1 or 2 watt. So not so much. To get SATA hotplug/hotswap have to do the following steps.
With ZFs on your "root" storage medium:
Edit the cmdline file with
and add
to end of the file. After this you have to update the initramfs with
On the end you have to reboot your node.
Without ZFS on "normal" filesystems like ext4 or btrfs:
Edit the grub file with
and add to this line
the following part
Now you have to update the initrramfs. With grub we have to use this command.
Like in the text above we also need a reboot of the node.
Nice to know! The parameter can have following sets of values:
| CONFIG_SATA_MOBILE_LPM_POLICY | description |
|---|---|
0 | vanilla settings from the firmware of the sata controller |
1 | maximum performance (also max. energy consumption) |
2 | medium power |
3 | medium power with device initiated power management enabled |
4 | minimum power |
I think the power socket is from the TE ELCON Micro Power series, specifically 1-2204801-8 (female) and 2204748-2 (male).
Instead of just the MB, I got whole Gigabyte G431-MM0 rig (figured I'd get all the cables with it and maybe convert it to NAS, it has 2.5x4 hotplug sata bays connected to SlimSAS 4i connector to start with and plenty of space to somehow mount 3.5 drives).
So it looks like SlimSAS 8i port has indeed support for all 8 lanes which are connected to ASM2824 switch, the later exposes 10 pci-e 1x ports and in one them I've put nvme drive (in ASUSx4 adapter) to see how switch is wired/configured.
Turns out downstream ports are capped at 2.5GTS, so switch board aren't really usable for 10G nic nor disks.
Consumption wise 'smartplug' shows the rig eats 40W at complete idle (default install Fedora 38) and 13W powered off (BMS only). (removing PCI-E switch with extra nvme saves only 3W).
It's better than DL380G9 (with 10G nic) at 85W idle but worse than DIY i3 KabyLake 2-core at 20W idle (also without 10G nic).
lspci -vv/lshw output related to SlimSAS 8i port and attached switch:
So it looks like SlimSAS 8i port has indeed support for all 8 lanes which are connected to ASM2824 switch, the later exposes 10 pci-e 1x ports and in one them I've put nvme drive (in ASUSx4 adapter) to see how switch is wired/configured.
Turns out downstream ports are capped at 2.5GTS, so switch board aren't really usable for 10G nic nor disks.
Consumption wise 'smartplug' shows the rig eats 40W at complete idle (default install Fedora 38) and 13W powered off (BMS only). (removing PCI-E switch with extra nvme saves only 3W).
It's better than DL380G9 (with 10G nic) at 85W idle but worse than DIY i3 KabyLake 2-core at 20W idle (also without 10G nic).
lspci -vv/lshw output related to SlimSAS 8i port and attached switch:
Code:
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge (prog-if 00 [Normal decode])
Subsystem: Gigabyte Technology Co., Ltd Device 1000
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin ? routed to IRQ 30
IOMMU group: 7
Bus: primary=00, secondary=06, subordinate=13, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: ee700000-ee7fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #1, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <64us
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8
TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #1, PowerLimit 0W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
RootCap: CRSVisible+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
AtomicOpsCap: Routing- 32bit+ 64bit+ 128bitCAS-
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, ARIFwd-
AtomicOpsCtl: ReqEn- EgressBlck-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee00000 Data: 0000
Capabilities: [c0] Subsystem: Gigabyte Technology Co., Ltd Device 1000
Capabilities: [c8] HyperTransport: MSI Mapping Enable+ Fixed+
Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
RootCmd: CERptEn+ NFERptEn+ FERptEn+
RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
Capabilities: [270 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [2a0 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
Capabilities: [370 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1_PM_Substates+
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
L1SubCtl2:
Capabilities: [380 v1] Downstream Port Containment
DpcCap: INT Msg #0, RPExt+ PoisonedTLP+ SwTrigger+ RP PIO Log 6, DL_ActiveErr+
DpcCtl: Trigger:1 Cmpl+ INT+ ErrCor- PoisonedTLP- SwTrigger- DL_ActiveErr-
DpcSta: Trigger- Reason:00 INT- RPBusy- TriggerExt:00 RP PIO ErrPtr:1f
Source: 0000
Capabilities: [3c4 v1] Designated Vendor-Specific: Vendor=1022 ID=0001 Rev=1 Len=44 <?>
Kernel driver in use: pcieport
06:00.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 34
IOMMU group: 19
Bus: primary=06, secondary=07, subordinate=13, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: ee700000-ee7fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [80] Express (v2) Upstream Port, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <64us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR+
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-
AtomicOpsCap: Routing-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: EgressBlck-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis+
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [c0] Subsystem: Device 0000:0000
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 14, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 05000001 0000210f 07080007 00000000
Capabilities: [140 v1] Power Budgeting <?>
Capabilities: [160 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [180 v1] Multicast
McastCap: MaxGroups 64, ECRCRegen-
McastCtl: NumGroups 1, Enable-
McastBAR: IndexPos 0, BaseAddr 0000000000000000
McastReceiveVec: 0000000000000000
McastBlockAllVec: 0000000000000000
McastBlockUntransVec: 0000000000000000
McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000
Capabilities: [1c0 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [200 v1] Vendor Specific Information: ID=001a Rev=0 Len=000 <?>
Kernel driver in use: pcieport
07:0f.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 48
IOMMU group: 31
Bus: primary=07, secondary=13, subordinate=13, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: [disabled] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D3 NoSoftRst- PME-Enable+ DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee00000 Data: 0000
Capabilities: [80] Express (v2) Downstream Port (Slot+), MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #15, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
ClockPM+ Surprise+ LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #0, PowerLimit 0W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR+
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- ARIFwd+
AtomicOpsCap: Routing-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, ARIFwd-
AtomicOpsCtl: EgressBlck-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis+, Selectable De-emphasis: -3.5dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [c0] Subsystem: Device 0000:0000
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 14, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 04000001 0000210f 13000013 00000000
Capabilities: [140 v1] Power Budgeting <?>
Capabilities: [160 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [180 v1] Multicast
McastCap: MaxGroups 64, ECRCRegen-
McastCtl: NumGroups 1, Enable-
McastBAR: IndexPos 0, BaseAddr 0000000000000000
McastReceiveVec: 0000000000000000
McastBlockAllVec: 0000000000000000
McastBlockUntransVec: 0000000000000000
McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000
Capabilities: [1c0 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [200 v1] Vendor Specific Information: ID=001a Rev=0 Len=000 <?>
Kernel driver in use: pcieport
08:00.0 Non-Volatile memory controller: Silicon Motion, Inc. SM2263EN/SM2263XT (DRAM-less) NVMe SSD Controllers (rev 03) (prog-if 02 [NVM Express])
Subsystem: Biwin Storage Technology Co., Ltd. Device 2263
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 34
IOMMU group: 20
Region 0: Memory at ee700000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable+ Count=16 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00002100
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [158 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [178 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [180 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=10us
Kernel driver in use: nvme
Kernel modules: nvme
lshw:
*-pci:4
description: PCI bridge
product: Family 17h (Models 00h-0fh) PCIe GPP Bridge
vendor: Advanced Micro Devices, Inc. [AMD]
physical id: 3.1
bus info: pci@0000:00:03.1
version: 00
width: 32 bits
clock: 33MHz
capabilities: pci pm pciexpress msi ht normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:30 memory:ee700000-ee7fffff
*-pci
description: PCI bridge
product: ASM2824 PCIe Gen3 Packet Switch
vendor: ASMedia Technology Inc.
physical id: 0
bus info: pci@0000:06:00.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pm msi pciexpress normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:34 memory:ee700000-ee7fffff
*-pci:0
description: PCI bridge
product: ASM2824 PCIe Gen3 Packet Switch
vendor: ASMedia Technology Inc.
physical id: 0
bus info: pci@0000:07:00.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pm msi pciexpress normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:35 memory:ee700000-ee7fffff
*-nvme
description: NVMe device
product: HP SSD EX900 250GB
vendor: Silicon Motion, Inc.
physical id: 0
bus info: pci@0000:08:00.0
logical name: /dev/nvme1
version: T0825A0
serial: HBSE21030300220
width: 64 bits
clock: 33MHz
capabilities: nvme pm msi pciexpress msix nvm_express bus_master cap_list
configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:1dee1deeHBSE21030300220 HP SSD EX900 250GB state=live
resources: irq:34 memory:ee700000-ee703fff
...
*-pci:1
description: PCI bridge
product: ASM2824 PCIe Gen3 Packet Switch
vendor: ASMedia Technology Inc.
physical id: 2
bus info: pci@0000:07:02.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pm msi pciexpress normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:37
*-pci:2
description: PCI bridge
product: ASM2824 PCIe Gen3 Packet Switch
vendor: ASMedia Technology Inc.
physical id: 3
bus info: pci@0000:07:03.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pm msi pciexpress normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:39
*-pci:3
description: PCI bridge
product: ASM2824 PCIe Gen3 Packet Switch
vendor: ASMedia Technology Inc.
physical id: 4
bus info: pci@0000:07:04.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pm msi pciexpress normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:40
*-pci:4
description: PCI bridge
product: ASM2824 PCIe Gen3 Packet Switch
vendor: ASMedia Technology Inc.
physical id: 6
bus info: pci@0000:07:06.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pm msi pciexpress normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:41
*-pci:5
description: PCI bridge
product: ASM2824 PCIe Gen3 Packet Switch
vendor: ASMedia Technology Inc.
physical id: 7
bus info: pci@0000:07:07.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pm msi pciexpress normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:42
*-pci:6
description: PCI bridge
product: ASM2824 PCIe Gen3 Packet Switch
vendor: ASMedia Technology Inc.
physical id: 8
bus info: pci@0000:07:08.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pm msi pciexpress normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:43
*-pci:7
description: PCI bridge
product: ASM2824 PCIe Gen3 Packet Switch
vendor: ASMedia Technology Inc.
physical id: a
bus info: pci@0000:07:0a.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pm msi pciexpress normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:44
*-pci:8
description: PCI bridge
product: ASM2824 PCIe Gen3 Packet Switch
vendor: ASMedia Technology Inc.
physical id: b
bus info: pci@0000:07:0b.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pm msi pciexpress normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:45
*-pci:9
description: PCI bridge
product: ASM2824 PCIe Gen3 Packet Switch
vendor: ASMedia Technology Inc.
physical id: c
bus info: pci@0000:07:0c.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pm msi pciexpress normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:46
*-pci:10
description: PCI bridge
product: ASM2824 PCIe Gen3 Packet Switch
vendor: ASMedia Technology Inc.
physical id: e
bus info: pci@0000:07:0e.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pm msi pciexpress normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:47
*-pci:11
description: PCI bridge
product: ASM2824 PCIe Gen3 Packet Switch
vendor: ASMedia Technology Inc.
physical id: f
bus info: pci@0000:07:0f.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pm msi pciexpress normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:48Hi, I'm new here and also bought this board, because of the ecc and ipmi support.
I mean 2.5 GT/s --> 20 GB/s would be still fast enough in my case.
My targeted setup would be:
6x 2TB SSDs
1 - 2x 2,5 GBps NICs (depends on the capabilietes of the board, like this one Link)
The M.2 slot is used by a NVMe SSD for the OS.
4 SATA slots would be used from the SSDs and the other 2 would come from the SlimSAS 4i slot (leaving 2 SATA in reserve).
For now, the 8i port would be used for the NIC.
Regarding your description, I would need the additional board (red box in the picture) or something similiar, to get access to some PCI-e slots.
I mean 2.5 GT/s --> 20 GB/s would be still fast enough in my case.
My targeted setup would be:
6x 2TB SSDs
1 - 2x 2,5 GBps NICs (depends on the capabilietes of the board, like this one Link)
The M.2 slot is used by a NVMe SSD for the OS.
4 SATA slots would be used from the SSDs and the other 2 would come from the SlimSAS 4i slot (leaving 2 SATA in reserve).
For now, the 8i port would be used for the NIC.
If it turns out this SlimSAS 8i connector cannot be bifurcated I will just use it as a x8 wide port with this adapter:
SlimSAS PCIe gen4 Device Adapter 2* 8i to x16 – C-Payne PCB Design
(just connecting one SlimSAS cable for x8)
and then using it with a 10GbE network card (most of which are x8).
Has anyone attempted to use the SlimSAS 4i for 4x SATA yet? I hope this works. Seems like there is some doubt around this.
Do I need to crosflash the BIOS for this to work or does the stock bios that comes with this EC1 variant support enabling SATA mode for this port?
SlimSAS PCIe gen4 Device Adapter 2* 8i to x16 – C-Payne PCB Design
(just connecting one SlimSAS cable for x8)
and then using it with a 10GbE network card (most of which are x8).
Has anyone attempted to use the SlimSAS 4i for 4x SATA yet? I hope this works. Seems like there is some doubt around this.
Do I need to crosflash the BIOS for this to work or does the stock bios that comes with this EC1 variant support enabling SATA mode for this port?
I'd say 2.5 GTs -> 2.5 Gbit, so just barely enough for a 2.5Gbit nic. I think I saw 1 slot on switch board to have 2x lanes advertised (but I might be mistaken).
Board marked with red is a power distribution pcb, PCIe switch is beneath metal plate (see attached picture).
to use 8i slot for 1 PCI board, you'd just need a cable + an adapter board to PCIe (a few mentioned on this thread (search on aliexpress for cheaper options)).
Also reading whole thread it seems that folks weren't able to enable bifurcation on the slot so far. So to share 8 lanes with several PCI slots, one will need a PCIe switch board. Depending on your needs the one that comes with original server potentially might serve 10 spinning drives, for ssd/nvme one 2.5GTs link is too slow (assuming one cares about performance) and all attached drives will share 8lane uplink to motherboard.
Another option could be:
SlimSAS 8i to PCIe adapter + a PCIe switch to 2 slots each 4x PCIe3 links (or more slots (it all depends on use case) which would share bandwidth of common uplink)
this should be able to serve a 10Gb nic and an HBA board for additional disks.
looks like ones I've seen on aliexpress for ~2 cheaper
another cheaper adapter with more choice of power connectors.
4i slot is connected to 4xSATA SSD backplane in original server, so it should work.
Can confirm that 4i to 4x SATA cable works out of the box with the stock BIOS