SFP+ cards with ASPM support?

FucaBala · Sep 20, 2023

Guys, I think this will be my final update.
I tested 2 cards, model MC4121A-ACAT Huawey brand.

I used 4 different PCs:

12600k on z690 aorus elite
5950X on X570 Aorus Master
9500 on HP Prodesk
10500 on HP Elite Desk

I used the most updated firmware/BIOS avaiable.

Tried to play around with the paramters above and there was absolutelly no change at all.

I tried a fresh install of Ubuntu, without Mellanox tools and with the tools. No change in behaviour P States.

The only findings I can share:
AUTO_POWER_SAVE_LINK_DOWN_P0=1 and AUTO_POWER_SAVE_LINK_DOWN_P0=1 does not change the P state, but reduced power usage in roughly 4 watts. This was consistent.

Funny thing is that when connecting the card on a x8 slot, the PState drops to a minimum of 3
When putting the card on a x4 slot, the Pstate increased to 3, saving another 4 watts (but you will cut the bandwith and limit the card)

No matter what I have done I could never make the system going to a PState higher than 3.

My last attempt would be to try flashing the Mellanox firmware insted of the Huawey one, but this appears to be tricky and I don;t know how to do it.

Hope these could help.

FucaBala · Sep 22, 2023

Hello folks.
Some something to share:

On my machine I have 2 NICs, a 1GB Realtek and a Connectx-4, so I thought about comparing the lspci of both cards.

The realtek one allows my PC to reach Pstate8.
Mellanox only Pstate2.

Below is the full lspci of each card.

Realtek

Bash:

admin@truenas[~]$ sudo lspci -s 07:00.0 -vvv
07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
        Subsystem: Gigabyte Technology Co., Ltd RTL8125 2.5GbE Controller
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 18
        IOMMU group: 23
        Region 0: I/O ports at 4000 [size=256]
        Region 2: Memory at 43800000 (64-bit, non-prefetchable) [size=64K]
        Region 4: Memory at 43810000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 01
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 256 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Via message/WAKE#, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp+ ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00000800
        Capabilities: [d0] Vital Product Data
pcilib: sysfs_read_vpd: read failed: No such device
                Not readable
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [148 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
        Capabilities: [178 v1] Transaction Processing Hints
                No steering table available
        Capabilities: [204 v1] Latency Tolerance Reporting
                Max snoop latency: 3145728ns
                Max no snoop latency: 3145728ns
        Capabilities: [20c v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=150us PortTPowerOnTime=150us
                L1SubCtl1: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
                           T_CommonMode=0us LTR1.2_Threshold=306176ns
                L1SubCtl2: T_PwrOn=150us
        Capabilities: [21c v1] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
        Kernel driver in use: r8169
        Kernel modules: r8169

Mellanox Connectx-4

Bash:

admin@truenas[~]$ sudo lspci -s 01:00.0 -vvv           
01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
        Subsystem: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        IOMMU group: 18
        Region 0: Memory at 5c000000 (64-bit, prefetchable) [size=32M]
        Expansion ROM at 43500000 [disabled] [size=1M]
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x8
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABC, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [48] Vital Product Data
                Product Name: CX4121A - ConnectX-4 LX SFP28
                Read-only fields:
                        [PN] Part number: MCX4121A-ACAT_C12   
                        [EC] Engineering changes: A1
                        [V0] Vendor specific: PCIeGen3 x8       
                        [RV] Reserved: checksum good, 0 byte(s) reserved
                End
        Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
                Vector table: BAR=0 offset=00002000
                PBA: BAR=0 offset=00003000
        Capabilities: [c0] Vendor Specific Information: Len=18 <?>
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 04, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 1
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration- 10BitTagReq- Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
                IOVSta: Migration-
                Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 2, stride: 1, Device ID: 1016
                Supported Page Size: 000007ff, System Page Size: 00000001
                Region 0: Memory at 0000000060000000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [1c0 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [230 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core

In summary, both cards support ASPM L1

Realtek:
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

Mellanox :
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

The only difference I found is that the Realtek has this configuration below and the Mellanox has nothing similar:
Realtek card

Bash:

        Capabilities: [20c v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=150us PortTPowerOnTime=150us
                L1SubCtl1: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
                           T_CommonMode=0us LTR1.2_Threshold=306176ns

Does anyone has any idea of how to possibly try to force those and see what happens ?

SnJ9MX · Sep 25, 2023

Somewhat on-topic: do RJ45 (10GbE) cards suffer from the same issues? People always say they take more power but if they allow a lower idle state, it may be a wash.

FucaBala · Sep 26, 2023

Thats something I would like to know as well, but never had a 10Gb.

Does anyone which owns a Intel or Aquantia could provide a quick update of which P state your build goes?

SnJ9MX said:
Somewhat on-topic: do RJ45 (10GbE) cards suffer from the same issues? People always say they take more power but if they allow a lower idle state, it may be a wash.

Fnurrop · Sep 30, 2023

It seems that my Solarflare card managed to enable ASPM and still work.

The cpu package is still only at C2 and C3 though.

After UEFI config was updated I used the enable-aspm script: http://ftp.dei.uc.pt/pub/linux/kernel/people/mcgrof/aspm/enable-aspm

Code:

$ sudo lspci -s 01:00.0 -vvv
01:00.0 Ethernet controller: Solarflare Communications SFC9020 10G Ethernet Controller
    Subsystem: Solarflare Communications SFN5122F-R7 SFP+ Server Adapter
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 16
    Region 0: I/O ports at 5100 [size=256]
    Region 2: Memory at a2000000 (64-bit, non-prefetchable) [size=16M]
    Region 4: Memory at a3090000 (64-bit, non-prefetchable) [size=64K]
    Expansion ROM at a3040000 [disabled] [size=256K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Address: 0000000000000000  Data: 0000
        Masking: 00000000  Pending: 00000000
    Capabilities: [70] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <8us
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W
        DevCtl:    CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
        LnkCap:    Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
        LnkCtl:    ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 5GT/s, Width x8
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
        Vector table: BAR=4 offset=00000000
        PBA: BAR=4 offset=00008000
    Capabilities: [d0] Vital Product Data
        Product Name: Solarstorm SFN5122F SFP+ Server Adapter
        Read-only fields:
            [PN] Part number: SFN5122F

or hmm, I do spot some issues in dmesg. Speedtests still show 10G working as normal however.

Code:

[ 3366.922075] pcieport 0000:00:01.0: AER: Corrected error received: 0000:01:00.0
[ 3366.922122] sfc_siena 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
[ 3366.922128] sfc_siena 0000:01:00.0:   device [1924:0803] error status/mask=00001000/00002000
[ 3366.922135] sfc_siena 0000:01:00.0:    [12] Timeout              
[ 3376.050244] r8169 0000:04:00.0 right: Link is Down
[ 3454.145486] pcieport 0000:00:01.0: AER: Corrected error received: 0000:01:00.0
[ 3454.145533] sfc_siena 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
[ 3454.145539] sfc_siena 0000:01:00.0:   device [1924:0803] error status/mask=00001000/00002000
[ 3454.145545] sfc_siena 0000:01:00.0:    [12] Timeout              
[ 3462.615570] pcieport 0000:00:01.0: AER: Corrected error received: 0000:00:01.0
[ 3462.615597] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
[ 3462.615603] pcieport 0000:00:01.0:   device [8086:4c01] error status/mask=00001000/00002000
[ 3462.615609] pcieport 0000:00:01.0:    [12] Timeout              
[ 3482.839387] pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:00:01.0
[ 3482.839416] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 3482.839421] pcieport 0000:00:01.0:   device [8086:4c01] error status/mask=00000001/00002000
[ 3482.839428] pcieport 0000:00:01.0:    [ 0] RxErr                  (First)
[ 3482.839434] pcieport 0000:00:01.0: AER:   Error of this Agent is reported first
[ 3482.839478] sfc_siena 0000:01:00.1: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
[ 3482.839483] sfc_siena 0000:01:00.1:   device [1924:0803] error status/mask=00001000/00002000
[ 3482.839489] sfc_siena 0000:01:00.1:    [12] Timeout              
[ 3482.843304] pcieport 0000:00:01.0: AER: Corrected error received: 0000:00:01.0
[ 3482.843326] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
[ 3482.843332] pcieport 0000:00:01.0:   device [8086:4c01] error status/mask=00001000/00002000
[ 3482.843338] pcieport 0000:00:01.0:    [12] Timeout              
[ 3487.233456] pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:00:01.0
[ 3487.233512] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 3487.233518] pcieport 0000:00:01.0:   device [8086:4c01] error status/mask=00000041/00002000
[ 3487.233524] pcieport 0000:00:01.0:    [ 0] RxErr                  (First)
[ 3487.233530] pcieport 0000:00:01.0:    [ 6] BadTLP               
[ 3487.233535] pcieport 0000:00:01.0: AER:   Error of this Agent is reported first

DaveLTX · Oct 1, 2023

FucaBala said:
Thats something I would like to know as well, but never had a 10Gb.

Does anyone which owns a Intel or Aquantia could provide a quick update of which P state your build goes?

I don't know about X550 but I was testing my X540 a while ago and despite idling on windows it drew enough power to overheat while on 1G... on a single port...

And the i7 7700 attached was stuck at C3

I have 57810S, emulex BE3, CX3P and CX3P to test though

tronic · Nov 16, 2023

firemeteor said:
Thanks for the information, @glow. Unfortunately the price of CX4 is beyond my budget :-(

For X520 I could see contradicting information:
According to the 82599 datasheet, it should have support to PCIE link state.
But there are other information indicating ASPM related bugs here and Errata 37 of the spec.

Late to the party, but here is the output for my x520

Code:

5:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    Subsystem: Intel Corporation Ethernet Server Adapter X520-2
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 32 bytes
    Interrupt: pin A routed to IRQ 43
    NUMA node: 0
    Region 0: Memory at 387fffe80000 (64-bit, prefetchable) [size=512K]
    Region 2: I/O ports at 4020 [disabled] [size=32]
    Region 4: Memory at 387ffff04000 (64-bit, prefetchable) [size=16K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Address: 0000000000000000  Data: 0000
        Masking: 00000000  Pending: 00000000
    Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
        Vector table: BAR=4 offset=00000000
        PBA: BAR=4 offset=00002000
    Capabilities: [a0] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
        DevCtl:    CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <1us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
        LnkCtl:    ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 5GT/s (ok), Width x8 (ok)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [100 v1] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
        UESvrt:    DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        CEMsk:    RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+
        AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [140 v1] Device Serial Number 90-e2-ba-ff-ff-2a-a0-b8
    Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
        ARICap:    MFVC- ACS-, Next Function: 1
        ARICtl:    MFVC- ACS-, Function Group: 0
    Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
        IOVCap:    Migration-, Interrupt Message Number: 000
        IOVCtl:    Enable- Migration- Interrupt- MSE- ARIHierarchy+
        IOVSta:    Migration-
        Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
        VF offset: 128, stride: 2, Device ID: 10ed
        Supported Page Size: 00000553, System Page Size: 00000001
        Region 0: Memory at 00000000ab100000 (64-bit, non-prefetchable)
        Region 3: Memory at 00000000ab200000 (64-bit, non-prefetchable)
        VF Migration: offset: 00000000, BIR: 0
    Kernel driver in use: ixgbe
    Kernel modules: ixgbe

tamuin · Nov 17, 2023

Gio said:
This is a $70 TrendNet 10GB SFP+ card from eBay. I force set aspm=force in kernel and then later using a script to force the bit to enable LinkCtl to Enable L1.

If you could post some basic instructions and the script for this it would be very helpful. Thanks!

meltbox · Dec 9, 2023

tamuin said:
If you could post some basic instructions and the script for this it would be very helpful. Thanks!

en:users:documentation:aspm [Linux Wireless]

wireless.wiki.kernel.org

Might be more than you bargained for but...

For what its worth I have tried force enabling this for SAS cards which claim L1 support and it made zero difference. It seems the flags can be set in memory but they aren't always honored by that hardware. In some cases they just cause lockups from what I have gathered.

Although I need to check that chiphell post because maybe its my asrock server motherboard hiding ASPM support. I don't have any such message from the kernel though.

EDIT: Worse than that though, some of those devices seem to overwrite their own ASPM to disabled the second you touch it. Honestly the whole thing feels like a mess from a user perspective. Don't know if its a linux driver thing or just hardware manufacturers have tons of ASPM related bugs and end up hard disabling it on lots of hardware.

tamuin · Dec 9, 2023

meltbox said:
en:users:documentation:aspm [Linux Wireless]

wireless.wiki.kernel.org

Might be more than you bargained for but...

It is but it appears straightforward, thank you.

richardm · Jan 28, 2024

FWIW, I have a QLogic Fastlinq QL-41162 copper version and my AMD system idles in C6 in Windows. Unfortunately I don't have a bootable Linux partition at the moment.

I have another of these in an ESXi system and I've no clue how it idles. Next time I fire it up I'll try to remember to check.

cromo · Mar 6, 2024

I am testing on a W680 with Mellanox 4 LX, idling. Without the card, the system is at 6.5-7.5 W. With the card and nothing in the ports, it goes up to 19.5-20.5. ASPM L1 enabled. The system remains 99% in C10 state. I even enabled auto power save on link down. mlxconfig -e q shows no other deviation from defaults. Firmware is up to date. Card is sitting in a x4 slot. So while I am able to have pretty much everything enabled, I am still loosing 13-14W. Plus an extra 1W with DAC in and the interface up.

From what I read others are seeing half these numbers.

janek202 · Mar 6, 2024

Intel X710-DA2 also supports ASPM and allows C7 to be reached on my Tiny P330 system. It added 3W in total to idle power consumption with nothing connected to SFP+ ports.

cromo · Mar 7, 2024

janek202 said:
Intel X710-DA2 also supports ASPM and allows C6 to be reached on my Tiny P330 system. It added 3W in total to idle power consumption with nothing connected to SFP+ ports.

And how much with a DAC connected, if you have one handy?

janek202 · Mar 11, 2024

cromo said:
And how much with a DAC connected, if you have one handy?

Hi, sorry for a delay, I finally measured it.

P330 tiny, i5, headless, idle + powertop --auto-tune.
Total system consumption, measured at the wall.
No SFP+: 5.9W
1x 3m Copper DAC: 6.0W

cromo · Mar 11, 2024

janek202 said:
Hi, sorry for a delay, I finally measured it.

P330 tiny, i5, headless, idle + powertop --auto-tune.
Total system consumption, measured at the wall.
No SFP+: 5.9W
1x 3m Copper DAC: 6.0W

Thanks. But woah, only 0.1W with DAC attached? Is the network interface actually up, per dmesg/ifconfig/LEDs in the back?

janek202 · Mar 11, 2024

cromo said:
Thanks. But woah, only 0.1W with DAC attached? Is the network interface actually up, per dmesg/ifconfig/LEDs in the back?

Yes, I was able to ping the other host, the interface was up in Ubuntu.
It’s also possible that my cheap power meter is not very accurate, on such small loads.

cromo · Mar 11, 2024

janek202 said:
It’s also possible that my cheap power meter is not very accurate, on such small loads.

Ah, in such cases I used an extension cord and an additional device of a fixed power draw, like a bulb, and then subtract the Watt measured.

kinoapparatom · Mar 14, 2024

cromo said:
Ah, in such cases I used an extension cord and an additional device of a fixed power draw, like a bulb, and then subtract the Watt measured.

@janek202 yeah this is a great way to somewhat improve the precision of cheap power meters, please let us know if you can ;-) btw which i5 do you have inside your p330 tiny exactly? which os/distro? thanks!!!!!

janek202 · Mar 14, 2024

kinoapparatom said:
@janek202 yeah this is a great way to somewhat improve the precision of cheap power meters, please let us know if you can ;-) btw which i5 do you have inside your p330 tiny exactly? which os/distro? thanks!!!!!

Sorry, I don't have a good constant load I can use.
I measured my other system (Dell Wyse 5070) and my results matched other values I've seen online (~4W).

My P330 has i5-8500T, 2x8GB RAM and 3 nvme SSDs (one in a 2230 WiFi slot). Right now it runs Ubuntu Desktop 22.04 LTS for testing purposes, but I'll move it to debian.

SFP+ cards with ASPM support?

New Member

New Member

Active Member

New Member

New Member

Active Member

New Member

Member

New Member

Member

New Member

Member

New Member

Member

New Member

Member

New Member

Member

New Member

New Member