SFP+ cards with ASPM support?

[firemeteor]

I have a Mellanox CX312A card which I grabbed for reasonably low power spec 3.8W typical && 4.55W max (Page 48 of the manual).
However, the measured power consumption change before and after the card installation is rather huge: ~10W without vs ~17.5 with the NIC.
So this is a ~7.5W power tax at idle without cable plugged in. Why would the measurement go beyond the number listed in the spec?

One thing I noticed is that the CX3 NIC refuses to work with ASPM while it claims the cap and the config is enabled in the BIOS:
From lspci -vvv;

LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

Without the NIC, powertop reports very good package idle state:

Package;0
C2 (pc2); 6.7%
C3 (pc3); 0.1%
C6 (pc6); 0.0%
C7 (pc7); 0.0%
C8 (pc8); 91.5%
C9 (pc9); 0.0%
C10 (pc10); 0.0%

With the NIC, powertop reports that the package is no longer able to idle as well:

Package;0
C2 (pc2); 5.5%
C3 (pc3); 92.7%
C6 (pc6); 0.0%
C7 (pc7); 0.0%
C8 (pc8); 0.0%
C9 (pc9); 0.0%
C10 (pc10); 0.0%

I suspect the difference between PC3 && PC8 may explain the gap between the measured number and the rating on the spec.
I also suspect ASPM feature may help the package to go into deeper idle state.

I wonder if anybody has more conclusive information on this topic?
i.e. if ASPM should help improve package C-state? If CX3 can be configured to enable ASPM?
If there are other SFP+ NICS on the market that are known to behave better on this regard?

 

[abq]

Nice catch, with real results & analyses! Thank You for sharing info on this popular Mellanox CX312A card.

 

[glow]

My ConnectX 4 Lx seems to have it, but I cannot really prove it based on the limited expressions of Windows (powercfg /energy only shows if it is disabled, but it won't say if it's working).

Also a lspci dump of a CX4Lx from here seems to indicate it has ASPM L1 support.

Mellanox ConnectX-4/5/6 Bitfehler ab Linux Kernel 5.13 – Thomas-Krenn-Wiki

Bei Linux Systemen mit NVIDIA Mellanox Netzwerkkarten der ConnectX-4/5/6 Serien kommt es mit den Linux Kernel Versionen 5.13.*, 5.14.* und 5.15.0 - 5.15.19 zu Bitfehlern sofern optische Transceiver verwendet werden. Die Ausgabe von ethtool -m [INTERFACE] zeigt eine Laser wavelength von 22560nm...

www.thomas-krenn.com

My X520s don't seem to support it, but I'll have to dig further.

It is something I am interested in, too, since I'm idly trying to squeeze every watt of efficiency I can with my home network setup. Not severe measures, mostly trying to save power where I can.

 

[firemeteor]

Thanks for the information, @glow. Unfortunately the price of CX4 is beyond my budget :-(

For X520 I could see contradicting information:
According to the 82599 datasheet, it should have support to PCIE link state.
But there are other information indicating ASPM related bugs here and Errata 37 of the spec.

 

[firemeteor]

I tried some dirty trick from this post to manually force enable the ASPM L0S mode.
Unfortunately this lead to floods of error message in the kernel log:

[ 597.337196] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
[ 597.337197] pcieport 0000:00:01.0: device [8086:1901] error status/mask=00001000/00002000
[ 597.337198] pcieport 0000:00:01.0: [12] Timeout
[ 597.339789] pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:00:01.0
[ 597.339825] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
[ 597.339826] pcieport 0000:00:01.0: device [8086:1901] error status/mask=00001100/00002000
[ 597.339826] pcieport 0000:00:01.0: [ 8] Rollover
[ 597.339827] pcieport 0000:00:01.0: [12] Timeout
[ 597.339829] pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:00:01.0
[ 597.339854] pcieport 0000:00:01.0: AER: can't find device of ID0008

I suspect the claim on ASPM support from this CX3 could be fake or at least buggy ...
Given the end-of-life status of this card I think we probably shouldn't expect a fix... :-(

 

[Mithril]

I've got a handful of SFP+ cards I'm not currently using, I'll try to check them this weekend and see if they claim ASPM in lspci and if that seems to work, and include what my killawatt claims over an hour of idle.

 

[firemeteor]

@Mithril, thanks for the help! Looking forward to your result!

 

[Mithril]

Hmm, so my test machine might be too old for valid results, as the only one I have free is an i5 2500 which only goes down to C6 (maybe C7?)

That being said, dmsg claims Ubutnu has ASPM enabled. I'll either need to free up another machine or verify *any* card of any kind will enter ASPM. It's possible this is a bios issue since it's a consumer board. I've got some power figures but now I'm not sure I trust them. If theres an OS that does ASPM better I'd be happy to fire it up on a spare SSD.

 

[firemeteor]

It's true that the motherboard is playing a major role here. I just realize that my Asrock consumer board forcefully disable this ASPM feature through FADT. This even defeat the pcie_aspm=force Linux kernel command-line:

$ dmesg|grep ASPM
[ 0.030562] PCIe ASPM is forcibly enabled
[ 0.112759] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[ 0.344160] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[ 0.344439] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration
[ 0.436502] pci 0000:03:00.0: Disabling ASPM L0s/L1

On the other hand, the Gigabyte C246N-WU2 (a workstation board) is doing great on this front.

 

[Mithril]

So I have tried *every single pcie card I have* that is not in use in my test rig under Ubuntu, not a single one enables ASPM, many claim to have it. trying to force enable it seems to at best not work. I'm not sure if it's due to the age of the board or being a consumer board. At this point, I'd like to see someone with an example of working ASPM with a PCIe card as it feels like theres too many variables here.

 

[firemeteor]

The link from @glow suggests that CX4 may be able to work with ASPM enabled. So probably newer cards have higher chance to work...
I also remember the Intel x710 datasheet describing ASPM functionality, however I haven't seen any real world evidence from kernel log...

 

[firemeteor]

So I have tried *every single pcie card I have* that is not in use in my test rig under Ubuntu, not a single one enables ASPM, many claim to have it. trying to force enable it seems to at best not work. I'm not sure if it's due to the age of the board or being a consumer board. At this point, I'd like to see someone with an example of working ASPM with a PCIe card as it feels like theres too many variables here.

BTW, it might be worthy to name those cards that do not work with ASPM according to your experiment.
Other people who care may benefit from the list. And other card owners who had different experience can also chime-in...

 

[Mithril]

BTW, it might be worthy to name those cards that do not work with ASPM according to your experiment.
Other people who care may benefit from the list. And other card owners who had different experience can also chime-in...

Right now I have no working set, so I can't rule out motherboard or OS. I'd rather not muddy the waters with unconfirmed stuff.

 

[firemeteor]

Trying to contribute a data-point from side-band.
As part of an effort to debug a NVME (SN570 1TB) pass-through problem (FLR causing AER / DPC error related to link low power state), I attempted to force my NVME out of the ASPM L1 mode. This bumps the machine power by ~3W at the outlet.

PS: With NVME in ASPM idle mode, the addtional power consumption is not easily measurable. I can only say that it worth << 1W.
PS2: This time the test bench comes with more devices plugged, instead of being bare-minimal as my original post was measuring.
So the numbers cannot be compared directly. Most importantly, this time I'm with my CX3 installed so I guess the difference of the 3W mostly comes from the NVME SSD itself, instead from the change in package C State.
Maybe I can do a negative experiment -- roll back to the minimal setup without CX3 and compare the impact on power. I suppose this time the impact of deeper package C State can be isolated...

Maybe when I get some spare time...

 

[victhor393]

I have a Broadcom BCM957404A4, a Chelsio T520-SO-CR and a system with claimed ASPM support, sadly, no way to measure power consumption.

BCM97404A4 reports ASPM is supported and enabled, T520-SO-CR reports ASPM is not supported.

This is the output from lspci -vvv for the cards, the link is downgraded to Gen1 due to a misconfiguration on my end. It doesn't seem to affect ASPM in any way, even after fixing said misconfiguration.

01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
    Subsystem: Broadcom Inc. and subsidiaries Device 4141
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 80
    IOMMU group: 8
    Region 0: Memory at 441200000 (64-bit, prefetchable) [size=64K]
    Region 2: Memory at 440000000 (64-bit, prefetchable) [size=1M]
    Region 4: Memory at 4412a0000 (64-bit, prefetchable) [size=8K]
    Expansion ROM at fcf20000 [disabled] [size=128K]
    Capabilities: [48] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
    Capabilities: [50] Vital Product Data
        Product Name: Broadcom Adv. Dual 25Gb Ethernet
        Read-only fields:
            [PN] Part number: BCM957414
            [MN] Manufacture ID: 1028
            [V0] Vendor specific: FFV20.06.02.08
            [V1] Vendor specific: DSV1028VPDR.VER2.1
            [V2] Vendor specific: NPY2
            [V3] Vendor specific: PMTD
            [V4] Vendor specific: NMVBroadcom Corp
            [V5] Vendor specific: DTINIC
            [V6] Vendor specific: DCM1001FFFFFF1202FFFFFF1403FFFFFF1604FFFFFF1805FFFFFF1A06FFFFFF1C07FFFFFF1E08FFFFFF2101FFFFFF2302FFFFFF2503FFFFFF2704FFFFFF2905FFFFFF2B06FFFFFF2D07FFFFFF2F08FFFFFF
            [V7] Vendor specific: L1D1
            [RV] Reserved: checksum good, 85 byte(s) reserved
        End
    Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [a0] MSI-X: Enable+ Count=74 Masked-
        Vector table: BAR=4 offset=00000000
        PBA: BAR=4 offset=00001000
    Capabilities: [ac] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
        DevCtl:    CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+ FLReset-
            MaxPayload 512 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
        LnkCap:    Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
            ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl:    ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 2.5GT/s (downgraded), Width x8
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
             10BitTagComp- 10BitTagReq- OBFF Via WAKE#, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
        LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
             EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [100 v1] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [13c v1] Device Serial Number 00-0a-f7-ff-fe-b7-1d-90
    Capabilities: [150 v1] Power Budgeting <?>
    Capabilities: [160 v1] Virtual Channel
        Caps:    LPEVC=0 RefClk=100ns PATEntryBits=1
        Arb:    Fixed- WRR32- WRR64- WRR128-
        Ctrl:    ArbSelect=Fixed
        Status:    InProgress-
        VC0:    Caps:    PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
            Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
            Ctrl:    Enable+ ID=0 ArbSelect=Fixed TC/VC=01
            Status:    NegoPending- InProgress-
    Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=020 <?>
    Capabilities: [1b0 v1] Latency Tolerance Reporting
        Max snoop latency: 1048576ns
        Max no snoop latency: 1048576ns
    Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
        ARICap:    MFVC- ACS-, Next Function: 1
        ARICtl:    MFVC- ACS-, Function Group: 0
    Capabilities: [1c0 v1] Single Root I/O Virtualization (SR-IOV)
        IOVCap:    Migration- 10BitTagReq- Interrupt Message Number: 000
        IOVCtl:    Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
        IOVSta:    Migration-
        Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
        VF offset: 16, stride: 1, Device ID: 16dc
        Supported Page Size: 000005ff, System Page Size: 00000001
        Region 0: Memory at 0000000441220000 (64-bit, prefetchable)
        Region 2: Memory at 0000000440100000 (64-bit, prefetchable)
        Region 4: Memory at 0000000441240000 (64-bit, prefetchable)
        VF Migration: offset: 00000000, BIR: 0
    Capabilities: [230 v1] Transaction Processing Hints
        Interrupt vector mode supported
        Device specific mode supported
        Steering table in MSI-X table
    Capabilities: [300 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn- PerformEqu-
        LaneErrStat: 0
    Capabilities: [200 v1] Precision Time Measurement
        PTMCap: Requester:+ Responder:- Root:-
        PTMClockGranularity: Unimplemented
        PTMControl: Enabled:- RootSelected:-
        PTMEffectiveGranularity: Unknown
    Kernel driver in use: bnxt_en
    Kernel modules: bnxt_en

01:00.0 Ethernet controller: Chelsio Communications Inc T520-SO Unified Wire Ethernet Controller
    Subsystem: Chelsio Communications Inc Device 0000
    Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 79
    IOMMU group: 8
    Region 0: Memory at c1000000 (64-bit, non-prefetchable) [size=512K]
    Region 2: Memory at c1080000 (64-bit, non-prefetchable) [size=512K]
    Region 4: Memory at c1a80000 (64-bit, non-prefetchable) [size=8K]
    Expansion ROM at c1100000 [disabled] [size=512K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
        Address: 0000000000000000  Data: 0000
        Masking: 00000000  Pending: 00000000
    Capabilities: [70] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
        DevCtl:    CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 512 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 8GT/s, Width x8, ASPM not supported
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl:    ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 2.5GT/s (downgraded), Width x8
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp+ ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
             EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [b0] MSI-X: Enable- Count=34 Masked-
        Vector table: BAR=4 offset=00000000
        PBA: BAR=4 offset=00001000
    Capabilities: [d0] Vital Product Data
        Product Name: T520-SO       
        Read-only fields:
            [PN] Part number: 110118850B0   
            [EC] Engineering changes: 0000000000000000
            [SN] Serial number: PT17200571           
            [RV] Reserved: checksum good, 165 byte(s) reserved
        Read/write fields:
            [RW] Read-write area: 249 byte(s) free
        End
    Capabilities: [100 v2] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [140 v1] Virtual Channel
        Caps:    LPEVC=0 RefClk=100ns PATEntryBits=1
        Arb:    Fixed- WRR32- WRR64- WRR128-
        Ctrl:    ArbSelect=Fixed
        Status:    InProgress-
        VC0:    Caps:    PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
            Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
            Ctrl:    Enable+ ID=0 ArbSelect=Fixed TC/VC=01
            Status:    NegoPending- InProgress-
        VC1:    Caps:    PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
            Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
            Ctrl:    Enable- ID=1 ArbSelect=Fixed TC/VC=00
            Status:    NegoPending- InProgress-
    Capabilities: [170 v1] Device Serial Number 00-00-00-00-00-00-00-00
    Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
        ARICap:    MFVC- ACS-, Next Function: 1
        ARICtl:    MFVC- ACS-, Function Group: 0
    Capabilities: [1a0 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn- PerformEqu-
        LaneErrStat: 0
    Capabilities: [1c0 v1] Single Root I/O Virtualization (SR-IOV)
        IOVCap:    Migration- 10BitTagReq- Interrupt Message Number: 000
        IOVCtl:    Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
        IOVSta:    Migration-
        Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
        VF offset: 8, stride: 4, Device ID: 5807
        Supported Page Size: 00000553, System Page Size: 00000001
        Region 0: Memory at 00000000c1b0e000 (64-bit, non-prefetchable)
        Region 2: Memory at 00000000c1880000 (64-bit, non-prefetchable)
        Region 4: Memory at 00000000c1a82000 (64-bit, non-prefetchable)
        VF Migration: offset: 00000000, BIR: 0
    Capabilities: [200 v1] Transaction Processing Hints
        Interrupt vector mode supported
        Steering table in MSI-X table
    Kernel driver in use: cxgb4
    Kernel modules: cxgb4

By the way, this T520 I own has a nasty bug that makes the ASIC just randomly start heating up for no apparent reason. I don't know if it affects all of them. It will sit happily at like 60-70C with one port connected (or 50C-60C with no links), often for hours at a time, then, out of a sudden, start slowly warming up all the way to 100C, even with no ports connected. I reported this to Chelsio (as this model was still under support at the time), but they tried to place the blame on me not having airflow over the card instead...
It usually cools itself down if the driver is reloaded, failing that, suspending or rebooting the system will fix it, temporarily.
Anyway, they say the limit is 125C, which is reassuring, but I presume that this isn't supposed to happen, regardless if I have a fan or not.

 

[Gio]

I wonder if anybody has more conclusive information on this topic?
i.e. if ASPM should help improve package C-state? If CX3 can be configured to enable ASPM?
If there are other SFP+ NICS on the market that are known to behave better on this regard?

Did you ever figure this out?

I just got a ConnectX-3 dual SFP+ card and I am seeing 10 watt power usage with nothing connected to the ports (no transceivers either). ASPM is enabled in BIOS and aspm=force is set on linux kernel yet the consumption isn't what I expected (3w per port, x2 =~6w when in use - much less at idle as it is now where I see 10w minimum)

 

[glow]

My ConnectX 4 Lx seems to have it, but I cannot really prove it based on the limited expressions of Windows (powercfg /energy only shows if it is disabled, but it won't say if it's working).

Also a lspci dump of a CX4Lx from here seems to indicate it has ASPM L1 support.

Mellanox ConnectX-4/5/6 Bitfehler ab Linux Kernel 5.13 – Thomas-Krenn-Wiki

Bei Linux Systemen mit NVIDIA Mellanox Netzwerkkarten der ConnectX-4/5/6 Serien kommt es mit den Linux Kernel Versionen 5.13.*, 5.14.* und 5.15.0 - 5.15.19 zu Bitfehlern sofern optische Transceiver verwendet werden. Die Ausgabe von ethtool -m [INTERFACE] zeigt eine Laser wavelength von 22560nm...

www.thomas-krenn.com

My X520s don't seem to support it, but I'll have to dig further.

It is something I am interested in, too, since I'm idly trying to squeeze every watt of efficiency I can with my home network setup. Not severe measures, mostly trying to save power where I can.

HWInfo 7.30 shows it's available, but not enabled. This is with Link State Power Management set to Moderate, the default setting with the Ryzen Balanced profile. The ConnectX-4 is plugged into the MB's 2nd M.2 Key M slot, via a riser (mITX motherboard). I don't know if that affects things.

EDIT: the GPU states L0s and L1 supported, but also disabled. I actually cannot find a single PCIe device that states Enabled for ASPM. Looking into what this means, since it seems a bit wrong. I'll note my GPU also states it's running at 2.5GT/s, which is PCIe 1 speeds. If I load it up, it reports 8.0GT/s, but I need to restart HWINFO to see that change. GPUz notes it immediately. Off topic at this point. I had previously thought this link rate changing was part of ASPM, but apparently not? Or is this reporting Disabled, as in "it's not being used at the usec HWINFO was polling for info?"

x

 

[joeribl]

I read on Reddit the Aquantia AQ113 has L0 & L1ASPM support. Would be nice to see some actual test though.

 

[nasi]

It's not about NICs but at least a hint that ASPM "is a thing" used by Intel: enable an idle low power consumption profile for Intel Arc Graphics Desktop cards

 

[Gio]

This is a $70 TrendNet 10GB SFP+ card from eBay. I force set aspm=force in kernel and then later using a script to force the bit to enable LinkCtl to Enable L1.

0000:00:1c.0/03:00.0 Ethernet controller [0200]: Aquantia Corp. AQC100 10G Ethernet MAC controller [AQtion] [1d6a:00b1] (rev 02)
        Subsystem: TRENDnet AQC100 10G Ethernet MAC controller [AQtion] [20f4:101b]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        IOMMU group: 11
        Region 0: Memory at 70c40000 (64-bit, non-prefetchable) [size=64K]
        Region 2: Memory at 70c50000 (64-bit, non-prefetchable) [size=4K]
        Region 4: Memory at 70800000 (64-bit, non-prefetchable) [size=4M]
        Expansion ROM at 70c00000 [disabled] [size=256K]
        Capabilities: [40] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 25.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+

Power observations:
- idle with ASPM enable (<2 watt - single port)
- iperf3 10GB max with ASPM (14 watts - single port)