SFP+ cards with ASPM support?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

kinoapparatom

New Member
Mar 14, 2024
2
0
1
Sorry, I don't have a good constant load I can use.
I measured my other system (Dell Wyse 5070) and my results matched other values I've seen online (~4W).

My P330 has i5-8500T, 2x8GB RAM and 3 nvme SSDs (one in 2230 WiFi slot). Right now it runs Ubuntu Desktop 22.04 LTS for testing purposes, but I'll move it to debian.
Thanks a lot! No worries
You mentioned 6w with sfp 10g card and that it added 3w at idle so you mean you have just 3w at idle with 3x ssds but without sfp 10g card????
Btw which nvme ssds do you have installed?
 

janek202

New Member
Feb 4, 2023
27
21
3
Thanks a lot! No worries
You mentioned 6w with sfp 10g card and that it added 3w at idle so you mean you have just 3w at idle with 3x ssds but without sfp 10g card????
Btw which nvme ssds do you have installed?
You can find more details in this post:

I got these results with display enabled. Running headless saves a few Watts. I added Kioxia BG4 256GB as a boot drive since then.
I didn't measure my system without a 10G card running headless.
 
Last edited:

f1r3

New Member
Jul 25, 2024
1
0
1
I am testing on a W680 with Mellanox 4 LX, idling. Without the card, the system is at 6.5-7.5 W. With the card and nothing in the ports, it goes up to 19.5-20.5. ASPM L1 enabled. The system remains 99% in C10 state. I even enabled auto power save on link down. mlxconfig -e q shows no other deviation from defaults. Firmware is up to date. Card is sitting in a x4 slot. So while I am able to have pretty much everything enabled, I am still loosing 13-14W. Plus an extra 1W with DAC in and the interface up.

From what I read others are seeing half these numbers.
That are very good idle values for a W680 Board. Would you mind sharing which one you are using?
 

Mithril

Active Member
Sep 13, 2019
440
150
43
Does the intel X710 suffer from the "restrictive SFP optics" issue that some other Intel cards do, and if so can that be disabled and alternatively does the supermicro version of the card have that issue?
 

joeribl

Active Member
Jun 6, 2021
173
60
28
Does the intel X710 suffer from the "restrictive SFP optics" issue that some other Intel cards do, and if so can that be disabled and alternatively does the supermicro version of the card have that issue?
Depends on the brand of X710. The original Intel ones have the restriction. But can be unlocked: https://forums.servethehome.com/ind...ing-any-brand-sfp-modules-on-intel-x710.29040

I think i read in one post the Lenovo ones don't care. But i cannot seem to find the post in the forum.
 

Mithril

Active Member
Sep 13, 2019
440
150
43
Depends on the brand of X710. The original Intel ones have the restriction. But can be unlocked: https://forums.servethehome.com/ind...ing-any-brand-sfp-modules-on-intel-x710.29040

I think i read in one post the Lenovo ones don't care. But i cannot seem to find the post in the forum.
Ok, I seem to recall someone complaining one of the (newer?) Intel ones could no longer be unlocked.

After looking at issues other people are having with some of the N100 boards in the mini PCs that have also SFP+ ports, I am considering one of the N100 itx boards and dropping a ASPM compatible SFP+ card in it, since the X710 is gen 3 that means even 2x lanes would be fairly OK up to 10Gb internet, and most likely due to cost if I do bump my speed it would be "only" 5Gb any time in the next year or 2
 

nereith

New Member
Mar 23, 2019
7
3
3
Just bought Chelsio T520-CR and QLogic QL41262HLCU. Updated to the latest firmware from Chelsio and Marvell respectively.

1st one supports ASPM, but not the 2nd.

2nd one is an odd case since the datasheet mentions " Supports PCIe upconfigure to reduce link width to conserve power ".
 
  • Like
Reactions: blunden

blunden

Active Member
Nov 29, 2019
736
241
43
Ok, I seem to recall someone complaining one of the (newer?) Intel ones could no longer be unlocked.
I believe that was the e810. Intel claimed it wasn't vendor locked though if I recall correctly, so that's a bit weird.
 

nasi

Member
Feb 25, 2020
65
20
8
I think i read in one post the Lenovo ones don't care. But i cannot seem to find the post in the forum.
I have a Lenovo card which refused to work with several 1Gbit-SFPs, my guess was it is not supporting any other speed than 10Gbit.
But it works with a 10Gbit-SFP+ from Flexoptix, configured to emulate a Intel module as well as a DAC from Mellanox.
 

Muppet17

New Member
Mar 21, 2024
8
5
3
I have a Lenovo card which refused to work with several 1Gbit-SFPs, my guess was it is not supporting any other speed than 10Gbit.
But it works with a 10Gbit-SFP+ from Flexoptix, configured to emulate a Intel module as well as a DAC from Mellanox.
I have a supermicro AOC-STG-i4S (XL710) which also refuses to sync with a 1G optic (admittedly a 1G GPON module). 10G DAC and oddly a 1G RJ45 converter is fine though. Still need to pull the lenovo tiny it sits in to investigate ASPM implementation.
 

joeribl

Active Member
Jun 6, 2021
173
60
28
I have a supermicro AOC-STG-i4S (XL710) which also refuses to sync with a 1G optic (admittedly a 1G GPON module). 10G DAC and oddly a 1G RJ45 converter is fine though. Still need to pull the lenovo tiny it sits in to investigate ASPM implementation.
XL710 is one of the few that behave nicely with ASPM
 

luckylinux

Member
Mar 18, 2012
69
21
8
For those with an Error in dmesg like this:
Code:
ACPI FADT declares the system doesn't support PCIe ASPM, so disable
I wrote a small Script to deploy on Multiple Systems to "patch" it.

I'm also trying to write a small Script to Debug ASPM since I am getting Stuck a lot left and right and I'd like to keep some Notes on how to properly Troubleshoot as I learn on this journey.

Note that right now it only patches the FACP Table by incrementing the Oem Revision Number (needed to make sure that the Patched ACPI Tables are loaded instead of the Default ones !) and set the PCIe ASPM Not Supported (V4) : 0 Attribute

Check that everything is working with:
Code:
# Check that ASPM is not doing something weird
dmesg | grep -i aspm
And:
Code:
# List PCIe Devices with ASPM Disabled
lspci -vvv | grep --color -B40 -A40 -i "ASPM Disabled"
And:
Code:
# Display Statistics using turbostat every 0.5 Seconds (poll even every 0.05 Seconds if you want, polling every 5 Seconds might always "fall" into Spots where PC6/PC7 are 0%)
turbostat --show Avg_MHz,Busy%,Bzy_MHz,TSC_MHz,POLL,POLL%,C1%,C1E%,C3%,C6%,C7s%,CPU%c1,CPU%c3,CPU%c6,CPU%c7,Pkg%pc2,Pkg%pc3,Pkg%pc6,Pkg%pc7,PkgWatt,CorWatt,IPC,IRQ,SMI --interval 0.5
Also make sure to ACTIVATE ASPM in Linux after boot
Code:
# PowerSave Mode
powertop --auto-tune
cpupower frequency-set --governor ondemand

# Activate ASPM
echo -n powersave > /sys/module/pcie_aspm/parameters/policy
Important Caveats:
- Sometimes no Matter what you do, everything seems right (ASPM Forcefully Enabled, nothing weird in lspci / dmesg / etc, but CPU does NOT enter Package State PC6/PC7 at all. This happened on a Supermicro X10SLL-F (Intel Xeon E3-1275 v3 - Graphics is not enabled/powered anyways so it's NOT that !) with a Mellanox ConnectX-4 NIC in one of the 2 CPU-connected PCIe Slots. The PCH Connected PCIe Slot (PCIe 2.0 x4 in x8) was allowing ASPM to somewhat work, not hugely, but maybe 10% PC6 and 10% PC7). This is weird since a brief testing on a Supermicro X11SSL-F (Intel Xeon E3-1230 v5) showed PC6/PC7 in both the CPU-connected PCIe Slot as well as the PCH-connected PCIe Slot

- Do NOT set the governor to [IQUOTE]powersave[/IQUOTE] unless this is a really lightly used System, as it might cause the load to go up since the Frequency is always low. Governor ondemand and conservative yield similar Results and seemed OK in my brief Testing

- You might need to DISABLE the ACPI Tables temporarily (see this), boot the System, then regenerate the ACPI Tables, should you change any Setting in the BIOS or maybe just move a PCIe Card from one Slot to another Slot (or install a new PCIe Card, etc)

EDIT 1: On Proxmox VE Systems, wait at least 2-5 Minutes after Boot (as reported by uptime) since the CPU Package is unlikely to even reach PC2 in the early Phases due to all VMs spinning up (let alone PC6 or PC7).

EDIT 2: On Supermicro X10SLL-F, it seems that setting the following helped re-enable ASPM with an Intel X710-DA2 in the CPU-connected PCIe slot:
- C1 Auto Demotion -> Disabled
- C3 Auto Demotion -> Disabled
- C-State Pre-Wake -> Disabled
- Package C-State limit -> C7
- LakeTiny Feature -> Enabled

I suspect it's more the "Demotion" and "Pre-Wake" Settings that did the Trick.

Furthermore the Governor conservative in a very short Test seemed now to work better than the ondemand Governor with regards to PC6 (PC7 disabled as soon as the Intel X710-DA2 NIC is in one of the CPU-connected PCIe Slots)

EDIT 3: for more Advanced Troubleshooting and Possibilities, my other Repository might be of Help. It outlines a "Checklist" to check the most common Issues as well as more advanced Stuff such as Disassembling the BIOS and setting hidden BIOS Options from UEFI Shell.
 
Last edited:

luckylinux

Member
Mar 18, 2012
69
21
8
I am testing on a W680 with Mellanox 4 LX, idling. Without the card, the system is at 6.5-7.5 W. With the card and nothing in the ports, it goes up to 19.5-20.5. ASPM L1 enabled. The system remains 99% in C10 state. I even enabled auto power save on link down. mlxconfig -e q shows no other deviation from defaults. Firmware is up to date. Card is sitting in a x4 slot. So while I am able to have pretty much everything enabled, I am still loosing 13-14W. Plus an extra 1W with DAC in and the interface up.

From what I read others are seeing half these numbers.
On a much older Platform (Supermicro X10SLL-F with Intel Xeon E3-1230 v3) I could ONLY get the Mellanox ConnectX-4 LX [EN] *really* working with ASPM if it was put in a PCH-connected PCIe Slot. Then IIRC I could reach PC6.

Do you mean you have this Motherboard/System ? Or some other W680 Chipset ?

1735485091147.png

Did you put the Mellanox ConnectX-4 LX in the CPU-connected PCIe Slots or PCH-connected (Chipset/DMI) PCIe Slots ?
 

luckylinux

Member
Mar 18, 2012
69
21
8
I don't know about X550 but I was testing my X540 a while ago and despite idling on windows it drew enough power to overheat while on 1G... on a single port...

And the i7 7700 attached was stuck at C3

I have 57810S, emulex BE3, CX3P and CX3P to test though
Did you have a chance to test the Broadcom/QLogic 57810s or at least can you provide a `lspci -vvv` Output for it ?

I'm trying to look for some alternatives because the Intel X710-DA2, while quite good, is also "rare" and relatively expensive.
 

Mithril

Active Member
Sep 13, 2019
440
150
43
Did you have a chance to test the Broadcom/QLogic 57810s or at least can you provide a `lspci -vvv` Output for it ?

I'm trying to look for some alternatives because the Intel X710-DA2, while quite good, is also "rare" and relatively expensive.
Same boat, especially since its in the "I would get an ASPM card for several side projects but nothing I NEED right now".

Alternatively, what are some OEM versions of that card?
 

luckylinux

Member
Mar 18, 2012
69
21
8
Reading online I kinda have a bad Feeling though in terms of Drivers Support and Firmware Upgrades.

I guess the major Players are/were Mellanox, Intel and Chelsio to some extent. Broadcom/QLogic I never heard much about (personally). Maybe I've been living under a Rock though :rolleyes:.
 

luckylinux

Member
Mar 18, 2012
69
21
8
By the Way, the Mellanox ConnectX-4 CX416A EN Dual Port 100GbE QSFP28 MCX416A-CCAT does NOT Support ASPM. It doesn't even contain most of the Registers that the ConnectX-4 LX displays in mlxconfig, so there is basically zero Way to configure that.

The ConnectX-4 LX CX4121A ConnectX-4 EN 25GbE SFP28 MCX4121A-ACAT kinda supports ASPM, but only in the PCH/DMI/Chipset connected PCIe Slot. Which is probably fine for a DMI 3.0 x 4 Link and "not too much" SATA Controller Usage at the same Time (32gbps Theoretical Bandwidth), more on the Edge for a DMI 2.0 x 4 Link and/or more SATA Bandwidth used (16gbps Theoretical Bandwidth), although most likely still WAY better than 1GbE ;) .

Or, better put, the CPU Package C-State is stays stuck at PC2/PC3 when the Mellanox ConnectX-4 LX NIC is plugged in a CPU-connected PCIe Slot, even though it still claims to support ASPM. In my Experience, only when connected in the PCH/DMI/Chipset PCIe Slot, the Package C-State will be able to achieve a higher State (better Power Savings). The Intel X710-DA2 on the other Hand achieve high Package C-States in both CPU-connected and PCH-connected PCIe Slots (although slightly better in the PCH-connected PCIe slot, IIRC).

I'm still not sure why, whether the PCH/Chipset can act as a "Buffer" or some sort of "Standardized" or "Smoothed" Interface or what else ... I suspect there might be some kind of Issue like with the Intel 82599 Errata, but I couldn't find any Errata Document for the Mellanox ConnectX-4 LX. Could anybody find some ?
 

DaveLTX

Active Member
Dec 5, 2021
190
51
28
Did you have a chance to test the Broadcom/QLogic 57810s or at least can you provide a `lspci -vvv` Output for it ?

I'm trying to look for some alternatives because the Intel X710-DA2, while quite good, is also "rare" and relatively expensive.
Code:
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 530FLR-SFP+ Adapter
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        IOMMU group: 15
        Region 0: Memory at 6001800000 (64-bit, prefetchable) [size=8M]
        Region 2: Memory at 6001000000 (64-bit, prefetchable) [size=8M]
        Region 4: Memory at 6002010000 (64-bit, prefetchable) [size=64K]
        Expansion ROM at 50880000 [disabled] [size=512K]
        Capabilities: [48] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] Vital Product Data
                Product Name: HP Ethernet 10Gb 2-port 530FLR-SFP+ Adapter
                Read-only fields:
                        [PN] Part number: 647579-001
                        [EC] Engineering changes: F-5345
                        [MN] Manufacture ID: 103C
                        [V0] Vendor specific: 12W PCIeGen2
                        [V1] Vendor specific: 7.14.65
                        [V3] Vendor specific: 7.13.75
                        [V5] Vendor specific: C0
                        [V6] Vendor specific: 7.14.5
                        [V7] Vendor specific: 530FLR-SFP+
                        [SN] Serial number: CN7451V3WH
                        [V2] Vendor specific: 5451
                        [V4] Vendor specific: 3863BB2ED418
                        [RV] Reserved: checksum good, 179 byte(s) reserved
                End
        Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [a0] MSI-X: Enable+ Count=32 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00001000
        Capabilities: [ac] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x8
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [13c v1] Device Serial Number 38-63-bb-ff-fe-2e-d4-18
        Capabilities: [150 v1] Power Budgeting <?>
        Capabilities: [160 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
        Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 1
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [220 v1] Physical Resizable BAR
                BAR 2: current size: 8MB, supported: 8MB
        Capabilities: [300 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Kernel driver in use: bnx2x
        Kernel modules: bnx2x
does that help?
Same boat, especially since its in the "I would get an ASPM card for several side projects but nothing I NEED right now".

Alternatively, what are some OEM versions of that card?
BCM57810S in flom 530FLR (slap a adaptor on it and off you go)