SFP+ cards with ASPM support?

[luckylinux]

01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 530FLR-SFP+ Adapter
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        IOMMU group: 15
        Region 0: Memory at 6001800000 (64-bit, prefetchable) [size=8M]
        Region 2: Memory at 6001000000 (64-bit, prefetchable) [size=8M]
        Region 4: Memory at 6002010000 (64-bit, prefetchable) [size=64K]
        Expansion ROM at 50880000 [disabled] [size=512K]
        Capabilities: [48] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] Vital Product Data
                Product Name: HP Ethernet 10Gb 2-port 530FLR-SFP+ Adapter
                Read-only fields:
                        [PN] Part number: 647579-001
                        [EC] Engineering changes: F-5345
                        [MN] Manufacture ID: 103C
                        [V0] Vendor specific: 12W PCIeGen2
                        [V1] Vendor specific: 7.14.65
                        [V3] Vendor specific: 7.13.75
                        [V5] Vendor specific: C0
                        [V6] Vendor specific: 7.14.5
                        [V7] Vendor specific: 530FLR-SFP+
                        [SN] Serial number: CN7451V3WH
                        [V2] Vendor specific: 5451
                        [V4] Vendor specific: 3863BB2ED418
                        [RV] Reserved: checksum good, 179 byte(s) reserved
                End
        Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [a0] MSI-X: Enable+ Count=32 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00001000
        Capabilities: [ac] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x8
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [13c v1] Device Serial Number 38-63-bb-ff-fe-2e-d4-18
        Capabilities: [150 v1] Power Budgeting <?>
        Capabilities: [160 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
        Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 1
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [220 v1] Physical Resizable BAR
                BAR 2: current size: 8MB, supported: 8MB
        Capabilities: [300 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Kernel driver in use: bnx2x
        Kernel modules: bnx2x

does that help?

BCM57810S in flom 530FLR (slap a adaptor on it and off you go)

Thank you for your Reply !

Yeah, I guess so, still I wonder why it's DISABLED when it lists the Latency for both L0s and L1 as non-infinite, both for LinkCap and DevCap:

DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-

Do you possible have ASPM disabled globally in the System (e.g. BIOS or you never actually ACTIVATED ASPM) ?

 

[DaveLTX]

Thank you for your Reply !

Yeah, I guess so, still I wonder why it's DISABLED when it lists the Latency for both L0s and L1 as non-infinite, both for LinkCap and DevCap:


Do you possible have ASPM disabled globally in the System (e.g. BIOS or you never actually ACTIVATED ASPM) ?

I figured so but the 8125BGs have ASPM enabled. Weird.

 

[frogtech]

Any word on Intel 82599 or X520 ASPM support or is X710 the "gold standard" here? Trying to avoid any adapters that require significant scripting or OS modifications to provide the support.

 

[joeribl]

Any word on Intel 82599 or X520 ASPM support or is X710 the "gold standard" here? Trying to avoid any adapters that require significant scripting or OS modifications to provide the support.

It has "support" but it is bugged. And i think in the end Intel decided the fix was to disable it in the end.

 

[luckylinux]

Any word on Intel 82599 or X520 ASPM support or is X710 the "gold standard" here? Trying to avoid any adapters that require significant scripting or OS modifications to provide the support.

Not sure if you read my Posts here on in another Thread.

TLDR: the Intel X710-DA2 should be the one that gives you the least amount of EXTRA Problems. Emphasis on **EXTRA**.

Check my Repository for the full Story and some "Checklist".

And check that your System supports ASPM **AT ALL** before even wondering about the NIC. If your BIOS is buggy, FADT Tables are messed up, no matter which NIC you install, it won't allow the CPU to go lower than PC2/PC3.

X520 does NOT support ASPM from what I read. Neither does X540 and X550 IIRC.

Intel 82599 had an Errata (Page 35, Section 37) saying that some Signal must NOT be sent in order for ASPM to work correctly.

 

[luckylinux]

I figured so but the 8125BGs have ASPM enabled. Weird.

Based on my Experience culprit #1 is the buggy BIOS / FADT Tables (if the BIOS option is activated of course - the PCIe Link/Root Port must support ASPM of course).

You can check my Repository if you are interested in finding out why it doesn't seem to work.

It's a bit of Mess here and there, but hopefully it can help somewhat .

 

[luckylinux]

One thing I saw before on Intel platforms and CPU root PCIE buses was this Multi-VC config..

I also saw that. But apparently, that's only on (very) recent Intel Platforms.

If you disassemble the BIOS using my Procedure, then you should be able to just grep -i multi through the Files in the ifrextractor Folder and see if there are some Settings hidden in the BIOS.

For both the Supermicro X10SLL-F (Intel Xeon e3 v3 Platform) and Supermicro X11SSL-F (Intel Xeon e3 v5/v6 Platform) no such Setting exists ...

EDIT 1: The only Settings that come up (on a Supermicro X10SLM+-F Motherboard for Intel Xeon e3 v3 Platform) are Snoop Latency Multiplier and Non Snoop Latency Multiplier. But these look like different Things ...

 

[gaplek]

.

X520 does NOT support ASPM from what I read. Neither does X540 and X550 IIRC.

X550 have aspm support

Intel X550-T2: no ASPM support listed

 

[luckylinux]

X550 have aspm support

Intel X550-T2: no ASPM support listed

Good to know.

But I see they are RJ45 10gbps. Not sure if 2.5gbps/5.0gbps are supported. Thus you need a RJ45 Switch ([relatively] expensive) or some SFP+ to RJ45 Adapters (EXPENSIVE and high Power Consumption). So I don't really see a use case for those.

For Instance the Mikrotik CRS304-4XG-IN is almost 50% more expensive than the Mikrotik CRS305-1G-4S+IN (Review STH Article).

Overall it seems to use a bit more power (8W +[2-3 W]x4 = 16W ... 20W) vs (13 W) of the Mikrotik CRS305-1G-4S+IN, but you also get a more Powerful ARM 64 bit CPU, 32MB of Flash (why is Mikrotik always saving on pennies here ???? - you cannot even install Packages on RouterOS otherwise you cannot upgrade !) and possibly support for 2.5 gbps and 5.0 gbps Networking.

On the other Hand you can get deals at SFP+ DACs for like 2-3 $ / each at Auctions when buying bulk items, so the whole SFP+ DAC being much more expensive than Cat6/Cat6a/Cat7/etc Cables doesn't really hold much.

 

[joeribl]

Good to know.

But I see they are RJ45 10gbps. Not sure if 2.5gbps/5.0gbps are supported. Thus you need a RJ45 Switch ([relatively] expensive) or some SFP+ to RJ45 Adapters (EXPENSIVE and high Power Consumption). So I don't really see a use case for those.

For Instance the Mikrotik CRS304-4XG-IN is almost 50% more expensive than the Mikrotik CRS305-1G-4S+IN (Review STH Article).

Overall it seems to use a bit more power (8W +[2-3 W]x4 = 16W ... 20W) vs (13 W) of the Mikrotik CRS305-1G-4S+IN, but you also get a more Powerful ARM 64 bit CPU, 32MB of Flash (why is Mikrotik always saving on pennies here ???? - you cannot even install Packages on RouterOS otherwise you cannot upgrade !) and possibly support for 2.5 gbps and 5.0 gbps Networking.

On the other Hand you can get deals at SFP+ DACs for like 2-3 $ / each at Auctions when buying bulk items, so the whole SFP+ DAC being much more expensive than Cat6/Cat6a/Cat7/etc Cables doesn't really hold much.

X550 supports 2.5 and 5 Gbit. X540 only 1 and 10 Gbit

 

[Almighty]

Supermicro AOC-STG-i4S 4-port low profile X710 (based on XL710-BM2 chipset) doesn't seem to support ASPM, even though I updated it to the latest SM firmware 9p40 and even cross-flashed to Intel stock firmware 9p53, resetting vendor defaults. The card runs at low ~4W wattage and from what I can tell doesn't prevent my server from entering deeper C6 state. Has anyone had success enabling ASPM on that specific card?

 

[gigi-biji]

I am looking at getting eBay x710-DA-2 to be used with 10G DAC cable (only utilizing 1 port). Anyone can comment on Dell vs Lenovo or other versions? Scalping the Internet I saw that someone on Reddit reported an issue with Dell version that was keeping the PC from properly shutting down/rebooting.

 

[blunden]

I am looking at getting eBay x710-DA-2 to be used with 10G DAC cable (only utilizing 1 port). Anyone can comment on Dell vs Lenovo or other versions? Scalping the Internet I saw that someone on Reddit reported an issue with Dell version that was keeping the PC from properly shutting down/rebooting.

I have a Dell X710-DA2 and it works fine. I've used it in two different AMD based systems (X570 and X670E chipsets). You can find the Dell firmware updates on their support site without any form of support contract or account.

I have no issues with shutting down or rebooting with it connected, but perhaps there was a firmware bug in some old firmware or something. It's not something I've heard of before though.

You can also supposedly cross-flash them to the generic Intel firmware if you want. There is a guide on the Level1Techs forum. I imagine that's true for both the Dell and the Lenovo cards. I never bothered doing so though.

 

[WhiteNoise]

Does anybody know what is the situation with the ConnectX-6 LX?

ASPM, C-states and more generally does it run hot?