HP NC523SFP / QLogic QLE3242 10Gb NICs SFP+/DAC compatibilities and Linux?

[BLinux]

Does anyone have any thing to share regarding HP NC523SFP / QLogic QLE3242 10Gb NICs and their SFP+/DAC compatibility? And how about driver support in Linux? Is the native kernel driver okay? are there benefits to use 3rd party drivers from QLogic or HP? Are there any firmware issues?

 

[_alex]

i got my hands on a couple of 3442, both rj45 and sfp+
haven't had a chance for a closer Look, but these were used under a 3.1 Kernel with - how i got it - in-tree drivers for iscsi.
so i'd guess in-tree drivers forthe older 3242 should be mature.

for DAC/sfp's i'm not sure, can try with finisar sfp's and a mixed bag of dac's / breakout-cables

 

[BLinux]

i got my hands on a couple of 3442, both rj45 and sfp+
haven't had a chance for a closer Look, but these were used under a 3.1 Kernel with - how i got it - in-tree drivers for iscsi.
so i'd guess in-tree drivers forthe older 3242 should be mature.

for DAC/sfp's i'm not sure, can try with finisar sfp's and a mixed bag of dac's / breakout-cables

do you know which linux driver these use? is it the qlcnic? do you know if that driver supports sr-iov?

please let me know if/when you test them out and if you find that it's picky about SFP+ or DACs. thanks...

 

[_alex]

no, they are in proprietary os atm, just had a quick Look over dmesg that was accessible. but will install smth more convenient soon. guess the 9442 should do sr-iov, can try this. but not sure for 9242, best would be looking at specs and Kernel Module parameters to get an idea.

will need some more days to get things sorted, the nics are only small part of the whole thing i'm currently digging through ...

 

[_alex]

do you know which linux driver these use? is it the qlcnic?

Obviously bnx2x is used there ...

 

[BLinux]

Obviously bnx2x is used there ...

maybe i'm missing something here, but it's not obvious to me it is bnx2x.

from what i can tell searching, it does appear to be instead, qlcnic:

http://driverdownloads.qlogic.com/Q...t.aspx?ProductCategory=325&Product=1159&Os=65

Drivers & Software - HPE Support Center.

And, although I don't know if this info is current, looks like sr-iov support in the driver is in "tech preview":

Network Adapter Feature Support in RHEL - Red Hat Customer Portal

At least on RHEL/CentOS.

I'm curious about the QLE3242 because I've seen it for quite cheap lately and wondering how good of an option it would be for 10GbE in a VM environment, as an alternative to Mellanox options which cost a bit more.

 

[arglebargle]

maybe i'm missing something here, but it's not obvious to me it is bnx2x.

from what i can tell searching, it does appear to be instead, qlcnic:

http://driverdownloads.qlogic.com/Q...t.aspx?ProductCategory=325&Product=1159&Os=65

Drivers & Software - HPE Support Center.

And, although I don't know if this info is current, looks like sr-iov support in the driver is in "tech preview":

Network Adapter Feature Support in RHEL - Red Hat Customer Portal

At least on RHEL/CentOS.

I'm curious about the QLE3242 because I've seen it for quite cheap lately and wondering how good of an option it would be for 10GbE in a VM environment, as an alternative to Mellanox options which cost a bit more.

I picked one of these up for $15 on eBay last week to test SR-IOV, unfortunately I haven't found vendor driver source anywhere so I'll need to install RHEL on a USB drive to really test it.

One note first, I'd like to jump right into testing this card but it runs hot and I mean H. O. T. -- I burned my hand swapping it out after it'd run for 15 minutes last night. I need to find some way to cool it before I do any real testing. The heatsink literally smells like a space heater does in winter, it's absurd how much heat this card produces.

That PCIe 2.0 19W note in the dump is 100% accurate, the card idled at 16-17W with no DAC or module plugged in.

I dumped some info in the short time I had the card booted and @BLinux was right, this card uses qlcnic:

01:00.0 Ethernet controller: QLogic Corp. cLOM8214 1/10GbE Controller (rev 54)
    Subsystem: Hewlett-Packard Company NC523SFP 10Gb 2-port Server Adapter
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 37
    Region 0: Memory at fe600000 (64-bit, non-prefetchable) [size=2M]
    Region 4: Memory at fe890000 (64-bit, non-prefetchable) [size=64K]
    Expansion ROM at fe840000 [disabled] [size=256K]
    Capabilities: [40] MSI-X: Enable+ Count=32 Masked-
        Vector table: BAR=0 offset=00090000
        PBA: BAR=0 offset=00090800
    Capabilities: [80] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [88] Vital Product Data
        Product Name: HP NC523SFP 10Gb 2-port Server Adapter
        Read-only fields:
            [PN] Part number: 593715-001
            [EC] Engineering changes: A-5115
            [SN] Serial number: MY51170CXU
            [V0] Vendor specific: 19W PCIe Gen2
            [V2] Vendor specific: 5117
            [MN] Manufacture ID: 4e 45 33 32 31 30 34 30 34 2d 32 31 20 20 46
            [RM] Unknown: 30 31 30 38 30 32
            [YA] Asset tag: NA
            [RV] Reserved: checksum good, 0 byte(s) reserved
        End
    Capabilities: [a0] MSI: Enable- Count=1/32 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [c0] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
        DevCtl:    Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
            MaxPayload 128 bytes, MaxReadReq 256 bytes
        DevSta:    CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 5GT/s, Width x8, ASPM not supported, Exit Latency L0s <64ns, L1 <1us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
        LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
            Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
            Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
            EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
    Capabilities: [100 v1] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt:    DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap:    First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
    Capabilities: [140 v1] Device Serial Number 59-69-46-61-6e-48-73-75
    Kernel driver in use: qlcnic
    Kernel modules: qlcnic
01:00.1 Ethernet controller: QLogic Corp. cLOM8214 1/10GbE Controller (rev 54)
    Subsystem: Hewlett-Packard Company NC523SFP 10Gb 2-port Server Adapter
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 37
    Region 0: Memory at fe400000 (64-bit, non-prefetchable) [size=2M]
    Region 4: Memory at fe880000 (64-bit, non-prefetchable) [size=64K]
    Expansion ROM at fe800000 [disabled] [size=256K]
    Capabilities: [40] MSI-X: Enable+ Count=32 Masked-
        Vector table: BAR=0 offset=00091000
        PBA: BAR=0 offset=00091800
    Capabilities: [80] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [88] Vital Product Data
        Product Name: HP NC523SFP 10Gb 2-port Server Adapter
        Read-only fields:
            [PN] Part number: 593715-001
            [EC] Engineering changes: A-5115
            [SN] Serial number: MY51170CXU
            [V0] Vendor specific: 19W PCIe Gen2
            [V2] Vendor specific: 5117
            [MN] Manufacture ID: 4e 45 33 32 31 30 34 30 34 2d 32 31 20 20 46
            [RM] Unknown: 30 31 30 38 30 32
            [YA] Asset tag: NA
            [RV] Reserved: checksum good, 0 byte(s) reserved
        End
    Capabilities: [a0] MSI: Enable- Count=1/32 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [c0] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
        DevCtl:    Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
            MaxPayload 128 bytes, MaxReadReq 256 bytes
        DevSta:    CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 5GT/s, Width x8, ASPM not supported, Exit Latency L0s <64ns, L1 <1us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
        LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
            EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
    Capabilities: [100 v1] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt:    DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap:    First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
    Capabilities: [140 v1] Device Serial Number 59-69-46-61-6e-48-73-75
    Kernel driver in use: qlcnic
    Kernel modules: qlcnic

ethtool -i <nic>
driver: qlcnic
version: 5.3.66
firmware-version: 4.8.20
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
root@ted:~# modinfo qlcnic
filename:       /lib/modules/4.15.18-4-pve/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko
firmware:       phanfw.bin
version:        5.3.66
license:        GPL
description:    QLogic 1/10 GbE Converged/Intelligent Ethernet Driver
srcversion:     1ACB70F0C897B41034DC5BC
alias:          pci:v00001077d00008440sv*sd*bc02sc00i00*
alias:          pci:v00001077d00008040sv*sd*bc02sc00i00*
alias:          pci:v00001077d00008C30sv*sd*bc02sc00i00*
alias:          pci:v00001077d00008830sv*sd*bc02sc00i00*
alias:          pci:v00001077d00008430sv*sd*bc02sc00i00*
alias:          pci:v00001077d00008030sv*sd*bc02sc00i00*
alias:          pci:v00001077d00008020sv*sd*bc02sc00i00*
depends:
retpoline:      Y
intree:         Y
name:           qlcnic
vermagic:       4.15.18-4-pve SMP mod_unload modversions
parm:           qlcnic_mac_learn:Mac Filter (0=learning is disabled, 1=Driver learning is enabled, 2=FDB learning is enabled) (int)
parm:           use_msi:MSI interrupt (0=disabled, 1=enabled) (int)
parm:           use_msi_x:MSI-X interrupt (0=disabled, 1=enabled) (int)
parm:           auto_fw_reset:Auto firmware reset (0=disabled, 1=enabled) (int)
parm:           load_fw_file:Load firmware from (0=flash, 1=file, 2=POST in fast mode, 3= POST in medium mode, 4=POST in slow mode) (int)

If you have a machine that supports ARI forwarding on the upstream PCIe switches and bridges you might want to look at the NC552SFP instead. It's the same price and the in-box linux kernel driver supports SR-IOV, the only catch being that it requires ARIFwd+ for SR-IOV to function. The NC552 does use the bnx2x driver, BTW.