Mellanox Connect X3 Pro not recognized by T730

[Sealside]

Hi!

I have gotten a couple of Mellanox X3 Pro (MCX312B-XCCT). I can't get it to be recognized
by my HP Thin T730. It is like BIOS is not detecting the card at all. If I plug it into my i7 server it
shows up. I have flash latest bios (2.42.500), as far as I can tell it does not
have IB feature, and should run ethernet mode.

This is the info I get when I have it plugged into the I7:

Device #1:
----------

Device type:    ConnectX3Pro   
Device:         02:00.0         

Configurations:                              Next Boot
         SRIOV_EN                            True(1)         
         NUM_OF_VFS                          8               
         LOG_BAR_SIZE                        3               
         BOOT_OPTION_ROM_EN_P1               True(1)         
         BOOT_VLAN_EN_P1                     False(0)       
         BOOT_RETRY_CNT_P1                   0               
         LEGACY_BOOT_PROTOCOL_P1             None(0)         
         BOOT_VLAN_P1                        1               
         BOOT_OPTION_ROM_EN_P2               True(1)         
         BOOT_VLAN_EN_P2                     False(0)       
         BOOT_RETRY_CNT_P2                   0               
         LEGACY_BOOT_PROTOCOL_P2             None(0)         
         BOOT_VLAN_P2                        1

mstflint -d 02:00.0 q full
Image type:            FS2
FW Version:            2.42.5000
FW Version(Running):   2.33.5100
FW Release Date:       5.9.2017
MIC Version:           2.0.0
Config Sectors:        2
PRS Name:              cx3pro_MCX312B_2mb.prs
Product Version:       02.42.50.00
Rom Info:              type=PXE version=3.4.752
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                 ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
MACs:                                       f452147529f0     f452147529f1
VSD:                   
PSID:                  MT_1200111023

2:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
    Subsystem: Mellanox Technologies Device 000c
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 16
    Region 0: Memory at f7d00000 (64-bit, non-prefetchable) [size=1M]
    Region 2: Memory at f0000000 (64-bit, prefetchable) [size=8M]
    Expansion ROM at f7c00000 [disabled] [size=1M]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [48] Vital Product Data
        Product Name: CX312B - ConnectX-3 Pro SFP+
        Read-only fields:
            [PN] Part number: MCX312B-XCCT         
            [EC] Engineering changes: A4
            [SN] Serial number: MT1445X02924           
            [V0] Vendor specific: PCIe Gen3 x8   
            [RV] Reserved: checksum good, 0 byte(s) reserved
        Read/write fields:
            [V1] Vendor specific: N/A   
            [YA] Asset tag: N/A                     
            [RW] Read-write area: 101 byte(s) free
            [RW] Read-write area: 253 byte(s) free
            [RW] Read-write area: 253 byte(s) free
            [RW] Read-write area: 253 byte(s) free
            [RW] Read-write area: 253 byte(s) free
            [RW] Read-write area: 253 byte(s) free
            [RW] Read-write area: 253 byte(s) free
            [RW] Read-write area: 253 byte(s) free
            [RW] Read-write area: 253 byte(s) free
            [RW] Read-write area: 253 byte(s) free
            [RW] Read-write area: 253 byte(s) free
            [RW] Read-write area: 253 byte(s) free
            [RW] Read-write area: 253 byte(s) free
            [RW] Read-write area: 253 byte(s) free
            [RW] Read-write area: 253 byte(s) free
            [RW] Read-write area: 252 byte(s) free
        End
    Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-
        Vector table: BAR=0 offset=0007c000
        PBA: BAR=0 offset=0007d000
    Capabilities: [60] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 116.000W
        DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap:    Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
             AtomicOpsCtl: ReqEn-
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
    Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
        ARICap:    MFVC- ACS-, Next Function: 0
        ARICtl:    MFVC- ACS-, Function Group: 0
    Capabilities: [148 v1] Device Serial Number f4-52-14-03-00-75-29-f0
    Capabilities: [108 v1] Single Root I/O Virtualization (SR-IOV)
        IOVCap:    Migration-, Interrupt Message Number: 000
        IOVCtl:    Enable- Migration- Interrupt- MSE- ARIHierarchy-
        IOVSta:    Migration-
        Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
        VF offset: 1, stride: 1, Device ID: 1004
        Supported Page Size: 000007ff, System Page Size: 00000001
        Region 2: Memory at 0000000000000000 (64-bit, prefetchable)
        VF Migration: offset: 00000000, BIR: 0
    Capabilities: [154 v2] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt:    DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [18c v1] #19
    Kernel driver in use: mlx4_core
    Kernel modules: mlx4_core

What could be the issue that it is not recognized by the HP T730? I have an Intel XA520 10G that works fine in the T730.

 

[necr]

Do you get any lights while the card is in T730? Is it getting warm after some minutes?
Do you have some special BIOS settings related to PCIe?

 

[Sealside]

Do you get any lights while the card is in T730? Is it getting warm after some minutes?
Do you have some special BIOS settings related to PCIe?

Yes lights are on when I connect it to a switch and it gets warm. T730 bios is really limited, not much to configure for pcie.
Seems a bit wiered to new since a lot of people are using mellanox for t730.

Best regards s

 

[Labs]

One thing I noticed is this:

FW Version:            2.42.5000
FW Version(Running):   2.33.5100

Looks like you upgraded the firmware of the card but still running some older firmware.

What do you mean not detected by the BIOS?

Only in UEFI BIOS you can see the network cards but ONLY in the case the card has the UEFI ROM.
From what I see you have no UEFI ROM because it is written "Rom Info: type=PXE version=3.4.752"

What about the OS level, can you see it there? Did you check it with "lspci" command? If the card is visible in lspci command you are good to go.

 

[Sealside]

One thing I noticed is this:

FW Version:            2.42.5000
FW Version(Running):   2.33.5100

I ran this just after a fw burn, so next boot it shows the same.

What do you mean not detected by the BIOS?

The XA520 will be listed as bootable with pxe, the Mellanox does not show up.
When plugging it into my i7 pc it shows up as bootable.

What about the OS level, can you see it there? Did you check it with "lspci" command? If the card is visible in lspci command you are good to go.

Does not show up under lspci.

 

[fohdeesha]

try setting "SRIOV_EN" to false in the card firmware config, something like "mlxconfig -d /dev/mst/mt4099_pci_cr0 set SRIOV_EN=false". run mst start then mst status to get whatever dev/mst devie your card is, it's probably slightly different than my plain cx3 example

 

[Sealside]

I still have the same issue sriov_en didn't change anything. I'll see if I can get hold of a.plain crx3 card instead. Guess the pro is incompatible with t730. Thanks for suggestions!

Best regards s

 

[Sealside]

For anyone finding this post, I ended up getting a lenovo tiny instead. The t730 I had (might be a faulty one) was unstable on reboots with the Intel da 520 card, sometimes not booting). I still have the t730 and use it for other purposes , replacing some rpis and it is working great with a regular Intel rj45 card.