Unable to get link between 2 HP 649281-B21

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

DBordello

Member
Jan 11, 2013
43
0
6
I recently purchased 3 HP 649281-B21:

HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21+ QSFP /SFP's | eBay

649281-B21 656089-001 661685-001 HP IB FDR/EN 10/40GB 2P 544QSFP ADAPTER

I am connecting all 3 together with 3 cables:

Amazon.com: QSFP+ to QSFP+ 40Gb Passive DAC Copper Infiniband Cable Direct Attach QSFP 31 inch 80cm: Computers & Accessories

Two nodes are HP DL 380 G7s running ESXi. They fired right up, and got link between each other.

The other node is a custom built FreeNas machine. Each ESXi box is connected to the FreeNas machine. However, I am unable to get link between either ESXi node and the FreeNas node. FreeNas does recognize the NIC:

[root@filejunky] ~# ifconfig mlxen0
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:02:c9:fb:a3:30
inet 10.3.3.1 netmask 0xffff0000 broadcast 10.3.255.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
[root@filejunky] ~# ifconfig mlxen1
mlxen1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:02:c9:fb:a3:31
inet 10.4.3.1 netmask 0xffff0000 broadcast 10.4.255.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier


This NIC is the "odd man out". The other two were purchased as a pair, this one was not.

Any thoughts on getting link between the FreeNas machine and ESXi?
 

DBordello

Member
Jan 11, 2013
43
0
6
I am wondering if this cards is in IB mode. For what it is worth:

[root@filejunky] /dev# lspci -s 01:00.0 -vvvvvvv
01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
Subsystem: Hewlett-Packard Company InfiniBand FDR/EN 10/40Gb Dual Port 544QSFP Adapter
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at df700000 (64-bit, non-prefetchable)
Region 2: Memory at dc000000 (64-bit, prefetchable)
Expansion ROM at df600000 [disabled]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] Vital Product Data
Not readable
Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-
Vector table: BAR=0 offset=0007c000
PBA: BAR=0 offset=0007d000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 116.000W
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis+, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
 

CookiesLikeWhoa

Active Member
Sep 7, 2016
112
26
28
35
Assuming that NIC does 40GBe (looks like a Melanox Contectx 3 NIC so it should), I doubt there is a lot of support for it in FreeNAS. Melanox cards aren't the most supported, and then cover them with HP firmware and I suspect that's the issue.

I could be completely wrong here though.
 

DBordello

Member
Jan 11, 2013
43
0
6
There goes the IB mode theory:

[root@filejunky] ~# sysctl sys.device.mlx4_core0.mlx4_port1
sys.device.mlx4_core0.mlx4_port1: eth
[root@filejunky] ~# sysctl sys.device.mlx4_core0.mlx4_port2
sys.device.mlx4_core0.mlx4_port2: eth
 

DBordello

Member
Jan 11, 2013
43
0
6
It appears I may be barking up the wrong tree.

ESXi <-> ESXi Gets Link
FreeNas <-> FreeNas (loopback) Gets link
ESXi <-> FreeNas NO LINK

I wonder if I have the ESXi cards stuck in IB? They are 40GB auto-negotiate.
 

DBordello

Member
Jan 11, 2013
43
0
6
Well, I have been chasing the wrong problem. This is not software/configuration related.

I have 2 cards (purchased as a pair) that will connect to each other.

The other one will not connect to the other 2. But, it will connect with itself (Port 1 to Port 2). I have swapped the card with the other cards, and the issue follows the NIC.

I am fairly confident it is in "eth" mode. I have tried it in FreeNas, and ESXi.

Any thoughts on the best way to "reset" this card to the defaults? I have a hard time believing it is defective, since it presents itself to the OS, and will get link with loopback.
 

techtoys

Active Member
Feb 25, 2016
189
50
28
58
I have these cards in a working IB setup 40G.
I am confused about your problem.
You want them to work in 40G Ethernet mode?
The cards may work in 10G ethernet mode but I don't have that setup since I would need a 10G switch.
I have a 40G IB switch,

as far a reset of the card,
I immediately flashed the cards with Mellanox firmware.
 
Last edited:

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
I'd also suggest to flash to MLX firmware first . This will also 'reset' the card;)
 

DBordello

Member
Jan 11, 2013
43
0
6
As an update on this. I was blaming the wrong card :)

Two cards had very old firmware. I updated them all to the latest MLX, and everything is working great.