C6100 - Server 2016 with Storage Spaces Direct, Infiniband Help

[DannyDB]

Hello Techies

So diving into a home lab with the PowerEdge C6100.
I have purchased a Voltaire 4036 and installed Server 2016 on my hosts.

I am running into an issue of no link lights on the IB Switch.

Hardware:
MCX384A-BCAA Dual Port Mezzanine - Mellanox ConnectX-3 VPI (MT04099) Network Adapter
Voltaire 4036
HPC QDR Cables

I have checked the Switch and initialized the subnet manager.

When I query the Port on the Switch I am seeing no link and the ports are not disabled
# Port info: Lid 1 port 6
LinkState:.......................Down
PhysLinkState:...................Polling
Lid:.............................0
SMLid:...........................0
LMC:.............................0
LinkWidthSupported:..............1X or 4X
LinkWidthEnabled:................1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedActive:.................undefined (7)

HAs anyone experienced this issue that can help me with it?

Thank You

Danny

 

[DannyDB]

Update, So after updating the Firmware on my cards and Software on the Voltaire 4036 I still am unable to establish any connectivity.

I think my issue is that the Voltaire is an Infiniband Only Switch and the MCX384A-BCAA are hard coded for Ethernet.

Looks like the settings on my MCX384A-BCAA are hard coded for Ethernet Only:


Have any of you been able to change these settings to enable IB?

 

[i386]

This is an "EN" card, not "VPI" so it's ethernet only.

You could try to flash a "vpi" firmware, but can't say if this will work or not. <- this could also destroy the nic! flash at your own risk!

 

[DannyDB]

Do you know of anyone that has done that successfully? I need the cards for east-west traffic so the ethernet functionality is not of that great importance. I also purchased the Voltaire, and I don't see too many good options for a QDR or QSFP Eth switch. Any suggestions are appreciated.

Thank You

 

[i386]

Do you know of anyone that has done that successfully?

Not with the "en" cards, but with cx3 vpi cards, somewhere here in the forum.

I don't see too many good options for a QDR or QSFP Eth switch

If you want a "cheap" 40gbe switch look for arista 7050 on ebay, there are many listings for <$800 with the bo button