Infiniband HBA ports show as "cable unplugged" in Windows 2016?

[Agrikk]

I am setting up a new IB storage network for my home lab and I need help establishing a connection with my Windows Server 2016 box.

The hardware is:
HP DL180 G6 server with Windows Server 2016 x64
HP Infiniband 4X DDR Connect-X PCI-e Dual Port HBA
Infiniband SFF-8740 cable
Topspin 120 24-port IB switch

I have upgraded the firmware on the Topspin switch to 2.9.0/build 170 and have installed the Mellanox WinOF VPI (v5.35) on my server. I have set the MTU on both ports of the HBA to 2048 to match the switch settings.

The card detects as a Mellanox ConnectX interface card in the OS, but the network adapter appears as "cable unplugged" for both ports. I have used WinOF to configure the ports as IPoIB and as Ethernet just to see if one works over the other, but both are down.

There is a link light on the switch for both connections as well as a link light on the HBA on both ports. The ports on the switch show Up/Up. The ports are set to auto-negotiate and show up at 4x-SDR(10Gbps).

What have I missed? How come the adapters show "cable unplugged" in the OS?

 

[Agrikk]

More information:

The driver version of the adapter is 5.35.12978.0. Firmware version is 2.7.0

Part number is 448397-B21 / 452372-001

 

[Agrikk]

After lots of googling, I found that a "Network Cable Unplugged" status on a Mellanox HBA means that a Subnet Manager doesn't exist on that segment.

What I don't understand is that the Topspin 120 has a subnet manager in it, so how come it doesn't register?

 

[cesmith9999]

Is the subnet manager service running?

Chris

 

[i386]

Maybe it's not running?

 

[Agrikk]

Is the subnet manager service running?

Chris

Where? There is no subnet manager service installed on the Windows box, and the Topspin device has a subnet manager with a subnet prefix and a guid assigned with a oper-status of Master.

I also see that under Infiniband->Services there are none.

I'm not sure what any of this means.

 

[cesmith9999]

Infiniband always needs a subnet manager installed some where in the various subnets

That service need along with the inability to route is why we never used it. We use RoCEv2 instead. Much easier to manage.

Cisco High Performance Subnet Manager for Infiniband Server Switches, Release 1.2 - Fundamentals of the Cisco High-Performance Subnet Manager [Cisco SFS High Performance InfiniBand Subnet Manager]

New to infiniband, can't get a working connection. | Mellanox Interconnect Community

I hope these links help.

Chris

 

[Agrikk]

Infiniband always needs a subnet manager installed some where in the various subnets

That service need along with the inability to route is why we never used it. We use RoCEv2 instead. Much easier to manage.

Cisco High Performance Subnet Manager for Infiniband Server Switches, Release 1.2 - Fundamentals of the Cisco High-Performance Subnet Manager [Cisco SFS High Performance InfiniBand Subnet Manager]

New to infiniband, can't get a working connection. | Mellanox Interconnect Community

I hope these links help.

Chris

Thanks for the reply. The Cisco option looks interesting, but that simply installs a subnet manager on a seperate box instead of using the SM in the Topspin switch, right?

 

[Agrikk]

After a lot of futzing, (including upgrading the firmware on the HP adapters to 2.9.1 using MFT 3.5.0 and installing the Mellanox VPI suite WinOF v5.3.5):

I have been able to set up a direct connection between two Windows Server 2016 servers when I put the adapters into Ethernet mode, and when I connect the adapters to the Topspin 120 switch and put them into IPoIB mode, get get a good green light on the adapter and switch, and the amber light is lit on each adapter, indicating a good connection.

In addition, the switch itself shows the two servers as up and active nodes in the IB section of the management console, and the IB HBAs show up as active ports and functional neighbors. So the subnet manager on the switch is functioning as it should, I guess.

So everything seems good and all, but the HBAs still show as Network Cable unplugged within Windows and I am not able to send any traffic through them.

What have I missed? Why are these darned adapters not coming up in windows when everything else shows them as operational?

 

[i386]

Do you have an ip address assigned on the infiniband port on the server?
When I change my vpi cards from ethernet to infiniband the ports don't keep the old settings.

 

[Agrikk]

A missing IP address wouldn't show a cable unplugged error but yes, when I switch back and forth between IPoIB and Ethernet modes I check to make sure there is an IP address.

 

[Agrikk]

Anyone have any more ideas?

 

[TeleFragger]

so almost a year later and I'm curious where you went with this...

I have this... and windows drivers don't give me the tab to select Ethernet or ipoib...
switch - Flextronics F-x430066 8 Port 4x SDR Infiniband
cable - Mellanox Mcc4l30-300 Microgigacn Latch 0.3 M Infiniband Cable
cards - HPE InfiniBand 4X DDR Conn-X PCI-E G2 Dual Port HCA (483514-B21)

so I'm tying to mess around and figure it out as well...

 

[TeleFragger]

ive learned so much since my post...hah...
I didn't check to see when the last time @Agrikk posted so not sure if even on forums anymore...

 

[Agrikk]

Yeah, I'm still here. Two years later. Did you manage to get it working?

I could never get it to work and ultimately I moved away from IB to 10g fiber using a Force10 switch and Mellanox ConnectX-2 Ethernet Adapters.

 

[TeleFragger]

Yeah, I'm still here. Two years later. Did you manage to get it working?

I could never get it to work and ultimately I moved away from IB to 10g fiber using a Force10 switch and Mellanox ConnectX-2 Ethernet Adapters.

WOW I missed your reply...

LOL... 7 months later ..... for me..

I am running in ethernet mode and got it running at 10gb..
ConnectX-1 cards and 1x ConnectX-2 cards.

I have
Windows 10 Gamer
Plex
Server 2019 Storage
ESXI 6.7

all running 10gb and good. It is funny that today, I started to try to go IB and yup still no clue on this. I have a serial cable plugged into the switch and after googling, it is said that you need to turn the subnet manager on in side the switch... well i have no clue how to get inside. I am reading that I need IBADM software but cannot find it anywhere to download so that hurts.
Then found the manual for the switch and it is not that great...

When I posted in Dec 2018, I was not sure how to do it but got it all going via ethernet and then i am like, well now how do I go about getting my machines to get away from the 1gb connection to the web and I was directed to a 48 port brocade switch, given command lines, and BOOM!!!!
only 10gb nics!!!

fyi - i am looking for at least one more connectx-2 cards if you have any you wish to sell... They support RDMA and connectx-1 does not..I have one now but need at least 2 more...