Mellanox Connect X2

Falloutboy

Member
Oct 23, 2011
116
13
18
I have some Mellanox Connect X2 IPoIB nics that I am trying to get running under Windows 10 x64 Build 2004, I believe I have correctly updated the firmware for them but Windows network connections are reporting them as "Network Cable Unplugged".
For completeness the network cables are labeled Mellanox P/N MC2206310-005 and the NICs are MHQH19B-XTR ( rev A1-A3).
If someone could give me some guidance to get these working it would be appreciated.

Thanks.
 

Freebsd1976

Active Member
Feb 23, 2018
202
30
28
ib network need subnet managers, you need run opensm (Include in mellanox drivers package )on your one pc
 

Falloutboy

Member
Oct 23, 2011
116
13
18
ib network need subnet managers, you need run opensm (Include in mellanox drivers package )on your one pc
Thanks for that, I am trying that now on one of the machines and am about to run up the other, I am not sure that anything is correct at this point so this is a bit of a learning exercise.
 

Falloutboy

Member
Oct 23, 2011
116
13
18
O.K I now have connection lights up, the cable is found and the network cards are shown under Control Panel->Network & Internet->Network connections ; however they can not ping each other. I have configured one with 192.168.10.5 and the other with 192.168.10.10 both with netmask 255.255.255.0 no gatway, no dns.

Scratch that, the machine running opensm can not ping the one that is not but the other way is fine, is this a server -> client thing or should they both be able to ping each other?
 

Falloutboy

Member
Oct 23, 2011
116
13
18
ib network need subnet managers, you need run opensm (Include in mellanox drivers package )on your one pc
Any more to this? I can run opensm on either machine and I can then ping the machine itself and the remote ; however the remote can not ping the local, if I reverse it and run opensm on the other machine the same thing happens.
 

Freebsd1976

Active Member
Feb 23, 2018
202
30
28
The nic now run vpi mode , you can change it to eth , download mft tools and run
mlxconfig -d /dev/mst/mt**** set LINK_TYPE_P1=2 LINK_TYPE_P2=2
cold reboot , now your nic will run in eth mode , no ib and no opensm needed

edit :
on windows

mlxconfig -d mt**** set LINK_TYPE_P1=2 LINK_TYPE_P2=2

you need run
mst status

to got mt*** exactly
 

Falloutboy

Member
Oct 23, 2011
116
13
18
The nic now run vpi mode , you can change it to eth , download mft tools and run
mlxconfig -d /dev/mst/mt**** set LINK_TYPE_P1=2 LINK_TYPE_P2=2
cold reboot , now your nic will run in eth mode , no ib and no opensm needed

edit :
on windows

mlxconfig -d mt**** set LINK_TYPE_P1=2 LINK_TYPE_P2=2

you need run
mst status

to got mt*** exactly
Will that Eth connection be at 40Gb or 10Gb /s - also any specific version of MFT tools or will any do?
 

Falloutboy

Member
Oct 23, 2011
116
13
18
The nic now run vpi mode , you can change it to eth , download mft tools and run
mlxconfig -d /dev/mst/mt**** set LINK_TYPE_P1=2 LINK_TYPE_P2=2
cold reboot , now your nic will run in eth mode , no ib and no opensm needed

edit :
on windows

mlxconfig -d mt**** set LINK_TYPE_P1=2 LINK_TYPE_P2=2

you need run
mst status

to got mt*** exactly
O.K I did exactly as you said.
MST status responds with.
MST devices:
------------
mt26428_pci_cr0
mt26428_pciconf0


C:\Program Files\Mellanox\WinMFT>mlxconfig -d mt26428_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

Device #1:
----------

Device type: ConnectX2
PCI device: mt26428_pci_cr0

-E- Failed to query device: mt26428_pci_cr0. Device not supported.

So I tried the pciconf0 version and just the version of the card I.E mt26428 and got the same sort of
result.
 

Falloutboy

Member
Oct 23, 2011
116
13
18
While I am aware it's not considered good form to do a general reply to your own post .. I have this working and all I can say is IPoIB is weird, I am hoping I still can do what I want to do with this setup.

Thanks to some helpful people here I learnt that you had to run opensm on one of the PC's to get the interfaces up but I struck problems until I realized what was going on.

I configured my two PC's with a class C subnet 192.168.10.5 and 192.168.10.15 respectively and one could ping the other and itself but the 2nd could only ping itself... Not what I am used to.

BUT. I can transfer files between the two PC's I can only surmise that opensm is doing the work for me, I don't have an hosts file at the moment but can still enter the name of the RemotePC if it has a share on it and transfer one or more files. I am still working on getting opensm running as a windows service it does not want to seem to do that but transfers are happening, i managed about 325 meg per second until I ran out of file to send.

If anyone can provide more information it would be appreciated especially with regards to what I want to do which is use a dual processor PC with a bunch of these to make a switch using open source software for connecting my two servers together and 3 clients.

Thanks all for the help thus far. If I can get these into Ethernet mode and retain the 40gb rate that would be even better.
 

Freebsd1976

Active Member
Feb 23, 2018
202
30
28
connectx-2 card is old , so maybe need Mellanox WinOF Rev 4.80.50000 and old winmft
No ping Maybe firewall related?
Use iperf3 or other to test speed
 

i386

Well-Known Member
Mar 18, 2016
1,932
503
113
31
Germany
Thanks all for the help thus far. If I can get these into Ethernet mode and retain the 40gb rate that would be even better.
After installing WInOF open the device manager, go to system devices and there should be a mellanox device listed. Open the properties for that device and there should be an entry for port type configuration.
 
  • Like
Reactions: Falloutboy