Dell C6100 and Windows Server 2012 no RDMA

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
Just for completeness, grab the .ini file, put it into the WinMFT folder along with the downloaded firmware file, and try to flash again without forcing. Let us know what happens.

I got all the files from the post. I guess I can go back and try to create my own firmware file.

PS C:\Users\Administrator.XXX> mst status
MST devices:
------------
mt26428_pci_cr0
mt26428_pciconf0


C:\Users\Administrator.XXX>flint -d mt26428_pci_cr0 query
Image type: ConnectX
FW Version: 2.10.720
Device ID: 26428
Description: Node Port1 Port2 Sys image
GUIDs: 14feb59097fee724 14feb59097fee725 14feb59097fee726 14feb59097fe
e727
MACs: 14feb5fee725 14feb5fee726
VSD:
PSID: DEL09A0000009
 

zane

Member
Aug 22, 2013
70
0
6
I re-downloaded and replaced the ini and mlx file putting them back into the C:\Program Files\Mellanox\WinMFT dir. Still producing the same results and will not install without the -conf flag.

Just for completeness, grab the .ini file, put it into the WinMFT folder along with the downloaded firmware file, and try to flash again without forcing. Let us know what happens.
I'm running 2012 Datacenter I doubt that has anything to do with it. I reinstalled the WinOF VPI 4.40 and that seemed to fix it on one node. This has happening on 2 of my nodes both had the same firmware and cmd results. I "forced" the upgrade with the -conf flag on both and they seem to be working. Ill continue to set things up and let you know if I run into any issues. Thanks for the help!

PS C:\Users\Administrator> Get-SmbServerNetworkInterface

Scope Name Interface Index RSS Capable RDMA Capable Speed IpAddress
---------- --------------- ----------- ------------ ----- ---------
* 20 False True 40 Gbps 169.254.51.162
* 20 False True 40 Gbps fe80::a4aa:622e:...
* 19 False True 40 Gbps 169.254.246.84
* 19 False True 40 Gbps fe80::ec48:e44d:...
* 13 False False 1 Gbps 192.168.2.228
* 13 False False 1 Gbps fe80::cc16:1732:...
 

zane

Member
Aug 22, 2013
70
0
6
So the adapters show up as Network Cable Unplugged. I have brand new cables MC2206130-003 QSFP MC2206130-003 30 Passive Copper IB FDR10
Are the cables ok with this card? The cards will not even ping their own IP.
 
Last edited:

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
With regards to your "cable unplugged" issue: Do you have a subnet manager running? Mine show up as unplugged, even when they are configured for IPoIB, until they are under the influence of a subnet manager. RDMA also shows up as "False" until the same.
 

zane

Member
Aug 22, 2013
70
0
6
:confused: Sorry I'm a total noob with infiniband. But to answer your question no I don't have a subnet manager running. Do I have to purchase this software or does Mellanox provide a free edition?

With regards to your "cable unplugged" issue: Do you have a subnet manager running? Mine show up as unplugged, even when they are configured for IPoIB, until they are under the influence of a subnet manager. RDMA also shows up as "False" until the same.
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
:confused: Sorry I'm a total noob with infiniband. But to answer your question no I don't have a subnet manager running. Do I have to purchase this software or does Mellanox provide a free edition?
If you have a switch, it may have a subnet manager built in. If not, the Mellanox driver install includes a subnet manager. On Windows, it gets installed as a service, but by default it is not running.
 

zane

Member
Aug 22, 2013
70
0
6
Is this service OpenSM? I don't see a service by this name or anything with Mellanox in the name. Man I'm really lost here lol. Thanks for your help DBA!

If you have a switch, it may have a subnet manager built in. If not, the Mellanox driver install includes a subnet manager. On Windows, it gets installed as a service, but by default it is not running.
 

zane

Member
Aug 22, 2013
70
0
6
Steps are here http://forums.servethehome.com/proc...00-windows-server-2012-no-rdma.html#post13613 you have to create the service in PowerShell

SC.EXE delete OpenSM
New-Service –Name "OpenSM" –BinaryPathName "`"C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\opensm.exe`" --service -L 128" -DisplayName "OpenSM" –Description "OpenSM" -StartupType Automatic

Is this service OpenSM? I don't see a service by this name or anything with Mellanox in the name. Man I'm really lost here lol. Thanks for your help DBA!
 

Sielbear

Member
Aug 27, 2013
31
2
8
Here's a question for you IB peeps. I'm using the 1078 SAS daughter-card in all 4 nodes I'm running. Is there another recommended / compatible IB card that will sit in the PCIe slot on this system? I've only got 1 C6100 chassis, but would love to explore using IB for replication and backups across nodes. Bonus (lightning round) question: is there a recommended 8-port (or thereabouts) IB QDR switch that's commonly available?
 

zane

Member
Aug 22, 2013
70
0
6
Is connectx-3 backwards compatible with connectx-2 QDR's 40gb? Windows 2012 SMB IPoIB.
 

zane

Member
Aug 22, 2013
70
0
6
:confused:
Can any one point me in the right direction on why this would not work(see image). All three servers are running 2012 datacenter, dual port QDR connectx2 cards, and firmware is up to date. I can not get the Yellow link to come up says "Network Cable Unplugged" I know the cable is good. Do I have to have a switch for this to work?

 

Smalldog

Member
Mar 18, 2013
62
2
8
Goodyear, AZ
:confused:
Can any one point me in the right direction on why this would not work(see image). All three servers are running 2012 datacenter, dual port QDR connectx2 cards, and firmware is up to date. I can not get the Yellow link to come up says "Network Cable Unplugged" I know the cable is good. Do I have to have a switch for this to work?

You need a subnet manager running if you are running not using a managed switch.

EDIT: I didn't see your picture originally, but now I see it. I believe that you need to run a subnet manager on the other node(s). I thought the subnet manager would only bind to a single port on the dual port cards. Either way, I don't think it would hurt to run a subnet manager on each node as it just goes idle if there it detects another subnet manager already running.

Jeff
 
Last edited:

zane

Member
Aug 22, 2013
70
0
6
I have OpemSM running on all of the nodes and that did does not fix it. I just don't see why this should not work but I really dont know much about IB products. Is there another subnet manager that I should try out that is free or cheap?

You need a subnet manager running if you are running not using a managed switch.

EDIT: I didn't see your picture originally, but now I see it. I believe that you need to run a subnet manager on the other node(s). I thought the subnet manager would only bind to a single port on the dual port cards. Either way, I don't think it would hurt to run a subnet manager on each node as it just goes idle if there it detects another subnet manager already running.

Jeff
 
Last edited:

Smalldog

Member
Mar 18, 2013
62
2
8
Goodyear, AZ
I have OpemSM running on all of the nodes and that did does not fix it. I just don't see why this should not work but I really dont know much about IB products. Is there another subnet manager that I should try out that is free or cheap?
Check to see which port the subnet manager is bound. I thought it would only bind to 1 port, and I am not sure which port is the "first" port.
 

zane

Member
Aug 22, 2013
70
0
6
I believe that you are correct that it is only binding to one port. So i manually ran in two CMD windows:
Get the port GUID with:
ibstat -p
opensm -g 0x14feb59097fee725 -daemon
opensm -g 0x14feb59097fee726 -daemon

This brings up both ports but I think there should be a way to bring up both port with the service. For Linux there is a setting in a config file that you can set "PORTS=ALL". Anyone know how or where I can do this in Windows? Do I need to create a custom config file and if so where should it go?

There is a ton of good doc for linux but not much of anything for win.

Check to see which port the subnet manager is bound. I thought it would only bind to 1 port, and I am not sure which port is the "first" port.
 

sag

Member
Apr 26, 2013
34
6
8
I believe that you are correct that it is only binding to one port. So i manually ran in two CMD windows:
Get the port GUID with:
ibstat -p
opensm -g 0x14feb59097fee725 -daemon
opensm -g 0x14feb59097fee726 -daemon

This brings up both ports but I think there should be a way to bring up both port with the service. For Linux there is a setting in a config file that you can set "PORTS=ALL". Anyone know how or where I can do this in Windows? Do I need to create a custom config file and if so where should it go?

There is a ton of good doc for linux but not much of anything for win.
If you go to section 4.3 in the following link it will tell you how to setup 2 OpenSM services on windows and bind them to separate ports. This is the guide I followed and it works like a charm.
Deploying Windows Server 2012 with SMB Direct (SMB over RDMA) and the Mellanox ConnectX-2/ConnectX-3 using InfiniBand – Step by Step - Jose Barreto's Blog - Site Home - TechNet Blogs