Help: Testing Mellanox ConnectX-3 - not connecting

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
D

DolphinsDan

Member
Sep 17, 2013
90
6
8
I need to find a picture host but here is the config:

Physical environment
Server with a ConnectX-3 354A card (IB + Ethernet one)
FDR QSFP Cable connecting Port 0 to Port 1 on the card
Link established and verified via lights on the card

Hypervisor Environment
Windows Server 2012 R2 Standard base system. Hyper-V installed. ConnectX-3 354A's are detected by the OS and
I created two external switches one on the first port, one on the second port

Test VMs
Ubuntu 13.10 server x2
Here are the Ubuntu configs:
Code: 
auto eth1
iface eth1 inet static
netmask 255.255.255.0
address 172.17.0.5
network 172.17.0.0
broadcast 172.17.0.255

and 

auto eth1
iface eth1 inet static
netmask 255.255.255.0
address 172.17.0.6
network 172.17.0.0
broadcast 172.17.0.255
When I ping the adapter's own address (e.g. .5) I get time = 0.04ms. When I try to ping the other port/ machine (e.g. .6) I get "From 172.17.0.6 icmp_seq=1 Destination Host Unreachable"

What am I missing here? Help appreciated as I do want to share speeds I get.
 
D

DolphinsDan

Member
Sep 17, 2013
90
6
8
I'm assuming Ethernet mode using late December 2013 drivers. Link speed is 40gbps and they show up in Windows as Ethernet 5/ Ethernet 6 Mellanox IPoIB Adapters.
 
D

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA
DolphinsDan said:
I'm assuming Ethernet mode using late December 2013 drivers. Link speed is 40gbps and they show up in Windows as Ethernet 5/ Ethernet 6 Mellanox IPoIB Adapters.
Click to expand...


If they have "IPoIB" in their Windows names, then they are almost certainly in Infiniband mode. Check Device Manager to confirm this.

You might want to switch the cards to Ethernet mode instead. If you do want to stay in Infiniband mode and use IPoIB, you'll need a subnet manager running. Can you confirm that you have one installed and that it's running?
 
D

DolphinsDan

Member
Sep 17, 2013
90
6
8
No subnet manager running. I do not even see a setting to switch from IB to Ethernet. Which is easier to do? Switch to Ethernet mode or install subnet manager? Do I need to run subnet manager on windows or can I run in a Linux vm?
 
C

Chuntzu

Active Member
Jun 30, 2013
383
98
28
You will find the subnet manager under program file/mellanox/vpi/in/tools or command line opensm.exe. To adjust from ib to Ethernet its under device manager, system devices, the name of your mellanox card, right click properties and then the settings are in one of those tabs. Guy name Jose did a good write up on tech net in 2012 with all the power shell commands. Should be one of the first articles on Google if you search how to set up infiniband on server 2012.
 
britinpdx

britinpdx

Active Member
Feb 8, 2013
367
184
43
Portland OR
Lots of good advice so far, I'll throw in my 2 cents worth. My knowledge is limited to ConnectX-2 cards working in a Server 2012 environment in IPoIB mode. I cannot talk with any level of confidence about Ubuntu, so I won't even try.

It sounds as if you are running a Mellanox ConnectX-3 MCX354A-FCB card. This is a VPI (Virtual Protocol Interconnect) card, and the card can operate in Infiniband or Ethernet mode depending on how it is setup. In ALL of the ConnectX-2 cards that I have installed under Server2012, the default operation from the driver has always been to run in IB mode.

In an IB environment, a subnet manager (SM) is required to map identifiers (LID's) to the ports connected to the IB fabric so that a routing table can be created. SM can be hardware (switch) or software based. If you are not running a switch with an onboard SM then in you need to run a software SM. In Windows, this is OpenSM and it is setup as a service.

There's a couple of resources available to assist on Infiniband setup, both a Mellanox White Paper as well as a great blog by Jose Baretto that walk you through the steps of setting up IB in Server 2012. They are both similar (not sure who copied who) but I've always used the Jose Baretto guide and copied powershell commands as needed.

If you are unsure about IB setup then I suggest you read through the links and start there.
 
D

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA
DolphinsDan said:
No subnet manager running. I do not even see a setting to switch from IB to Ethernet. Which is easier to do? Switch to Ethernet mode or install subnet manager? Do I need to run subnet manager on windows or can I run in a Linux vm?
Click to expand...
On Windows, you can view and switch the card mode via windows device manager. Your card will have three entries in device manager. You'll find one for each port under network devices and a third one either under network devices or sometimes hidden under system devices. This third entry has a UI to switch modes, with an option for IB, Ethernet, or "Auto".

On Linux, you can check the configuration with /sbin/connectx-port-config --show and change it with /sbin/connectx-port-config.

A useful URL from my notes: http://www.mellanox.com/related-doc...dware _for_VPI_Operation_Application_Note.pdf

On Windows, the IB driver installed a subnet manager as a service for you, but didn't enable it. Check services. You can also run it on Linux, of course.

For you, right now, enabling the subnet manager in Windows services will probably be the fastest way to get running. In the future, however, using Ethernet mode is definitely easier for a system without a managed switch since you don't have to worry about the subnet manager at all. With ConnectX-2, that would limit you to 10GbE, but with ConnectX-3 you get 40Gbit Ethernet, which is the same raw speed as 56Gbit Infiniband running IPoIB. With your cards, I'd use Ethernet mode for your two-node system.
 
Last edited:
A

Aluminum

Active Member
Sep 7, 2012
431
46
28
dba said:
On Windows, you can view and switch the card mode via windows device manager. Your card will have three entries in device manager. You'll find one for each port under network devices and a third one either under network devices or sometimes hidden under system devices. This third entry has a UI to switch modes, with an option for IB, Ethernet, or "Auto".

On Linux, you can check the configuration with /sbin/connectx-port-config --show and change it with /sbin/connecx-port-config.

A useful URL from my notes: http://www.mellanox.com/related-doc...dware _for_VPI_Operation_Application_Note.pdf

On Windows, the IB driver installed a subnet manager as a service for you, but didn't enable it. Check services.

For you, right now, enabling the subnet manager in Windows services will probably be the fastest way to get running. In the future, however, using Ethernet mode is definitely easier in a two-node system since you don't have to worry about the subnet manager at all. With ConnectX-2, that would limit you to 10GbE, but with ConnectX-3 you get 40Gbit Ethernet, which is the same raw speed as 56Gbit Infiniband running IPoIB. With your cards, I'd use Ethernet mode for your two-node system.
Click to expand...
Yep, I have two of those dual port X-3 cards along with a EN, which is same hardware but firmware locked to 10/40GbE only. If you are just doing peer to peer links I would highly recommend changing them over.

Go to device manager, find the actual card itself under system devices (NOT the network interfaces) and in its properties change the port you are using to Ethernet mode. Then they will work exactly like any other NIC you are used to.
 
D

DolphinsDan

Member
Sep 17, 2013
90
6
8
WOW! Thank you everyone for the help!!!!! That is awesome. Even a main site help guide?!?

I followed the main site guide and instructions above. One thing that you could add is that after the change you need to re-enter static IPs for the Mellanox HCAs.

Now the bad news... it still does not work. I even changed the windows vswitches to 172.17.0.1 and 172.177.0.2 with a 255.255.255.0 subnet (no gateway though?)

Now on Ethernet I'm guessing this is a network address/ routing issue or that I'm doing something impossible i.e. an External loopback.

I would ping from Windows but "ping 172.17.0.1" directly to the Mellanox External vSwitch port isn't going to help since windows can see both adapters.
 
D

DolphinsDan

Member
Sep 17, 2013
90
6
8
Mini breakthrough - It is a networking setting. I just do not know which one. I just tried to ping the two Ubuntu VM's over the 172.17.0.x address space and I could ping both of them. They just cannot talk to each other over the Mellanox card ports.
 
D

DolphinsDan

Member
Sep 17, 2013
90
6
8
Success!!!!

$ iperf -c 172.17.0.6 -P 4
------------------------------------------------------------
Client connecting to 172.17.0.6, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[ 6] local 172.17.0.5 port 33764 connected with 172.17.0.6 port 5001
[ 3] local 172.17.0.5 port 33761 connected with 172.17.0.6 port 5001
[ 4] local 172.17.0.5 port 33762 connected with 172.17.0.6 port 5001
[ 5] local 172.17.0.5 port 33763 connected with 172.17.0.6 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-10.0 sec 632 MBytes 530 Mbits/sec
[ 4] 0.0-10.0 sec 636 MBytes 533 Mbits/sec
[ 5] 0.0-10.0 sec 638 MBytes 535 Mbits/sec
[ 3] 0.0-10.0 sec 636 MBytes 533 Mbits/sec
[SUM] 0.0-10.0 sec 2.48 GBytes 2.13 Gbits/sec


THANK YOU ALL!
 
D

dswartz

Active Member
Jul 14, 2011
610
79
28
That tcp window size looks awfully small for a link that size. Can you run it again with maybe 64K windows?
 
D

DolphinsDan

Member
Sep 17, 2013
90
6
8
64K made numbers go down. Here' s a crazy one - when I swapped the client and server nodes (making .5 the server) I got better results:
$ iperf -c 172.17.0.5 -P 4 -t 30
------------------------------------------------------------
Client connecting to 172.17.0.5, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[ 6] local 172.17.0.6 port 39395 connected with 172.17.0.5 port 5001
[ 3] local 172.17.0.6 port 39392 connected with 172.17.0.5 port 5001
[ 4] local 172.17.0.6 port 39393 connected with 172.17.0.5 port 5001
[ 5] local 172.17.0.6 port 39394 connected with 172.17.0.5 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-30.0 sec 2.31 GBytes 661 Mbits/sec
[ 3] 0.0-30.0 sec 2.08 GBytes 595 Mbits/sec
[ 4] 0.0-30.0 sec 2.14 GBytes 613 Mbits/sec
[ 5] 0.0-30.0 sec 2.13 GBytes 609 Mbits/sec
[SUM] 0.0-30.0 sec 8.65 GBytes 2.48 Gbits/sec


Looks like it might actually be maxing out a single core and that is what is killing performance.
 
You must log in or register to reply here.
Top Bottom