Change port type on Connectx-2 Dual Port Cards

[PnoT]

I have a Dell C6100 with Connectx-2 daughter cards that are supposed to do IB in one port and Ethernet in another. The documentation for my card states that the only IB/Ethernet scenario supported is Port 1 IB and Port 2 ETH. I have my connections setup properly yet I'm having mixed results in the cards rediscovering their setup for ETH now. I have 1 server, after a reboot, show up with 1 port IB and the other listed as Ethernet but the other 3 nodes are not auto discovering properly.

The documentation says to go into device manager and under the properties of the adapter there should be a "Port Configuration" tab... ugh NOT. I ran across this post where the guy has no tab either but was able to change the settings via mlxconfig. MCX354A-FCBT does not show "Port Protocol" tab ... | Mellanox Interconnect Community

Mlxconfig says it does not support my device when trying to query it or manage it in any way. The documentation seems to back that up as well as my device is MT26428.

Can someone shed some light on how I can get these ports configured properly?

 

[PnoT]

I'm still looking for the answer on how to set the ports via cmd line or another method BUT I did manage to get things working.

The manual states:





So according to the above I should have IB in Port 1 and Ethernet in Port 2 but it's the exact opposite as that setup completely sends my servers into a panic and nothing works or comes online. If I switch them so Ethernet is Port 1 and IB is Port 2 everything works fine.

In Device Manager IB is Location 1 and Eth Location 2 so they must have screwed up the picture above unless someone else has an explanation that I've missed.

 

[RacerX]

I know this post is ancient but I never really seen an answer to the question. Overall, the Connectx-2 MT26428 can be a real pain to install correctly. Last night I stumbled into the fact that it works with Windows 2012 R2.

First get the latest firmware from Mellanox.
Install Windows 20012 R2
Device Manager displays
The top port supports Infiniband.
The bottom port supports 10GB Ethernet.

I tested the bottom port and it works properly at 10GB

 

[edrozenberg]

The easiest way to change port type is via the Mellanox script connectx_port_config which is buried deep in the Mellanox OFED distribution downloads (http://www.mellanox.com/page/products_dyn?product_family=26). As background, the way the script changes the port type is by writing a value to the PCI device entry for your card in /sys/devices.

I'm attaching a copy of the script below (remove the .txt extension after downloading). It assumes it is running in /sbin but you could place it elsewhere such as /usr/local/sbin and change the script contents for the new location.

Before running the script, create the following dir which the script will use:

# mkdir /etc/infiniband

After the script runs, it will create a file:
/etc/infiniband/connectx.conf

This file will contain something like (different PCI device ID depending on your own setup):

# ConnectX Port Configuration for 0000:02:00.0
/sbin/connectx_port_config -d 0000:02:00.0 -c eth

You can use the command it writes to the file to set up the port type as part of your boot process for example, in one of your boot scripts. You also have the option of writing the appropriate value to the pci device in /sys/devices yourself as part of the boot process - you can find out the correct approach by reading how the script does it.

Note: the port type needs to be set again each time the machine boots - it always resets to the default when the machine reboots.

Similar info here:
Mellanox ConnectX-2 with Ubuntu 18.04? : homelab

Mellanox automation scripts for python/ruby/puppet:
Mellanox/config-tools

 

[ndroftheline]

another approach to configuring the ports in these older cards is to use the method discussed here. 0014419: Mellanox ConnectX cards refuse to work in Ethernet mode with kernel kernel-3.10.0-693.11.6 onwards - CentOS Bug Tracker which adds an option to the mlnx4_core module setting the port mode. that is,

# echo options mlx4_core port_type_array=2,2 >> /etc/modprobe.d/mlx4.conf

NOTE: 2,2 means set option 2 on port 0, and option 2 on port 1. if you have only 1 port, only put one value instead of using the comma. if you have multiple connectx cards i think you can specify the device ID, i didn't have to do that because only 1 card.

then regenerate initramfs. on debian, instructions are Linux kernel module options on Debian
that is,

# update-initramfs -u -k all

another option also discussed there is to pass it as an argument in GRUB like so (i didn't try this)

setting the following in /etc/default/grub:

GRUB_CMDLINE_LINUX="mlx4_core.port_type_array=2,2"

and then updating the grub config:

update-grub2

hope this helps save someone some time in the future.