Running "crossover" ethernet between ConnectX 3 cards

[LasseKongo]

Hi,

Does it work to run switchless ethernet between two ConnectX-3 cards using a regular QSFP cable ?

My plan is to use my dual-port cards with one IB port connected to an IB switch, and one ethernet port directly connected between the servers. I have added "options mlx4_core port_type_array=1,2" to a modprobe.d file on both servers.
If I boot the servers without the cable attached, one server reports ethernet as the link layer, and the other reports infiniband as the link layer. If I connect the cable, both servers report Infiniband as link layer.

I´m suspecting driver issues here, the server that always reports Infiniband is an older Opensuse 12.2 installation, while the other one is an Ubuntu 15.04 with a significantly newer mlx4 driver.

I just would like to confirm that it is actually possible to do a back-to-back connection with these cards when the port is set to ethernet as the link layer.

 

[Chuckleb]

Yes, you can run back-to-back Ethernet with them. I has that running for a while and it worked well. Update the driver and make sure you set it to Ethernet type in the driver config. Also note that in older drivers you could only run dual ports in certain configurations for each port.

 

[LasseKongo]

Yes, you can run back-to-back Ethernet with them. I has that running for a while and it worked well. Update the driver and make sure you set it to Ethernet type in the driver config. Also note that in older drivers you could only run dual ports in certain configurations for each port.

Great, then I know it is possible at least. The Mellanox driver in Opensuse 12.2 is dated 2008 and I´m pretty sure that is the problem, but I don´t want to tear it down if i´m not sure it can work.

 

[andrewbedia]

Why in the name of whomever would you run openSUSE 12.2 where support for it ended in January 2014? Especially in light of all of the SSL/OPENSSH vulnerabilities as of recent?

 

[LasseKongo]

Why in the name of whomever would you run openSUSE 12.2 where support for it ended in January 2014? Especially in light of all of the SSL/OPENSSH vulnerabilities as of recent?

I have a Xen installation with VGA passthrough to 2 desktops among other virtual machines. When I have tried updating it in the past, I have always messed up the passthrough part. The other system mentioned in my post is an Ubuntu 15.04 on which I intend to run KVM with VFIO passthrough, and to migrate all my VM:s to. SSH vulnerabilities isn´t much of a problem on home servers sitting behind an up to date Pfsense firewall.