Mellanox ConnectX-3 on Proxmox 7.3

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
I

inf3rno

Member
Mar 24, 2021
36
1
8
Hi everyone!
I have a Proxmox 7.4-17 with a Debian 11.6 (“Bullseye”) on my computer.
I have the following network card:
Code: 
  Device Type:      ConnectX3Pro
  Part Number:      ...
  Description:      HP InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+QSFP Adapter
  PSID:             HP_1370110017
  PCI Device Name:  /dev/mst/mt4103_pci_cr0
  Port1 MAC:        ...
  Port2 MAC:        ...
  Versions:         Current        Available    
     FW             2.42.5052      N/A          
     CLP            8025           N/A          
     PXE            3.4.0754       N/A          
     UEFI           14.11.0049     N/A
I cannot find drivers for Debian 11.6 here: Linux Ethernet Drivers
Is there a way to make it work somehow?
 
N

NPS

Active Member
Jan 14, 2021
147
44
28
What do you want to achieve? Drivers for all basic functionality are built into linux.
 
O

oneplane

Well-Known Member
Jul 23, 2021
845
484
63
inf3rno said:
They don't install.
Click to expand...
How so? In any case, you can also switch to the community DEB, but like NPS wrote, unless you're doing something special, the drivers are already included.

The card might not have the correct firmware loaded, but that's not driver-specific. Another thing is that when you have the correct firmware you'd still need to setup the interfaces for them to appear in ip/ifconfig.
 
  • Like
Reactions: inf3rno
I

inf3rno

Member
Mar 24, 2021
36
1
8
oneplane said:
How so? In any case, you can also switch to the community DEB, but like NPS wrote, unless you're doing something special, the drivers are already included.

The card might not have the correct firmware loaded, but that's not driver-specific. Another thing is that when you have the correct firmware you'd still need to setup the interfaces for them to appear in ip/ifconfig.
Click to expand...
Thanks! I am completely new to configuring interfaces for it. I can see the ports in proxmox, so in theory the correct driver is loaded and it is in ETH mode, but I don't know what is the next step or how to do it. What I see in the proxmox gui that the ports are not active, but they are recognized as Mellanox HP network card. Is there any tutorial about how to configure the interfaces? Another problem that I don't have any videocard, so I need something safe, because if I lose connection on the 1Gbps card, then I have no way atm. to get it back.
 
I

inf3rno

Member
Mar 24, 2021
36
1
8
I solved it meanwhile.
Needed

nano /etc/default/grub
Code: 
GRUB_CMDLINE_LINUX_DEFAULT=" pci=realloc=off"
And I added bridges for both ports:

nano /etc/network/interfaces
Code: 
auto lo
iface lo inet loopback

iface enp9s0 inet manual

iface wlp8s0 inet manual

auto enp10s0
iface enp10s0 inet static
        address 192.168.2.124/24

auto enp10s0d1
iface enp10s0d1 inet static
        address 192.168.2.125/24

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.124/24
        gateway 192.168.1.1
        bridge-ports enp9s0
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 192.168.2.124/24
        bridge-ports ensp10s0
        bridge-stp off
        bridge-fd 0
 
I

inf3rno

Member
Mar 24, 2021
36
1
8
Well it looks like I don't know what I am doing. Somebody wrote what I did is just doing memory - memory transfer and doesn't use the network cards at all. The main problem that they are not active in proxmox. :S
 
Stephan

Stephan

Well-Known Member
Apr 21, 2017
944
712
93
Germany
To test the link performance as a loop back on a single machine you need some trickery, otherwise Linux kernel will realize it is just talking to itself and will optimize transmit and receive over the external cable out ("memcpy"). See https://forums.servethehome.com/ind...um-p-8124-asrock-epc621d8a.35843/#post-349009 for one solution. Don't do this if you don't have out-of-band IPMI KVM or are not in same building. One typo or problem and you need a reset.
 
  • Like
Reactions: inf3rno
O

oneplane

Well-Known Member
Jul 23, 2021
845
484
63
You can also use ethtool to get more information about the ports if you want to check what the OS knows about the network card. It's possible it's simply not detecting any link, or it's configured for the wrong type of link.
 
I

inf3rno

Member
Mar 24, 2021
36
1
8
Stephan said:
To test the link performance as a loop back on a single machine you need some trickery, otherwise Linux kernel will realize it is just talking to itself and will optimize transmit and receive over the external cable out ("memcpy"). See https://forums.servethehome.com/ind...um-p-8124-asrock-epc621d8a.35843/#post-349009 for one solution. Don't do this if you don't have out-of-band IPMI KVM or are not in same building. One typo or problem and you need a reset.
Click to expand...
Thanks! Then I rather don't do it, I cannot reinstall it, the computer doesn't have a videocard or IPMI.
 
I

inf3rno

Member
Mar 24, 2021
36
1
8
tsteine said:
I notice you have the Infiniband/Ethernet variant. Is the port mode in the firmware set to Infiniband perhaps?

See
HowTo Change Port Type in Mellanox ConnectX-3 Adapter (nvidia.com)
Click to expand...
One of the ports is in IB mode, the other in ETH.
OFED tools don't exist, because afaik. proxmox has only inbox drivers. In the case of inbox driver the option 1 is working, but it is not persistent, the option 2 is not working, I don't have RDMA, I have not idea what it is or how to enable it.
 
Last edited:
I

inf3rno

Member
Mar 24, 2021
36
1
8
oneplane said:
You can also use ethtool to get more information about the ports if you want to check what the OS knows about the network card. It's possible it's simply not detecting any link, or it's configured for the wrong type of link.
Click to expand...
Thanks! I tried ethtool, the following is the result:

Code: 
Settings for enp10s0:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseKX4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                56000baseCR4/Full
                                56000baseSR4/Full
                                1000baseX/Full
                                10000baseCR/Full
                                10000baseSR/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseKX4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                1000baseX/Full
                                10000baseCR/Full
                                10000baseSR/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  40000baseCR4/Full
                                             56000baseCR4/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 40000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: no
Code: 
Settings for enp10s0d1:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseKX4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                56000baseCR4/Full
                                56000baseSR4/Full
                                1000baseX/Full
                                10000baseCR/Full
                                10000baseSR/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseKX4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                1000baseX/Full
                                10000baseCR/Full
                                10000baseSR/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  40000baseCR4/Full
                                             56000baseCR4/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 40000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: no
 
I

inf3rno

Member
Mar 24, 2021
36
1
8
When put both in ETH mode and do the following:
Code: 
ip link set dev enp10s0 up
ip link set dev enp10s0d1 up
then ethtool writes that "link detected". Does this mean that it works properly?

I have another weird thing in dmesg:

Code: 
[    1.092182] mlx4_core: Mellanox ConnectX core driver v4.0-0
[    1.092213] mlx4_core: Initializing 0000:0a:00.0
[    9.512279] mlx4_core 0000:0a:00.0: DMFS high rate steer mode is: disabled performance optimized steering
[    9.512482] mlx4_core 0000:0a:00.0: 15.752 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:03.1 (capable of 63.008 Gb/s with 8.0 GT/s PCIe x8 link)
[    9.613786] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
So according to this only 2 PCIE lanes are used, which is weird, because it is in a fully operating PCIE x16 slot. Maybe I am wrong, but this cards appears to be dead if this is true. I mean I sold my old 10Gbps cards to replace them with 40 or 56Gbps, not with 15Gbps...
 
I

inf3rno

Member
Mar 24, 2021
36
1
8
Stephan said:
To test the link performance as a loop back on a single machine you need some trickery, otherwise Linux kernel will realize it is just talking to itself and will optimize transmit and receive over the external cable out ("memcpy"). See https://forums.servethehome.com/ind...um-p-8124-asrock-epc621d8a.35843/#post-349009 for one solution. Don't do this if you don't have out-of-band IPMI KVM or are not in same building. One typo or problem and you need a reset.
Click to expand...
I tried this, it writes only 7.6 Gbps SUM. Looks like the x2 speed is really true with this card or it is even slower. Not sure how that happened, or why it doesn't use all 8 lanes.
 
You must log in or register to reply here.
Top Bottom