SR-IOV for Mellanox ConnectX-2

SebastianJ

New Member
Dec 3, 2016
3
0
1
I'm very surprised anyone recommends this card at all. That being said, if someone else is stuck with this card you have to install Mellanox crap at least once to write the firmware. Once you write the firmware it should work with the regular kernel modules for most purposes.

These two posts were very useful in my struggle:
Setting SR-IOV num_vfs for ConnectX-2 card - Setting SR-IOV num_vfs for ConnectX-2 card | Mellanox Interconnect Community
1. Enable SRIOV in BIOS.
2. Enable SRIOV in Linux Kernel:
Activate Intel VT-d in the kernel by appending the intel_iommu=on parameter to the kernel line of the kernel line in the /boot/grub/grub.conf file.​
3. Enable SRIOV in the firmware:
A. Copy the INI file locally or dump it from the device by running flint -d <mstdevice> dc >tmp/ini.ini.
B. Edit the INI file and add these params under HCA :
[root@]# vim /tmp/MCX354A-FCB_A1_SRIOV.ini
--> Add the following lines under [HCA] section:
num_pfs = 1
total_vfs = 64
sriov_en = true
C. reburn the FW by running mlxburn -d mstdevice -fw <fw.mlx> -c /tmp/ini.ini
4. Enable SRIOV in the driver:
A. Create the file /etc/modprobe.d/mlx4_core.conf and add the following content:
options mlx4_core num_vfs=64 port_type_array=2,2 (which will open the driver with 64 VFs and Port1 = eth, port2=eth)
If you need IB port_type shall be = 1​
B. Restart the driver.​
5. Reboot the Machine.
6. How to check that SRIOV is working: Use LSPCI.​

Also some good info here: More custom Mellanox firmware tips

If your card is unsupported (older than x-3) then you must do all customizations direct in the ini and write it to the device.

That being said. Don't buy Mellanox.
 

_alex

Active Member
Jan 28, 2016
874
96
28
Bavaria / Germany
i tried with my cards, too some month ago and found that SR-IOV will only work if you set the ports to ethernet for connectx-2, and with customized/modified Firmware. Not sure if opensm makes sense then, besides you go roce then.
Official support for sr-iov i guess was introduced from the connectx-3 onwards.
 

Rand__

Well-Known Member
Mar 6, 2014
6,086
1,460
113
Ah ok.
Then I'll have to try for myself, i got some x3 cards but have not gotten around testing the adjusted driver:)
If that would work it would be a neat way to run opensm without the need for a managed switch
 

_alex

Active Member
Jan 28, 2016
874
96
28
Bavaria / Germany
with the connectx-3 it should be way easier and might work when ports are in ib-mode. Good luck, and curious to hear how in turned out in the end.