SR-IOV for Mellanox ConnectX-2

[SebastianJ]

I've not dealt with anything but Intel cards in the past so I picked up a Mellanox to play with. Is this real life?

These are the official instructions to enable SR-IOV: HowTo Configure SR-IOV for ConnectX-3 with KVM ... | Mellanox Interconnect Community

I find it hard to believe that one needs to install so much shit to get this working?! What a nightmare.

Does anyone have a normal sane way to do it without all their crap?

 

[SebastianJ]

I'm very surprised anyone recommends this card at all. That being said, if someone else is stuck with this card you have to install Mellanox crap at least once to write the firmware. Once you write the firmware it should work with the regular kernel modules for most purposes.

These two posts were very useful in my struggle:
Setting SR-IOV num_vfs for ConnectX-2 card - Setting SR-IOV num_vfs for ConnectX-2 card | Mellanox Interconnect Community

1. Enable SRIOV in BIOS.
2. Enable SRIOV in Linux Kernel:

Activate Intel VT-d in the kernel by appending the intel_iommu=on parameter to the kernel line of the kernel line in the /boot/grub/grub.conf file.​

3. Enable SRIOV in the firmware:
A. Copy the INI file locally or dump it from the device by running flint -d <mstdevice> dc >tmp/ini.ini.
B. Edit the INI file and add these params under HCA :
[root@]# vim /tmp/MCX354A-FCB_A1_SRIOV.ini
--> Add the following lines under [HCA] section:
num_pfs = 1
total_vfs = 64
sriov_en = true
C. reburn the FW by running mlxburn -d mstdevice -fw <fw.mlx> -c /tmp/ini.ini
4. Enable SRIOV in the driver:

A. Create the file /etc/modprobe.d/mlx4_core.conf and add the following content:

options mlx4_core num_vfs=64 port_type_array=2,2 (which will open the driver with 64 VFs and Port1 = eth, port2=eth)
If you need IB port_type shall be = 1​

B. Restart the driver.​

5. Reboot the Machine.
6. How to check that SRIOV is working: Use LSPCI.​

Also some good info here: More custom Mellanox firmware tips

If your card is unsupported (older than x-3) then you must do all customizations direct in the ini and write it to the device.

That being said. Don't buy Mellanox.

 

[Rand__]

So have you tried running open-sm on a sr-iov'ed vf yet?

 

[_alex]

i tried with my cards, too some month ago and found that SR-IOV will only work if you set the ports to ethernet for connectx-2, and with customized/modified Firmware. Not sure if opensm makes sense then, besides you go roce then.
Official support for sr-iov i guess was introduced from the connectx-3 onwards.

 

[Rand__]

Ah ok.
Then I'll have to try for myself, i got some x3 cards but have not gotten around testing the adjusted driver
If that would work it would be a neat way to run opensm without the need for a managed switch

 

[_alex]

with the connectx-3 it should be way easier and might work when ports are in ib-mode. Good luck, and curious to hear how in turned out in the end.