Proxmox issues with SR-IOV and Mellanox Connectx-3

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

skelleton

Member
Nov 4, 2015
47
10
8
124
I hope this is the right board for this issue, if not feel free to move the topic over to the Linux board.

I am using Proxmox 4.1 with all current updates and today I encountered an issue that I can't quite explain.
I enabled SR-IOV on my connectx-3 card and after the reboot I was not able to access my VMs from the network.
I was able to reach the Proxmox Host without any issues. From the Promox host itself I was was able to ping the VMs and could not see any problems with that
The VMs had issues reaching each other, there was often excessive packet loss or not connection at all.

All VMs were connected over a Linux bridge that is on one port of the Mellanox card. Proxmox also uses this bridge as its only connection into my network.

Once I disable the SR-IOV again there are no issues.
I used following kernel module options to enable SR-IOV(as per documentation the Mellanox site):
options mlx4_core port_type_array=2,2 num_vfs=0,8,0 probe_vf=0,8,0

I also tried putting the Virtual Functions on the other Port of the card and the result was the same.


I almost forgot:
- The card is connected to a DLINK DGS1510-28x that is still new and running in default configuration.
- lspci shows SR-IOV capabillities for the card
- when SR-IOV is enabled I can see the virtual functions of the card in lspci so it seems to be working, did not test further due to above mentioned issue.