Proxmox SR-IOV for ConnectX3

[acmcool]

Anyone using mellanox connectX3 40gbps cards in mellanox? I am trying to get SR-IOV working..
Not able to mst to start..
Getting following..

Starting MST (Mellanox Software Tools) driver set
Loading MST PCI modulemodprobe: FATAL: Module mst_pci not found in directory /lib/modules/4.10.17-2-pve
- Failure: 1
Loading MST PCI configuration modulemodprobe: FATAL: Module mst_pciconf not found in directory /lib/modules/4.10.17-2-pve
- Failure: 1
Create devices

mst_pci driver not found
Unloading MST PCI module (unused)modprobe: FATAL: Module mst_pci not found.
- Failure: 1
Unloading MST PCI configuration module (unused)modprobe: FATAL: Module mst_pciconf not found.
- Failure: 1
root@proxmox1:~#

 

[acmcool]

This issue is now resolved..Driver was not installed correctly..
Now off to configuring SR-IOV

 

[_alex]

nice, i'm also playing with sr-iov with them on proxmox.
do you passthrough vf's, and if so have a quick guide how to?
would like to have ib in some vm's, but not the whole hca or a whole port...

 

[acmcool]

nice, i'm also playing with sr-iov with them on proxmox.
do you passthrough vf's, and if so have a quick guide how to?
would like to have ib in some vm's, but not the whole hca or a whole port...

I could not get it to work.So for now gave up on this and moved this host back to ESXI

 

[_alex]

Oh, sad to hear.
Will give it a try, i got sri-iov working for Ports in EN-Mode and saw separate pcie-devices, too.
So passthrough of one of them to a VM shouldn't be that hard ?
Will give it a try and post if it works.

 

[acmcool]

Oh, sad to hear.
Will give it a try, i got sri-iov working for Ports in EN-Mode and saw separate pcie-devices, too.
So passthrough of one of them to a VM shouldn't be that hard ?
Will give it a try and post if it works.

Sure let me know

 

[acmcool]

Oh, sad to hear.
Will give it a try, i got sri-iov working for Ports in EN-Mode and saw separate pcie-devices, too.
So passthrough of one of them to a VM shouldn't be that hard ?
Will give it a try and post if it works.

Hey any luck?

 

[_alex]

not yet, was busy mounting my mlx switches on rails that arrived today, rearranging nodes and switches in the rack, recabling the whole fabric and stuff like this today.
guess i'll get ist sorted / tested by wednesday evening. You want to use ib or en vf's? i'll test both, so could begin with the part you are interested in.

 

[acmcool]

not yet, was busy mounting my mlx switches on rails that arrived today, rearranging nodes and switches in the rack, recabling the whole fabric and stuff like this today.
guess i'll get ist sorted / tested by wednesday evening. You want to use ib or en vf's? i'll test both, so could begin with the part you are interested in.

I am using them in ethernet mode.. Looking to use VF's

 

[_alex]

ok, that might be faster, have two nodes that have EN vf's already configured and visible in lspci. currently one of the vf's on the hca's is already in use for vxlan - so they basically work.

how far did you make it ?
cards setup?
iommu enabled in BIOS + Kernel Parameter added?
kernel-module configured?
vf's visible in lspci?

 

[acmcool]

I was able to see vf's in lspci..
Then i added to passthrough to my windows VM...Windows would not boot..went to BSD..

 

[acmcool]

i had this in in VM config...
Machine=Q35
hostpci0: 02:00.1,pcie=1

 

[_alex]

ok, will give it a try to passthrough one of the vf, but have no windows vm but should be the same passing to Linux.

Any errors when the vm started/tried another guest-os?

 

[acmcool]

No...only tried Windows VM

 

[_alex]

No Problem with Debian, tried it in a jessie - vm i had at hand, both worked:

hostpci0: 02:00.1

and

machine: q35
hostpci0: 02:00.1,pcie=1

OS recognizes the nic and loads mlx4_en
Also dmesg shows 'Detected virtual function - running in slave mode'
What is a bit odd is the fact that it assigned the VF to eth0, but can live with this.
Also ethtool shows the right drivers and speeds.

Not done any more testing, benchmarking and such.

 

[acmcool]

No Problem with Debian, tried it in a jessie - vm i had at hand, both worked:

hostpci0: 02:00.1

and

machine: q35
hostpci0: 02:00.1,pcie=1

OS recognizes the nic and loads mlx4_en
Also dmesg shows 'Detected virtual function - running in slave mode'
What is a bit odd is the fact that it assigned the VF to eth0, but can live with this.
Also ethtool shows the right drivers and speeds.

Not done any more testing, benchmarking and such.

mine worked with debian as well..
windows still bsd..

Sent from my SM-G955U using Tapatalk

 

[_alex]

ok, strange ...
sorry, but can't help with windows, but maybe someone else knows where to look at.

 

[acmcool]

Anyone else tried this with windows?
I tried Windows 10. Now will try with windows 2016 server

 

[gigatexal]

subd, this is really cool.

 

[_alex]

besides i have no idea why it doesn't work with windows i managed to get 4vf ib on port1 and 4vf en on port2 concurrently. will try to pass one of each to my vm tomorrow.
in the end i'd like to have 2x ib + 2x en, from two hca for mpio plus an hba in that vm.