Need help with Mellanox InfinBand SR-IOV and RDMA

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Ted Sinabian

New Member
Nov 19, 2015
9
0
1
47
Hi,

We are currently trying to install and configure SR-IOV on ESXi 5.5 hosts. So far so good, I can see the VF's and I can assign them to the VM's but here comes the problem. The cards show yellow exclamation marks and won't activate. The error code in windows device manager is 10....

Goal: To assign VF's to the VM's and enable RDMA on VM's

Drivers installed on the ESxi is OFED 2.3.3.1 and they are working
Firmware is the latest from HP (the IB cards are HP OEM) Part nr: 764285-B21 firmware: 2.34.5000

Any help is appreciated!
 

33_viper_33

Member
Aug 3, 2013
204
3
18
I have been out of the community for awhile due to a new, very demanding job. I know as of a year ago, the drivers didn't support SR-IOV in ESXi. I'm both curious and hopeful that this has changed.
 

Ted Sinabian

New Member
Nov 19, 2015
9
0
1
47
Then I have good news for you. It is supported now but not for ESXi 6.0. Planned release for ESXi 6.x is Q2 2016.
 

33_viper_33

Member
Aug 3, 2013
204
3
18
Cool! If I have time this weekend, I'll reinstall ESXi 5.5 and see if I can get it working with my CX3 cards.
 

Ted Sinabian

New Member
Nov 19, 2015
9
0
1
47
You need to uninstall the inbox drivers and install ofed 2.3.3.1. This is the version that supports SR-IOV and RDMA (according to Mellanox support)
 

33_viper_33

Member
Aug 3, 2013
204
3
18
Thanks Ted, I'll take a look over the weekend and see what I get. I'm familiar with the process and did quite a bit of testing awhile back. Just haven't had time to do my hobbies.... I'll let you know what I experience. does it work using VT-d pass-through?
 

33_viper_33

Member
Aug 3, 2013
204
3
18
If VT-d Passthrough works, we can narrow it down to an SR-IOV function. Is the windows install a bare metal install or a VM?
 

Ted Sinabian

New Member
Nov 19, 2015
9
0
1
47
If VT-d Passthrough works, we can narrow it down to an SR-IOV function. Is the windows install a bare metal install or a VM?
It works with regular passthrough without VF enabled. I've even tried RoCE and it also works great.
The Windows install is a VM.
 

Ted Sinabian

New Member
Nov 19, 2015
9
0
1
47

Ted Sinabian

New Member
Nov 19, 2015
9
0
1
47
Are there any errors in your logs under ESXi? How about Windows?
Version=1
EventType=PnPDeviceProblemCode
EventTime=130924964528377393
Consent=1
ReportIdentifier=009bf7b1-8f83-11e5-80c6-005056973b84
Response.type=4
Sig[0].Name=Architecture
Sig[0].Value=x64
Sig[1].Name=Hardware Id
Sig[1].Value=MLX4\ConnectX-3_vEth
Sig[2].Name=Setup class GUID
Sig[2].Value={4d36e972-e325-11ce-bfc1-08002be10318}
Sig[3].Name=PnP problem code
Sig[3].Value=0000000A
Sig[4].Name=Driver name
Sig[4].Value=mlx4eth63.sys
Sig[5].Name=Driver version
Sig[5].Value=5.10.11345.0
Sig[6].Name=Driver date
Sig[6].Value=09-10-2015
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=6.3.9600.2.0.0.272.7
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=1053
FriendlyEventName=Could not load driver software
ConsentKey=PnPDeviceProblemCode
AppName=Mellanox ConnectX-3 Virtual Function Ethernet Adapter
AppPath=C:\Windows\System32\drvinst.exe
ReportDescription=Windows was able to successfully install device driver software, but the driver software encountered a problem when it tried to run. The problem code is 10.
ApplicationIdentity=00000000000000000000000000000000
 

Ted Sinabian

New Member
Nov 19, 2015
9
0
1
47
Ok, it's solved. The drivers only support SR-IOV and RDMA in InfiniBand mode for Windows guest VM's. Ethernet is supported on Linux dists.
I need to use a gateway to communicate with my ethernet network. It's going to be interesting to see how this works!
 

33_viper_33

Member
Aug 3, 2013
204
3
18
Ive used PFSense in a VM to act as a gateway in the past. My SX6036 isn't licensed for gateway... ESXi Virtual Switch and PFSense is an acceptable workaround for me.

I didn't have a chance to play with the server over the weekend. This news only makes me more excited. I shall try with ESXi 6...
 

Ted Sinabian

New Member
Nov 19, 2015
9
0
1
47
Ive used PFSense in a VM to act as a gateway in the past. My SX6036 isn't licensed for gateway... ESXi Virtual Switch and PFSense is an acceptable workaround for me.

I didn't have a chance to play with the server over the weekend. This news only makes me more excited. I shall try with ESXi 6...
Using PFSense as GW is very interesting! I'll definitely try it.

Regarding ESXi 6. It's not supported, only ESXi 5.5 is supported with OFED 2.3.3.1 and 4.95 drivers for Windows and 2.4 drivers for Linux dist. Support for ESXi 6.x will come in Q2 2016 according to Mellanox.
 

33_viper_33

Member
Aug 3, 2013
204
3
18
Using PFSense as GW is very interesting! I'll definitely try it.

Regarding ESXi 6. It's not supported, only ESXi 5.5 is supported with OFED 2.3.3.1 and 4.95 drivers for Windows and 2.4 drivers for Linux dist. Support for ESXi 6.x will come in Q2 2016 according to Mellanox.

Is it not working or just not supported? I'm surprised there would be that big of a difference that it wouldn't work with some manhandling..
 

Ted Sinabian

New Member
Nov 19, 2015
9
0
1
47
Is it not working or just not supported? I'm surprised there would be that big of a difference that it wouldn't work with some manhandling..

I think it's not working; I'll leave that one up to you. Maybe you can get it to work with some "man handling" (aka hammer?) hehe


Do you know if it's possible to run subnet manager in VM's? I'm trying to run SM in a VM with a VF as IB nic. Unfortunately, the gateway can't be a GW and SM at the same time.
 

33_viper_33

Member
Aug 3, 2013
204
3
18
I've done it 2ways in the past. I either let the ESXi driver be the SM or set up my physical Windows server to be an am, usually both for redundancy. I'm sure a VM could be set up as SM, especially when using pass through. I have a mellanox sx 6036 that handeles SM duties these days. I use a PFsense VM for the gateway.