Can't get passthrough to work on esxi 6.0 with connectx-3

Discussion in 'VMware, VirtualBox, Citrix' started by DavidVermillion, May 31, 2016.

  1. DavidVermillion

    Joined:
    Sep 22, 2015
    Messages:
    60
    Likes Received:
    17
    I can't get my Mellanox ConnectX-3 cards to passthrough to esxi VMs on my Intel C2600CP. It's added to the list of Passthrough devices, and I've got it assigned to the VM, but it never successfully initializes. In both FreeBSD and UbuntuServer guest it complains about being unable to assign an IRQ.

    My current suspicion is that it's related to the PCI reset. I've updated to the latest firmware for the NIC and messed around in the /etc/vmware/passthru.map with all the different reset methods (flr,d3d0,link,bridge) but I still get no joy.

    I'm about ready to give up, as everything I've seen on the internet leads to a dead end. Does anyone have any ideas? Or at least have an example of this working with similar hardware?
     
    #1
  2. CreoleLakerFan

    CreoleLakerFan Active Member

    Joined:
    Oct 29, 2013
    Messages:
    466
    Likes Received:
    169
    I'm not familiar with your motherboard, but I have had issues in the past with a Supermicro board and pass-through because the PCI slot I was using was behind a bridge which split 16 lanes into two 8x slots. Check the block diagram of the board for a bridge or switch between the slot you are using and the CPU.
     
    #2
  3. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,764
    Likes Received:
    858
    Or try another PCI-e slot 'maybe'...grasping at straws here, ya got another 10G nic ya can throw in to test, cheap Intel X520-DA1 or similar?
     
    #3
  4. DavidVermillion

    Joined:
    Sep 22, 2015
    Messages:
    60
    Likes Received:
    17
    Thanks for the suggestions. I tried another PCI slot and checked the diagram and got the same behavior. Sadly, I don't have any other 10gb nics I can use.

    Perhaps I'll try again later. I have a feeling this is probably a Mellanox problem.
     
    #4
  5. groove

    groove Member

    Joined:
    Sep 21, 2011
    Messages:
    63
    Likes Received:
    10
    Hi David,

    Were you able to resolve this issue ? I was able to get the cards to passthrough on a Tyan S5070, but performance drops to a point that make it completely unusable.

    Even a simple ping has very high latency :
    This is between 2 ConnectX-3 cards connected directly to each other. Both of them are installed on hosts running esxi 6.0 u2. On one host the card is passthrough to a Solaris 11.3 vm and the other is configured as a VMKernel. If I configure both cards as VMKernel then the ping latency is within a few uS (~ .500 ms).

    I'm going to try to passthrough this card to a Linux VM and check the performance.

    Could it be a Mellanox driver issue again ?
     
    #5
  6. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,764
    Likes Received:
    858
    Do you have latest firmware for the CX3 card applied? I found some anomalies when old FW was used.
     
    #6
  7. groove

    groove Member

    Joined:
    Sep 21, 2011
    Messages:
    63
    Likes Received:
    10
    Thanks for the reply whitey - yes, both cards have the latest firmware that I could find for them - well there are 2 different ConnectX-3 cards - one is a 'Mellanox ConnectX-3 FDR InfiniBand + 40GigE MCX354A - that's got f/w ver 2.40.5000. This is one that I'm trying to pass through to the Solaris VM. The other is a MCX353A-QCAT - and that has f/w ver 2-33-5100 (the latest I could find) - this one is being used by the VMKernel on the second esxi 6.0 node.

    I did get a chance to test the MCX354A by passing it through to a Linux VM and it worked fine (ping latency was down to ~ 0.600 ms). So it's just an issue when pass through the Solaris VM.

    As of now I have the MCX354A attached to a non-VMKernel switch and the Solaris vm is using it through a VMXNET3. I would like to get it pass through directly to the Solaris VM so that I can test iSer / SRP between the second ESXi host and the Solaris VM.
     
    #7
Similar Threads: Can't passthrough
Forum Title Date
VMware, VirtualBox, Citrix ESXi 6.7 can't change settings, don't survive reboot (passthrough issues, too!) May 1, 2018
VMware, VirtualBox, Citrix Can't get Ubuntu to install in ESXI 6 Sep 11, 2016
VMware, VirtualBox, Citrix ESXi 5.5 - Can't connect to vSphere Data Protection Appliance Apr 8, 2015
VMware, VirtualBox, Citrix Can't shut down ESXi 5.5 host fully Feb 5, 2014
VMware, VirtualBox, Citrix Audio Problem - nVidia 8400GS Passthrough - Windows 7 - ESXi 6.7.0 U1 Nov 23, 2019

Share This Page