SR-IOV with Supermicro x9scm-f & Connectx-2?

[Darkytoo]

I didn't know if this was even possible. I ended up upgrading the motherboard to a supermicro after seeing their FAQ on their site stating that all x9 boards support SR-IOV. I installed the motherboard and then I re-burned the firmware with some additional switches for sriov support that I found on the mellanox site, and then when I do a get-vmhost |fl on the host, I get the following error:

{To use SR-IOV on this system, the system BIOS must be updated to allow
Windows to control PCI Express. Contact your system manufacturer for an
update., SR-IOV cannot be used on this system as the PCI Express hardware
does not support Access Control Services (ACS) at any root port. Contact
your system vendor for further information.}

I have vt-d enabled in the BIOS, which has been updated to the latest and greatest and the error will not go away. I contacted supermicro support and one support agent says it will not work, the other agent said it will work just not the onboard nics. Has anyone gotten this to work?

 

[dexvx]

To my knowledge, none of the desktop based boards (with the exception of the X79) properly support SR-IOV. VT-d is a requirement for direct I/O passthrough. The requirements of SR-IOV are VT-d along with ACS and proper Interrupt Remapping support (e.g. a superset of VT-d).

If you want SR-IOV, it is best to get a Romley (C6xx) based board. For work purposes, the primary systems I use SR-IOV on are the Dell PowerEdge 12G (R620/R720). The 11G PowerEdge (R410/710) support SR-IOV with revision C2 chipset and later, but you need to do a security override on Hyper-V. I've also tried the Supermico X9DAi that works. The Intel Copper Pass (S2600CO) has PCI-E compliance issues, but it will mostly work.

However, these are all with Intel based network adapters (82576, 82599 (X520), i350, and X540).

 

[Darkytoo]

Interesting. I have 2 Xeon e5 boards that give me no issue with sr-iov, it's only the two x9scm boards that do. The hard part of this is supermicro support loves to contradict themselves. I've had one engineer tell me no, and another engineer tell me that it will, but the adapters on it don't. From reading through the site, it appears it will support it as long as the card is plugged into a CPU PCIe slot, not a PCH PCIe slot, which my connectX card is plugged into. My suspicion is that SR-IOV has not officially been enabled in the windows drivers as it seems to work under linux and vmware fine.

 

[dexvx]

All Xeon E5 boards are based on the Romley or Tylersburg chipset, which should support SR-IOV (again, it depends on what the OEM does with the BIOS). Also, VM-Ware and KVM/Xen are much more lenient on allowing SR-IOV to work. Hyper-V has checks for proper VT-d Interrupt Remapping and IOMMU support. You can check this article here:

Everything you wanted to know about SR-IOV in Hyper-V Part 8 - John Howard - Senior Program Manager in the Hyper-V team at Microsoft - Site Home - TechNet Blogs

It boils down to the the fact that pre-Romley systems have hardware errata that may allow a malicious VF device to crash the host - unless the NIC has some kind of MDD (Malicious Driver Detection) at the hardware level.

Also, the Supermicro engineer is correct about PCH SR-IOV. Currently, no PCH based PCI-E slots will do SR-IOV. This is due to lack of ACS support on those slots. Even a server PCH is just the client PCH die plus some additional server specific components..