Inexpensive LGA1150 motherboard w/ advanced virtualization features

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
245
43
Can anyone recommend a relatively inexpensive LGA1150 board with the full suite of SR-IOV niceties? I'm looking for something that supports ARI forwarding and ACS if possible.

I'd prefer ITX but I'll take any rec. at this point, I've tested on a handful of workstation platforms and none of them have ARI/ACS support.
 

fossxplorer

Active Member
Mar 17, 2016
556
98
28
Oslo, Norway
Just a head up, already mentioned by another member here ( i can't remember which thread) if you gonna work on V3 E3s of LGA1150 platform.
The Haswell based Xeon E3 V3s, have a bug related to ACS/SR-IOV/VT-d: https://www.intel.com/content/dam/w...cation-updates/xeon-e3-1200v3-spec-update.pdf
"It is recommended to avoid device direct assignment to Virtual Machines in virtualized
environments with this processor due to the lack of Access Control Services (ACS)
support in PCI-Express root ports. Some Operating Systems may check for ACS support
and potentially disable direct device assignment (that is, affects SR-IOV setup/
configuration within the server) as well."
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
245
43
Sigh, of course they do. Thanks for the heads up.

Do you know if the Haswell i7s have the same issue? IIRC SR-IOV was supported on the high end consumer chips as well, but i'm having a really hard time finding platforms that actually work.
 
Last edited:

zir_blazer

Active Member
Dec 5, 2016
358
129
43
Intel Processors based on the consumer platforms (Core i5/i7, Xeon E3) do NOT support ACS in the Processor integrated PCIe Controller. If you were to use a SR-IOV NIC, Linux by default will build the IOMMU Groups with both the NIC PF and VFs as part of the same group (Even if it is a single card and you don't bifurcate to 8x/8x), which means that unless you use the ACS patch (Considered an unsafe hack), you can't passthrough a NIC VF to a VM using QEMU (Other Hypervisors should present similar issues). You can technically get full functionally if you use Chipset PCIe Slots since they do support ACS, at the cost of latency and perhaps bandwidth since Chipset slots with 4 lanes are rare (I never saw more than 4 lanes in a Chipset slot and I don't know if it is technically possible).
Basically, you need a LGA 2011/2011-3/2066 platform if you want full features. You also get APICv. Core i7 HED works, but avoid Skylake-X, that while I don't recall seeing proof that they don't have ACS/APICv support in LGA 2066 but I wouldn't risk it anyways since logic says they don't support it since its the same silicon.
 
  • Like
Reactions: fossxplorer

fossxplorer

Active Member
Mar 17, 2016
556
98
28
Oslo, Norway
Great insight that i didn't know before now!
Do you know anything about AMD CPUs? Ryzen vs Threadripper vs Epyc (or older ones in HP T620 GX-420CA and T730 RX-427BB)?
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
245
43
Intel Processors based on the consumer platforms (Core i5/i7, Xeon E3) do NOT support ACS in the Processor integrated PCIe Controller. If you were to use a SR-IOV NIC, Linux by default will build the IOMMU Groups with both the NIC PF and VFs as part of the same group (Even if it is a single card and you don't bifurcate to 8x/8x), which means that unless you use the ACS patch (Considered an unsafe hack), you can't passthrough a NIC VF to a VM using QEMU (Other Hypervisors should present similar issues). You can technically get full functionally if you use Chipset PCIe Slots since they do support ACS, at the cost of latency and perhaps bandwidth since Chipset slots with 4 lanes are rare (I never saw more than 4 lanes in a Chipset slot and I don't know if it is technically possible).
Basically, you need a LGA 2011/2011-3/2066 platform if you want full features. You also get APICv. Core i7 HED works, but avoid Skylake-X, that while I don't recall seeing proof that they don't have ACS/APICv support in LGA 2066 but I wouldn't risk it anyways since logic says they don't support it since its the same silicon.
Ah, thanks! I went back and re-read the post discussing support on the i7's and it referenced the high end LGA 2011 chips, that's where my confusion came from.