Quick followup - after a kernel update and a reboot I'm unable to get separate IOMMU grouping for the CX2 VFs on two machines unless I use the ACS hack (pcie_acs_override=downstream added to the boot command line.)
This occurs on two machines, one with recent Mellanox OFED drivers and one with the "in-box" linux kernel driver. I can't tell if this is a Mellanox driver issue or if the motherboard doesn't entirely support ACS. I thought it'd be five minutes to get SR-IOV working on an I350 and test but it's taking a bit longer.
Passthrough works great when there's solid driver support, which for Mellanox cards means it works great on Linux hosts with Linux guests as long as everyone has very recent kernels. Literally everything else is a crapshoot, I can bring up a passed through VF on FreeBSD, but only on FreeBSD 12 -- support for Mellanox VFs doesn't exist in the pre 12 driver -- and the device isn't freed on VM shutdown, so I get one shot per VF per boot. Not great for a reliable pfSense host
Good news: I get full line rate in a Ubuntu 18.04 guest using the stock linux kernel mellanox driver with a VF. CPU load is no higher than if iperf were running on the host itself. VF comes up as normal on boot, releases as expected on VM shutdown and is available for re-use if the VM is cycled multiple times. Basically Linux guests on a Linux host is *awesome* as long as the VF drivers are reliable.
I really want another vendor card to test with, but I'm not sure what my inexpensive dual port 10GbE + solid SR-IOV options are, I was going to start researching that today (there has to be something no one is using that's cheap on eBay.)
@WANg You mentioned your solarflare cards were overheating earlier, did you find the manual fan minimum speed option in the bios? It's under one of the headings on the second tab from the right, Boot Options or Power or something like that. My 40Gb Mellanox cards were running 60C+ so I cranked fan speed up to 4 or 5 (it's actually kind of loud) and now they're holding steady around 45-55C.
I think my long term plan with these high performance VPI cards involves finding a 10mm thick 5V fan that's moderately quiet and soldering it on to a USB plug and (ab)using the internal USB3 port. Another cool idea

cool

would be to use an Arduino to connect to the i2c header on the nic and act as a fan controller depending on card temp.
I'm going to try and get SR-IOV working on this I350 so I can figure out if the IOMMU grouping issue is motherboard or mellanox related.