HP t730 Thin Client as an HP Microserver Gen7 Upgrade

[WANg]

I'm actually testing right now to determine whether something that popped up is a Mellanox driver issue or a motherboard iommu support issue, i didn't want to jump the gun without testing a second vendor. I'll post in a few minutes.

Edit: search eBay for an HP nc365t, they're as cheap as $15 and are geniune I350 hardware.

Oh oh. It's not actually sending/receiving packets in the Ubuntu VF, is it?
Thanks for the heads-up - just ordered one for about $25 including shipping. I'm going to pre-load Proxmox on this M.2 and then see how far this rabbit-hole goes. Let the insanity begin!

 

[arglebargle]

Quick followup - after a kernel update and a reboot I'm unable to get separate IOMMU grouping for the CX2 VFs on two machines unless I use the ACS hack (pcie_acs_override=downstream added to the boot command line.)

This occurs on two machines, one with recent Mellanox OFED drivers and one with the "in-box" linux kernel driver. I can't tell if this is a Mellanox driver issue or if the motherboard doesn't entirely support ACS. I thought it'd be five minutes to get SR-IOV working on an I350 and test but it's taking a bit longer.

I did a bunch of passthrough testing but it's all basically mellanox driver support stuff and I don't want to hijack your thread with it beyond a tl;dr. Passthrough works great when there's solid driver support which for Mellanox cards means it works great on Linux hosts with Linux guests as long as everyone has very recent kernels. Literally everything else is a crapshoot, I can bring up a passed through VF on FreeBSD, but only on FreeBSD 12 -- support for Mellanox VF guests doesn't exist in the pre 12 driver -- and the device isn't properly freed on VM shutdown and remains unusable after that. So I get one shot per VF per boot, not great for a reliable pfSense host

Good news: I get full line rate from a Ubuntu 18.04 guest VF using the stock linux kernel mellanox driver. CPU load is no higher than if iperf were running on the host itself; VF comes up as normal on boot, releases as expected on VM shutdown and is available for re-use if the VM is cycled multiple times. Basically Linux VF guests on a Linux host are *awesome* as long as the drivers handle VFs reliably.

I really want another vendor card to test with, but I'm not sure what my inexpensive dual port 10GbE + solid SR-IOV options are, I was going to start researching that today (there has to be something no one is using that's cheap on eBay.)

@WANg You mentioned your solarflare cards were overheating earlier, did you find the manual fan minimum speed option in the bios? It's under one of the headings on the second tab from the right, Boot Options or Power or something like that. My 40Gb Mellanox cards were running 60C+ so I cranked fan speed up to 4 or 5 (it's actually kind of loud) and now they're holding steady around 45-55C.

I think my long term plan with these high performance VPI cards involves finding a 10mm thick 5V fan that's moderately quiet and soldering it on to a USB plug and (ab)using the internal USB3 port. Another cool idea would be to use an Arduino to connect to the i2c header on the nic and act as a fan controller depending on card temp.

I'm going to try and get SR-IOV working on this I350 so I can figure out if the IOMMU grouping issue is motherboard or mellanox related.

 

[arglebargle]

Alright, I managed to get VF assignment working on the I350 and all of its devices are grouped into a single IOMMU grouping as well. I had to use pci=assign-busses on the kernel command line to even be able to assign VFs and I'm not sure if that skews the outcome.

I might test with a more recent kernel and see if anything has changed, I think I have Ubuntu on a USB stick on my desk somewhere.

 

[WANg]

Quick followup - after a kernel update and a reboot I'm unable to get separate IOMMU grouping for the CX2 VFs on two machines unless I use the ACS hack (pcie_acs_override=downstream added to the boot command line.)

This occurs on two machines, one with recent Mellanox OFED drivers and one with the "in-box" linux kernel driver. I can't tell if this is a Mellanox driver issue or if the motherboard doesn't entirely support ACS. I thought it'd be five minutes to get SR-IOV working on an I350 and test but it's taking a bit longer.

Passthrough works great when there's solid driver support, which for Mellanox cards means it works great on Linux hosts with Linux guests as long as everyone has very recent kernels. Literally everything else is a crapshoot, I can bring up a passed through VF on FreeBSD, but only on FreeBSD 12 -- support for Mellanox VFs doesn't exist in the pre 12 driver -- and the device isn't freed on VM shutdown, so I get one shot per VF per boot. Not great for a reliable pfSense host

Good news: I get full line rate in a Ubuntu 18.04 guest using the stock linux kernel mellanox driver with a VF. CPU load is no higher than if iperf were running on the host itself. VF comes up as normal on boot, releases as expected on VM shutdown and is available for re-use if the VM is cycled multiple times. Basically Linux guests on a Linux host is *awesome* as long as the VF drivers are reliable.

I really want another vendor card to test with, but I'm not sure what my inexpensive dual port 10GbE + solid SR-IOV options are, I was going to start researching that today (there has to be something no one is using that's cheap on eBay.)

@WANg You mentioned your solarflare cards were overheating earlier, did you find the manual fan minimum speed option in the bios? It's under one of the headings on the second tab from the right, Boot Options or Power or something like that. My 40Gb Mellanox cards were running 60C+ so I cranked fan speed up to 4 or 5 (it's actually kind of loud) and now they're holding steady around 45-55C.

I think my long term plan with these high performance VPI cards involves finding a 10mm thick 5V fan that's moderately quiet and soldering it on to a USB plug and (ab)using the internal USB3 port. Another cool idea cool would be to use an Arduino to connect to the i2c header on the nic and act as a fan controller depending on card temp.

I'm going to try and get SR-IOV working on this I350 so I can figure out if the IOMMU grouping issue is motherboard or mellanox related.

HM...Okay, thanks for the SRIOV probing - saves me some downtime on the t730.

First - when it comes to VF freeing - I just need clarification...This is assuming a Proxmox host and a FreeBSD guest, right? Once the VM shuts down don't you want the VF to stay associated with the guest VM and not returned back to the hypervisor for re-assignment? Or is it not freed in the sense that the VF is flagged as busy/used and blocks re-use on subsequent guest VM reboots, and you'll need a VM host reboot to re-allocate the VF? Also, this sounds like a bug that should be reported to the FreeBSD guys.

Second - a cheap SRIOV capable 10GbE card? SolarFlare SFN6122F - often sold by vendors on eBay as the S6102 (so it doesn't show up correctly on searches and therefore command lower prices), they are SRIOV compatible and should cost about $30/card. Make sure you ask for a card with low profile brackets. I don't see any mentions specifically indicating issues with SRIOV in FreeBSD.

Third - Eh, honestly, I didn't have too long to play around with the SolarFlare SFN7322 Flareon before I had to pull it from the chassis and substitute a SFN5122F in its place. I might push the fan RPM (and noise) up a bit and see how the card copes. I do not have great hopes, however. That card genuinely runs hot.

 

[arglebargle]

Not released as in the device remains locked in an unusable state until host reboot. Reloading the host driver might work too, I haven't tested. It's definitely a Mellanox driver issue and not something wrong with the t730 though.

 

[tomaash]

search eBay for an HP nc365t, they're as cheap as $15 and are geniune I350 hardware.

@WANg Beware, these are in fact Intel 82580 (i340) based and don't support SR-IOV.
HP 361T (dualport) and 366T (quadport) are I350 NICs and are somewhat more expensive.

 

[WANg]

Alright, I managed to get VF assignment working on the I350 and all of its devices are grouped into a single IOMMU grouping as well. I had to use pci=assign-busses on the kernel command line to even be able to assign VFs and I'm not sure if that skews the outcome.

I might test with a more recent kernel and see if anything has changed, I think I have Ubuntu on a USB stick on my desk somewhere.

Hm, strange - in SolarFlare (or I would imagine Mellanox) there is usually a utility that the vendor would provide that would allow you to turn on SRIOV, create PF/VFs and then assign them out. Doesn't Intel have a similar utility? That being said, if the I350 only support up to 8 VFs, it might not support that kind of configuration.

 

[Marsh]

What @tomaash said is true.

 

[arglebargle]

Oh man, I'm sorry. I mixed up the i340 and i350 models. I had it in my head that the nc364t was the i340 (it's not.) I'm ... apparently really bad with details

Hm, strange - in SolarFlare (or I would imagine Mellanox) there is usually a utility that the vendor would provide that would allow you to turn on SRIOV, create PF/VFs and then assign them out. Doesn't Intel have a similar utility? That being said, if the I350 only support up to 8 VFs, it might not support that kind of configuration.

VFs on the I350 are handled by kernel module parameters, it's literally `load igb num_vfs=n,n,n,n` and you're done. There was some issue with the bus (02?) the card was attached to being unavailable to allocate VFs without that pci kernel parameter.

 

[WANg]

Oh lord. Looks like my shopping around for an I350 opened up a can of worms involving counterfeit cards, which looks rather endemic. At this stage...maybe I should stick with SolarFlare SRIOV and wire it up to the Mikrotik (when I get around to ordering it). Looks like that I340-T4 card already shipped, but since it's only $30, I am not exactly going crazy over it.

 

[arglebargle]

Yeah, I dealt with that when I shopped for mine. The big tell (as far as I know) is whether the delta is embossed on the transformers, and the heatsink on a lot of the counterfeit cards is a bit shorter than normal.

This looks legit, but after my earlier mistake I'd appreciate it if someone would provide a second set of eyes:
Intel I350-T4 PCI-Express PCI-E Four RJ45 Gigabit Ports Server Adapter | eBay

I ended up paying around $27 for mine, though I did wait about a week for a cheap auction to pop up.

 

[TLN]

Hm.. that topic sounds interesting. Does this client support all the stuff for Vmware ESXI? VT-D, SR-IOV? Etc?

 

[arglebargle]

VT-d (or AMD's version) works but I'm struggling through IOMMU issues at the moment. IOMMU ACS doesn't seem to be fully supported without using the insecure ACS override patch and I'm not yet sure exactly why that is. It could be that we need chipset quirks added to support ACS or that the chipset doesn't support IOMMU ACS on the pcie port.

Using the patch I have SR-IOV VFs passed to Linux guests working at full line rate reliably and everything is stable over guest reboots. BSD support is on a per-vendor basis. I suspect Intel works and I don't know anything at all about the other vendors. Mellanox is glitchy, the guest VFs only survive one VM boot before they get stuck and don't work anymore.

 

[Patrick]

@WANg I am back but leaving again in 36 hours. What is this fun to join in?

 

[WANg]

VT-d (or AMD's version) works but I'm struggling through IOMMU issues at the moment. IOMMU ACS doesn't seem to be fully supported without using the insecure ACS override patch and I'm not yet sure exactly why that is. It could be that we need chipset quirks added to support ACS or that the chipset doesn't support IOMMU ACS on the pcie port.

Using the patch I have SR-IOV VFs passed to Linux guests working at full line rate reliably and everything is stable over guest reboots. BSD support is on a per-vendor basis. I suspect Intel works and I don't know anything at all about the other vendors. Mellanox is glitchy, the guest VFs only survive one VM boot before they get stuck and don't work anymore.

Hmm...the acs_override patch basically tells the hardware to ignore hardware DMA segregation on the IOMMU (because the assumption is that the IOMMU is not setup correctly in the chipset) and trust whatever is exposed in the hardware via SMBIOS, right? So what happens if you don't use acs_override? The SRIOV doesn't work, or it's just buggy?

 

[WANg]

@WANg I am back but leaving again in 36 hours. What is this fun to join in?

Oh, we are trying to see if we can get SRIOV NIC partitioning working on the t730 thin clients using Intel, Mellanox and SolarFlare PCIe cards. Looks like we can do it but it'll involve overriding the PCIe ACS (access control services). I might have to check the t730 again for proper ACS support on the PCIe root complex in Debian.

Update: Nope - just tested the t730 on Debian. lspci -vv | grep ACSCtl (the definitive way to tell whether ACS capability is built into the PCIe root complex or not) does not show anything. Guess the RX427BB does not/will not do ACS - however, most desktop class hardware does not/will not - I already verified that on a few machines in my house - would be interesting to see if my i7-5775R will have full ACS support, since I plan to get KVMGT working there on Proxmox. If any of you have a MicroServer G10, see if the Opteron X3216 supports it - I am pretty sure that the Xeon E3v2s in the MicroServer G8s do.

 

[arglebargle]

Hmm...the acs_override patch basically tells the hardware to ignore hardware DMA segregation on the IOMMU (because the assumption is that the IOMMU is not setup correctly in the chipset) and trust whatever is exposed in the hardware via SMBIOS, right? So what happens if you don't use acs_override? The SRIOV doesn't work, or it's just buggy?

I'm going to butcher it if I try to explain it myself, I only just learned all of the terminology yesterday. This is a dense read but very informative: VFIO tips and tricks: IOMMU Groups, inside and out

The ACS override basically lies about the upstream (root?) port's ability to isolate devices so that we can pass them around individually using VFIO instead of needing to pass the whole iommu group. As I understand it the hardware we're working with almost certainly lacks proper ACS support so we need the override to be able to pass individual VFs using VFIO in a "probably-safe" fashion.

There's also a chance that the root port *does* support ACS but it isn't implemented perfectly to spec and isolation could be achieved by passing kernel quirks but that's way out of my depth. I was thinking of reaching out to the blog author above and asking how I'd go about confirming one way or the other.

I just booted my other t730 from a portable Ubuntu drive and installed 4.19rc1, the iommu groups still need the ACS override so this isn't specific to Proxmox or 4.15.

 

[WANg]

I'm going to butcher it if I try to explain it myself, I only just learned all of the terminology yesterday. This is a dense read but very informative: VFIO tips and tricks: IOMMU Groups, inside and out

The ACS override basically lies about the upstream (root?) port's ability to isolate devices so that we can pass them around individually using VFIO instead of needing to pass the whole iommu group. As I understand it the hardware we're working with almost certainly lacks proper ACS support so we need the override to be able to pass individual VFs using VFIO in a "probably-safe" fashion.

There's also a chance that the root port *does* support ACS but it isn't implemented perfectly to spec and isolation could be achieved by passing kernel quirks but that's way out of my depth. I was thinking of reaching out to the blog author above and asking how I'd go about confirming one way or the other.

I just booted my other t730 from a portable Ubuntu drive and installed 4.19rc1, the iommu groups still need the ACS override so this isn't specific to Proxmox or 4.15.

Yeah, I read the article earlier and got a slightly different understanding - basically it sounded like if the ACS is not there, the kernel will read SMBIOS to figure out how the PCIe devices is connected, and then "mush" various devices into IOMMU groups using their topography on the bus (which might or might not allow valid transfers between devices in those groups as there is no guesses what they are and whether they can talk to each other). The whole idea of ACS (it sounds like) is to perform sanity checks on those PCIe transfers and then kill the invalid ones. Think of it as like something sanity checking/housekeeping the bus before someone takes a leap of faith and sends a crapload of data to an IOMMU group containing both a network card and an SD card reader. In a way, bypassing it sounds like both a good and a bad idea.

So here's the difficulty that we are facing - this machine has an AMD CPU with an AMD IOMMU, and I discover to my chagrin there's really not that much good documentation on how the IOMMU works, and there are not too many people messing with stuff like SRIOV VF partitioning or GPU passthrough on AMD consumer machines (those who tried it in the more recent Ryzen machines seem to complain about issues with VFIO being slow unless RVI (AMD's MMU virtualization) is deactivated). From what I understand, AMD's IOMMU is more towards allowing the CPU and the GPU to share memory content (zero-copy) via their heterogenous systems architecture (which is actually present on the t730) than to allow safe segregation of VFIOs. The write-ups that I've seen so far are primarily on Intel machines, and how ACS seems to be implemented only on top-of-the-range machines (or Xeons).

That being said, if I read between the lines on this SuperMicro support page, it looks like they recommend disabling ACS even when present if you want fast peer-to-peer connectivity between GPUs, and describes how to check for ACS presence in Linux (lspci -vvv | grep ACSCtl). Interesting.

 

[arglebargle]

Ah, yeah, I should probably read up on this again tomorrow when my brain isn't mush. I only have a very high level understanding of bits and pieces and, you're right, there really isn't much info about how iommu is implemented on these APUs vs server parts. I really think it'd be worthwhile to reach out to someone who really knows AMD's implementation to get their input.

 

[arglebargle]

Hey, quick question -- Did you ever get SR-IOV working on Proxmox/Debian with your Solarflare cards? I'm having a hell of a time figuring out what isn't working on an SFN5000 series card: it's failing to initialize SR-IOV at boot and the kernel messages are super unhelpful.

As I understand it the process should just involve toggling SR-IOV, setting the number of VFs and one or two other parameters with sfboot and it should work.

This is the same machine that runs SR-IOV just fine with my Mellanox cards.
---
Edit: SR-IOV functionality on these Solarflare cards, and almost all others that I tested, is tied to ARIFwd functionality. Without functional ARI the driver refuses to enable SR-IOV. Mellanox drivers /will/ work without ARI and will let you use VF/PFs addressed from 0 to 7.