Epyc Rome: Unable to pass-through SATA ports to VM. FLR bug?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

llowrey

Active Member
Feb 26, 2018
167
138
43
I am not having any success passing through any of the built-in peripherals on my Epyc 7502 (SATA or USB).

When I start a VM that has the built-in SATA or USB as PCI pass-through devices, I get a sequence of...
Code:
vfio-pci 0000:45:00.0: not ready 1023ms after FLR; waiting
vfio-pci 0000:45:00.0: not ready 2047ms after FLR; waiting
...
vfio-pci 0000:45:00.0 not ready 65535ms after FLR; giving up
... followed by the vm getting stuck and sometimes other PCI devices will end up stuck. In the end, I have to hard power cycle in order to recover.

I am able to pass-through other devices, such as an Adaptec controller which does not support FLR (FLReset-) and the motherboard's Intel gigE controllers and a Mellanox CX3 that do (FLReset+). So, there does not appear to be a problem with FLR, just with the AMD peripherals.

I'm running CentOS Linux release 8.1.1911 (Core) with kernel 4.18.0-147.8.1.el8_1.x86_64. Motherboard is an ASRock Rack EPYCD8 with BIOS 2.10.

Search results suggest that this is a known issue with no workaround other than a sketchy kernel patch that has been rejected.

Has anyone successfully used either the built-in USB or SATA controllers as pass-though devices?
 

greybizay

New Member
Oct 11, 2021
4
4
3
Shame. Seems a waste of power to add a pcie card when there's a perfectly good SATA controller on die. I'll try the proxmox forums...
 

greybizay

New Member
Oct 11, 2021
4
4
3
I get exactly the same error that OP did.
i.e. when I attempt to start the VM with the passed through SATA controller, i have to hard reset the machine to revert a lock-up.
FWIW i'm able to pass through a 9211-8i with zero headache; so again another finger pointing at AMD.

SP3 drivers? Do you have any more info?
Same goes for ASRock and Supermicro acknowledging the issue?
My google-fu must be off...
 

RolloZ170

Well-Known Member
Apr 24, 2016
5,158
1,549
113
SP3 drivers? Do you have any more info?
the corresponding thing is intel INF driver.
you need a driver for all processors, they are included in the chipset or socket drivers.
you can get them in the ASRock Rack or supermicro drivers download sections.
ASRock provides only a "AMD all in 1 driver"
but supermicro provides a AMD SP3 Driver
note that both EPYCD8 and ROMED8 is officialy not compatible with Windows 10.
for linux usage you need (chipset) drivers too, ask for them in linux forums.
 

zir_blazer

Active Member
Dec 5, 2016
355
128
43
I recall identical issues on AM4 and these were related to AGESA version (Thus BIOS version). I know some people managed to passthrough the built-in Ryzen functions, but not in detail cause as I don't have such platform I don't remember the winning combo. Nor how the same plays on EPYC, because it is far rarer.
If I recall correctly, it was related to Ryzen advertising FLR when it didn't worked properly, or something like that. At least it was for the USB and Audio PCI Devices, but may add SATA as well: https://www.reddit.com/r/VFIO/comments/eba5mh
 
  • Like
Reactions: RolloZ170

ectoplasmosis

Active Member
Jul 28, 2021
117
53
28
I had issues when trying to pass through the native SATA ports on ROMED8-2T with both Rome & Milan CPUs.

Ended up just sticking a HBA in and passing that through.
 
  • Like
Reactions: area51

ectoplasmosis

Active Member
Jul 28, 2021
117
53
28
system on chip.
the ports come from PCIe lanes that are switchable from SATA to NVME, maybe that can become an issue.
Jumpers set to dedicate lanes to each controller. All devices enumerated on bare-metal OS, just cannot pass through SATA controllers reliably.
 

zir_blazer

Active Member
Dec 5, 2016
355
128
43
Jumpers set to dedicate lanes to each controller. All devices enumerated on bare-metal OS, just cannot pass through SATA controllers reliably.
If you want to be a Guinea's Pig, could you try the above Linux Kernel patch but adding the Rome/Milan SATA Controller PCI ID? I can't help you with the patching and compile procedure, but I'm interesed in the results.
I think no one bothered to test it before to check if the FLR bug applies or not because in AM4 the lanes that multiplex to SATA (Actually only 2 of them do) are pretty much always used for a M-Key M.2 Slot thus there shouldn't be many Motherboards to actually test this. And not many Rome/Milan users doing passthrough, either.
 

RolloZ170

Well-Known Member
Apr 24, 2016
5,158
1,549
113
Jumpers set to dedicate lanes to each controller. All devices enumerated on bare-metal OS, just cannot pass through SATA controllers reliably.
there is no extra SATA controller only the switch, the Rome/Milan CPU itself serves SATA or NVMe.

ROMED8-2T.jpgROMED8-2T
 

greybizay

New Member
Oct 11, 2021
4
4
3
FWIW, Proxmox 7.2 was released today. I've upgraded to it, and used the newly exposed /sys/bus/pci/devices/x:x:x.x/reset_method functionality to remove the FLR method from all 4x SATA devices on-die after boot with a quick systemd script.

This appears to have rectified the issue; and i've now ditched the HBA card (that was pulling 12-15W unnecessarily).

Thanks for pointers all.