Most of my interest these days are almost exclusively in the Passthrough area, or for those that aren't in the know, the ability of a Hypervisor/VMM to pass a PCI Device to a VM and use it from there, with its native Drivers, capabilities and all.
Some users wants to pass a SATA Controller to a VM so that they can do their ZFS File Server black magic thingy, as using an emulated Controller omits info that needs bare metal access like SMART and such, which the passthroughed Controller can get directly from the HD/SSDs. Typically, the integrated SATA Controller is a bad candidate for Passthrough since if you have a SATA boot drive, unplugging the controller from the host tends to crash it, so only people that boot from an USB Flash Drive, network boot, or maybe copy the host OS to a RAMDisk (Assuming that the SATA Controller can be properly reset without rebooting the computer), can try to pass the Chipset SATA Controller. There are some exceptions, like the Intel X99 platform that had two independent SATA Controllers (With 6 and 4 SATA Ports), or AMD Ryzen, which besides the Chipset SATA Controller, also has a 2 Port SATA Controller in Ryzen itself, but its availability depends on the Motherboard topology. I suppose that Threadripper and Epyc can have up to 2 or 4 SATA Controllers with their own 2 SATA Ports each, assuming that the Motherboard designer wanted to use them as SATA Ports instead of PCIe Lanes. Getting the perfect Motherboard for Ryzen seems an impossible task.
While many Motherboards also come with third party SATA Controllers like Marvell or ASMedia, I tend to not like these since either their performance is subpar compared to the Chipset Controller, they have less features, or even they have compatibility issues with something else (I recall a Marvell SATA Controller that couldn't work with the IOMMU turned on). Users that have no other possibility and want for any reason have a real SATA Controller in a VM, typically buy cheap SATA Controllers that also uses these chips. However, considering that SAS Controllers can be used with SATA Drives, should be of much higher quality, and can be found for cheap if going for used parts, why not using these as SATA Controllers replacements?
I have been lurking around the SAS Controllers Threads and there are soo many different Controllers and features that choosing good ones for this use case seems hard, so I prefer if someone with good knowledge about these things can point me and other people interesed in SAS Controllers in the right direction.
So far, points that I considered:
IT Mode vs IR Mode
IT Mode was supposed to be the SAS Controller working a dumb mode where all the drives connected to it are handed as JBOD, so it was the preferred method to use SAS for Software RAID. Does IT Mode benefits from huge amounts of RAM for the SAS Controller, or a faster one, or it doesn't matter?
What about IR Mode, is there any reason why people would prefer it over Software RAID? I read that in many cases, to switch between modes you have to flash the Firmware, so I suppose that SAS Controller card selection should be heavily dependent on this point.
Amount of Ports and cabling
As far that I know, SAS cabling is expensive. I know that a SAS Controller can use SATA drives, but not if they can be connected directly (There was a SAS Port type that could take them, but not sure if its common or not), or if the SAS Controller just uses one of those convoluted 1-to-4 cables and you need a special SATA version of them. I'm assuming that everything will be internal.
Card PCIe size
PCIe 1x seems too small and bandwidth starved, 4x the sweet point, 8x/16x... overkill. One of the problems is that in Intel consumer platforms, the 16x PCIe Slots are typically coming from the Processor PCIe Controller bifurcated to either 8x/8x or 8x/4x/4x, and all them fall in the same IOMMU Group. Again, Motherboard selection is critical so that you have a physical slot that is PCIe 4x or higher using Chipset PCIe Lanes so that you can plug the SAS Controller and get it in its own IOMMU Group (For QEMU-VFIO). Ryzen could actually be better for this since it can easily get a 8 lane slot coming from the Processor with its own exclusive group, which seems to be the typical size for SAS Controllers.
SCSI Passthrough
According to some documentation I recall having read about the QEMU paravirtualized device VirtIO SCSI, it seems that it is possible to do some form of "SCSI Passthrough" so that the SCSI commands travels further up the chain, maybe up to the SAS Controller or the SAS drive itself. I suppose that this may not work for SATA since SATA in SAS required some form of tunneling support. Anyone has more info about this? With a full SAS stack including SAS Drives, does SCSI Passthrough lowers latency or something? This one is actually useful for people that wants to add more Drives but NOT use the Controller for Passthrough.
Drive size limit
I recall that SAS2 Controllers had a 2 TB (Or TiB?) size limit for individual drives. Does this applies to ALL SAS2 Controllers, or earlier ones? Can this be fixed by flashing newer Firmwares? Does this means that unless people intend to use them for low capacity HDs/SSDs, I have to look at just SAS3 Controllers?
My idea is to attempt to push used SAS Controllers as replacement for SATA for the reasons I mentioned at the beginning of the Thread, I see them as a superior, higher quality choice. But the idea needs some refining, and to figure out what card models to pick. Expect to see this Thread linked often.
Some users wants to pass a SATA Controller to a VM so that they can do their ZFS File Server black magic thingy, as using an emulated Controller omits info that needs bare metal access like SMART and such, which the passthroughed Controller can get directly from the HD/SSDs. Typically, the integrated SATA Controller is a bad candidate for Passthrough since if you have a SATA boot drive, unplugging the controller from the host tends to crash it, so only people that boot from an USB Flash Drive, network boot, or maybe copy the host OS to a RAMDisk (Assuming that the SATA Controller can be properly reset without rebooting the computer), can try to pass the Chipset SATA Controller. There are some exceptions, like the Intel X99 platform that had two independent SATA Controllers (With 6 and 4 SATA Ports), or AMD Ryzen, which besides the Chipset SATA Controller, also has a 2 Port SATA Controller in Ryzen itself, but its availability depends on the Motherboard topology. I suppose that Threadripper and Epyc can have up to 2 or 4 SATA Controllers with their own 2 SATA Ports each, assuming that the Motherboard designer wanted to use them as SATA Ports instead of PCIe Lanes. Getting the perfect Motherboard for Ryzen seems an impossible task.
While many Motherboards also come with third party SATA Controllers like Marvell or ASMedia, I tend to not like these since either their performance is subpar compared to the Chipset Controller, they have less features, or even they have compatibility issues with something else (I recall a Marvell SATA Controller that couldn't work with the IOMMU turned on). Users that have no other possibility and want for any reason have a real SATA Controller in a VM, typically buy cheap SATA Controllers that also uses these chips. However, considering that SAS Controllers can be used with SATA Drives, should be of much higher quality, and can be found for cheap if going for used parts, why not using these as SATA Controllers replacements?
I have been lurking around the SAS Controllers Threads and there are soo many different Controllers and features that choosing good ones for this use case seems hard, so I prefer if someone with good knowledge about these things can point me and other people interesed in SAS Controllers in the right direction.
So far, points that I considered:
IT Mode vs IR Mode
IT Mode was supposed to be the SAS Controller working a dumb mode where all the drives connected to it are handed as JBOD, so it was the preferred method to use SAS for Software RAID. Does IT Mode benefits from huge amounts of RAM for the SAS Controller, or a faster one, or it doesn't matter?
What about IR Mode, is there any reason why people would prefer it over Software RAID? I read that in many cases, to switch between modes you have to flash the Firmware, so I suppose that SAS Controller card selection should be heavily dependent on this point.
Amount of Ports and cabling
As far that I know, SAS cabling is expensive. I know that a SAS Controller can use SATA drives, but not if they can be connected directly (There was a SAS Port type that could take them, but not sure if its common or not), or if the SAS Controller just uses one of those convoluted 1-to-4 cables and you need a special SATA version of them. I'm assuming that everything will be internal.
Card PCIe size
PCIe 1x seems too small and bandwidth starved, 4x the sweet point, 8x/16x... overkill. One of the problems is that in Intel consumer platforms, the 16x PCIe Slots are typically coming from the Processor PCIe Controller bifurcated to either 8x/8x or 8x/4x/4x, and all them fall in the same IOMMU Group. Again, Motherboard selection is critical so that you have a physical slot that is PCIe 4x or higher using Chipset PCIe Lanes so that you can plug the SAS Controller and get it in its own IOMMU Group (For QEMU-VFIO). Ryzen could actually be better for this since it can easily get a 8 lane slot coming from the Processor with its own exclusive group, which seems to be the typical size for SAS Controllers.
SCSI Passthrough
According to some documentation I recall having read about the QEMU paravirtualized device VirtIO SCSI, it seems that it is possible to do some form of "SCSI Passthrough" so that the SCSI commands travels further up the chain, maybe up to the SAS Controller or the SAS drive itself. I suppose that this may not work for SATA since SATA in SAS required some form of tunneling support. Anyone has more info about this? With a full SAS stack including SAS Drives, does SCSI Passthrough lowers latency or something? This one is actually useful for people that wants to add more Drives but NOT use the Controller for Passthrough.
Drive size limit
I recall that SAS2 Controllers had a 2 TB (Or TiB?) size limit for individual drives. Does this applies to ALL SAS2 Controllers, or earlier ones? Can this be fixed by flashing newer Firmwares? Does this means that unless people intend to use them for low capacity HDs/SSDs, I have to look at just SAS3 Controllers?
My idea is to attempt to push used SAS Controllers as replacement for SATA for the reasons I mentioned at the beginning of the Thread, I see them as a superior, higher quality choice. But the idea needs some refining, and to figure out what card models to pick. Expect to see this Thread linked often.
Last edited: