SAS Controller as SATA replacement for PCI Passthrough

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

zir_blazer

Active Member
Dec 5, 2016
355
128
43
Most of my interest these days are almost exclusively in the Passthrough area, or for those that aren't in the know, the ability of a Hypervisor/VMM to pass a PCI Device to a VM and use it from there, with its native Drivers, capabilities and all.
Some users wants to pass a SATA Controller to a VM so that they can do their ZFS File Server black magic thingy, as using an emulated Controller omits info that needs bare metal access like SMART and such, which the passthroughed Controller can get directly from the HD/SSDs. Typically, the integrated SATA Controller is a bad candidate for Passthrough since if you have a SATA boot drive, unplugging the controller from the host tends to crash it, so only people that boot from an USB Flash Drive, network boot, or maybe copy the host OS to a RAMDisk (Assuming that the SATA Controller can be properly reset without rebooting the computer), can try to pass the Chipset SATA Controller. There are some exceptions, like the Intel X99 platform that had two independent SATA Controllers (With 6 and 4 SATA Ports), or AMD Ryzen, which besides the Chipset SATA Controller, also has a 2 Port SATA Controller in Ryzen itself, but its availability depends on the Motherboard topology. I suppose that Threadripper and Epyc can have up to 2 or 4 SATA Controllers with their own 2 SATA Ports each, assuming that the Motherboard designer wanted to use them as SATA Ports instead of PCIe Lanes. Getting the perfect Motherboard for Ryzen seems an impossible task.
While many Motherboards also come with third party SATA Controllers like Marvell or ASMedia, I tend to not like these since either their performance is subpar compared to the Chipset Controller, they have less features, or even they have compatibility issues with something else (I recall a Marvell SATA Controller that couldn't work with the IOMMU turned on). Users that have no other possibility and want for any reason have a real SATA Controller in a VM, typically buy cheap SATA Controllers that also uses these chips. However, considering that SAS Controllers can be used with SATA Drives, should be of much higher quality, and can be found for cheap if going for used parts, why not using these as SATA Controllers replacements?

I have been lurking around the SAS Controllers Threads and there are soo many different Controllers and features that choosing good ones for this use case seems hard, so I prefer if someone with good knowledge about these things can point me and other people interesed in SAS Controllers in the right direction.
So far, points that I considered:

IT Mode vs IR Mode
IT Mode was supposed to be the SAS Controller working a dumb mode where all the drives connected to it are handed as JBOD, so it was the preferred method to use SAS for Software RAID. Does IT Mode benefits from huge amounts of RAM for the SAS Controller, or a faster one, or it doesn't matter?
What about IR Mode, is there any reason why people would prefer it over Software RAID? I read that in many cases, to switch between modes you have to flash the Firmware, so I suppose that SAS Controller card selection should be heavily dependent on this point.

Amount of Ports and cabling
As far that I know, SAS cabling is expensive. I know that a SAS Controller can use SATA drives, but not if they can be connected directly (There was a SAS Port type that could take them, but not sure if its common or not), or if the SAS Controller just uses one of those convoluted 1-to-4 cables and you need a special SATA version of them. I'm assuming that everything will be internal.

Card PCIe size
PCIe 1x seems too small and bandwidth starved, 4x the sweet point, 8x/16x... overkill. One of the problems is that in Intel consumer platforms, the 16x PCIe Slots are typically coming from the Processor PCIe Controller bifurcated to either 8x/8x or 8x/4x/4x, and all them fall in the same IOMMU Group. Again, Motherboard selection is critical so that you have a physical slot that is PCIe 4x or higher using Chipset PCIe Lanes so that you can plug the SAS Controller and get it in its own IOMMU Group (For QEMU-VFIO). Ryzen could actually be better for this since it can easily get a 8 lane slot coming from the Processor with its own exclusive group, which seems to be the typical size for SAS Controllers.

SCSI Passthrough
According to some documentation I recall having read about the QEMU paravirtualized device VirtIO SCSI, it seems that it is possible to do some form of "SCSI Passthrough" so that the SCSI commands travels further up the chain, maybe up to the SAS Controller or the SAS drive itself. I suppose that this may not work for SATA since SATA in SAS required some form of tunneling support. Anyone has more info about this? With a full SAS stack including SAS Drives, does SCSI Passthrough lowers latency or something? This one is actually useful for people that wants to add more Drives but NOT use the Controller for Passthrough.

Drive size limit
I recall that SAS2 Controllers had a 2 TB (Or TiB?) size limit for individual drives. Does this applies to ALL SAS2 Controllers, or earlier ones? Can this be fixed by flashing newer Firmwares? Does this means that unless people intend to use them for low capacity HDs/SSDs, I have to look at just SAS3 Controllers?


My idea is to attempt to push used SAS Controllers as replacement for SATA for the reasons I mentioned at the beginning of the Thread, I see them as a superior, higher quality choice. But the idea needs some refining, and to figure out what card models to pick. Expect to see this Thread linked often.
 
Last edited:

ttabbal

Active Member
Mar 10, 2016
743
207
43
47
If your card is in IT mode, I don't see much point in onboard cache/battery etc.. I'm not even sure they do anything in IT mode. Treat it like you would a dumb SATA controller. The ZFS guys have this pretty well sorted out, there are a bunch of cards that work well in IT mode with simple firmware flashing. Dell H210 are the ones I've used and they work great.

Typically, the cards in the used market are 8 port, with 2 SAS connectors on the card. I used SAS->SATA breakout cables to connect 8 drives. Depending on the chassis, some can use SAS connections and do the breakout on the backplane. Cables aren't terribly expensive online. If you buy from Dell etc, yes, they probably charge a premium. 16 port cards are available, but rare and expensive.

The cards I've seen are generally PCIe 4x. How that relates to IOMMU groups is up to the motherboard and BIOS/EFI.

SAS1 cards have the 2TB limit. SAS2 and SAS3 do not.

You mention QEMU which makes me thing you're in Linux. On Linux, I use the host for ZFS, eliminating any IOMMU issues, then pass filesystems to VM/Containers for whatever they need. So I can't really discuss specifics on passthrough.
 

i386

Well-Known Member
Mar 18, 2016
4,217
1,540
113
34
Germany
What about IR Mode, is there any reason why people would prefer it over Software RAID?
Yes, it's still the best solution for "legacy" systems that can't boot from software raid.
Does IT Mode benefits from huge amounts of RAM for the SAS Controller, or a faster one, or it doesn't matter?
Oboard ram doesn't matter if you are not using the raid capabilities.
What do you mean with "faster"? SAS2 vs sas3?
Drive size limit
I recall that SAS2 Controllers had a 2 TB (Or TiB?) size limit for individual drives. Does this applies to ALL SAS2 Controllers, or earlier ones? Can this be fixed by flashing newer Firmwares? Does this means that unless people intend to use them for low capacity HDs/SSDs, I have to look at just SAS3 Controllers?
That was a limitation (32bit for block adressing) in the sas1 specification (sas1 specs were releases 2004), it's not possible to fix it with a firmware update.

Max drive size for sas1 = 2^32 * blocksize


All sata devices report to the hba/raid controller 512 byte block size, some sas devices report 4096 or 8192 byte block sizes > more than 2tb with sas1 possible.
 

zir_blazer

Active Member
Dec 5, 2016
355
128
43
If your card is in IT mode, I don't see much point in onboard cache/battery etc.. I'm not even sure they do anything in IT mode. Treat it like you would a dumb SATA controller. The ZFS guys have this pretty well sorted out, there are a bunch of cards that work well in IT mode with simple firmware flashing.
Yes, it's still the best solution for "legacy" systems that can't boot from software raid.
So, basically, IT Mode is the way to go, be it either with a native dumb HBA SAS Controller card, or a RAID one that can be flashed to IT.
What would be a scenario where you can't boot from Software RAID? As far that I know, Linux can boot from ZFS and some types of Software RAID, so it should be for something else... but then, you can also use a boot drive like a SATA DOM, USB Flash Drive, or even a small SSD with an OS installed so that it can load the Software RAID after booting...


Oboard ram doesn't matter if you are not using the raid capabilities.
What do you mean with "faster"? SAS2 vs sas3?
By faster, I was talking about a faster onboard CPU for the SAS Controller. So, neither onboard RAM nor CPU matter if you're using it in IT Mode? I suppose that a pure HBA/IT Mode SAS Controller should be cheaper by design then.

Seriously, considering all the good things that Software RAID can do all by itself, what is the point in Hardware RAID/IR Mode at all? I recall hearing horror stories about people whose RAID Controller died or something, and that a RAID was pretty much bound to a specific controller, so migrating them from one machine to another was a pain in the butt. The only positive thing would be that the RAID CPU overhead cost is done by the card itself, but everything else seems to be a solid win for dumb HBAs with Software RAID. And should be cheaper to boot. Go figure...


Typically, the cards in the used market are 8 port, with 2 SAS connectors on the card. I used SAS->SATA breakout cables to connect 8 drives. Depending on the chassis, some can use SAS connections and do the breakout on the backplane. Cables aren't terribly expensive online. If you buy from Dell etc, yes, they probably charge a premium. 16 port cards are available, but rare and expensive.
Since the idea about the SAS Controller is to "plug it as an addon card into your standard Desktop", I doubt that backplanes matters.
When using breakout cables, is the SAS2/3 600 MB p/s / 1200 MB p/s theorical bandwidth per Drive, or per Port, so that in a breakout cable, total bandwidth will have to be divided between 4 Drives? It may not matter for HDs, but should for SSDs.


The cards I've seen are generally PCIe 4x. How that relates to IOMMU groups is up to the motherboard and BIOS/EFI.

You mention QEMU which makes me thing you're in Linux. On Linux, I use the host for ZFS, eliminating any IOMMU issues, then pass filesystems to VM/Containers for whatever they need. So I can't really discuss specifics on passthrough.
Yes, this is for Linux. For home users wanting to use Passthrough for a Windows gaming VM, QEMU-KVM-VFIO is the most popular option.
The IOMMU Groups does matter regardless if you're going to Passthrough that specific card or not. For example, suppose that you have an Intel consumer platform (LGA 1155/1150/1151). The Processor has a PCIe Controller with 16 Lanes, that can be used as a single 16x slot, or bifurcated to 8x/8x or 8x/4x/4x with an appropiated Chipset and Motherboard. The Processor PCIe Controller does NOT support ACS, everything plugged to it gets into the same IOMMU Group. If you plug the Video Card that you want to Passthrough into one of these, and along you plug the SAS Controller, the IOMMU Group becomes inviable, because you have to pass both of them to the same VM. You can't leave the SAS Controller in the hand of the host in that case. Typically, in these Intel consumer platforms you can find Motherboards that got a physical 16x sized slot with 4 Chipset PCIe Lanes, but I never saw with 8. If the SAS Controller actually needs that bandwidth, it will be bottlenecked, as 8x lanes are only available from a Processor PCIe Slot. But if 4x is the standard size, then there shouldn't be any issues.


SAS1 cards have the 2TB limit. SAS2 and SAS3 do not.
That was a limitation (32bit for block adressing) in the sas1 specification (sas1 specs were releases 2004), it's not possible to fix it with a firmware update.

Max drive size for sas1 = 2^32 * blocksize


All sata devices report to the hba/raid controller 512 byte block size, some sas devices report 4096 or 8192 byte block sizes > more than 2tb with sas1 possible.
I though that limitation was for SAS2, not SAS1. Since SAS1 should be obsolete by now, then there is no reason to buy such an old card on the first place.
The Block Addressing limitation made me remember that MBR and FAT32 had a similar 2 TiB limitation, that could technically be extended if you used a HD with bigger sectors (4 KiB instead of 512 Bytes).



Also, as some further info, the thing I called "SCSI Passthrough" is also know as "LUN Passthrough":
virtio-scsi LUN passthrough • r/VFIO
Raw-device-mapping-support - OpenStack

I know that LUN is a mostly a SCSI/SAS term, but I don't know if this applies to SATA, or other media type, for that matter. What I try to say is that I'm not sure if that works only with a SAS Controller and/or SAS Drives or LUN is a broader term.
 

rune-san

Member
Feb 7, 2014
81
18
8
If your card is in IT mode, I don't see much point in onboard cache/battery etc.. I'm not even sure they do anything in IT mode.
It's unfortunate, but depending on your Controller, sometimes you need to, to be able to have full functionality. Many variants from OEMs of the LSI 3108 controller will limit controller queue depth to ~240 unless the Cache Module is installed, at which point it will increase to ~900.
 

zir_blazer

Active Member
Dec 5, 2016
355
128
43
Reviving this, through there were a few questions pending anyways. I have recently hear about SR-IOV support in SAS Controllers (Not sure if HBA and/or RAID) like the LSI SAS 3108. However, according to this (Dated from 2013)...

The SAS controller used in the experiments was a LSISAS 3108, a 3rd generation SAS controller made by LSI Corporation. This controller can be either a standard non-SR-IOV controller or a SR-IOV controller based upon the firmware installed. When testing SR-IOV, the controller was using SR-IOV firmware and in the bare metal and virtio environments, the controller was in non-SR-IOV mode to accurately represent the real-world usage of all three of the test cases. In both configurations, the firmware version was 02.00.01.00. The number of virtual functions presented to the system can also be configured, along with the resource allocation between the functions, both physical and virtual. Except where noted, the number of virtual functions was set to four and the available resources were equal across the virtual functions. The physical function received reduced resources because it was not needed for regular I/O operations. The resources in question are the depth of the request queues provided to the driver to submit requests to the controller. Each function has up to 8 MSI-x vectors available to it. The driver used was version 3.00.00.00 for nonSR-IOV mode and 10.165.01.00 for the SR-IOV hypervisor driver. It should be noted this SR-IOV driver and firmware are pre-production versions. Both the non-SR-IOV firmware and driver are production versions.
When in SR-IOV mode, the controller must be configured with how to divide its available resources between the virtual functions. It was decided to configure the controller to provide each virtual function with 21% of the resources and the physical function would have 16% of the resources for its own usage. This division is rather arbitrary, and is discussed further in Chapter 6. A small amount of testing was done changing the resource allocation and is discussed later in this chapter.
...you need a card that has a SR-IOV capable Firmware. So in addition to IT or IR Mode, there is another thing that an user may need to check. You also need that the Linux Driver (If using KVM and maybe Xen) supports and wants to enable SR-IOV, in addition to Motherboard support for SR-IOV to give PCI Bridges ample resources.

Since I didn't bothered to read the full thing, I don't know if its possible to map a Virtual Function to a specific Port/Drive so that you can do PCI Passthrough of the VF and use the native card Drivers in the VM for a single Drive, or otherwise what else the guest VM is supposed to see if you pass it a VF. Seems to be much more flexible than passthroughing a boring SATA Controller.
Regardless, SR-IOV + PCI Passthrough looks as if it can be even more fun that the standard thing... So why the hell I don't see this mentioned more often???
 
Last edited: