SAS Controller as SATA replacement for PCI Passthrough

Discussion in 'RAID Controllers and Host Bus Adapters' started by zir_blazer, Feb 26, 2018.

  1. zir_blazer

    zir_blazer Active Member

    Dec 5, 2016
    Likes Received:
    Most of my interest these days are almost exclusively in the Passthrough area, or for those that aren't in the know, the ability of a Hypervisor/VMM to pass a PCI Device to a VM and use it from there, with its native Drivers, capabilities and all.
    Some users wants to pass a SATA Controller to a VM so that they can do their ZFS File Server black magic thingy, as using an emulated Controller omits info that needs bare metal access like SMART and such, which the passthroughed Controller can get directly from the HD/SSDs. Typically, the integrated SATA Controller is a bad candidate for Passthrough since if you have a SATA boot drive, unplugging the controller from the host tends to crash it, so only people that boot from an USB Flash Drive, network boot, or maybe copy the host OS to a RAMDisk (Assuming that the SATA Controller can be properly reset without rebooting the computer), can try to pass the Chipset SATA Controller. There are some exceptions, like the Intel X99 platform that had two independent SATA Controllers (With 6 and 4 SATA Ports), or AMD Ryzen, which besides the Chipset SATA Controller, also has a 2 Port SATA Controller in Ryzen itself, but its availability depends on the Motherboard topology. I suppose that Threadripper and Epyc can have up to 2 or 4 SATA Controllers with their own 2 SATA Ports each, assuming that the Motherboard designer wanted to use them as SATA Ports instead of PCIe Lanes. Getting the perfect Motherboard for Ryzen seems an impossible task.
    While many Motherboards also come with third party SATA Controllers like Marvell or ASMedia, I tend to not like these since either their performance is subpar compared to the Chipset Controller, they have less features, or even they have compatibility issues with something else (I recall a Marvell SATA Controller that couldn't work with the IOMMU turned on). Users that have no other possibility and want for any reason have a real SATA Controller in a VM, typically buy cheap SATA Controllers that also uses these chips. However, considering that SAS Controllers can be used with SATA Drives, should be of much higher quality, and can be found for cheap if going for used parts, why not using these as SATA Controllers replacements?

    I have been lurking around the SAS Controllers Threads and there are soo many different Controllers and features that choosing good ones for this use case seems hard, so I prefer if someone with good knowledge about these things can point me and other people interesed in SAS Controllers in the right direction.
    So far, points that I considered:

    IT Mode vs IR Mode
    IT Mode was supposed to be the SAS Controller working a dumb mode where all the drives connected to it are handed as JBOD, so it was the preferred method to use SAS for Software RAID. Does IT Mode benefits from huge amounts of RAM for the SAS Controller, or a faster one, or it doesn't matter?
    What about IR Mode, is there any reason why people would prefer it over Software RAID? I read that in many cases, to switch between modes you have to flash the Firmware, so I suppose that SAS Controller card selection should be heavily dependent on this point.

    Amount of Ports and cabling
    As far that I know, SAS cabling is expensive. I know that a SAS Controller can use SATA drives, but not if they can be connected directly (There was a SAS Port type that could take them, but not sure if its common or not), or if the SAS Controller just uses one of those convoluted 1-to-4 cables and you need a special SATA version of them. I'm assuming that everything will be internal.

    Card PCIe size
    PCIe 1x seems too small and bandwidth starved, 4x the sweet point, 8x/16x... overkill. One of the problems is that in Intel consumer platforms, the 16x PCIe Slots are typically coming from the Processor PCIe Controller bifurcated to either 8x/8x or 8x/4x/4x, and all them fall in the same IOMMU Group. Again, Motherboard selection is critical so that you have a physical slot that is PCIe 4x or higher using Chipset PCIe Lanes so that you can plug the SAS Controller and get it in its own IOMMU Group (For QEMU-VFIO). Ryzen could actually be better for this since it can easily get a 8 lane slot coming from the Processor with its own exclusive group, which seems to be the typical size for SAS Controllers.

    SCSI Passthrough
    According to some documentation I recall having read about the QEMU paravirtualized device VirtIO SCSI, it seems that it is possible to do some form of "SCSI Passthrough" so that the SCSI commands travels further up the chain, maybe up to the SAS Controller or the SAS drive itself. I suppose that this may not work for SATA since SATA in SAS required some form of tunneling support. Anyone has more info about this? With a full SAS stack including SAS Drives, does SCSI Passthrough lowers latency or something? This one is actually useful for people that wants to add more Drives but NOT use the Controller for Passthrough.

    Drive size limit
    I recall that SAS2 Controllers had a 2 TB (Or TiB?) size limit for individual drives. Does this applies to ALL SAS2 Controllers, or earlier ones? Can this be fixed by flashing newer Firmwares? Does this means that unless people intend to use them for low capacity HDs/SSDs, I have to look at just SAS3 Controllers?

    My idea is to attempt to push used SAS Controllers as replacement for SATA for the reasons I mentioned at the beginning of the Thread, I see them as a superior, higher quality choice. But the idea needs some refining, and to figure out what card models to pick. Expect to see this Thread linked often.
    Last edited: Feb 26, 2018
  2. ttabbal

    ttabbal Active Member

    Mar 10, 2016
    Likes Received:
    If your card is in IT mode, I don't see much point in onboard cache/battery etc.. I'm not even sure they do anything in IT mode. Treat it like you would a dumb SATA controller. The ZFS guys have this pretty well sorted out, there are a bunch of cards that work well in IT mode with simple firmware flashing. Dell H210 are the ones I've used and they work great.

    Typically, the cards in the used market are 8 port, with 2 SAS connectors on the card. I used SAS->SATA breakout cables to connect 8 drives. Depending on the chassis, some can use SAS connections and do the breakout on the backplane. Cables aren't terribly expensive online. If you buy from Dell etc, yes, they probably charge a premium. 16 port cards are available, but rare and expensive.

    The cards I've seen are generally PCIe 4x. How that relates to IOMMU groups is up to the motherboard and BIOS/EFI.

    SAS1 cards have the 2TB limit. SAS2 and SAS3 do not.

    You mention QEMU which makes me thing you're in Linux. On Linux, I use the host for ZFS, eliminating any IOMMU issues, then pass filesystems to VM/Containers for whatever they need. So I can't really discuss specifics on passthrough.
  3. i386

    i386 Well-Known Member

    Mar 18, 2016
    Likes Received:
    Yes, it's still the best solution for "legacy" systems that can't boot from software raid.
    Oboard ram doesn't matter if you are not using the raid capabilities.
    What do you mean with "faster"? SAS2 vs sas3?
    That was a limitation (32bit for block adressing) in the sas1 specification (sas1 specs were releases 2004), it's not possible to fix it with a firmware update.

    Max drive size for sas1 = 2^32 * blocksize

    All sata devices report to the hba/raid controller 512 byte block size, some sas devices report 4096 or 8192 byte block sizes > more than 2tb with sas1 possible.
  4. zir_blazer

    zir_blazer Active Member

    Dec 5, 2016
    Likes Received:
    So, basically, IT Mode is the way to go, be it either with a native dumb HBA SAS Controller card, or a RAID one that can be flashed to IT.
    What would be a scenario where you can't boot from Software RAID? As far that I know, Linux can boot from ZFS and some types of Software RAID, so it should be for something else... but then, you can also use a boot drive like a SATA DOM, USB Flash Drive, or even a small SSD with an OS installed so that it can load the Software RAID after booting...

    By faster, I was talking about a faster onboard CPU for the SAS Controller. So, neither onboard RAM nor CPU matter if you're using it in IT Mode? I suppose that a pure HBA/IT Mode SAS Controller should be cheaper by design then.

    Seriously, considering all the good things that Software RAID can do all by itself, what is the point in Hardware RAID/IR Mode at all? I recall hearing horror stories about people whose RAID Controller died or something, and that a RAID was pretty much bound to a specific controller, so migrating them from one machine to another was a pain in the butt. The only positive thing would be that the RAID CPU overhead cost is done by the card itself, but everything else seems to be a solid win for dumb HBAs with Software RAID. And should be cheaper to boot. Go figure...

    Since the idea about the SAS Controller is to "plug it as an addon card into your standard Desktop", I doubt that backplanes matters.
    When using breakout cables, is the SAS2/3 600 MB p/s / 1200 MB p/s theorical bandwidth per Drive, or per Port, so that in a breakout cable, total bandwidth will have to be divided between 4 Drives? It may not matter for HDs, but should for SSDs.

    Yes, this is for Linux. For home users wanting to use Passthrough for a Windows gaming VM, QEMU-KVM-VFIO is the most popular option.
    The IOMMU Groups does matter regardless if you're going to Passthrough that specific card or not. For example, suppose that you have an Intel consumer platform (LGA 1155/1150/1151). The Processor has a PCIe Controller with 16 Lanes, that can be used as a single 16x slot, or bifurcated to 8x/8x or 8x/4x/4x with an appropiated Chipset and Motherboard. The Processor PCIe Controller does NOT support ACS, everything plugged to it gets into the same IOMMU Group. If you plug the Video Card that you want to Passthrough into one of these, and along you plug the SAS Controller, the IOMMU Group becomes inviable, because you have to pass both of them to the same VM. You can't leave the SAS Controller in the hand of the host in that case. Typically, in these Intel consumer platforms you can find Motherboards that got a physical 16x sized slot with 4 Chipset PCIe Lanes, but I never saw with 8. If the SAS Controller actually needs that bandwidth, it will be bottlenecked, as 8x lanes are only available from a Processor PCIe Slot. But if 4x is the standard size, then there shouldn't be any issues.

    I though that limitation was for SAS2, not SAS1. Since SAS1 should be obsolete by now, then there is no reason to buy such an old card on the first place.
    The Block Addressing limitation made me remember that MBR and FAT32 had a similar 2 TiB limitation, that could technically be extended if you used a HD with bigger sectors (4 KiB instead of 512 Bytes).

    Also, as some further info, the thing I called "SCSI Passthrough" is also know as "LUN Passthrough":
    virtio-scsi LUN passthrough • r/VFIO
    Raw-device-mapping-support - OpenStack

    I know that LUN is a mostly a SCSI/SAS term, but I don't know if this applies to SATA, or other media type, for that matter. What I try to say is that I'm not sure if that works only with a SAS Controller and/or SAS Drives or LUN is a broader term.
  5. rune-san

    rune-san Member

    Feb 7, 2014
    Likes Received:
    It's unfortunate, but depending on your Controller, sometimes you need to, to be able to have full functionality. Many variants from OEMs of the LSI 3108 controller will limit controller queue depth to ~240 unless the Cache Module is installed, at which point it will increase to ~900.
  6. zir_blazer

    zir_blazer Active Member

    Dec 5, 2016
    Likes Received:
    Reviving this, through there were a few questions pending anyways. I have recently hear about SR-IOV support in SAS Controllers (Not sure if HBA and/or RAID) like the LSI SAS 3108. However, according to this (Dated from 2013)... need a card that has a SR-IOV capable Firmware. So in addition to IT or IR Mode, there is another thing that an user may need to check. You also need that the Linux Driver (If using KVM and maybe Xen) supports and wants to enable SR-IOV, in addition to Motherboard support for SR-IOV to give PCI Bridges ample resources.

    Since I didn't bothered to read the full thing, I don't know if its possible to map a Virtual Function to a specific Port/Drive so that you can do PCI Passthrough of the VF and use the native card Drivers in the VM for a single Drive, or otherwise what else the guest VM is supposed to see if you pass it a VF. Seems to be much more flexible than passthroughing a boring SATA Controller.
    Regardless, SR-IOV + PCI Passthrough looks as if it can be even more fun that the standard thing... So why the hell I don't see this mentioned more often???
    Last edited: Mar 21, 2018
Similar Threads: Controller SATA
Forum Title Date
RAID Controllers and Host Bus Adapters Controller(s) for 12 sata dvd-roms Jul 2, 2019
RAID Controllers and Host Bus Adapters How do you use these? LSI SAS9200-16E 16-Port 6Gb/s PCI Express SATA+SAS HBA Controller H3-25140-02 Apr 3, 2019
RAID Controllers and Host Bus Adapters Recommended (current) RAID controller for both PCI-E 2 and 3 ESXi hosts with SAS and SATA SSDs Sep 17, 2018
RAID Controllers and Host Bus Adapters SAS/SATA Controller recommendation, TQ backplane Feb 27, 2018
RAID Controllers and Host Bus Adapters Request for recommendations: Desktop SATA controller Oct 29, 2017

Share This Page