HBA Controler vs Mobo vs Cached Controller in IT mode, ZFS usage

[vulcan4d]

I'm quite confused based on my own tests and some of the information provided about controllers in this forum.

I started my home server running ZFS with drives connected directly to the motherboard. The performance was fantastic. Then I wanted to expand my array but ran out of SATA interfaces, so I got a H200 controller and flashed it to IT mode. The performance was absolutely horrible, the server IO-Wait jumped through the roof. Then I learned this is because the HBA controller disables cache of the drives, ok that's fine so now I need a controller with cache. After additional research, I'm reading that the cache won't work on controller cards flashed in IT mode. So what is the point of getting one with cache mode when relying on something like ZFS vs hardware raid? How does one get an expansion card, use ZFS and allow cache to continue to work. The server has a UPS so power failure is not an issue. Any suggestions are greatly appreciated!

 

[ttabbal]

A lot of us are running SATA drives with HBAs. I've never seen indication that the disk cache is disabled. Even if it were, the host should be providing cache in RAM, this would be the ARC in ZFS-speak. I don't have an H200, but I do have 3x H310 and can saturate 10Gbe with the 10x2 mirror array.

You want ZFS to have the lowest level access to the disk you can. Caching controllers are a bad idea with ZFS. Perhaps you are seeing something else. Can you test with the same disk? Connected to onboard SATA then on the HBA? With a standard workload. Perhaps just a single-disk filesystem, could be ZFS, with something like Bonnie++?

If power failure is a non-issue, you can also try sync=disabled. You should read up on that though, it can cause issues. For testing, it might be worth a try though. It's also possible you have a bad card or cable causing problems. An iffy drive can cause high IO-wait as well, which is why I suggested testing with the same drive.

 

[DedoBOT]

Lsi9300 hba ,oem or not, IT mode - hdd cache enable/disable it option per port in the lsi's bios or mb's if in uefi mode..

 

[Spartacus]

HBAs/it-mode controllers are inherently non-raid, non-caching, thats just how they work they, don't need the bbu because theres no cache on the adapter. I pretty sure the cache only comes into play when a raid array is used. That being said ZFS has direct access to the drive, IIRC its best to leave the drive cache even if you have a SLOG device otherwise you'll see performance issues.

Lsi9300 hba ,oem or not, IT mode - hdd cache enable/disable it option per port in the lsi's bios or mb's if in uefi mode..

Even the LSI2008 (9211-8i) should be more than enough for 8 platter drives, much less a LSI2308 (9207-8i), the LSI9300 just seems wasteful on price and performance unless hes using a bunch of SSD and/or a drive expander to a bunch of drives.