Are drives on multiple controllers better than one?

[snakyjake]

I want to have 8 SATA drives, software RAID.

Some motherboards have 1 controller with 8 ports, some have 2 controllers from different manufacturers for 4x4 ports, or the other option of 1 motherboard controller + LSI SAS/SATA.
All of these choices support at least 8 drives.

Is it better to have all the drives on a single controller?
Then maybe any drive-to-drive communication won't have to go to the bus?
Better queuing?
Better at managing overhead?

Is it better to have the drives distributed across controllers?
Maybe better if multiple controllers can handle the workload better?

Is there a difference between all drives on a single controller versus distributed on 2+ controllers.

 

[pricklypunter]

Ultimately, it will come down to the PCI bus capability to service the data and the rest of your system to process it I would imagine. That being said, you still need to look at compatibility for whatever you plan on doing with it. Some onboard controllers don't support VT-d very well, pretty much most fake raid type controllers are likely going to be problematic. In terms of raid, you have negated a lot of the "what's best" questions just by opting for a software raid implementation as this will then be determined by whichever software you use

 

[TuxDude]

The differences just come down to bandwidth and failure-domain.

If you are using spinning disks, you don't have to worry about bandwidth with only 8 drives - any controller is going to have enough PCIe bandwidth to let all of the drives work at full speed at the same time. If you have enough drives to saturate the PCIe link to the controller, then adding a second controller will give you more PCIe bandwidth to spread your disks across.

As for failure-domain, I wouldn't worry about it for home use. If this was a large enterprise deployment where the cost of downtime is easily more expensive than the cost of all the extra hardware, then adding additional controllers can make everything more reliable. Either dual SAS controllers that are connected in a redundant way to dual-port SAS drives, or dual controllers of any type configured with RAID10 such that if a controller fails it takes out 1 disk in each mirror, will allow a system to continue running in the event of a controller failure - combine with hot-swap PCIe and you can also recover/repair the system with no downtime.

Also - drives don't communicate with each other. The only way to keep any kind of storage traffic off the PCIe bus is to use a hardware RAID card - then the RAID engine can do any IO it wants without using the PCIe bus and only data read/written by the OS is using up bandwidth there. Under software RAID your drives will all do DMA to communicate directly with system memory without involving the CPU, but that will still always go across the PCIe bus.