I noticed that a lot of those PCIe expanders have a molex connector. Does it mean that they can be used outside of a PCIe slot? I am not sure what they use the PCIe slot for in the first place, I assumed power (and frankly if you have to sacrifice a PCIe slot, what is the point of an expander over an additional controller?)
The SAS expanders use the PCIe slot mostly for the 12V power, not the PCIe lanes. It's not a PCIe device. You communicate with it via your SAS controller. The SAS expander boards that have a molex port can be used outside of a PCIe slot by providing the power it needs via that molex port. Even for boards without the molex port, you can use a PCIe powered extender as a way to provide the 12V it needs.
There are some combinations of controllers + expander, and specific firmware versions, that can result in problems. But generally speaking, compatibility is pretty good. If you're using a SAS-3 expander, it is recommended to use a SAS-3 capable controller.
One of the drawbacks of a SAS controller + SAS expander setup is that diagnosing I/O problems can be difficult and tricky because the I/O can take multiple paths. Sometimes it will look like all your drives are randomly getting errors, but it is usually not all the drives at fault. To identify the problem usually requires swapping parts in a process of elimination, as well as analyzing diagnostic information like driver error messages in the logs. I'm not recommending against a SAS expander setup, as it has benefits too, but this is one aspect of it that trips some people up when they encounter errors and aren't experienced at diagnosing the problem.
