All right, I finally got this monster working and stable.
The controllers and their enclosure:
I ended up using 6x StarTech PEXUSB3S44V USB Cards. Each one of these has 4 usb ports, but each port has it's own dedicated controller.
For each usb port, I used a DATOptic eS3U31 usb to esata...