Short version:
Is there any good reason why the 9211-8i, HP SAS expander and Hitachi 5k3000 disks seems to be a perfectly happy combo, even at high IO stress levels, but is a fail for OI/SE11? Am I missing some magic config or an updated driver that I just can't locate?
Long Version:
I know the ZFS insiders discourage SAS expanders with SATA drives, but is there any reason this config won't work?
Supermicro X8SIA-F
32 GB Kingston ECC RDIM (4x8GB quad rank DIMMS)
Intel x3460
LSI 9211-8i with IT firmware (8.0.0.0-IT)
HP SAS expander firmware rev 2.06
10x Hitachi 5k3000 2TB drives
2x Hitachi 7200 RPM Travelstar 2.5" for system drives
10GBe Dual NIC
1000base-T quad NIC
ESXi 4.1 + OpenIndiana or Solaris 11 Express
Everything seems to be working with the hardware. MB/CPU passed a 24 hour test with Memtest86+. All 10 drives were exercised with a long-mode self test. 9211-8i was flashed and tested stand-alone, and it finds all 10 drives at boot time (actually, it finds all 11 since I have a spare in the system).
My plan was to load ESXi and do a ZFS all-in-one following Gea's instructions. I learned along the way that you can't only do PCI passthrough on this motherboard using the 2 PCIe x8 slots because they are sitting behind a PCI switch. This isn't a problem because I can put the SAS card in the "x8 in x16" slot.
ESXi installed no problem. Nice and easy, like always, except that it really, really, really wants to use that 10GBe NIC for its management network - unfortunately, its not hooked to anything quite yet. Easy enough to fix - but getting into the console requires pressing "F2" and SuperMicro's IPMI console keeps wanting to eat F2s...and there is nothing in the macro menu to send an F2. Off to get a USB keyboard and take a walk into the garage...
But...when installing OpenIndiana to the system it stalls forever at "examining disks". Try #2 was using Solaris 11 Express LiveCD install. Same problem. Try #3 was doing a "text only" install of Solaris 11 Express. This time it got to the same place, but the text-baed UI let a large number of timeout errors show up. Hmmm. Back to checking all my hardware, cables, everything... Not troubles found but the OS would still not install. After several hours of trying I gave up.
In order to find where the fault is I decided to load up a Linux guest, pass the 9211-8i through to it and start exercising some things. First .iso I found was Fedora-14 x64. Seemed good enough. Loaded it up as a guest. Easy. Passed through the SAS card - found all the drives right away. Ran a few bare-drive benchmarks on individual drives. No errors. Hmmm. Built up a 10-drive MDRaid raid-6 array. Worked perfectly. Took 18 hours to initialize, but it did work. Ran some bare drive benches on that array and looked really good (>1GByte/Sec reads, though writes seemed to be limited to just better than single-drive speed). Ran a few stress tests, no problems at all...no matter what I do I can't seem to get this config to fault.
So - here's the question again: is there any good reason why the 9211-8i, HP SAS expander and Hitachi 5k3000 disks seems to be a perfectly happy combo, even at high IO stress levels, but is a fail for OI/SE11? Am I missing some magic config or an updated driver that I just can't locate?
Is there any good reason why the 9211-8i, HP SAS expander and Hitachi 5k3000 disks seems to be a perfectly happy combo, even at high IO stress levels, but is a fail for OI/SE11? Am I missing some magic config or an updated driver that I just can't locate?
Long Version:
I know the ZFS insiders discourage SAS expanders with SATA drives, but is there any reason this config won't work?
Supermicro X8SIA-F
32 GB Kingston ECC RDIM (4x8GB quad rank DIMMS)
Intel x3460
LSI 9211-8i with IT firmware (8.0.0.0-IT)
HP SAS expander firmware rev 2.06
10x Hitachi 5k3000 2TB drives
2x Hitachi 7200 RPM Travelstar 2.5" for system drives
10GBe Dual NIC
1000base-T quad NIC
ESXi 4.1 + OpenIndiana or Solaris 11 Express
Everything seems to be working with the hardware. MB/CPU passed a 24 hour test with Memtest86+. All 10 drives were exercised with a long-mode self test. 9211-8i was flashed and tested stand-alone, and it finds all 10 drives at boot time (actually, it finds all 11 since I have a spare in the system).
My plan was to load ESXi and do a ZFS all-in-one following Gea's instructions. I learned along the way that you can't only do PCI passthrough on this motherboard using the 2 PCIe x8 slots because they are sitting behind a PCI switch. This isn't a problem because I can put the SAS card in the "x8 in x16" slot.
ESXi installed no problem. Nice and easy, like always, except that it really, really, really wants to use that 10GBe NIC for its management network - unfortunately, its not hooked to anything quite yet. Easy enough to fix - but getting into the console requires pressing "F2" and SuperMicro's IPMI console keeps wanting to eat F2s...and there is nothing in the macro menu to send an F2. Off to get a USB keyboard and take a walk into the garage...
But...when installing OpenIndiana to the system it stalls forever at "examining disks". Try #2 was using Solaris 11 Express LiveCD install. Same problem. Try #3 was doing a "text only" install of Solaris 11 Express. This time it got to the same place, but the text-baed UI let a large number of timeout errors show up. Hmmm. Back to checking all my hardware, cables, everything... Not troubles found but the OS would still not install. After several hours of trying I gave up.
In order to find where the fault is I decided to load up a Linux guest, pass the 9211-8i through to it and start exercising some things. First .iso I found was Fedora-14 x64. Seemed good enough. Loaded it up as a guest. Easy. Passed through the SAS card - found all the drives right away. Ran a few bare-drive benchmarks on individual drives. No errors. Hmmm. Built up a 10-drive MDRaid raid-6 array. Worked perfectly. Took 18 hours to initialize, but it did work. Ran some bare drive benches on that array and looked really good (>1GByte/Sec reads, though writes seemed to be limited to just better than single-drive speed). Ran a few stress tests, no problems at all...no matter what I do I can't seem to get this config to fault.
So - here's the question again: is there any good reason why the 9211-8i, HP SAS expander and Hitachi 5k3000 disks seems to be a perfectly happy combo, even at high IO stress levels, but is a fail for OI/SE11? Am I missing some magic config or an updated driver that I just can't locate?