Supermicro X8DTU-F + AOC-USAS2-L8E + another LSI SAS2008 issue...

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

BLinux

cat lover server enthusiast
Jul 7, 2016
2,669
1,081
113
artofserver.com
Yesterday, I was testing out a few things and ran into a strange issue I can't explain.

I have a X8DTU-F system with a AOC-USAS2-L8E, which is basically a LSI SAS2008 based SAS controller in IT mode. This is a testing system and everything works as usual in this configuration. I use a USB flash drive to boot various OSes on this test system; in yesterday's case, it was CentOS 7 on USB drive.

Yesterday, I added another LSI SAS2008 based controller (H310 flashed to IT mode) in a PCI-E slot. The system POSTs just fine, but when it went to boot the USB drive with CentOS 7 on it, it just shows a blank screen with a cursor blinking; doesn't even show grub2 or anything else. At first I thought maybe my USB drive went bad, as i do use it a lot from machine to machine. But after much troubleshooting, the USB drive is fine, and so were a few others I tried, but none of them would boot this machine with the 2nd SAS2008 controller in it.

Once I removed the 2nd SAS2008 controller, everything goes back to normal and any of the USB drives will boot just fine. So, out of curiosity, I decide to try a swap, so I removed the AOC-USAS2-L8E card and just install the one H310/SAS2008 card alone. In this configuration, all of the USB drives boot just fine. The problem happens when I have both the AOC-USAS2-L8E + H310/LSI SAS2008 installed together.

What would cause this?
 

Terry Kennedy

Well-Known Member
Jun 25, 2015
1,140
594
113
New York City
www.glaver.org
What would cause this?
4 things I can think of:

1) Make sure both adapters are at the same firmware and BIOS revision levels and up-to-date.

2) If there are drive(s) hooked up to the controller, it may appear earlier in the boot order than your USB drive. Check the BIOS setup options for boot drives. It also may have "bumped" the USB drive completely out of the boot list.

3) Check the setup menu on the controllers (since they're both 2008s, they should share a single menu). You'll see a number (or blank) for boot order as well as "enabled BIOS and OS" or some other state. You can set them both to blank for boot order unless you plan on booting from a drive on them. Normally you'll get an "adapter boot order changed, suggest running setup" or similar message from the LSI BIOS if it detects the number of LSI cards changed.

4) Your system may not have enough BIOS address space to map both cards' BIOS. That normally happens on systems with 3 or more LSI cards, unless there are other option BIOSes (BIOI?) on other cards.
 

BLinux

cat lover server enthusiast
Jul 7, 2016
2,669
1,081
113
artofserver.com
4 things I can think of:

1) Make sure both adapters are at the same firmware and BIOS revision levels and up-to-date.
I may double check on that, but I believe they were at least very similar versions, P20 variant at least.
2) If there are drive(s) hooked up to the controller, it may appear earlier in the boot order than your USB drive. Check the BIOS setup options for boot drives. It also may have "bumped" the USB drive completely out of the boot list.
This was one of the things i checked too. To make sure I wasn't trying to boot some other device, i also hit F11 for the boot device menu and selected the USB drive it detected; same results as described above.
3) Check the setup menu on the controllers (since they're both 2008s, they should share a single menu). You'll see a number (or blank) for boot order as well as "enabled BIOS and OS" or some other state. You can set them both to blank for boot order unless you plan on booting from a drive on them. Normally you'll get an "adapter boot order changed, suggest running setup" or similar message from the LSI BIOS if it detects the number of LSI cards changed.
That's one thing I did not try.
4) Your system may not have enough BIOS address space to map both cards' BIOS. That normally happens on systems with 3 or more LSI cards, unless there are other option BIOSes (BIOI?) on other cards.
I'm not booting off these LSI cards, so you think I could work around this by disabling the OpROM for those cards in the BIOS?
 

Terry Kennedy

Well-Known Member
Jun 25, 2015
1,140
594
113
New York City
www.glaver.org
I'm not booting off these LSI cards, so you think I could work around this by disabling the OpROM for those cards in the BIOS?
You only need the BIOS on the first card if you're booting from it in non-UEFI mode or you want to see the list of devices at boot time. Additional cards in the same adapter family don't need a BIOS installed as it is redundant.
 

nthu9280

Well-Known Member
Feb 3, 2016
1,628
498
83
San Antonio, TX
Not 100% sure but another item to check with the H310s is to try mask B5-B6 pins. I only had one card in X8 board sometime ago. I didn't have boot issue but one of the DIMMs was disabled.

Sent from my Nexus 6 using Tapatalk
 

BLinux

cat lover server enthusiast
Jul 7, 2016
2,669
1,081
113
artofserver.com
You only need the BIOS on the first card if you're booting from it in non-UEFI mode or you want to see the list of devices at boot time. Additional cards in the same adapter family don't need a BIOS installed as it is redundant.
Yes, I know that; hence it got me thinking about disabling the OpRom setting on these PCI-E slots, as they are enabled by default (though, I should check). Just confirming that would address the BIOS out of address space issue you mentioned regarding have 3 or more HBAs?

Not 100% sure but another item to check with the H310s is to try mask B5-B6 pins. I only had one card in X8 board sometime ago. I didn't have boot issue but one of the DIMMs was disabled.
That's a thought... though the H310 card did work fine on it's own without covering bin B5/B6; but i guess there could be some strange interaction between it and the AOC-USAS2-L8E card? I may give that a try too..

thanks guys for the suggestions... really appreciate it.