Supermicro 6036ST-6LR SAS 2008 controller IT mode

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Status
Not open for further replies.

jlinoff

New Member
Feb 25, 2012
12
0
0
I am building a filer based on OpenIndiana/ZFS using the SuperMicro 6036ST-6LR super server because I need HA. To use ZFS efficiently I want to use I/T mode on the SAS controllers.

I flashed the controllers using PH11 from ftp://ftp.supermicro.com/driver/SAS/LSI/2008/IT/Firmware/ and everything seemed to work fine. The flash completed successfully but I could not see the SAS expander and the 8 SAS drives in the SAS BIOS topology list when I rebooted. I also expected to see them on the Boot page of the MB BIOS as well. The MB BIOS did show the SAS controller (but no drives).

I then re-flashed with the PH10 IT. Same thing.

I then re-flashed with the PH11 IR mode. Same thing.

I then re-flashed using ver 12 from LSI (9211-8i) but that removed the controller from the BIOS boot menu which leads me to believe that SuperMicro does some customization of their firmware so I flashed back to PH11 I/T mode.

I have not seen this before but my previous experience has been with SATA with a single head that used the LSI SAS 2008 HBA. It worked perfectly on the first try which leads me to believe that there is something that I am missing because I have not used SAS drives and SAS expanders before.

Can you suggest how I can get SAS working?

Thanks.
 

mobilenvidia

Moderator
Sep 25, 2011
1,956
212
63
New Zealand
I think you need to sort this out in the LSI OptionROM
Press what ever you need to get to the select boot device (at boot time)
Then when the LSI2008 BIOS splash screen comes up, press CTRL+C (I assume this is the same for SuperMIcro)
When the boot device menu appears press the LSI card
You should now be in the LSI OptionROM and here the cards and expander should show.
In IT mode there is very little to look at, but hopefully the drives and expander are there

The expander is connected to the SAS2008 card with both SFF8087 connections all done internally on your setup
Normally there are cables running from controller to expander which normally is also on the backplane.
 
Last edited:

jlinoff

New Member
Feb 25, 2012
12
0
0
I think you need to sort this out in the LSI OptionROM
Press what ever you need to get to the select boot device (at boot time)
Then when the LSI2008 BIOS splash screen comes up, press CTRL+C (I assume this is the same for SuperMIcro)
When the boot device menu appears press the LSI card
You should now be in the LSI OptionROM and here the cards and expander should show.
In IT mode there is very little to look at, but hopefully the drives and expander are there

The expander is connected to the SAS2008 card with both SFF8087 connections all done internally on your setup
Normally there are cables running from controller to expander which normally is also on the backplane.
Thank you for responsing. You described exactly what I am doing.

Ctrl-C is what is used by SuperMicro as well.

The problem is that when I get into the topology page from the LSI option ROM (they call it the SAS controller BIOS) I see the controller but I do not see the expander or the drives. This has not happened to me before (except once when I had a cabling problem).

I am trying to figure out how to track down the problem. The fact that sees the controller card is a good sign (I think) but I am stuck. Maybe the next step is to disassemble the chassis...
 

mobilenvidia

Moderator
Sep 25, 2011
1,956
212
63
New Zealand
I take it your Mobo has the LSI SAS2008 Mezzanine card ?
I also assume that there are no cables coming from this card (as would normally happen)
And the backplane attaches to the back of the Mobo, again no cables.
So everything should work, unless something is seated right.

The Expander chip is on the mobo, I'm not sure how this interfaces with the CPU or at all
Do have any PCIe cards installed ?
If you do try removing them, the SAS expander may share a slot

You are using SAS drives ?
I do believe you need a 'Supermicro Controller Card AOC-LSISS9252 6Gb s SAS to SATA' card for SATA drives to work.
 

jlinoff

New Member
Feb 25, 2012
12
0
0
Thanks again for your help.

I take it your Mobo has the LSI SAS2008 Mezzanine card ?
Yes, the MB has an LSI SAS2008 mezzanine card (AOM-SAS2-L8, LSI 9211-8i HBA).

I also assume that there are no cables coming from this card (as would normally happen)
And the backplane attaches to the back of the Mobo, again no cables.
So everything should work, unless something is seated right.
Yup, that is what i was thinking.

The Expander chip is on the mobo, I'm not sure how this interfaces with the CPU or at all
Good observation. I don't know either.

Do have any PCIe cards installed ?
If you do try removing them, the SAS expander may share a slot
Good suggestion. I removed them but the problem still occurs.

You are using SAS drives ?
Yes.

I do believe you need a 'Supermicro Controller Card AOC-LSISS9252 6Gb s SAS to SATA' card for SATA drives to work.
Inteteresting. That would certainly explain why it isn't working.

Do you think that I could test that theory by trying to add some SATA drives instead?
 

mobilenvidia

Moderator
Sep 25, 2011
1,956
212
63
New Zealand
Yes SATA drives need the interposer, why I asked the question in case you had SATA.

To me it looks like it should work, you've got everything as it should be.

I would contact Supermicro to see if they can diagnose you issue.
I would avoid mentioning flashing the non Supermicro firmware, incase they loose interest.
Keep us informed
 

jlinoff

New Member
Feb 25, 2012
12
0
0
To me it looks like it should work, you've got everything as it should be.
Thank you, I really appreciate the sanity check. I thought I was going nuts.

I would contact Supermicro to see if they can diagnose you issue.
I contacted them on Friday.

I would avoid mentioning flashing the non Supermicro firmware, incase they loose interest.
Blast! I wish I had contacted you sooner. I did mention that I flashed the Firmware but I did it using files they supplied so it might be okay. Also, I am looking to buy quite a number of these for the private cloud I manage so hopefully that will keep their interest.

Unfortunately, as you know, I really can't run ZFS with HW RAID enabled. I need a simple HBA.

Keep us informed
Will do. I really appreciate your feedback. I have been trying to crack this problem for days trying everything I could think of with no success. Once I get it working I will write it up and post a message here.

Thanks again.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
jlinoff, mobilenvidia is probably the LSI flashing master on here. Apologies for being slow to respond (getting ready for a really big piece I am writing for new chips coming along).

Do keep us posted. If you wanted to do a main site write-up let me know.

Regards,
Patrick
 

jlinoff

New Member
Feb 25, 2012
12
0
0
jlinoff, mobilenvidia is probably the LSI flashing master on here. Apologies for being slow to respond (getting ready for a really big piece I am writing for new chips coming along).

Do keep us posted. If you wanted to do a main site write-up let me know.

Regards,
Patrick
I really appreciated mobilenvidia's insights and the fast response.

I will keep you posted and I would be happy to do a write up if/when I get it working.
 

mobilenvidia

Moderator
Sep 25, 2011
1,956
212
63
New Zealand
No probs in helping, I'm learning about Expanders and there are a few different setups

Unfortunately, as you know, I really can't run ZFS with HW RAID enabled. I need a simple HBA.
ZFS will happily work with the SAS2008 in IR mode, as anydrives not in RAID mode are passed through.
ZFS will even work in LSI9240 mode and the drives set to JBOD, this does add an extra layer between drive and OS

But the best performance for ZFS is the controller in IT mode
 

mobilenvidia

Moderator
Sep 25, 2011
1,956
212
63
New Zealand
Just looking at the Supermicro 6036ST-6LR manual and seeing what this beast is all about.

Are you running both nodes ?
What CPU's are running in this ?

Redundant motherboard now that is the bees knees in protection.

Can you swap round the Mezzanine cards from one Mobo to the other, this could the issue ?
I wonder if it's possible to disable one motherboard so that the other motherboard is used to test the expander/SAS controller on this.
You could possibly swap the whole Motherboard unit out of the case into the other ?

Ive given you some home work to do :)
 

jlinoff

New Member
Feb 25, 2012
12
0
0
Just looking at the Supermicro 6036ST-6LR manual and seeing what this beast is all about.
Cool.

Are you running both nodes ?
I have tried it 3 ways: 1 board standalone, the other board standalone and both boards together.

For the standalone tests I tried just powering up one server while both were in the chassis and I tried physically removing one server.

What CPU's are running in this ?
2 Intel Xeon X5645 (low power since I am running this at home on a 110VAC circuit). There is only CPU per server board.

I have 6GB RAM per CPU because this is just for my lab.

Redundant motherboard now that is the bees knees in protection.
yup^2, that is why I bought the thing.

Can you swap round the Mezzanine cards from one Mobo to the other, this could the issue ?
I didn't try that but I did try removing it which, unfortunately, didn't help.

I wonder if it's possible to disable one motherboard so that the other motherboard is used to test the expander/SAS controller on this.
You could possibly swap the whole Motherboard unit out of the case into the other ?
I think that I did that. In my previous experiments I physically removed one of the servers so that there was only one available. Is that what you meant?

Ive given you some home work to do :)
And I really, really appreciate it. All ideas are welcome.

I am beginning to think that I flashed the wrong firmware version. I just can't tell.

Unfortunately I am traveling this week so I can't do any more experiments for a few days. I was supposed to present this filer to my company as the next generation storage server but since this lab version wasn't ready I am trying to talk about other things :)
 

jlinoff

New Member
Feb 25, 2012
12
0
0
ZFS will happily work with the SAS2008 in IR mode, as any drives not in RAID mode are passed through.
ZFS will even work in LSI9240 mode and the drives set to JBOD, this does add an extra layer between drive and OS
Thanks that is really helpful. I may try that if I can't get IT mode working just to prove the functionality of OpenIndiana/ZFS.

But the best performance for ZFS is the controller in IT mode
 

mobilenvidia

Moderator
Sep 25, 2011
1,956
212
63
New Zealand
Another thing you could try is in the system BIOS, load the default setting by pressing F9, save and reboot
Just incase there is something in the BIOS causing the issue

It's unlikely that both boards are faulty so something is afoot for it to not cooperate
 

jlinoff

New Member
Feb 25, 2012
12
0
0
Another thing you could try is in the system BIOS, load the default setting by pressing F9, save and reboot
Just incase there is something in the BIOS causing the issue

It's unlikely that both boards are faulty so something is afoot for it to not cooperate
That is a great idea. Thanks.
 

jlinoff

New Member
Feb 25, 2012
12
0
0
Solved!

I finally figured out the problem yesterday based on a suggestion by a SuperMicro support representative.

It turned out to be very simple, I had not seated my disks properly. They were too far forward to connect to the SAS controller bus. Once I fixed that everything else was fine. Even the IT mode worked.

I am now proceeding with formatting and building the array.
 
Status
Not open for further replies.