Intel S1200BTL, E3 v2 & multiple SAS controllers. A running log of events.

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
Hi,

This is intended as much as a log of issues encountered as much as it is a request of suggestions. Hopefully it will list out any issues encountered so others may be aware before trying this combo as there seems to be very little documented testing on an E3 v2 with a S1200BTL (not the BTLR) past the initial "will it / wont it" questions.

Brief Overview.
I am currently in the process of building a 24 bay vSphere Essentials server for a SMB customer. They are consolidating a number of low use "servers" on to one machine using proper server grade hardware (for the most part).

The originally planned hardware;
Intel E3-1230 v2
Intel S1200BTL motherboard (BTLR is not readily available from local distributors)
3x M1015 SAS controllers (Customers insistence). These are new sourced from the local distributor.
2x Intel ET dual port network controller.
Various hard drives.
Supermicro SC846A-R1200 24bay chassis with redundant PSU.
32GB ECC ram (Kingston and Intel certified)

So first things first, flashing the S1200BTL to the latest BIOS in order to support the V2 E3. I pulled an old i3-2100 off the shelf and put it in with the ram and heatsink (the CPU still had some thermal paste on it from when I had previously used it). Turned the server on and it spun up for a few second and then turned off. I cycled like this in a consent loop with no audible sign as to what was wrong.

Trouble shooting;
Checked the motherboard seating and connections - all ok.
Checked the ram in various combinations - no change.
Swapped the ram with non ECC ram - Three beeps repeated over and over.
Replaced the supported ram and checked the status lights on the back of the board - 01011001 was the code.
Checked the code in the tech specs document - not listed.
Finally took the CPU out in order to change for my own machines E3-1230 v1 when I thought of removing the thermal paste and checking it.... it was an i5-2500 and not an i3.

Resultant learning experience;
Unsupported memory gives 3 beeps
Unsupported processor give power cycling and a status code of 01011001 on the diagnostic LEDs at the rear of the board.
Check the CPU even if you need to clean off the thermal paste before using it :eek:

Putting in the i3-2120T from my HTPC and the system booted fine. I updated the BIOS (pretty easy process) and then installed the E3-1230v2 which booted fine.

As expected the system would not boot past the M1015 initialization screen.

I flashed the M1015 and after a lot of playing around ended up with the current LSI 9211-IR set of firmware (P14) and was able to boot and enter the 9211 IR config screen but only 1 controller was listed. The SAS controllers were seated in slots 4, 5 and 6. The ultimate goal is to flash to the LSI 9240 firmware as this allows the addition of new drives without being required to configure them in the SAS cards BIOS before they become available to any OS. I have, however, stuck with the 9211 IR firmware for now whilst I troubleshoot.

After swapping the controllers in and out trying different combinations I have found that only PCIe slot 6 and 3 are usable and using any other slots results in the motherboard not seeing the controllers. I have also found that using slots 3 and 6 for the LSI 9211 controllers also results in the Intel ET network card not being seen if placed in slot 4. I have not yet tried the network card in slot 5.

Now for a bit more for a dig in the technical specifications.... It seems that PCIe slot 3 is controlled by the PCH (C204), PCIe slot 6 is a dedicated x8, but PCIe slots 4 & 5 have a shared x4 and the final x4 lanes are for the dedicated SAS module (optional add-on). This sort of explains why I may have trouble using the SAS controllers in slots 4 and 5 at the same time but not why the network card is not picked up if used in slot 4 and nothing is in slot 5.

My concern is that although the E3 v2 will work on the board, there is some issues with the PCIe lanes if you wish to fully populate. Various reports mention the addition of a couple of resistors to the BTLR boards in order to make them E3 v2 compatible and if that is not just Intel smoke then maybe it was to rectify this issue.

I have the option to try with my E3-1230 v1 and see if I have any issues in this board so I will probably try that tonight. My own unit is fully populated with SAS controllers and network cards taking up all available slots. I have not had an issue with it so I am pretty surprised.

If I can get the Network card seen in either slots 4 or 5 then I can get the Intel AXXRMS2LL040 SAS module and use the motherboard sata for the remaining 4 drives (this was the original plan). If I cannot then I have to try the E3-1230v1 and see if that corrects the issue and then probably source a S1200BTLR which is not preferred as I will be bound to a 5 unit minimum by the distributor :(, or confirm with the customer that it would be ok to swap out for a Supermicro board, again, not preferred as I already have a full price M1015 I can no longer use for this build and so do not require a S1200BTL as well.

Any other constructive suggestions welcome.
Thanks
RB
 

Andreas

Member
Aug 21, 2012
127
1
18
Don't know if the MB utilizes all of them, but the E3-1230v2 CPU has 20 PCIv3 lanes off the CPU chip (vs. 16 on the consumer i7-37xx version). They can be configured in 3 ways:
1 x16 + 1 x4
2 x8 + 1 x4
1 x8 + 3 x4

Check if the MB has a PLX chip as an indicator of a shared slot setup causing the trouble with the expansion cards.

rgds,
Andy
 

mobilenvidia

Moderator
Sep 25, 2011
1,956
213
63
New Zealand
The E3's V2 have 16x PCie Gen3 lanes and 4x PCIe Gen2 lanes
The PCH has 8x PCIe Gen2 lanes
The above applies to chipsets C204, 206 and C216

With the C202 chipset the CPU seems to loose the 4x CPU PCIe Gen2 lanes, not sure why.

28 lanes all up.
The PCH usually uses a few for LAN, audio etc

Intel don't use PLX bridges, so it won't be that.

1200BTL manual Page 15 shows how the PCIe slots are dished out from CPU and PCH
 
Last edited:

Andreas

Member
Aug 21, 2012
127
1
18
Can't say I have a short and consise answer :)

According to ark.intel.com:
http://ark.intel.com/products/52276/Intel-Xeon-Processor-E3-1270-(8M-Cache-3_40-GHz)
E3-1200 series CPU have 2 x PCIv2 lanes

According to the data sheet for E3-1200v1 series:
http://www.intel.com/content/www/us/en/processors/xeon/xeon-e3-1200-family-vol-1-datasheet.html

page 10: graphics shows x16 and x4 port (all PCIv2)


page 12: Table 1-1
3 different setups are described:
Essential server, standard server and workstation/advanced server

For essential server:
Mode 2, 4 and 6 show that (x16 and x4) configs are not supported for Server/Workstation products.

16 external usable PCIv2 lanes in total from the CPU plus a DMI 2.0 connection to the PCH (equals x4 bandwidth of 2 GB/sec).
Supported configs:
1 x16
2 x8
1 x8 and 2 x4


For the other 2 product lines:
x16 and x4 is supported.

page28: chapter 2.2.3 of the E3-1200 datasheet manual states:
"Advanced Server/Workstation and Essential/Standard Server SKUs support an additional x4 port"

So, what makes a E3 CPU advanced or essential?
Further digging brought up this document:
http://www.intel.com/content/www/us/en/processors/xeon/xeon-e3-1200-product-family-brief.html

In short (page 5, bottom):
If any of the E3-1200v1 CPUs are connected to a C202 PCH -> 16 PCI lanes,
if connected to a C204 or C206 -> 20 PCI lanes.

What was not clear from the docs: Are the 4 additional lanes exiting directly from the CPU, or via the PCH?

Reading further into the datasheet, page 67 shows the PCI I/O pins of the E3-1200v1 series.
There it is, 16 + 4. Just make sure you got a C204 or C206 mobo :)

Andy
 
Last edited:

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
Thanks for all the digging around.

Current state of play is this;
M1015 cards (Flashed to LSI 9211-8i) work fine in slots 3 & 6.
The Intel ET card works fine in slot 5
Slot 4 seems to be unusable for some reason with either a SAS controller or the ET lan card.

As listed in the motherboard tech specs and the processor specs above, the BTL has 1x8, 1x4 (dedicated for the add-on SAS module available), 1x4 (shared by two slots) & 1x4 (from the C204 PCH).

The issue, and one that is not made clear on the initial specs, is that the x4 going to the dedicated SAS module socket cannot be used / re-routed for anything but the SAS module. If you are not using a SAS module then you are loosing those 4 lanes.

The ability to use slots 4 & 5 seems to be limited by the fact they are shared. I will double confirm that the third controller will not go in slot 5 tomorrow but even if it did it would limit the ability to use slot 4 for the lan card as both cards would potentially be fighting over the 4 lanes. In reality for this build that would probably not matter as it is an entry server with only about half the hard drives populated and very low usage.

I have now ordered a SAS module for the board and that will fit my needs but I am going to be left with a couple of sets of SFF-8087 -> SFF-8087 cables, a M1015 and 4 sets of SFF-8087 -> 4x 7-pin SATA forward breakout cables.

Oh and I still have to try flashing to the 9240-8i firmware and see if I can get in to the MegaRaid WebGui on boot as that was not working.

Regards
RB
 

mobilenvidia

Moderator
Sep 25, 2011
1,956
213
63
New Zealand
To get to the WebGUI, try pressing the Key combo at system boot for choosing a Boot drive.
Press the CTRL+H when M1015/LSI9240 splash screen comes up.
With choices to boot from pick the LSI card, should now got to GUI
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
To get to the WebGUI, try pressing the Key combo at system boot for choosing a Boot drive.
Press the CTRL+H when M1015/LSI9240 splash screen comes up.
With choices to boot from pick the LSI card, should now got to GUI
Hmm, on further investigation I see this is listed as a fix for not being able to access a raid controllers GUI on boot but am unclear as to why this would make the gui available when it usually is not.

To my understanding, the problem usually occurs due to the lack of available (CMOS ??) memory for the GUI to load in to at boot time. The memory space is usually taken up by config settings for integrated chipsets (i.e. USB controller / SATA controller / Audio). I know by disabling all but the most fundamental motherboard Bios options I can get in to the WebGUI for the M1015 on my old MSI C2D board. Server boards usually have more memory space available as they are designed to work with cards like raid controllers with their own configuration options at boot.

After reflashing the M1015 (now LSI 9211-8i) to a LSI 9240-8i (IR) I could access the WebGUI without an issue (not boot drive option step required). I can only think that having a controller in slot 4/5 or flashing with just the LSI 9240-8i rom but without cleearing the controller to a LSI 9211-8i may have caused some issue.

So in essence;
2x M1015 (flashed to either LSI 9211-8i (IR or IT) or LSI 9240-8i) work fine in slots 3 & 6
1x Intel ET dual port controller works fine in slot 5
E3 v2 Xeon works fine in the S1200BTL board with the lastest BIOS (i3 / GXXX or E3 v1 needed to do the initial flash).

I now have a RMS2LL040 on its way to me so I can make use of the x4 dedicated to the raid module socket and that should hopefully sort out the connectivity for all 24 drives and allow multiple dedicated passthroughs of specific drive sets to specific VMs.

Regards
RB