Supermicro SC-846E1 Build Log

[Patrick]

Eeeek! So here is a total let-down. I did a dual E5-2630L configuration:
http://linux-bench.com/parser.html using ID: 00051415159520

28GB of RAM. Now here is the extremely interesting part:
P1-DIMMA1 TEMPNormal34 degrees C
P1-DIMMB1 TEMPNormal32 degrees C
P1-DIMMC1 TEMPNormal37 degrees C
P1-DIMMD1 TEMPNormal35 degrees C
P2-DIMME1 TEMPN/ANot Present!
P2-DIMMF1 TEMPNormal35 degrees C
P2-DIMMG1 TEMPNormal34 degrees C
P2-DIMMH1 TEMPNormal34 degrees C

If I am reading that correctly, P2 = processor 2 DIMM E1 is missing!

 

[TD_Trader]

As I try things out, I wanted to keep a log of what did not work:

• Swapping PSUs for 1.4kw Supermicro 80Plus Gold versions - These did not lock in place and therefore did not power on system. Very strange.

Good to know!

• I tried using an Intel Xeon E5-2630L V2 ES - this did not work. QF8D - will try after new bios.

I saw some great deals on "ES" chips, and I was quite curious (and tempted) to buy a few. I'm glad to see you bought one, and I'm hoping that you can get it to run. If so, then I may pick up a few as well.

• All 4 DIMMs CPU1 populated = stuck at B7 post code. It did work with all 16 DIMMs I tried in pairs of 2.

Hmm... strange.

 

[TD_Trader]

Updated IPMI firmware and BIOS. QF8D works. Also, I added in the 4th DIMM and that works now!?! I visually inspected the socket but did not see anything wrong. Very strange indeed. I may try some of the other V2's tomorrow. Then it is time for Mellanox FDR and to move the AOC to a PCIe 3.0 slot.

Sweet! So the ES "QF8D" sample works? What BIOS and IPMI firmware did you update to?

 

[TD_Trader]

Eeeek! So here is a total let-down. I did a dual E5-2630L configuration:
http://linux-bench.com/parser.html using ID: 00051415159520

28GB of RAM. Now here is the extremely interesting part:
P2-DIMME1 TEMPN/ANot Present!

If I am reading that correctly, P2 = processor 2 DIMM E1 is missing!

Is it possible that one particular DIMM is different and/or possibly doesn't have a temperature sensor on the DIMM itself?

Maybe that "N/A" and "Not Present" means that the temperature sensor information is not being read correctly (or maybe possibly that the DIMM is defective/bad?)

Have you tried sticking a different DIMM into that slot/socket? Maybe swap two DIMMs and see what the temperature readings are. Maybe the DIMM is bad, or maybe it's just having a hard time reading the temp reading for that particular slot?

 

[Patrick]

Well it is also not showing up in the BIOS or OS so it is a bigger issue.

 

[Patrick]

Server processor set 4. Re-seated everything at the same time. Running linux-bench now.

CPU1 TempNormal30 degrees C
CPU2 TempNormal31 degrees C
System TempNormal32 degrees C
Peripheral TempNormal38 degrees C
PCH TempNormal54 degrees C
P1-DIMMA1 TEMPNormal35 degrees C
P1-DIMMB1 TEMPNormal34 degrees C
P1-DIMMC1 TEMPNormal41 degrees C
P1-DIMMD1 TEMPNormal39 degrees C
P2-DIMME1 TEMPNormal41 degrees C
P2-DIMMF1 TEMPNormal39 degrees C
P2-DIMMG1 TEMPNormal36 degrees C
P2-DIMMH1 TEMPNormal37 degrees C

 

[Patrick]

And the plot thickens. During Linux-Bench I am seeing:
CPU1 TempNormal43 degrees C
CPU2 TempNormal48 degrees C
System TempNormal35 degrees C
Peripheral TempNormal39 degrees C
PCH TempNormal55 degrees C
P1-DIMMA1 TEMPNormal42 degrees C
P1-DIMMB1 TEMPNormal40 degrees C
P1-DIMMC1 TEMPNormal46 degrees C
P1-DIMMD1 TEMPNormal45 degrees C
P2-DIMME1 TEMPNormal48 degrees C
P2-DIMMF1 TEMPNormal45 degrees C
P2-DIMMG1 TEMPNormal41 degrees C
P2-DIMMH1 TEMPNormal42 degrees C

Then.... at Execl Throughput (24 threads) it hangs and I see something like this:
CPU1 TempNormal38 degrees C
CPU2 TempNormal46 degrees C
System TempNormal32 degrees C
Peripheral TempNormal37 degrees C
PCH TempNormal52 degrees C
P1-DIMMA1 TEMPNormal44 degrees C
P1-DIMMB1 TEMPNormal43 degrees C
P1-DIMMC1 TEMPNormal49 degrees C
P1-DIMMD1 TEMPNormal47 degrees C
P2-DIMME1 TEMPN/ANot Present!
P2-DIMMF1 TEMPN/ANot Present!
P2-DIMMG1 TEMPN/ANot Present!
P2-DIMMH1 TEMPN/ANot Present!

Something is awry with this motherboard.

 

[Alfa147x]

I wonder if a capacitor is dead. My guess is under heavy load the RAM requires more power and the motherboard isn't able to deliver it.

That my guess.

 

[xnoodle]

Here's a run with 2xe5 2603's.

40071415358880

 

[Patrick]

OK well - I found a X9DRL-EF in the lab. Flashed that one and the Ubuntu server install worked:


One "minor" issue did arise. The X9DRL-EF has one less 4-pin fan header. I ended up using the expander backplane's 4-pin header to get around this.

 

[PigLover]

@PigLover did you happen to try a second SFF-8087 cable? There are two more ports on the backplane. If you could get an 8x lane back-haul that would be ideal.

@Patrick Question originally asked in the original deal thread but seems more appropriate to answer here.

I got a 9207-8i freed up today and tried dual-linking to the backplane. Works. Figured it would since the backplane expander is just a standard LSI 36 port expander chip, same one you find on the Chenbro, HP and Intel SAS expander boards that all support dual linking.

 

[Patrick]

@Patrick Question originally asked in the original deal thread but seems more appropriate to answer here.

I got a 9207-8i freed up today and tried dual-linking to the backplane. Works. Figured it would since the backplane expander is just a standard LSI 36 port expander chip, same one you find on the Chenbro, HP and Intel SAS expander boards that all support dual linking.

Awesome. Crazy question. Have you tried a dual ported drive also? It would be interesting if the backplane supported it.

 

[PigLover]

Have you tried a dual ported drive also?

Don't have any, sorry. But I think you have to have the EL2 backplane to properly support dual port operation.

 

[Patrick]

Don't have any, sorry. But I think you have to have the EL2 backplane to properly support dual port operation.

That is what I would think too. Maybe I can get a drive in there.

 

[Patrick]

I tried Linux-Bench on it... the four DIMMs on the second processor died again! Totally bizzare. This is a second motherboard too.

 

[Chuntzu]

Just found out the hard way about the power supplies.

 

[Patrick]

Just found out the hard way about the power supplies.

What happened?

 

[Chuntzu]

Bought a bunch of the 1400 watt power supplies for my cse-216 and cse-836 chassis and they didn't fit. The backplane connector is bigger on the 1400 watt supplies than the 1200 watt supplies. Lucky I was able to return them and order the 1200 watt supplies. I really should have checked my cse-747tq which already had the 1400 watt supplies but I was being a knuckle head and didn't check. Oh well lesson learned.

 

[firenity]

Spot on! These have the PCB that is more narrow. The 1.4kw and the 1280w-SQ are both the wider PCB.

If you're merely trying to get the PSU noise levels down and don't need the additional power, I think the PWS-920P-1R / PWS-920P-SQ (920W, Platinum) would be an option.