Intel Xeon E5-2670 Deal and Price Tracking

[JayG30]

The fans will ramp up when the P% Therm Margin falls below 20%... This is roughly ~80c core temp. Intel's approach is to balance the system noise with temps. There are various physical and virtual (aggregate) sensors all factoring into the BMC control of the cooling domains.

Stereodude had previously provided an excellent explanation of how he modified the SDR (sensor data records) to ramp the fans up sooner to maintain a larger P% Therm Margin. I myself haven't considered it necessary to do that.

Thanks. I assume you mean P1 Therm Ctrl % and P2 Therm Ctrl %. I haven't seen it report anything but 0 so far, lol.

80C does seem rather high considering the max temp looks like 82C. In the BIOS I noticed there was a section for "acoustic/power" (forget what it is called) that has like 4 preset behaviors. Does anyone know if this has any impact on this behavior? If not I might have to consider doing what Stereodude did as letting it go up to 80C doesn't make a lot of sense.

 

[jwegman]

Thanks. I assume you mean P1 Therm Ctrl % and P2 Therm Ctrl %. I haven't seen it report anything but 0 so far, lol.

No, I meant what I said. The Therm Ctrl % is the percentage of something being throttled. CPU throttling doesn't happen until the Therm Margin's get to 0% to protect the chip from over temp damage.

80C does seem rather high considering the max temp looks like 82C. ...

Intel only publishes the max tCase temp for this chip, not the max core temp. IMHO ~80c is not concerning for the heavily loaded core temps.

 

[JayG30]

No, I meant what I said. The Therm Ctrl % is the percentage of something being throttled. CPU throttling doesn't happen until the Therm Margin's get to 0% to protect the chip from over temp damage.

You said "P% Therm Margin" falls below 20%. I don't have a sensor with that name (P% Therm Margin). Margin sensor is expressed as degrees C. % sensor is called "Therm Ctrl", not Margin. The sensors on my system are;

P1 Therm Margin    Normal    OK    -67 degrees C
P2 Therm Margin    Normal    OK    -68 degrees C
P1 Therm Ctrl %    Normal    OK    0
P2 Therm Ctrl %    Normal    OK    0

I'm pretty sure I understand. As "P1 Therm Margin" approaches 0, "P1 Therm Ctrl %" will increase. That should cause an increase in fan speeds.

 

[jwegman]

You said "P% Therm Margin falls below 20%". I don't have a sensor with that name. The sensors on my system are called;

P1 Therm Margin    Normal    OK    -67 degrees C
P2 Therm Margin    Normal    OK    -68 degrees C
P1 Therm Ctrl %    Normal    OK    0
P2 Therm Ctrl %    Normal    OK    0

I'm pretty sure I understand. As "P1 Therm Margin" approaches 0, "P1 Therm Ctrl %" will increase. That should cause an increase in fan speeds.

Yeah, I was using P% not as a literal sensor name; but as a wildcard to represent P1|P2. I do see that I was stating 20% when I did mean -20deg.

When your CPUs starts heating up, the P1|P2 Thermal Margin will decrease from your idle value of ~-67 deg. When it reaches the~-20deg threshold, the BMC will start ramping up fans until things cool down and the margin rises above that threshold. On a healthy system, you shouldn't ever see the Therm Margins fall below the threshold for more than a few minutes while the BMC adjusts things to cool stuff down.

In a healthy system, I don't think that P1|P2 Therm Ctrl % should *ever* change from 0 unless something is seriously ****ed up with the cooling fans or something...

 

[LiQuiD[EViL]]

Ok, will update firmware and see how i can get it working. Java from Windows-VM usually works, at least on my supermicro boxes.

When everything fails there is still pxe or usb-cdrom to get an os on it. kvm and iso redirection is just so much simpler that it was naturally the first try.

You do have to use an older version of java. I have Java 7 Update 21 previously installed for other IPMI boards. It's working perfectly with the Intel S2600GL

 

[_alex]

You do have to use an older version of java. I have Java 7 Update 21 previously installed for other IPMI boards. It's working perfectly with the Intel S2600GL

thanks for the hint,
this is exactly the reason why i love Java that much ...

 

[jwegman]

FYI, for those interested; here's the temp/fan stat profile from a r2000 with dual e5-2670's running full bore (mprime) for 10 minutes:

top - 19:37:58 up 14 min,  4 users,  load average: 32.20, 29.65, 17.86
Tasks: 357 total,  1 running, 356 sleeping,  0 stopped,  0 zombie
%Cpu(s):  0.5 us,  0.2 sy, 99.2 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 65936676 total, 63495208 free,  1989772 used,  451696 buff/cache
KiB Swap:  0 total,  0 free,  0 used. 63472340 avail Mem

  PID USER  PR  NI  VIRT  RES  SHR S  %CPU %MEM  TIME+ COMMAND   
  5442 root  30  10 3857724 1.585g  4252 S  3179  2.5 391:59.87 mprime

Every 2.0s: sensors  Wed Apr 20 19:38:43 2016

coretemp-isa-0000
Adapter: ISA adapter
CPU Temp:  +79.0°C  (high = +90.0°C, crit = +100.0°C)
Core 0:  +78.0°C  (high = +90.0°C, crit = +100.0°C)
Core 1:  +78.0°C  (high = +90.0°C, crit = +100.0°C)
Core 2:  +78.0°C  (high = +90.0°C, crit = +100.0°C)
Core 3:  +75.0°C  (high = +90.0°C, crit = +100.0°C)
Core 4:  +75.0°C  (high = +90.0°C, crit = +100.0°C)
Core 5:  +77.0°C  (high = +90.0°C, crit = +100.0°C)
Core 6:  +77.0°C  (high = +90.0°C, crit = +100.0°C)
Core 7:  +74.0°C  (high = +90.0°C, crit = +100.0°C)

coretemp-isa-0001
Adapter: ISA adapter
Physical id 1:  +78.0°C  (high = +90.0°C, crit = +100.0°C)
Core 0:  +75.0°C  (high = +90.0°C, crit = +100.0°C)
Core 1:  +78.0°C  (high = +90.0°C, crit = +100.0°C)
Core 2:  +76.0°C  (high = +90.0°C, crit = +100.0°C)
Core 3:  +75.0°C  (high = +90.0°C, crit = +100.0°C)
Core 4:  +76.0°C  (high = +90.0°C, crit = +100.0°C)
Core 5:  +78.0°C  (high = +90.0°C, crit = +100.0°C)
Core 6:  +77.0°C  (high = +90.0°C, crit = +100.0°C)
Core 7:  +76.0°C  (high = +90.0°C, crit = +100.0°C)

i350bb-pci-0200
Adapter: PCI adapter
MB Temp:  +57.0°C  (high = +120.0°C, crit = +110.0°C)

Every 2.0s: ipmitool -H nas3-ipmi -U root -P xxxxxx...  Wed Apr 20 19:39:19 2016

SSB Therm Trip  | 0Dh | ok  |  7.1 |
BB P1 VR Temp  | 20h | ok  |  7.1 | 41 degrees C
Front Panel Temp | 21h | ok  | 12.1 | 15 degrees C
SSB Temp  | 22h | ok  |  7.1 | 49 degrees C
BB P2 VR Temp  | 23h | ok  |  7.1 | 39 degrees C
BB Vtt 2 Temp  | 24h | ok  |  7.1 | 44 degrees C
BB Vtt 1 Temp  | 25h | ok  |  7.1 | 33 degrees C
HSBP 1 Temp  | 29h | ok  |  7.1 | 24 degrees C
SAS Mod Temp  | 2Dh | ok  |  7.1 | 48 degrees C
Exit Air Temp  | 2Eh | ok  |  7.1 | 48 degrees C
LAN NIC Temp  | 2Fh | ok  |  7.1 | 57 degrees C
PS1 Temperature  | 5Ch | ok  | 10.1 | 31 degrees C
P1 Therm Margin  | 74h | ok  |  3.1 | -21 degrees C
P2 Therm Margin  | 75h | ok  |  3.2 | -21 degrees C
P1 Therm Ctrl %  | 78h | ok  |  3.1 | 0 percent
P2 Therm Ctrl %  | 79h | ok  |  3.2 | 0 percent
P1 DTS Therm Mgn | 83h | ok  |  3.1 | -21 degrees C
P2 DTS Therm Mgn | 84h | ok  |  3.2 | -21 degrees C
P1 VRD Hot  | 90h | ok  |  3.1 |
P2 VRD Hot  | 91h | ok  |  3.2 |
P1 Mem01 VRD Hot | 94h | ok  |  3.1 |
P1 Mem23 VRD Hot | 95h | ok  |  3.1 |
P2 Mem01 VRD Hot | 96h | ok  |  3.2 |
P2 Mem23 VRD Hot | 97h | ok  |  3.2 |
DIMM Thrm Mrgn 1 | B0h | ok  |  7.1 | -39 degrees C
DIMM Thrm Mrgn 2 | B1h | ok  |  7.1 | -38 degrees C
DIMM Thrm Mrgn 3 | B2h | ok  |  7.1 | -41 degrees C
DIMM Thrm Mrgn 4 | B3h | ok  |  7.1 | -42 degrees C
Agg Therm Mgn 1  | C8h | ok  |  7.1 | -16 degrees C
MIC 1 Margin  | C4h | ns  | 208.0 | Disabled
MIC 2 Margin  | C5h | ns  | 208.1 | Disabled

Every 2.0s: ipmitool -H nas3-ipmi -U root -P xxxxxx...  Wed Apr 20 19:39:38 2016

Fan Redundancy  | 0Ch | ok  | 29.1 | Fully Redundant
System Fan 1  | 30h | ok  | 29.1 | 5341 RPM
System Fan 2  | 32h | ok  | 29.2 | 5341 RPM
System Fan 3  | 34h | ok  | 29.3 | 5243 RPM
System Fan 4  | 36h | ok  | 29.4 | 5341 RPM
System Fan 5  | 38h | ok  | 29.5 | 5194 RPM
Fan 1 Present  | 40h | ok  | 29.1 | Device Present
Fan 2 Present  | 41h | ok  | 29.2 | Device Present
Fan 3 Present  | 42h | ok  | 29.3 | Device Present
Fan 4 Present  | 43h | ok  | 29.4 | Device Present
Fan 5 Present  | 44h | ok  | 29.5 | Device Present
PS1 Fan Fail  | A0h | ok  | 10.1 |

 

[f18ccx]

Just wanted to join in the conversation:

I picked up the Natex combo, and have it running in a SuperMicro 846, with the SQ power supplies, Noctua i4 heatsinks, and I've hand modified the S2600.spr bios file for the system fans so the machine is pretty quiet now.

My only issue is the PMBus cable coming out of the PSUs doesn't reach the PMBus connector on the S2600cp motherboard, it's too far away. Anyone know if it can be extended?

Other than that extremely happy with the setup!

 

[jdartnet]

Following up from my previous post:

Got the S2600CP combo from natex.us running on a Phanteks Entoo Pro Case. It's currently running Win 7 Ultimate.

So far, I am happy with the setup with the exception of two issues:

• LEDs are showing "b2h - DXE Legacy Option ROM init" and I can't find much info or understand what this could mean.
• I went ahead and updated the bmc/bios/fru/me update package from post #1161, but run into issues updating the fru. So far all my fans seem to run at full speed
  • are there any suggestions on where to plug in the case fans?
  • This case comes with a 140mm rear fan and 200mm front fan, and that's all I'm using.

For those using this case, I gave up on using the case's built in fan controller, since it seems like a better idea to have the motherboard control the fans rather than the controller, which draws all its power from the CPU1 connector.

 

[jdartnet]

Just wanted to join in the conversation:

I picked up the Natex combo, and have it running in a SuperMicro 846, with the SQ power supplies, Noctua i4 heatsinks, and I've hand modified the S2600.spr bios file for the system fans so the machine is pretty quiet now.

My only issue is the PMBus cable coming out of the PSUs doesn't reach the PMBus connector on the S2600cp motherboard, it's too far away. Anyone know if it can be extended?

Other than that extremely happy with the setup!

How did you go about hand modifying the s2600.spr file for the fans?

 

[nthu9280]

Just wanted to join in the conversation:

I picked up the Natex combo, and have it running in a SuperMicro 846, with the SQ power supplies, Noctua i4 heatsinks, and I've hand modified the S2600.spr bios file for the system fans so the machine is pretty quiet now.

My only issue is the PMBus cable coming out of the PSUs doesn't reach the PMBus connector on the S2600cp motherboard, it's too far away. Anyone know if it can be extended?

Other than that extremely happy with the setup!

Don't know whether you are in the L48 US.
If the cable is interchangeable with the one that come with Intel PDBs, I have a spare. I bought a PDB for spare and swapped the longer cable in p4216L chassis. PM me here.




Sent from my Nexus 6 using Tapatalk

 

[jdartnet]

I wasted few hours scratching my head as ESXi 6 wouldn't install saying that there were no NICs but I was able to get to BMC. Then I noticed both the NICs were disabled in BIOS.

All the 4 NICs were enabled in the S2600IP4 system I got from OEM XS.

Sent from my Nexus 6 using Tapatalk

So glad you mentioned this... my nics were disabled in the BIOS as well, and I hadn't considered to check for that! Man, first time working with a server board...I've learned a ton.

 

[nthu9280]

So glad you mentioned this... my nics were disabled in the BIOS as well, and I hadn't considered to check for that! Man, first time working with a server board...I've learned a ton.

I was able to get to the bmc web console & ipmi all on the same shared NIC. So I did not expect the NICs were disabled.

I downloaded esxi customizer tool ( William Lam / vmghetto ) to inject the drivers. Install still complained that there were no NICs

Sent from my Nexus 6 using Tapatalk

 

[f18ccx]

How did you go about hand modifying the s2600.spr file for the fans?

Download Intel® Server Board S2600CP Firmware Update Package for EFI

Download this, put it on a bootable flash drive. There's a S2600CP.sdr file in the top directory, it's a text file that defines all the sensors and fan ramp speeds. For a non-Intel chassis that's using the "slow ramp" fan option, the section you will want to modify is:

_SDR_TYPE C0
_SDR_TAG. 'OTHER'
_REC_LEN 0023

// Global Stepwise Curve Record
5B // Stepwise Curve ID

I modified the domain max to be 5 instead of 2, to match the Intel chassis's. That gives you 5 steps instead of 2. The format for the steps is temp on a line, followed by PWM in HEX between 00 and 63.

My setup is cool enough with the supermicro 80mm screamers that I used 00 (minimum) for all temperatures except the top one, I left that at 63 ( max speed) for safety.

Look on this site for more info:
Fan SDR Hacking for the Intel S1200V3RP Motherboard [updated] - Learning Bits

 

[mthompson176]

Following up from my previous post:

Got the S2600CP combo from natex.us running on a Phanteks Entoo Pro Case. It's currently running Win 7 Ultimate.

So far, I am happy with the setup with the exception of two issues:

• LEDs are showing "b2h - DXE Legacy Option ROM init" and I can't find much info or understand what this could mean.
• I went ahead and updated the bmc/bios/fru/me update package from post #1161, but run into issues updating the fru. So far all my fans seem to run at full speed
  • are there any suggestions on where to plug in the case fans?
  • This case comes with a 140mm rear fan and 200mm front fan, and that's all I'm using.

For those using this case, I gave up on using the case's built in fan controller, since it seems like a better idea to have the motherboard control the fans rather than the controller, which draws all its power from the CPU1 connector.

How is that case working out for you? I have been looking for a case for my Gigabyte GA-7PESH2 and was looking at this but was not sure if it would fit well. Since the S2600CP is EEB sized, it should be an option for me.

 

[jdartnet]

How is that case working out for you? I have been looking for a case for my Gigabyte GA-7PESH2 and was looking at this but was not sure if it would fit well. Since the S2600CP is EEB sized, it should be an option for me.

Yes, it's certainly an option. I did quite a bit of research for cases and I'm quite impressed at the quality and ease of cable management.

I only found two things that could be improved; one of the EEB holes is missing - not terrible, but something I noticed. Also, in my case I will need to order an extension for the EPS cable in my PSU. One of them (the one that goes to the #2 cpu) is too short to route behind the motherboard tray. Outside of these two issues, it is hands down the best case I've worked with for a home built workstation.

 

[jdartnet]

Download Intel® Server Board S2600CP Firmware Update Package for EFI

Download this, put it on a bootable flash drive. There's a S2600CP.sdr file in the top directory, it's a text file that defines all the sensors and fan ramp speeds. For a non-Intel chassis that's using the "slow ramp" fan option, the section you will want to modify is:

_SDR_TYPE C0
_SDR_TAG. 'OTHER'
_REC_LEN 0023

// Global Stepwise Curve Record
5B // Stepwise Curve ID

I modified the domain max to be 5 instead of 2, to match the Intel chassis's. That gives you 5 steps instead of 2. The format for the steps is temp on a line, followed by PWM in HEX between 00 and 63.

My setup is cool enough with the supermicro 80mm screamers that I used 00 (minimum) for all temperatures except the top one, I left that at 63 ( max speed) for safety.

Look on this site for more info:
Fan SDR Hacking for the Intel S1200V3RP Motherboard [updated] - Learning Bits

Wow, this is super helpful. Thank you!

I will follow up with results.

 

[f18ccx]

Yes, it's certainly an option. I did quite a bit of research for cases and I'm quite impressed at the quality and ease of cable management.

I only found two things that could be improved; one of the EEB holes is missing - not terrible, but something I noticed. Also, in my case I will need to order an extension for the EPS cable in my PSU. One of them (the one that goes to the #2 cpu) is too short to route behind the motherboard tray. Outside of these two issues, it is hands down the best case I've worked with for a home built workstation.

Yeah, for mine I needed both a 24pin main PSU extension and a 8 pin for CPU 2 power.

My case was used from eBay, had the original SAS expander, as I'm using 8TB drives I picked up a BPN-SAS2-846E1 as well from eBay. Was easy to swap. I liked the cleanness of having only 1 cable for all 24 drives.

 

[Roy68]

I have a Natex S2600CP board that shipped to UK from US that I'm in the process of building up now. I have just got the board to POST sitting on my desk with a newly acquired Fractal Design Newton 800w PSU and 64GB Samsung RAM. I only let the board run a few minutes since I was concerned about lack of airflow with the board sat on my desk with the supplied 1U copper CPU passive heatsinks, but knowing the board is OK has de-risked the project for me going forward. I have a couple of Cooler Master Hyper T4 heatsinks on order now.

Re the case, I ordered a B grade Phanteks Enthoo Pro case which seemed a bargain but arrived with front smashed so was returned. This turned out to be a blessing in disguise because what surprised me most was the fact this case is soooooo huge!!!! It is pretty much double the volume compared to the dual xeon HP Z600 workstation it will replace. I can live with a single SSD + HDD and a decent GPU, so I really don't want such a big case. I will use it for rendering in Blender plus general PC use.

Up to now my case search has focussed on SSI EEB form factor compatible cases, but since this is a server motherboard these cases are all huge, so I'm now also looking at E-ATX cases. I understand E-ATX also has 12"x 13" form factor as per SSI EEB, and generally all board locations are similar, apart from three of the middle board stand offs are in different locations, but this is easy enough to modify.

I like the Corsair Carbide Air 540 cubist style, apart from the window, but guess I could live with that.
Also Silverstone GD07 is quite small, but it's really a HTPC case.

Anyone else squeezed the S2600CP board into a smaller case??

 

[DavidVermillion]

Yes, it's certainly an option. I did quite a bit of research for cases and I'm quite impressed at the quality and ease of cable management.

I only found two things that could be improved; one of the EEB holes is missing - not terrible, but something I noticed. Also, in my case I will need to order an extension for the EPS cable in my PSU. One of them (the one that goes to the #2 cpu) is too short to route behind the motherboard tray. Outside of these two issues, it is hands down the best case I've worked with for a home built workstation.

Yeah, I noticed the missing EEB hole. I got around that by finding some old plastic standoffs and dremeling off the bottom post. Worked perfectly and the board is supported at that hole.