Intel S2600CP/P4000, 2 servers

[katit]

Build’s Name: Office servers (building 2 at the same time)
Operating System/ Storage Platform: Windows Server 2012 R2 and mix of stuff in Hyper-V
CPU: E5-2670
Motherboard: Intel S2600CP(2??)
Chassis: Intel P4000
Drives: TBD. OS host drives Intel SB3500 80Gb
RAM: 128GB
Add-in Cards: none
Power Supply: 550W
Other Bits: Remote management modules

I decided to document this build because I had to look for some things and had some questions. So I will put all I found here.

First of all - list of parts (x2 of everything)
Intel P4216XXMHEN Server Chassis 4U Rack, 550W New Bulk Packaging
Intel AXX2IOS I/O Shield for S2600CP2, S2400SC2, and S2400GP2 New Bulk Packaging
Refurbished S2600CP Server Motherboard with memory and CPU - seems like they out of stock now
Intel Heat Sink AUPSRCBTP-Newegg.com - x2
Intel FCPUPMAD Air Duct Spare for S2600CP In P4000 - Newegg.com
Intel Remote Management Module-Newegg.com
Intel S3500 80 GB,Internal, 2.5" (SSDSC2BB080G401) SSD

I decided I would start with one server, see how it goes and then build second one.

ASSEMBLY:
1. Installed RAM on motherboard. Boards came in with CPUs installed but all RAM (32 sticks) were in separate box
2. Installed standoffs, I/O shield and motherboard.
3. Installed heatsinks. Had to go to HD to get longer screwdriver...
4. Did all the wiring per Intel instructions for S2600CP board
5. Installed RMM. This document has all the information: http://www.intel.com/content/dam/su...rver/sb/intel_rmm4_bmc_ews_userguide_r2_8.pdf

That was it for assembly. Not much. For those boards 24pin power extension cable needed but it was provided with a air duct kit.

SETUP:
First what I didn't like. I spent some time on it and ended up hooking up KVM which I hoped to avoid by getting RMM..
So, I powered up server with only LAN cable in RMM LAN jack. I expected to see new device on network. There is LAN activity on port, but nothing in DHCP client list. Bummer. Had to move server to desk with VGA monitor and hook up keyboard...
I wondered around a little trying to understand what to update (firmware) where to get, etc. I did multi-step update to each module, etc, etc. So, here is straight instruction on how to do it quick and easy:

1. Download latest set of firmware: Download Intel® Server Board S2600CP Firmware Update Package for EFI
2. Put it on USB stick. Insert USB into USB port
3. In UEFI shell type
> map -r
> fs0:

Those commands will put you in USB drive (sorry folks I guess most of you know this stuff but for some it may save some time)

Use "cd S26..." to get into folder (if you copied whole folder)
And then just type
>Startup.nsh

This UEFI script will run all firmware updates needed.

4. GET RMM working. I linked RMM manual under ASSEMBLY. Basically, you need to set RMM to "Dynamic IP" so it get's IP assigned by DHCP. You will need to setup user as well. I really wish it was working by default so I don't have to carry servers out to other desk.

5. In BIOS set fans to "ACOUSTIC". They become much better. Not quiet. But better. Next I will try to edit .sdr file, but it will be another day..

TODO ITEMS:

1. Figure out WHY I get this "warning" triangle on front panel and on back. Is it because no storage hooked up yet?
2. See if I can get it to be more quiet
3. Setup some OS quickly and run memory/CPU tests for couple of days

SOME PICTURES OF BUILD:

 

[Zathras]

I built pretty much the exact same thing on Saturday, except I got the case with the dual 750W power supplies. I was pleasantly surprised with how easy it was, everything I needed was included and the update utility was nice and straightforward. Reaching some motherboard ports near the edges was tricky once it was installed, and I did the exact same thing you did with taping the fan leads out of the way while I put the board in.

I knew to expect that the NICs would be disabled in the BIOS to start with, and that the RMM4 would need to be configured, so I had already planned to have a keyboard/monitor handy. It would be nice if it worked out of the box like my SuperMicro boards, but it was a minor annoyance.

Not sure why you'd be getting a warning light, but then I had to do exactly zero troubleshooting with the system so I didn't get to practice diagnosing it. I almost hoped something would go wrong!

 

[katit]

Not sure why you'd be getting a warning light, but then I had to do exactly zero troubleshooting with the system so I didn't get to practice diagnosing it. I almost hoped something would go wrong!

1. Ok, this one is user error. This is "Status LED" on front panel. It can be Green/FlashingGreen/Yellow/FlashingYellow according to documentation.

It is solid green - means everything OK. Does you case have this light? I didn't read specs first so fact that it's triangle with exclamation mark made me thing something is not as it should be

2. I lowered Altitude and fans started to spin even slower. Not sure I will be able to do much more. There is some kind of weird pinch noise. Not loud but annoying. However, in office where it will be running I won't hear it so it is OK

3. MemTest went through just fine, did some CPU burn in tests, temps holding fine.


Still didn't install OS yet (waiting on drives). But don't anticipate much problems in that department
Yes, overall build went good. Still need to get cables better organized after storage is here..

 

[jwegman]

Did you use the little rubber motherboard supports that get 'sticky/stuck' to the bottom of the board? Most people don't notice these (they come supplied with the P4000 chassis in a little zip lock baggie). They are a very good idea to use as the board can flex quite a bit without them.

 

[katit]

I DID NOT! And thanks for reminding, because I was thinking what is this for
Where did you put them? Is there special spots on motherboards?

 

[TType85]

Once you get update the SDR and tell it what fans are there it gets quieter. I told mine it was a custom chassis and I used FAN4/5 and it quieted down a bunch.

 

[katit]

Once you get update the SDR and tell it what fans are there it gets quieter. I told mine it was a custom chassis and I used FAN4/5 and it quieted down a bunch.

Sorry, but what is SDR? And how do I get there? I think I used FAN1/2 as per manual

 

[TType85]

when you got through the uefi script after it flashes everything you are presented with a few options. I think it was the first option that lets you assign fans. If I tell it I have the P4000M case (one of the options) it doesn't set the fans right, I tell it custom and select what fans are plugged in.

 

[jwegman]

I DID NOT! And thanks for reminding, because I was thinking what is this for
Where did you put them? Is there special spots on motherboards?

Yes, there are little white circles on the bottom of the mobo as placement markers for the rubber supports... This is documented in step #3 of the s2600cp quick start guide

Quick Start Guide for Intel® Server Board S2600CP

 

[katit]

Hm. I ran Startup.nsh and there was no such option. I will try individual scripts and see..

 

[jwegman]

when you got through the uefi script after it flashes everything you are presented with a few options. I think it was the first option that lets you assign fans. If I tell it I have the P4000M case (one of the options) it doesn't set the fans right, I tell it custom and select what fans are plugged in.

This is most likely because you have your fans plugged into sockets other than what Intel expected you to use. If you have the (2)120mm fixed fan P4000 chassis, the fans need to be plugged into sockets 1 and 2. This is also documented in the quick start guide

 

[katit]

This is most likely because you have your fans plugged into sockets other than what Intel expected you to use. If you have the (2)120mm fixed fan P4000 chassis, the fans need to be plugged into sockets 1 and 2. This is also documented in the quick start guide

I got this part from quick start. But I missed rubber supports part. Guess need to re-read quick start guide

 

[TType85]

This is most likely because you have your fans plugged into sockets other than what Intel expected you to use. If you have the (2)120mm fixed fan P4000 chassis, the fans need to be plugged into sockets 1 and 2. This is also documented in the quick start guide

That damn quick start guide! Manuals, we don't need no stinking manuals! Next time I shut down the server I will move them and re-do the setup.

 

[jwegman]

That damn quick start guide! Manuals, we don't need no stinking manuals! Next time I shut down the server I will move them and re-do the setup.

...and the (2)120mm fans are color coded to the correct FAN1|FAN2 header they belong with... That's important as fan 1 is responsible for cooling the PCI cooling domain, while fan 2 is responsible for the CPU cooling domain (the cooling domains are controlled by the BMC and it's various sensors).

 

[katit]

Installed bumper pads on UNIT01, got all organized. Used SATA->SAS cables that came with it (they work!)

Now to the questions.. What do you see as far as fan speeds? This is what mine shows. System has slight "wining" noise. Does it show what yours show?

 

[TType85]

This is most likely because you have your fans plugged into sockets other than what Intel expected you to use. If you have the (2)120mm fixed fan P4000 chassis, the fans need to be plugged into sockets 1 and 2. This is also documented in the quick start guide

Got the fans switched to 1/2 and re-setup the chassis. It's even quieter now

 

[jwegman]

Now to the questions.. What do you see as far as fan speeds? This is what mine shows. System has slight "wining" noise. Does it show what yours show?

Technically, your PWM Fan Offset should be set to 0, otherwise it adds that PWM offset to the PWM value that the BMC is commanding.

Anyway, that's about right for my two dual 2670's in the p4000m chassis's (both have the (2) fixed 120mm fans)...

I'm currently running with Quiet Fan Idle Mode enabled, which can stop a fan if the BMC's aggregate therm sensor is below the threshold, resulting in a 0.00 reading or Not Available when they are not spinning; as I have some hot PCIe cards installed in both machines (couple of 8 port SAS2 HBAs, 10GBe, USB3, and Nvidia GPU), cooling domain 1 (fan 1) will typically always be spinning:

System 1 at idle:

george@ubuntuvm1:~$ ipmiutil sensor -c -g fan -N nas1-ipmi -U root -P xxxxxxxx
ipmiutil ver 2.97
isensor: version 2.97
-- BMC version 1.25, IPMI version 2.0
ID  | SDRType | Type            |SNum| Name             |Status| Reading
001a | Full    | Fan             | 30 | System Fan 1     | OK   | 1218.00 RPM
001b | Full    | Fan             | 31 | System Fan 2     | Init  | 0.00 RPM
ipmiutil sensor, completed successfully

System 2 at idle:

george@ubuntuvm1:~$ ipmiutil sensor -c -g fan -N nas2-ipmi -U root -P xxxxxxxx
ipmiutil ver 2.97
isensor: version 2.97
-- BMC version 1.25, IPMI version 2.0
ID  | SDRType | Type            |SNum| Name             |Status| Reading
0018 | Full    | Fan             | 30 | System Fan 1     | OK   | 1218.00 RPM
0019 | Full    | Fan             | 31 | System Fan 2     | Init  | 0.00 RPM
ipmiutil sensor, completed successfully

Here I loaded both the boxes down for ~15 minutes with prime95 on all 32 cores:

System 1:

Cpu(s):  3.8%us,  0.7%sy, 95.4%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  65936068k total, 55930656k used, 10005412k free,      228k buffers
Swap:        0k total,        0k used,        0k free, 28660888k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                       
185714 root      30  10 3766m 1.6g 4028 S 3017  2.5 467:34.02 mprime                                                                    
....
root@nas1:~# sensors
i350bb-pci-0700
Adapter: PCI adapter
MB Temp:      +58.0°C  (high = +120.0°C, crit = +110.0°C)

coretemp-isa-0000
Adapter: ISA adapter
CPU Temp:     +79.0°C  (high = +90.0°C, crit = +100.0°C)
Core 0:       +79.0°C  (high = +90.0°C, crit = +100.0°C)
Core 1:       +75.0°C  (high = +90.0°C, crit = +100.0°C)
Core 2:       +75.0°C  (high = +90.0°C, crit = +100.0°C)
Core 3:       +76.0°C  (high = +90.0°C, crit = +100.0°C)
Core 4:       +75.0°C  (high = +90.0°C, crit = +100.0°C)
Core 5:       +75.0°C  (high = +90.0°C, crit = +100.0°C)
Core 6:       +75.0°C  (high = +90.0°C, crit = +100.0°C)
Core 7:       +76.0°C  (high = +90.0°C, crit = +100.0°C)

coretemp-isa-0001
Adapter: ISA adapter
Physical id 1:  +74.0°C  (high = +90.0°C, crit = +100.0°C)
Core 0:         +70.0°C  (high = +90.0°C, crit = +100.0°C)
Core 1:         +73.0°C  (high = +90.0°C, crit = +100.0°C)
Core 2:         +72.0°C  (high = +90.0°C, crit = +100.0°C)
Core 3:         +73.0°C  (high = +90.0°C, crit = +100.0°C)
Core 4:         +70.0°C  (high = +90.0°C, crit = +100.0°C)
Core 5:         +74.0°C  (high = +90.0°C, crit = +100.0°C)
Core 6:         +73.0°C  (high = +90.0°C, crit = +100.0°C)
Core 7:         +73.0°C  (high = +90.0°C, crit = +100.0°C)

System 2:

top - 08:46:19 up 29 days, 20:22,  2 users,  load average: 36.50, 35.37, 23.87
Tasks: 418 total,   1 running, 417 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.7 us,  2.6 sy, 95.7 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 65936744 total, 17389884 free, 19772788 used, 28774072 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 44988648 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                 
132814 root      30  10 3857864 1.595g   4112 S  3056  2.5 478:58.10 mprime                                                          
...
coretemp-isa-0000
Adapter: ISA adapter
CPU Temp:     +77.0°C  (high = +90.0°C, crit = +100.0°C)
Core 0:       +74.0°C  (high = +90.0°C, crit = +100.0°C)
Core 1:       +76.0°C  (high = +90.0°C, crit = +100.0°C)
Core 2:       +76.0°C  (high = +90.0°C, crit = +100.0°C)
Core 3:       +73.0°C  (high = +90.0°C, crit = +100.0°C)
Core 4:       +76.0°C  (high = +90.0°C, crit = +100.0°C)
Core 5:       +77.0°C  (high = +90.0°C, crit = +100.0°C)
Core 6:       +76.0°C  (high = +90.0°C, crit = +100.0°C)
Core 7:       +75.0°C  (high = +90.0°C, crit = +100.0°C)

coretemp-isa-0001
Adapter: ISA adapter
Physical id 1:  +77.0°C  (high = +90.0°C, crit = +100.0°C)
Core 0:         +75.0°C  (high = +90.0°C, crit = +100.0°C)
Core 1:         +75.0°C  (high = +90.0°C, crit = +100.0°C)
Core 2:         +76.0°C  (high = +90.0°C, crit = +100.0°C)
Core 3:         +75.0°C  (high = +90.0°C, crit = +100.0°C)
Core 4:         +75.0°C  (high = +90.0°C, crit = +100.0°C)
Core 5:         +75.0°C  (high = +90.0°C, crit = +100.0°C)
Core 6:         +77.0°C  (high = +90.0°C, crit = +100.0°C)
Core 7:         +74.0°C  (high = +90.0°C, crit = +100.0°C)

george@ubuntuvm1:~$ echo "## System 1: " && ipmiutil sensor -c -g fan,temp -N nas1-ipmi -U root -P xxxxxx && echo "## System 2: " && ipmiutil sensor -c -g fan,temp -N nas2-ipmi -U root -P xxxxxx
## System 1:
ipmiutil ver 2.97
isensor: version 2.97
-- BMC version 1.25, IPMI version 2.0
ID  | SDRType | Type            |SNum| Name             |Status| Reading
000c | Compact | Temperature     | 0d | SSB Therm Trip   | OK   |
0010 | Full    | Temperature     | 20 | BB EDGE Temp     | OK   | 32.00 C
0011 | Full    | Temperature     | 21 | Front Panel Temp | OK   | 24.00 C
0012 | Full    | Temperature     | 22 | SSB Temp         | OK   | 47.00 C
0013 | Full    | Temperature     | 23 | BB BMC Temp      | OK   | 49.00 C
0014 | Full    | Temperature     | 24 | BB P2 VR Temp    | OK   | 62.00 C
0015 | Full    | Temperature     | 25 | BB MEM VR Temp   | OK   | 50.00 C
0016 | Full    | Temperature     | 29 | HSBP 1 Temp      | OK   | 26.00 C
0017 | Full    | Temperature     | 2a | HSBP 2 Temp      | OK   | 26.00 C
0018 | Full    | Temperature     | 2e | Exit Air Temp    | OK   | 31.00 C
0019 | Full    | Temperature     | 2f | LAN NIC Temp     | OK   | 58.00 C
001a | Full    | Fan             | 30 | System Fan 1     | OK   | 1204.00 RPM
001b | Full    | Fan             | 31 | System Fan 2     | OK   | 1960.00 RPM
0022 | Full    | Temperature     | 5c | PS1 Temperature  | OK   | 42.00 C
0023 | Full    | Temperature     | 5d | PS2 Temperature  | OK   | 35.00 C
0028 | Full    | Temperature     | 74 | P1 Therm Margin  | OK   | -21.00 C
0029 | Full    | Temperature     | 75 | P2 Therm Margin  | OK   | -26.00 C
...
ipmiutil sensor, completed successfully

## System 2:
ipmiutil ver 2.97
isensor: version 2.97
-- BMC version 1.25, IPMI version 2.0
ID  | SDRType | Type            |SNum| Name             |Status| Reading
000b | Compact | Temperature     | 0d | SSB Therm Trip   | OK   |
000f | Full    | Temperature     | 20 | BB EDGE Temp     | OK   | 33.00 C
0010 | Full    | Temperature     | 21 | Front Panel Temp | OK   | 26.00 C
0011 | Full    | Temperature     | 22 | SSB Temp         | OK   | 46.00 C
0012 | Full    | Temperature     | 23 | BB BMC Temp      | OK   | 50.00 C
0013 | Full    | Temperature     | 24 | BB P2 VR Temp    | OK   | 62.00 C
0014 | Full    | Temperature     | 25 | BB MEM VR Temp   | OK   | 49.00 C
0015 | Full    | Temperature     | 29 | HSBP 1 Temp      | OK   | 30.00 C
0016 | Full    | Temperature     | 2a | HSBP 2 Temp      | OK   | 28.00 C
0017 | Full    | Temperature     | 2f | LAN NIC Temp     | OK   | 60.00 C
0018 | Full    | Fan             | 30 | System Fan 1     | OK   | 1190.00 RPM
0019 | Full    | Fan             | 31 | System Fan 2     | OK   | 2180.00 RPM
001e | Full    | Temperature     | 74 | P1 Therm Margin  | OK   | -22.00 C
001f | Full    | Temperature     | 75 | P2 Therm Margin  | OK   | -23.00 C
...
ipmiutil sensor, completed successfully

 

[katit]

Those 2 settings did it!!
I don't have any add-on cards, in my case quiet shuts down FAN2(CPU fan) and FAN1 is at around 1150
System is very quiet now. Not like my purposely built desktop but it is really quiet.

When I run Prime 95 FAN2 turns on and get's as high as 1800RPM, but then calms down within a minute.

At this point I consider this build to be complete!

 

[katit]

Hm.. The more I read about - more I want to add something else to those servers (or at least one)
I do .NET development and wanted to move my work environment to one of those boxes. Currently I use MacPro with VM under Fusion..

I always thought that GPU is something not necessary on VM servers. But now I read about RemoteFX and it looks like this is EXACTLY what I need for my dev VM

Any suggestions for budget GPU that I can use on those servers? Preferably without fans And I need it for 1-2 users max..

 

[katit]

Been 8 years almost!! Those servers still run. But I need to do and update to software.
I run Windows 2012 R2 server with Hyper V and few VMs (one windows and few Linux)

Question to all: Is this server still OK for Windows server 2022?
I want to renew storage (new drives) but CPU/RAM totally OK by me. I am just not sure if this hardware will support 2022 running and if it's OK..