Intel S2600CP w/ Intel P4000 Case

[marcoi]

Build’s Name: TBD
Operating System/ Storage Platform: ESXI
CPU: 2 x E5-2670 SR0KX V1
Motherboard: Intel S2600CP
Chassis: Intel P4216XXMHGR
Drives: Replace 2.5 hhd with 3.5 bay for now
RAM: 128 (16x8gb) DDR3
Add-in Cards: M1015 crossflashed, intel RS25AB080 with ssd cache key
Power Supply: Intel 750 plat redundant
Other Bits:

Usage Profile: plans to be home test bed.
I flushed out that I will use this server as a development only box. I ended up signing up for VMUG to use some more advance vmware features and vcenter. Right now some of the key vms are:
WS2012R2 as a workstation with nested Virtualbox running.
OpenLDAP server - with gui front end for apps need ldap
pfSense server - provide internet dev network and limit access to my other network.
FreeNas Server with pci m1015 passthrough in IT mode
vCenter Server - manage the two home nodes dev box and prod box

Most of the vms will leverage the core vms listed aboveand the will be a mix bertween linux and windows environment.


Extras:
Intel FCPUPMAD Air Duct Required for Intel Server Board S2600CP
Intel AUPSRCBTP Single Passive Heat Sink
INTEL A4ULCP Local Control Panel 4U
Intel AXXRMM4R Remote Management Module Lite

Links:
*Parts:
Intel® Xeon® Processor E5-2670 (20M Cache, 2.60 GHz, 8.00 GT/s Intel® QPI) Specifications
Intel® Server Board S2600CP2 Specifications
Intel® Server Chassis P4216XXMHGR Specifications
Air Duct FCPUPMAD (Required for Intel® Server Board S2600CP In P4000M) Specifications
Heat Sink AUPSRCBTP Specifications
Local Control Panel A4ULCP Specifications
Remote Management Module AXXRMM4R Specifications

Build out:
3-21-16: I put the MB into the case today, updated the bios and started to figure out how to silence this beast down. So far I put two of inline resisters on the two included banshee fans. (Amazon.com: uxcell 5 Pcs 3 Pins Noise Reduction Cable Lead for PC Cooling Fan: Computers & Accessories)

3-25-16: Update: I got it all together now. Really nice case, I'm happy with it. Super quiet now with latest bios updates. I did run into issues with BMC, but got it resolved. I also ended up ordering full RMM4 module so will most likely sell the rmm4 lite. Now im testing out and deciding storage.

4-25-16: Updates: I had setup the new intel card with 3x3tB HDDS and ssd cache of 4x250gb ssd drives in raid 1x0. the 3 drives are setup in raid 5 with sdd cache enabled. This seems to work out great. I also just added 960gb ssd drive (need to stop visiting great deals) as extra storage area for VMs. On Freenas i have 2x2tb drives in raid 1 and 1 extra 1tb. I share out the space to linux boxes using nfs shares. Eventually I want to replace the 2 2tb drives with seagate 4tb 2.5 drives listed in the forums.

Pics so far:

MB with New heat sinks installed. Noticed they are wrong in this picture


MB and HS correctly installed in case.


Top down shot.

I got busy building so no more pics for now.

 

[epicurean]

Would you be interested to build another one exactly like this? And sell it ?

 

[marcoi]

Would you be interested to build another one exactly like this? And sell it ?

I would love too, but at the moment i have too much on my plate with my real job to consider doing this. That is why I'm doing the build here so others can gather the info and do it as well.

 

[Stereodude]

Did you update the BIOS, BMC, etc and do a FRUSDR update?

Why are you using resistors to slow the fans down? If you close the case, or defeat the chassis open switch they don't spin that fast. They are PWM controlled fans and if you set the BIOS to Acoustic and the lowest altitude setting they should both run under 33% PWM at idle unless the ambient temperature is very hot where you are.

Additionally, you can make them do pretty much anything you want if you edit the S2600CP.sdr file used by the UpdateFRUSDR.nsh update script. I actually made them spin faster under load (not idle) since I didn't like how hot the processors were allowed to get (~82C).

 

[marcoi]

Did you update the BIOS, BMS, etc and do a FRUSDR update?

Why are you using resistors to slow the fans down? If you close the case, or defeat the chassis open switch they don't spin that fast. They are PWM controlled fans and if you set the BIOS to Acoustic and the lowest altitude setting they should both run under 33% PWM at idle unless the ambient temperature is very hot where you are.

Additionally, you can make them do pretty much anything you want if you edit the S2600CP.sdr file used by the UpdateFRUSDR.nsh update script. I actually made them spin faster under load (not idle) since I didn't like how hot the processors were allowed to get (~82C).

Yes i applied latest FW package. I figured out the case switch triggers the fans after playing with it into the wee hours last night. I removed one resistor from the cpu fan as im testing today to see what happens. I may get into updating the sdr file after playing with the system a bite more.

IS there a way to control the startup spin up of fans both internal and in the psu?

 

[marcoi]

Inside the case, im experimenting with additional fans. I might get some smaller low RPM fans and attach them to the heat sink.



My 3 pin resistors on the two system fans. I did end up removing them as this thing is quiet under normal load, only a banshee on startup. Still need to run prime 95 to load it up and see how hot it gets.




Air hood in place


Rear view


The 3.5 hdd trays are nice. It includes bracket to use 2.5 hdds as well. Nice touch i wasnt expecting. I plan on using some 3.5hdd icy dock bays to fit 8 2.5 ssd in one of the free bays. So when im done, ill have spots for 8 3.5 hdds and 8 2.5 hdds.

 

[Stereodude]

FYI: You don't have the black plastic air duct installed correctly. It should sit on the CPU's heatsink, not be about 1" off it. Something is under the ducting holding it up. You also don't want to put another fan in there between the heatsinks. It will act as an obstruction to the airflow.

 

[marcoi]

FYI: You don't have the black plastic air duct installed correctly. It should sit on the CPU's heatsink, not be about 1" off it. Something is under the ducting holding it up. You also don't want to put another fan in there between the heatsinks. It will act as an obstruction to the airflow.

yeah i had the extra fan under there. i since removed it and the resistors. Only real noise is on boot up or restart for the fans and PSU. Not sure if there is a way to limit that?

 

[Stereodude]

Only real noise is on boot up or restart for the fans and PSU. Not sure if there is a way to limit that?

I'm not sure. They really do go to 100% duty. There might be something in the S2600CP.sdr file, but off the top of my head I'm not sure which value in which section might change it. I've changed some other characteristics of the cooling by modifying the .sdr file already. I will take a look at it later.

Edit: I tried the three settings I thought might change it. None of them did. It's possible there is some tweak that can be made to the .sdr file, but I don't know what it might be.

 

[marcoi]

Im starting to get the socket error - error opening video socket when i launch the kvm console. Any ideas on how to fix it?
So far I reflashed the bmc bios to latest, left the unit powered off completely for several hours, updated java, tried on multiple pcs.

 

[marcoi]

i think i corrupted the rmm4 lite chip, so i ordered the rmm4 package with its own nic. Here hoping it will resolve my issue with kvm.

 

[Stereodude]

The BMC over LAN will work without it. Just not the iKVM or remote media mounting. So, I'm not sure that will solve your problem.

 

[marcoi]

The BMC over LAN will work without it. Just not the iKVM or remote media mounting. So, I'm not sure that will solve your problem.

Will The Web console work without the rmm key? Should I pop out the battery or try clear bios once more?

 

[Stereodude]

Will The Web console work without the rmm key?

That's what I was told by @jwegman. I didn't try it myself.

 

[marcoi]

I got it working again. clearing the cmos battery didnt help. I had to use syscfg tool from efi bios. Below are the steps if anyone else runs into it.

 

[Stereodude]

I had to use syscfg to modify one of the BMC accounts on my S2600CP2J so I could log into it.

 

[marcoi]

Latest addition was a set of caster wheels added on with 3m double sided tape. so far i have moved it too much as im not sure how strong the 3m tape is. Testing if i like it on wheels vs sitting on my desk. No space for a rack so trying out different solutions.

 

[marcoi]

Next is testing, I'm also waiting on two ram replacements and new rmm4 module since i though my issues with bmc was related to the rmm4 lite.

I'm also considering storage. Right now in my current prod WS2012 VM Server I have several 3 tb hdds being used for data storage and backups. I also have a 500gb ssd consumer drive used for vm hosting on prod server. I plan on using ibm m5015 card ( if the stupid battery starts working on it) and M1015 flashed to IT mode. M5015 setup for esxi storage and the m1015 pass-through to WS2012 VM with HDD.

Right now I have 8 3.5 bays that would hold the larger HDDs for my home storage. At least 4 bays would would end up on the m1015 pass-through to WS2012, the other 4 might be used on the M5015 to create a large storage pool for VM using Raid 5. I also plan on getting a 4 or 8 bay ssd cage that fits in a 5.25 bay and use some consumer ssds to fill it. Right now on my main rig, i had 8 256gb ssd drives in two raid 0 pools (4 drives per pool) on m5015 card. I might move 4 of the ssd overs to the server and do all my vm work on it now that i have UNLIMITED RESOURCES ( not really but it does feel that way haha)

Below are ssd benchmarks. D drive is first set of drives. (better ssd drives) E drive is second set of drives ( older thus slower)



So any ideas on storage ( that wont break the bank) I plan on make a lot of VMs and leaving them around as I update software versions of the enterprise software i work with I want to keep samples around for testing.

 

[pricklypunter]

Large Storage Pool + Raid5 = Tears

 

[Stereodude]

Large Storage Pool + Raid5 = Tears

With non-enterprise HDDs. With SSDs, are unrecoverable bit errors still a big issue?