C7000 estimated power consumption for a specific setup

[ostlere]

Hi

I'm new to the forum but I'm reading the post a few weeks ago. I'm planning a small render farm on a very tight budget for 3D rendering. Currently I'm looking an HP C7000 Blade Centre from ebay with the following specs:

HP BladeSystem C7000 Enclosure G2 w/ 16 x BL460c G6 Server Blades 32 x QUAD CORE E5620 128 Cores / 256 Threads 512GB RAM /32GB per node / 16x146GBSAS

I read some horror stories of this units power consumption. Could you help me estimate the power consumption of this exact setup? The thypical workload of my 3D job are 100% load when running a frame, but I don't need them to be constantly running 24/7/365 maybe 2-4 day a week on full load with some idle aswell.

I would greatly appreciate any given advice, thanks in advance!

 

[Evan]

C7000 is really only a chassis, power consumption depends all on the nodes.

Now G6 generation is a power hog compared to Gen8/9, at least in lower loaded situations.
I could only give power info for a fully loaded e5 v2 setup at the oldest but what you could do is search and compare DL380 consumption at load with the different CPU generations.

(At low load the G6 must be using almost double the Gen8 guessing), at high load I suspect a lot less difference though.

 

[Deci]

A BL460c G6 blade with dual 5620 cpus and all dimm slots fully populated reports ~260w of allocated power in my C7000s.

Running prime95 to peg all 8c/16t at 100% shows the total power usage of the enclosure changes by ~180w between idle and 100%, my enclosures total power numbers are not directly comparable as a total though as they are all a mix of different generation and model/amd/intel blades, but if you assume those numbers of ~80w idle and ~260w loaded each blade you are looking at approximately 4200w full load and 1300w idle.

Im not sure if you have ever heard a C7000 when the fans ramp up under load? i assume you have somewhere that sound isnt an issue as it makes a lot of it.

 

[markarr]

This is what you are looking for it is the HP power advisor, it will give you a ball park on idle and load wattages on any of their equipment.

HPE Power Advisor

 

[Jb boin]

A C7000 chassis with 16 BL460c G6 with on each 2*L5630, 24Gb of RAM and 1 SAS 10KRPM disk with average CPU load (system load is between 1 and 2 on a daily average), 1 Cisco switch and 1 passthrough module does between 2000 and 2250w (at 220v) on the last 24 hours.
So per node it does an average between 125w and 140w.

On another chassis with the same configuration except filled with L5640 instead of L5630 and only 5 nodes started it draws between 900w and 960w on the last 24 hours.
So per node it does an average between 180w and 192w but its to note that they also recieve ~30% more queries than the L5630 nodes does.


In any case i would recommand to go for L56xx rather than E5620 as it might not even be faster and will be definitely more power hungry, L5640 (60w TDP) also have 12 threads instead of 8 for the L5630 (40w TDP) and E5620 (80w TDP) and still manages to have a turbo speed higher than the 2 others (but the max non-turbo speed is at 140mhz less). Last year the L5630 and L5640 were very cheap, a little bit less than the E5620 i think but that wouldnt have been more than a 10-20€ premium per CPU.

You can also use more powerful CPUs and use less nodes which might also be a better solution especially if you might not keep all the nodes at high load all the time.
For example E5645 have almost the same specs as the E5620 but 12 threads instead of 8 for the same TDP and also could be found at a good price.

 

[fractal]

I concur with the advise to consider E5645's instead of E5620's. I realize that E5620's are dirt cheap and E5645's are not but idle power will be about the same no matter what processor you use. E5645's are very well suited to dense packing and the extra cores are beneficial even when limited by DDR3.

I would not bother with L56xx's for a render box. Turn it off when you aren't using it. You want as much get up and go as you can when it is running. I replaced the L5640's in my 1u twin with E5645's and love it. Idle power did not increase much but compute power along with electric bill is noticeably larger when the pedal hits the metal. Given what I know today, I would probably leave the E5620's in a crunch box rather than the L56xx's when the work loads are predictable and I can turn the box off when not needed.

 

[lzvolensky]

bear in mind there is additional power consumption by enclosure fans - easily 500W - and FlexFabric modules etc. So you are easily looking at 5000W total for 16 blades with 128 physical cores. Additional cost will be tremendous amount of heat generated by this set : additional kilowatts have to be expected along the way. Count how many amperes you need - will you get them ?

To supply enclosure with such hefty amount of power, you need to have the most powerful power supplies, I believe those are 2650W each. Six of them. They can run in N+N or N+1 redundancy configuration with obvious advantages and disadvantages. And efficiency. Add ~300 to 500W at your plug.

DON'T FORGET THE NOISE.

What I would do : what software you use ? Instead of CPU rendering, can you possibly utilize GPU to render ? I'd buy a couple of graphic cards, hook 'em to workstation-class computer and that's it. Easily re-sell STILL IN WARRANTY when finished couple of months later.

Not so easy to resell old outdated unsupported blades. Shipping...

 

[Jb boin]

You can have a C7000 filled with 16 blades for 100-150€ per blade with 2*L5640 (or E5645 as their price is similar) and 24Gb of RAM on each, the chassis with a Cisco switch and a passthrough module, 10*fans and 6*2400w PSUs but if you dont need that much nodes you can also go for Dell C6100 chassis that will cost around the same per node but if you really need 16 nodes its not as easy to administer as each node have its own IPMI interface.

To add a detail to the informations from a few posts back, the chassis filled with L5630s does at the moment 2186w/8755w (its the input power) on AC Redundant mode (the 3 first PSUs are on a separate power line than the 3 last so the power is always shared beetween at least 2 PSUs, limiting the efficiency and the possible peak power) with only 2 of the 2400w PSUs used (Dynamic Power mode is enabled wo it only starts up the least number of required PSUs instead of exploiting all of them with less load on each).

 

[aleksdj]

Opening this old forum again, but I am really confounded since it’s all new to me and I am only trying to set up a render farm. At this point I have the c7000 enclosure in the rack with the switcher, but I’m curious to find out what kind of an outlet do I need to power it. It’s one single c7000 enclosure with 8 g7 blades. Each one has a dual 6-core Xeon. I have c19 to schuko cables only at this point, and I was gonna plug three in one and three in another power strip and just plug it in the wall. But, from the looks of it, I may need a piece of equipment I don’t have. Do I need to plug the enclosure into an adapter first, or can I just plug straight to the electricity. Power seems pretty solid at the place, we have 12 workstations running on it no problem, but it’s an office building. This enclosure is in its own office, and I’m just staring at it as I’m pretty scared to just plug it into the outlet. Any real advice on actual plugging it in would be appreciated.

 

[zunder1990]

If you have about the same power usage as the above listed of ranging up to 2250watts and in 120volt land putting all PSU on the same breaker will not end well. 2250 watts at 120 volt is 18.75 amps, code says for consistence load like this you would not go over 80% of the breaker limit. If you go over that 80% mark it can over heat the wire or breaker. Also remember that voltage swings up and down so if the voltage drops to say 114 volt(which can happen) is 19.73 amps. By code normal outlets in the USA can not go over 20 amps. IF you are in 120volt land then you should connect to 30amp outlet using a PDU or two different 20amp outlets.

 

[aleksdj]

If you have about the same power usage as the above listed of ranging up to 2250watts and in 120volt land putting all PSU on the same breaker will not end well. 2250 watts at 120 volt is 18.75 amps, code says for consistence load like this you would not go over 80% of the breaker limit. If you go over that 80% mark it can over heat the wire or breaker. Also remember that voltage swings up and down so if the voltage drops to say 114 volt(which can happen) is 19.73 amps. By code normal outlets in the USA can not go over 20 amps. IF you are in 120volt land then you should connect to 30amp outlet using a PDU or two different 20amp outlets.

Probably should’ve also mentioned I’m in Europe (hence the c19 to schuko), so the outlets are usually 220/240V and rated usually at 16Amps

 

[simoncorner]

It's a little complicated because the system also reserves power that it may need. So for example, on one of my systems, the fans are currently using 137W DC but the system has reserved 500W DC (in case the load increases, so it needs more cooling). The system then calculates how much power it thinks it needs and how much is available. If you plugged one power supply in, you would find that after turning on a certain number of blades, it wouldn't let you turn any more on.
From a physical plugging in point of view, you have to remember that each PSU can draw 16A, which is why it has the C19 connector rather than the traditional C13/C15. Consequently, each cable and socket/outlet has to be able to supply the same. My home system is connected to two sockets on one ring main (UK terminology for a circuit where the sockets are connected in a ring), which can supply 32A. If I wanted to plug a third power supply in, technically I would need to plug it into a different ring main or I could overload the circuit. The circuit breaker/fuse should protect everything.
The problem with only using a small(er) number of power supplies rather than all 6 is that if one failed or a cable came loose, the system would lose power to some blades, at random, which could corrupt your data.
If you are in an office building, I would ask the owner/manager/etc how things are wired as you don't want to cause any issues. To connect it up fully and properly you would request 6 new independent 16A feeds. A quick search suggests schuko can do 16A but a much better solution would be to use commando/IEC60309 connections.

 

[penrhos]

We're running 40 x C7000 chassis all full of Gen7/8/9 BL460C's and once the initial startup is done and the blades are up and running most chassis are showing around 2.5Kw draw.

They're not quiet and you'll need 6 x C19/20 power leads (so officially 6 x 16A commando sockets to power them or two PDU's with 3 x C19/20 connections) so forget plugging them into a single ring-main at home a chassis full of blades needs 4 x PSU's to run.

We've just binned 40 x BL460C Gen7 servers and after Xmas we'll be binning 6 x C7000 Flex-10 chassis and another 40 x BL460C gen 7/8 and a few BL660 Gen8's as we've switched to Synergy frames - all our racks have dual 32A 3-phase power distribution units installed. and we run 3-4 x C7000 per rack.