Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

[s0lid]

On looking closer at the label on the power supply, looks like it also takes 48V DC input. That I would guess the 48v part of the power supply would just be a regulator circuitry to convert to 12.5v DC output. And the 48V DC rails on the rack would supply the power in a data center.

Are there UPSs that provide 48v output available? If so, that would be an option for powering these if a UPS power is desired. I need UPS power. Somehow, I'm not convinced by using a Wall -> UPS -> Transformer -> Power Supply approach. Don't know if all UPSs would even work. Just lost even more.

The best solution would be to use a normal 120v switching power supply and hack the 12v inputs to these boards. Is the power supply standard size, does it measure the same as any standard 2U, 1U or ATX power supply? Or is the dimensions proprietary too?

It's probably DC-DC Step down switching converter for 48V to 12.5V, regulators aren't just efficient enough for high powers.
48v ups is fairly simple, all you need is power supply that can feed up to 57VDC and batteries in series of fours so the nominal voltage over single battery pack is 48VDC. Also DC circuit breakers are recommended. 57VDC because it's roughly the recommended recharging voltage for the batteries, 4*14.4V and even if the PSU on the device is rated for 48VDC input they usually have voltage range from mid-30V to 60V, so 48VDC is just nominal voltage.

 

[Jon Massey]

On looking closer at the label on the power supply, looks like it also takes 48V DC input. That I would guess the 48v part of the power supply would just be a regulator circuitry to convert to 12.5v DC output. And the 48V DC rails on the rack would supply the power in a data center.

From my reading it can either be run off 200-270V AC or 48V DC. Can you not just plug it in to the mains?

48V is pretty standard in telco racks, and more DCs are providing it. We used to use forklift batteries as a 2nd-tier UPS while we got the genny up and running.

 

[unwind-protect]

I wonder whether you can have a bigass 48 V power supply outside the house, rated for 100 degrees fahrenheit, and save on cooling inside.

 

[Nnyan]

Ok, I'm going to take @Patrick up on his offer if someone else hasn't already!
No experience required!

Bill of Materials

32 x 8GB Nanya 8GB 2Rx4 DDR3L PC3L-10600 1333MHz ECC RAM ($12ea, free shipping) - Source from eBay, see multiple STH threads. I picked this up in a lot of 50 from eBay's KingMemoryUSA2 because I was actually outfitting two of these servers.

1 x ELC T-1000 Toroidal Transformer ($48, Prime shipping) - Source from Amazon Prime
Amazon.com: ELC T-1000 1000-Watt Voltage Converter Transformer - Step Up/Down - 110V/220V - Circuit Breaker Protection: Electronics
Total Costs: $460 per node, for a two-node "1.5U" format box. Each node is a dual E5-2670 with 128GB RAM (so in total 4x E5-2670 and 256 GB RAM)

Ok so I've found a number of posts about EEC Ram on here but I haven't been able to find one that refers to the $12 price. Can someone point me in the right direction? I'm seriously starting to consider a few of these units.

 

[Jon Massey]

Ok so I've found a number of posts about EEC Ram on here but I haven't been able to find one that refers to the $12 price. Can someone point me in the right direction? I'm seriously starting to consider a few of these units.

That sort of price is generally only available when you're buying a reasonable quantity: kingmemoryusa2 | eBay they've got some decent 8gb DDR lots listed at the mo

 

[polyfractal]

Hey all. I just finished a 4-node OCP build thanks to the details in this thread,particularly NorCalm's guide posted earlier in this thread. Full details are over in the build thread (I opted a separate thread so I didn't clutter this one with pictures, etc): 4-node Open Compute Cluster Build

Thanks so much to everyone that has tinkered with these units and made the information available, my build was super painless thanks to your help

 

[polyfractal]

The startup sequence is a bit funky (and not entire predictable when it comes to timing, imo), but your's sounds a bit wrong. Here's my Wiwynn on a good run:

1. Flip switch
2. PSU fan revs up, light turns green, fan slows down to normal speed and light turns yellow. This means "Running, good". Weird, using yellow as "good", i know.
3. Lower LED next to the hard drives lights up Yellow, also means "good". I haven't seen the top LED light up yet. If you trace the wires, they both go back to the power distribution boards at the front of each motherboard. I suspect the top LED lights when something is wrong with either the fans or the distribution board
4. Node A fan revs up, blue led on board turns on.
5. Node A flickers BPLED yellow (the outer-most led). I believe this is the "boot" LED
6. Node A flashes HDLED yellow (the center led) a few times, along with activity on the drive. This is the HDD led
7. 30-60s later, Node B's fan revs up, blue light turns on, follows steps 4-6
8. When everything is running you should see:
   1. Yellow on PSU
   2. Yellow by HDD
   3. Blue on both boards
   4. Flickering yellow on center led from each board, indicating HDD activity

If a board powers up (e.g. fan runs) and has steady blue, but never flickers the center yellow HDLED, it means something is wrong. I've encountered it twice now: once when I was installing RAM (a stick was bad) and literally just now while checking, because it appears one of my HDD's just died (yay Seagate). After swapping the HDD the boot sequence resumed as normal.

It sometimes takes longer for the second node to power on. I suspect it relates to the last state the board was in. You can hold down the red button to cycle power. The grey is a reset I believe. Although both buttons seem more like "suggestions" than anything else, sometimes the board appears to just ignore them.

So... your board sounds unusual. I'd unplug one of the nodes, remove the graphics card and boot, watching for HD activity on the center LED. You could also try swapping the HDDs, and make sure the fans at the front are fully seated. Perhaps also try the second SATA port and power, just in case it's related to that. Is the coin battery fully seated? Are you sure the PSU is plugged into the 240v socket and not the 120 on the transformer?

Otherwise, I'm outta ideas

 

[Mkvarner]

Anybody gotten something like a 970/980 to work? Im interested in buying one if it is possible to use one node as a gaming PC.

 

[polyfractal]

Many thanks to polyfractal.

Those two suggestions saved my day. The first node never booted but the blue LED lit up solid, no further progress beyond that. I used the red button on the second node several times, pressing, holding for a few seconds etc. and voila, it powered up.

The actual problem was with node 1's BIOS, it was in a bad state. Had to pull the battery out and do a reset and then both nodes powered up happily one after the other.

Woah, awesome! The coin battery idea was just a whimsical thought, glad it helped! I didn't even think about the BIOS being in a bad state or something similar. How'd you reset the BIOS, just let the charge drain for a few minutes before adding the battery back?

Happy you got it working, and good to know about those cards.

Is there a wiki somewhere we can start recording this information? Might be nice for people who try to replicate this stuff later, without having to wade through the thread. And in particular, I want to try out some newer GPUs to see if they work on the board.

Alright folks, here is the guide for using an ATX power supply with this server.

Wow! This is awesome. Despite a bit of soldering, this could be a lot more convenient than a transformer. I gotta admit, besides the efficiency loss, the transformer makes me nervous for completely irrational reasons

Great work! Might have to try this out myself in the future.

 

[polyfractal]

Finally got WakeOnLan working. This also fixes the issues I was having where reboots and hibernate would hang.

I was playing around with different solutions, so I'm not quite sure what the minimal procedure is, but this worked for me (I suspect some of it is redundant). This is for Ubuntu 14.04 server:

1. Add the following to your interface in /etc/network/interfaces
   • up /sbin/ethtool -s $IFACE wol g
     post-up /sbin/ethtool -s $IFACE wol g
     post-down /sbin/ethtool -s $IFaCE wol g
     
2. Add the following to /etc/modprobe.conf/local.conf
   • install 8139too /sbin/modprobe -i 8139too; /sbin/ethtool -s eth0 wol g
     
3. Blacklist mei and mei_me, due to the bug listed here. Add the following to /etc/modprobe.d/blacklist.conf:
   • blacklist mei
     blacklist mei_me
     
     
   • Without this, WOL would "work", but cause a reboot which then hung. The bugtracker has discussion around updated kernels, but I haven't tried that yet. Haven't noticed a problem with blacklisting mei yet...
     
4. Power cycle. If you reboot, it'll hang because these settings weren't loaded.
   
5. Once powered up, sudo pm-hibernate to sleep the nodes
   
6. wakeonlan <mac-address> from another machine, node starts back up. ether-tool seems to work as well

Note: the PSU and fan remain active even if both nodes are suspended to disk.

 

[dba]

Looking for cheap 2011 systems, I ran into this Quanta Open Compute barebones server for $199.
2 x Dual LGA2011 nodes with 10GB Ethernet. $50/cpu socket, with 10 GBE, and 1024GB Total RAM

Looks rather loud but should be easy to swap in larger fans. I wonder how expense a basic build would be.
Seams like a good deal but I might be missing something.

• 2x QUANTA WINDMILL OPEN COMPUTE SYSTEM BOARDS BAREBONE EACH WITH THE FOLLOWING:
• Supports 2x LGA 2011 Socket CPU'S (4x TOTAL)
• Supports 16x DDR3 Memory Modules (32x TOTAL)
• 1x 1Gb Ethernet PORT
• 1x 10GbE Ethernet Mezzanine CARD PN# CX341A (SFP)
  
• 2x USB Ports, 2x Internal SATA ports
  
• 1x 16x PCIE Slot
• 1x 700W Power Supply Included

Home » Open Compute Project

~21" Width not standard 19" rackmount, 200-277v input (confirmed by DBA)

$99 from sale seller for single node motherboard (Dual LGA2011) only.
$1269 bundle with 4 x E5-2660 (2.2Ghz 8c) 16GB RAM, 2 x 250GB HDD.


EBay Picture:

If anyone knows of a source for racks that fit this specific generation of OCP servers, please share. I bought six racks of these (5K cpu cores!) but now they are out of the racks so I can't buy more. Plenty of chassis like the above photo, but not racks to put them in.

 

[dba]

Looking for cheap 2011 systems, I ran into this Quanta Open Compute barebones server for $199.
2 x Dual LGA2011 nodes with 10GB Ethernet. $50/cpu socket, with 10 GBE, and 1024GB Total RAM

Looks rather loud but should be easy to swap in larger fans. I wonder how expense a basic build would be.
Seams like a good deal but I might be missing something.

• 2x QUANTA WINDMILL OPEN COMPUTE SYSTEM BOARDS BAREBONE EACH WITH THE FOLLOWING:
• Supports 2x LGA 2011 Socket CPU'S (4x TOTAL)
• Supports 16x DDR3 Memory Modules (32x TOTAL)
• 1x 1Gb Ethernet PORT
• 1x 10GbE Ethernet Mezzanine CARD PN# CX341A (SFP)
  
• 2x USB Ports, 2x Internal SATA ports
  
• 1x 16x PCIE Slot
• 1x 700W Power Supply Included

Home » Open Compute Project

~21" Width not standard 19" rackmount, 200-277v input (confirmed by DBA)

$99 from sale seller for single node motherboard (Dual LGA2011) only.
$1269 bundle with 4 x E5-2660 (2.2Ghz 8c) 16GB RAM, 2 x 250GB HDD.


EBay Picture:

If anyone knows of a source for racks that fit this specific generation of OCP servers, please share. I bought six racks of these (5K cpu cores!) but now they are out of the racks so I can't buy more. Plenty of chassis like the above photo, but not racks to put them in.

 

[dba]

Looking for cheap 2011 systems, I ran into this Quanta Open Compute barebones server for $199.
2 x Dual LGA2011 nodes with 10GB Ethernet. $50/cpu socket, with 10 GBE, and 1024GB Total RAM

Looks rather loud but should be easy to swap in larger fans. I wonder how expense a basic build would be.
Seams like a good deal but I might be missing something.

• 2x QUANTA WINDMILL OPEN COMPUTE SYSTEM BOARDS BAREBONE EACH WITH THE FOLLOWING:
• Supports 2x LGA 2011 Socket CPU'S (4x TOTAL)
• Supports 16x DDR3 Memory Modules (32x TOTAL)
• 1x 1Gb Ethernet PORT
• 1x 10GbE Ethernet Mezzanine CARD PN# CX341A (SFP)
  
• 2x USB Ports, 2x Internal SATA ports
  
• 1x 16x PCIE Slot
• 1x 700W Power Supply Included

Home » Open Compute Project

~21" Width not standard 19" rackmount, 200-277v input (confirmed by DBA)

$99 from sale seller for single node motherboard (Dual LGA2011) only.
$1269 bundle with 4 x E5-2660 (2.2Ghz 8c) 16GB RAM, 2 x 250GB HDD.


EBay Picture:

If anyone knows of a source for racks that fit this specific generation of OCP servers, please share. I bought six racks of these (5K cpu cores!) but now they are out of the racks so I can't buy more. Plenty of chassis like the above photo, but not racks to put them in.

 

[dba]

Looking for cheap 2011 systems, I ran into this Quanta Open Compute barebones server for $199.
2 x Dual LGA2011 nodes with 10GB Ethernet. $50/cpu socket, with 10 GBE, and 1024GB Total RAM

Looks rather loud but should be easy to swap in larger fans. I wonder how expense a basic build would be.
Seams like a good deal but I might be missing something.

• 2x QUANTA WINDMILL OPEN COMPUTE SYSTEM BOARDS BAREBONE EACH WITH THE FOLLOWING:
• Supports 2x LGA 2011 Socket CPU'S (4x TOTAL)
• Supports 16x DDR3 Memory Modules (32x TOTAL)
• 1x 1Gb Ethernet PORT
• 1x 10GbE Ethernet Mezzanine CARD PN# CX341A (SFP)
  
• 2x USB Ports, 2x Internal SATA ports
  
• 1x 16x PCIE Slot
• 1x 700W Power Supply Included

Home » Open Compute Project

~21" Width not standard 19" rackmount, 200-277v input (confirmed by DBA)

$99 from sale seller for single node motherboard (Dual LGA2011) only.
$1269 bundle with 4 x E5-2660 (2.2Ghz 8c) 16GB RAM, 2 x 250GB HDD.


EBay Picture:

If anyone knows of a source for racks that fit this specific generation of OCP servers, please share. I bought six racks of these (5K cpu cores!) but now they are out of the racks so I can't buy more. Plenty of chassis like the above photo, but not racks to put them in.

 

[smithse79]

I think @dba needs some racks

 

[Gene]

I think @dba needs some racks

I think so haha

 

[J Hart]

Depending on how many racks you need, you might want to just get the racks made to order. Looks like you could probably buy them from taobao, but you'd have to deal with shipping them/importing them.

 

[ethbro]

Beginner question, sorry to have to ask but would rather be sure where things like power are concerned.

Per dba, NorCalm, polyfractal, et al. (btw, thanks for typing up and sharing information, all!) a standard step-up transformer (e.g. ELC T-X000) works for running these off US household 110/120v.

If I were to wire up 3x of these (6 nodes), what would power cabling look like? To start at the basics, I'm taking one 220-240V outlet from the transformer and ultimately hooking to 3x C14 sockets. Can I just split the cable, or would they interfere with each other? If the later, is this something a simple powerstrip (rated appropriately) would resolve?

As may be apparently, I am not an EE but would rather not torch hardware through my own ineptitude. Any answers greatly appreciated!

Based on polyfractal's power numbers (thread), I'm thinking I'd be looking at something like:

All four nodes running:

• idle: 232W
• 64 cores and all fans at max: 1260W / 11 amps

= ~315W / node - 40W / node (downgrade E5 2670s to 2660s) = ~275W / node * 6 = 1650W / 6-node cluster = 15 amps @ 110V (so should have headroom on a dedicated 20 amp circuit)

 

[YetAnotherMinion]

Running prime95, and using the ELC-1000 transformer, I see wall draw (measured with kill-a-watt) of around 390W per node with E5-2670.

As for your wiring, you need to hook the power wires up in parallel. Your wall outlets work the same way, each socket is wired in parallel to the supply wire. Wiring up 3 c14 cables in parallel is the same thing electrically as plugging 3 different appliances into a power strip. However cutting your own cables that will handle 15 amps is not the best thing to do if you have little experience with splicing. They make C14 cables in a Y configuration Amazon.com: Tripp Lite Computer Power Extension Cord Y Splitter Cable 10A, 18AWG (IEC-320-C14 to 2x IEC-320-C13) 6-ft.(P004-006-2C13): Electronics

 

[grfxlab]

Anybody gotten something like a 970/980 to work? Im interested in buying one if it is possible to use one node as a gaming PC.

Unless you hack in an ATX power supply, there is no PCI-E power from the existing power supply. You will not be able to run a graphics card the needs external power.

 

[grfxlab]

If I were to wire up 3x of these (6 nodes), what would power cabling look like? To start at the basics, I'm taking one 220-240V outlet from the transformer and ultimately hooking to 3x C14 sockets. Can I just split the cable, or would they interfere with each other? If the later, is this something a simple powerstrip (rated appropriately) would resolve?

You can get a 200-240v power distribution unit (power strip) with the correct plug for your needs. I bought one from ebay for $35 rated for 24amps. Each power supply states a draw of 4 amps so I could plug in 6 boxes / 12 nodes. Very easy to use.