Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

[ThisSpaceForRent]

Has anyone been able to get the IPMI SOL continue to work through a PXE boot Centos/Ubuntu installation? I can get to the kernel and when its supposed to grab the OS install files the SOL stops updating. Server looks to be hanging and nothing ever happens.

Since the Mac has an issue with LANPLUS I have a Vmbox Ubuntu I use the SOL from. Have i missed a configuration somewhere?

 

[svt3391]

Hi all, I'm new to the forum and I'm really interested in grabbing one of these. So, which one is better the WIWynn or Quanta? I don't need any 10Gbps network cards or any of the extra stuff cause my home network is not ready for any of that. The WIWynn one is new in box which means it could serve a few more years than the used ones. I'm leaning towards the WiWynn one since it is brand new in box so it would serve up a few more years than the used ones. But looks like with the Quanta there has been success in upgrading BIOS and installing latest OSs. Has anyone tried installing Quanta BIOS to the WiWynn board? If they are similar architectures with identical components then should there be a problem? Someone confirmed they both are AMI and they both are the same version.

Also, has anyone tried installing ESXi or HyperV or any other hypervisors and had success? My main intention to grab one of these is to run virtual machines.

Both came from big Taiwanese ODM companies. Quanta is much bigger in scale, but they have very comparable qualities.

I have Wiwynn for the same reason you listed -- new vs old. But I had hard time finding updated BIOS, because this entire batch was ordered by Riot Games (the publisher of League of Legends) through ODM-Direct. Therefore it's not easy to get the newer bios, even from the manufacturer.

But if you are interested in running ESXi, the SV7210 is fully capable with the current BIOS. I have set one up and run smoothly. It is also capable of passing through component. I tried a LSI 9211-8i card and I had no issue passing it through to a VM.

 

[svt3391]

Forgive for my interruption, but, I need four cheapest (and scaled to eight eventually) headless nodes that run C4D rendering (pure CPU, network attached render files, usb-booted core windows) are those quanta still the minimum and cheapest you can get dual 2670's to work?

IMHO those are pretty much the cheapest at this time. Last week there were some brand new Intel 4 node server at around $400+ but they were long gone.

Go with either one for $199 (Wiwynn takes BO of $199). They were designed to run headless, and at a very low noise level.

 

[grfxlab]

Since ArcyneTheFirst was able to update the bios and install a more recent windows version, I took the plunge and purchased a Quanta test machine. If anyone is able to find a bios update for the Wiwynn version I would be interested in testing that too. Unfortunately, I have not found Wiwynn bios/drivers on the web, nor was contacting them helpful.

 

[s0lid]

On looking closer at the label on the power supply, looks like it also takes 48V DC input. That I would guess the 48v part of the power supply would just be a regulator circuitry to convert to 12.5v DC output. And the 48V DC rails on the rack would supply the power in a data center.

Are there UPSs that provide 48v output available? If so, that would be an option for powering these if a UPS power is desired. I need UPS power. Somehow, I'm not convinced by using a Wall -> UPS -> Transformer -> Power Supply approach. Don't know if all UPSs would even work. Just lost even more.

The best solution would be to use a normal 120v switching power supply and hack the 12v inputs to these boards. Is the power supply standard size, does it measure the same as any standard 2U, 1U or ATX power supply? Or is the dimensions proprietary too?

It's probably DC-DC Step down switching converter for 48V to 12.5V, regulators aren't just efficient enough for high powers.
48v ups is fairly simple, all you need is power supply that can feed up to 57VDC and batteries in series of fours so the nominal voltage over single battery pack is 48VDC. Also DC circuit breakers are recommended. 57VDC because it's roughly the recommended recharging voltage for the batteries, 4*14.4V and even if the PSU on the device is rated for 48VDC input they usually have voltage range from mid-30V to 60V, so 48VDC is just nominal voltage.

 

[Jon Massey]

On looking closer at the label on the power supply, looks like it also takes 48V DC input. That I would guess the 48v part of the power supply would just be a regulator circuitry to convert to 12.5v DC output. And the 48V DC rails on the rack would supply the power in a data center.

From my reading it can either be run off 200-270V AC or 48V DC. Can you not just plug it in to the mains?

48V is pretty standard in telco racks, and more DCs are providing it. We used to use forklift batteries as a 2nd-tier UPS while we got the genny up and running.

 

[unwind-protect]

I wonder whether you can have a bigass 48 V power supply outside the house, rated for 100 degrees fahrenheit, and save on cooling inside.

 

[Nnyan]

Ok, I'm going to take @Patrick up on his offer if someone else hasn't already!
No experience required!

Bill of Materials

32 x 8GB Nanya 8GB 2Rx4 DDR3L PC3L-10600 1333MHz ECC RAM ($12ea, free shipping) - Source from eBay, see multiple STH threads. I picked this up in a lot of 50 from eBay's KingMemoryUSA2 because I was actually outfitting two of these servers.

1 x ELC T-1000 Toroidal Transformer ($48, Prime shipping) - Source from Amazon Prime
Amazon.com: ELC T-1000 1000-Watt Voltage Converter Transformer - Step Up/Down - 110V/220V - Circuit Breaker Protection: Electronics
Total Costs: $460 per node, for a two-node "1.5U" format box. Each node is a dual E5-2670 with 128GB RAM (so in total 4x E5-2670 and 256 GB RAM)

Ok so I've found a number of posts about EEC Ram on here but I haven't been able to find one that refers to the $12 price. Can someone point me in the right direction? I'm seriously starting to consider a few of these units.

 

[Jon Massey]

Ok so I've found a number of posts about EEC Ram on here but I haven't been able to find one that refers to the $12 price. Can someone point me in the right direction? I'm seriously starting to consider a few of these units.

That sort of price is generally only available when you're buying a reasonable quantity: kingmemoryusa2 | eBay they've got some decent 8gb DDR lots listed at the mo

 

[polyfractal]

Hey all. I just finished a 4-node OCP build thanks to the details in this thread,particularly NorCalm's guide posted earlier in this thread. Full details are over in the build thread (I opted a separate thread so I didn't clutter this one with pictures, etc): 4-node Open Compute Cluster Build

Thanks so much to everyone that has tinkered with these units and made the information available, my build was super painless thanks to your help

 

[polyfractal]

The startup sequence is a bit funky (and not entire predictable when it comes to timing, imo), but your's sounds a bit wrong. Here's my Wiwynn on a good run:

1. Flip switch
2. PSU fan revs up, light turns green, fan slows down to normal speed and light turns yellow. This means "Running, good". Weird, using yellow as "good", i know.
3. Lower LED next to the hard drives lights up Yellow, also means "good". I haven't seen the top LED light up yet. If you trace the wires, they both go back to the power distribution boards at the front of each motherboard. I suspect the top LED lights when something is wrong with either the fans or the distribution board
4. Node A fan revs up, blue led on board turns on.
5. Node A flickers BPLED yellow (the outer-most led). I believe this is the "boot" LED
6. Node A flashes HDLED yellow (the center led) a few times, along with activity on the drive. This is the HDD led
7. 30-60s later, Node B's fan revs up, blue light turns on, follows steps 4-6
8. When everything is running you should see:
   1. Yellow on PSU
   2. Yellow by HDD
   3. Blue on both boards
   4. Flickering yellow on center led from each board, indicating HDD activity

If a board powers up (e.g. fan runs) and has steady blue, but never flickers the center yellow HDLED, it means something is wrong. I've encountered it twice now: once when I was installing RAM (a stick was bad) and literally just now while checking, because it appears one of my HDD's just died (yay Seagate). After swapping the HDD the boot sequence resumed as normal.

It sometimes takes longer for the second node to power on. I suspect it relates to the last state the board was in. You can hold down the red button to cycle power. The grey is a reset I believe. Although both buttons seem more like "suggestions" than anything else, sometimes the board appears to just ignore them.

So... your board sounds unusual. I'd unplug one of the nodes, remove the graphics card and boot, watching for HD activity on the center LED. You could also try swapping the HDDs, and make sure the fans at the front are fully seated. Perhaps also try the second SATA port and power, just in case it's related to that. Is the coin battery fully seated? Are you sure the PSU is plugged into the 240v socket and not the 120 on the transformer?

Otherwise, I'm outta ideas

 

[Mkvarner]

Anybody gotten something like a 970/980 to work? Im interested in buying one if it is possible to use one node as a gaming PC.

 

[polyfractal]

Many thanks to polyfractal.

Those two suggestions saved my day. The first node never booted but the blue LED lit up solid, no further progress beyond that. I used the red button on the second node several times, pressing, holding for a few seconds etc. and voila, it powered up.

The actual problem was with node 1's BIOS, it was in a bad state. Had to pull the battery out and do a reset and then both nodes powered up happily one after the other.

Woah, awesome! The coin battery idea was just a whimsical thought, glad it helped! I didn't even think about the BIOS being in a bad state or something similar. How'd you reset the BIOS, just let the charge drain for a few minutes before adding the battery back?

Happy you got it working, and good to know about those cards.

Is there a wiki somewhere we can start recording this information? Might be nice for people who try to replicate this stuff later, without having to wade through the thread. And in particular, I want to try out some newer GPUs to see if they work on the board.

Alright folks, here is the guide for using an ATX power supply with this server.

Wow! This is awesome. Despite a bit of soldering, this could be a lot more convenient than a transformer. I gotta admit, besides the efficiency loss, the transformer makes me nervous for completely irrational reasons

Great work! Might have to try this out myself in the future.

 

[polyfractal]

Finally got WakeOnLan working. This also fixes the issues I was having where reboots and hibernate would hang.

I was playing around with different solutions, so I'm not quite sure what the minimal procedure is, but this worked for me (I suspect some of it is redundant). This is for Ubuntu 14.04 server:

1. Add the following to your interface in /etc/network/interfaces
   • up /sbin/ethtool -s $IFACE wol g
     post-up /sbin/ethtool -s $IFACE wol g
     post-down /sbin/ethtool -s $IFaCE wol g
     
2. Add the following to /etc/modprobe.conf/local.conf
   • install 8139too /sbin/modprobe -i 8139too; /sbin/ethtool -s eth0 wol g
     
3. Blacklist mei and mei_me, due to the bug listed here. Add the following to /etc/modprobe.d/blacklist.conf:
   • blacklist mei
     blacklist mei_me
     
     
   • Without this, WOL would "work", but cause a reboot which then hung. The bugtracker has discussion around updated kernels, but I haven't tried that yet. Haven't noticed a problem with blacklisting mei yet...
     
4. Power cycle. If you reboot, it'll hang because these settings weren't loaded.
   
5. Once powered up, sudo pm-hibernate to sleep the nodes
   
6. wakeonlan <mac-address> from another machine, node starts back up. ether-tool seems to work as well

Note: the PSU and fan remain active even if both nodes are suspended to disk.

 

[dba]

Looking for cheap 2011 systems, I ran into this Quanta Open Compute barebones server for $199.
2 x Dual LGA2011 nodes with 10GB Ethernet. $50/cpu socket, with 10 GBE, and 1024GB Total RAM

Looks rather loud but should be easy to swap in larger fans. I wonder how expense a basic build would be.
Seams like a good deal but I might be missing something.

• 2x QUANTA WINDMILL OPEN COMPUTE SYSTEM BOARDS BAREBONE EACH WITH THE FOLLOWING:
• Supports 2x LGA 2011 Socket CPU'S (4x TOTAL)
• Supports 16x DDR3 Memory Modules (32x TOTAL)
• 1x 1Gb Ethernet PORT
• 1x 10GbE Ethernet Mezzanine CARD PN# CX341A (SFP)
  
• 2x USB Ports, 2x Internal SATA ports
  
• 1x 16x PCIE Slot
• 1x 700W Power Supply Included

Home » Open Compute Project

~21" Width not standard 19" rackmount, 200-277v input (confirmed by DBA)

$99 from sale seller for single node motherboard (Dual LGA2011) only.
$1269 bundle with 4 x E5-2660 (2.2Ghz 8c) 16GB RAM, 2 x 250GB HDD.


EBay Picture:

If anyone knows of a source for racks that fit this specific generation of OCP servers, please share. I bought six racks of these (5K cpu cores!) but now they are out of the racks so I can't buy more. Plenty of chassis like the above photo, but not racks to put them in.

 

[dba]

Looking for cheap 2011 systems, I ran into this Quanta Open Compute barebones server for $199.
2 x Dual LGA2011 nodes with 10GB Ethernet. $50/cpu socket, with 10 GBE, and 1024GB Total RAM

Looks rather loud but should be easy to swap in larger fans. I wonder how expense a basic build would be.
Seams like a good deal but I might be missing something.

• 2x QUANTA WINDMILL OPEN COMPUTE SYSTEM BOARDS BAREBONE EACH WITH THE FOLLOWING:
• Supports 2x LGA 2011 Socket CPU'S (4x TOTAL)
• Supports 16x DDR3 Memory Modules (32x TOTAL)
• 1x 1Gb Ethernet PORT
• 1x 10GbE Ethernet Mezzanine CARD PN# CX341A (SFP)
  
• 2x USB Ports, 2x Internal SATA ports
  
• 1x 16x PCIE Slot
• 1x 700W Power Supply Included

Home » Open Compute Project

~21" Width not standard 19" rackmount, 200-277v input (confirmed by DBA)

$99 from sale seller for single node motherboard (Dual LGA2011) only.
$1269 bundle with 4 x E5-2660 (2.2Ghz 8c) 16GB RAM, 2 x 250GB HDD.


EBay Picture:

If anyone knows of a source for racks that fit this specific generation of OCP servers, please share. I bought six racks of these (5K cpu cores!) but now they are out of the racks so I can't buy more. Plenty of chassis like the above photo, but not racks to put them in.

 

[dba]

Looking for cheap 2011 systems, I ran into this Quanta Open Compute barebones server for $199.
2 x Dual LGA2011 nodes with 10GB Ethernet. $50/cpu socket, with 10 GBE, and 1024GB Total RAM

Looks rather loud but should be easy to swap in larger fans. I wonder how expense a basic build would be.
Seams like a good deal but I might be missing something.

• 2x QUANTA WINDMILL OPEN COMPUTE SYSTEM BOARDS BAREBONE EACH WITH THE FOLLOWING:
• Supports 2x LGA 2011 Socket CPU'S (4x TOTAL)
• Supports 16x DDR3 Memory Modules (32x TOTAL)
• 1x 1Gb Ethernet PORT
• 1x 10GbE Ethernet Mezzanine CARD PN# CX341A (SFP)
  
• 2x USB Ports, 2x Internal SATA ports
  
• 1x 16x PCIE Slot
• 1x 700W Power Supply Included

Home » Open Compute Project

~21" Width not standard 19" rackmount, 200-277v input (confirmed by DBA)

$99 from sale seller for single node motherboard (Dual LGA2011) only.
$1269 bundle with 4 x E5-2660 (2.2Ghz 8c) 16GB RAM, 2 x 250GB HDD.


EBay Picture:

If anyone knows of a source for racks that fit this specific generation of OCP servers, please share. I bought six racks of these (5K cpu cores!) but now they are out of the racks so I can't buy more. Plenty of chassis like the above photo, but not racks to put them in.

 

[dba]

Looking for cheap 2011 systems, I ran into this Quanta Open Compute barebones server for $199.
2 x Dual LGA2011 nodes with 10GB Ethernet. $50/cpu socket, with 10 GBE, and 1024GB Total RAM

Looks rather loud but should be easy to swap in larger fans. I wonder how expense a basic build would be.
Seams like a good deal but I might be missing something.

• 2x QUANTA WINDMILL OPEN COMPUTE SYSTEM BOARDS BAREBONE EACH WITH THE FOLLOWING:
• Supports 2x LGA 2011 Socket CPU'S (4x TOTAL)
• Supports 16x DDR3 Memory Modules (32x TOTAL)
• 1x 1Gb Ethernet PORT
• 1x 10GbE Ethernet Mezzanine CARD PN# CX341A (SFP)
  
• 2x USB Ports, 2x Internal SATA ports
  
• 1x 16x PCIE Slot
• 1x 700W Power Supply Included

Home » Open Compute Project

~21" Width not standard 19" rackmount, 200-277v input (confirmed by DBA)

$99 from sale seller for single node motherboard (Dual LGA2011) only.
$1269 bundle with 4 x E5-2660 (2.2Ghz 8c) 16GB RAM, 2 x 250GB HDD.


EBay Picture:

If anyone knows of a source for racks that fit this specific generation of OCP servers, please share. I bought six racks of these (5K cpu cores!) but now they are out of the racks so I can't buy more. Plenty of chassis like the above photo, but not racks to put them in.

 

[smithse79]

I think @dba needs some racks

 

[Gene]

I think @dba needs some racks

I think so haha

 

[J Hart]

Depending on how many racks you need, you might want to just get the racks made to order. Looks like you could probably buy them from taobao, but you'd have to deal with shipping them/importing them.