Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

[dba]

By the way, I did buy one just to investigate. I have plenty of 208V power available, and serial port servers to manage them. Patrick, would be glad to write up something for the site.

Also: Looks 19" to me. The triple-wide version is definitely 21" though.

UPDATE: 19" wide but won't fit in a 19" rack, which have 17 3/4" openings between the rails. See later post for more details about these quirky servers.

I have my sample unit installed and powered on. It took more time than expected because they initially forgot to include the heatsinks. That's fixed now, and I have it loaded with dual Xeon E5-2670 CPUs and 64GB of RAM in each of the two sleds. Now I need to PXE boot an OS onto those sleds, then run my tests.

I have eight racks of these things "on hold" in case the testing works out favorably. That's almost 14,000 HyperThread cores!

These servers are shockingly quiet and cool at idle for a server. Gotta love the Xeon E5s.

Details, and photos, when I'm a bit further along.

 

[J Hart]

Something I found out about the power on these and how Facebook sorta puts these together. The power is 480V 3 phase and then one rack takes a phase to ground. They put 3 racks together and use a single 3 phase power drop to power all three racks. They call this setup a "triplet".

This motherboard's design is on this page. Server/SpecsAndDesigns - OpenCompute It is the "Facebook server Intel Xeon motherboard v2" The one thing that might be useful is the debug header. That little 14 pin connector near the very front gives you a serial port for console access in addition to POST codes and all.

You can see more about the debug port in this hackathon thing http://cdn.oreillystatic.com/en/ass...enCompute Hardware Hacking Presentation 1.pdf

 

[PnoT]

Nice work dba! I'm anxiously awaiting the results because I'm in a slump after selling the C6100 and need a replacement.

 

[dba]

Something I found out about the power on these and how Facebook sorta puts these together. The power is 480V 3 phase and then one rack takes a phase to ground. They put 3 racks together and use a single 3 phase power drop to power all three racks. They call this setup a "triplet".

This motherboard's design is on this page. Server/SpecsAndDesigns - OpenCompute It is the "Facebook server Intel Xeon motherboard v2" The one thing that might be useful is the debug header. That little 14 pin connector near the very front gives you a serial port for console access in addition to POST codes and all.

You can see more about the debug port in this hackathon thing http://cdn.oreillystatic.com/en/assets/1/event/95/Project Cheesy Fingers_ OpenCompute Hardware Hacking Presentation 1.pdf

These don't look like the triplet racks to me. The photos show a single rack with two power cords. According to the eBay seller, each of these racks has two PDUs with one power feed each, and the PDUs are 24A 208V 3-phase.

The labels on the PDUs say: "Sentry Smart Cabinet Distribution Unit". The Input is 120/208V 3W+N+PE 50/60hz 24A. The output is 208V C13<= 12A each.

 

[liqserv]

I got mine Windmill up and running too.

Yes, this is actually 21.1" and it's really quiet unit comparing to 1U servers. Average CPU0/1 temp is around 41-42C.
I was able to find all the drivers from manufacturers website and install them to Windows 2008 R2 OS. The problem is I was not able to install any edition of Windows 2012/Windows 8.1 yet. None of NT6.3 systems worked for me. It just halts seconds after logo screen appears during setup. "Your PC ran into a problem".
Probably updating BIOS version to a newer one would help, but it's not clear whether it exists or not.

American Megatrends
03/15/11
63-0100-000001-00101111-031511-Chipset-F03_3000
BIOS Date: 03/02/12 13:08:57 V
Chipset: Intel 3C00 rev 7

 

[dba]

I just received my test unit yesterday. The chassis is 19" wide overall. Of course a standard "19 inch" rack is just 17 3/4" wide between the rails, so these servers will not fit in a standard rack - not even close.

The non-standard width isn't a problem for me, since if I like them I'm going to buy the servers and their racks, but any home users should know that these are seriously non-standard.

Interestingly, the unit that I received included rigid plastic airflow shields over each motherboard. This is important for tinkering, since it means that you can run a single server without airflow problems - remember that these chassis are open top and normally rely on having a server above it to serve as the lid.

This chassis is seriously quirky in other ways as well. If you are looking at the images from the auction as reference:

1) You can see a short C-14 female power plug sticking out of the far right side. This is the only power cord you get! It's a custom cable that attaches to the power supply inside of the chassis with an odd connector and is routed out the back of the server where it dangles as a C-14 plug. If you pull on that C-14, it will slide out up to about 16". If you need more cable than that then get out your tools.

2) The disks mount in the space to the right that looks like it would contain the power supply. The two 3.5" disks mount with included tool-free sleds that are shockingly minimal, but do include a small amount of rubber dampening.

3) The small cables dangling out the back of the chassis are in fact the disk cables. The disks mount with their connectors sticking out the back of the chassis, and the cables just plug right in. No backplane, no strain relief, no nothing.

4) The 10Gbe mezzanine board is SFP format, not Base-T, and does not include any strain relief. It is very very wiggly.

5) There are plugs on the motherboard for a second disk for each motherboard, but you'll need to wire your own special cables, and you'll need to fabricate some sort of mounting bracket for the drive itself, which better be an SSD for weight and vibration reasons.

looks like these are Wiwynn SV7210 nodes. See:
http://www.h3platform.com/download/Datasheet_SV7210.pdf

BTW: Does anyone have a source for OCP debug cards?

 

[liqserv]

Unfortunately h3platform stores only datasheets (no firmware or other downloads) and wiwynn removed SV7210 from their products

 

[Tobias]

I have my sample unit installed and powered on. It took more time than expected because they initially forgot to include the heatsinks. That's fixed now, and I have it loaded with dual Xeon E5-2670 CPUs and 64GB of RAM in each of the two sleds. Now I need to PXE boot an OS onto those sleds, then run my tests.

I have eight racks of these things "on hold" in case the testing works out favorably. That's almost 14,000 HyperThread cores!

These servers are shockingly quiet and cool at idle for a server. Gotta love the Xeon E5s.

Details, and photos, when I'm a bit further along.

May i ask how you got console access ? is there IPMI or did you add a graphicscard or use serial ?

 

[dba]

May i ask how you got console access ? is there IPMI or did you add a graphicscard or use serial ?

In my case, I didn't, at least at first. During my testing I just PXE booted the nodes using Foreman and then logged in remotely using SSH.

More recently, having bought a pile of these servers and knowing that I'll need to debug one some day, I called up Penguin Computing in Fremont and bought an "Open Compute Debug Kit". The kit, which should show up this Friday, supposedly includes the serial port cable matching the quirky motherboard debug port and a PCI "debug card" that is very likely just a low-cost graphics card. The whole package was $460, is probably worth $30, but then again this is a corporate project where time is worth more than money. The serial cable alone was only $15 or so.

If you are on a budget, check out the Open Compute Windmill motherboard docs from Intel, which describe the serial port header on the motherboard, etc.

 

[RobertFontaine]

Watching this one carefully. I noticed the wiwynn sv7210's on fleabay last night and thought that they could be very handy. Esiso has bare single motherboards at $98USd. The limited expandability seems to make them pretty much either a pure Xeon Compute board or a nice litte VM Host. 1 of these units ,4 cpus would make a very nice location for hosting every miscellaneous server VM I could personally want to host for the rest of my life. The power issue is a bit of a problem for a single solution. If one of you fine electrical engineers could build a hobby board that lets us use ATX power supplies optionally and someone finds a cheap enclosure with a pair of redundant power supplies I suspect that these would sell like lemonade in the Sahara. A great home for 4 of our 2670's.

 

[liqserv]

You can also add video adapter as I did with my Windmill. Should be the simplest way. Probably you will need some older graphics card series. I used Radeon 3450 HD and Radeon 45xx HD series which worked fine for me. When I tried a newer GeForce 730 it just showed AMI logo and that's it, I was not able to enter BIOS or boot any further as the system just halts.

 

[RobertFontaine]

I found the board spec. The pinouts for power delivery are on page 19. Can someone that knows which end of a voltmeter to lick tell me why one couldn't make a custom cable and use an atx power supply? and better yet can you draw me a piksher so I ken make won.

Yes electricity is magic to me but I see 12. 5 volts, 5 volts, and ground so it seems like sticking the right wires in the right holes should work.

http://www.opencompute.org/assets/download/Open-Compute-Project-Intel-Motherboard-v2.0.pdf

 

[smithse79]

http://www.opencompute.org/assets/d...l4FXNGvMeAg3DD7MA&sig2=DQXhvhPHMsrW9x6hKA5VHg

I'm getting a 403 Forbidden error on that page

 

[RobertFontaine]

Link fixed.

 

[smithse79]

Link fixed.

This actually seems VERY do-able. It looks like the board only uses a 12.5V input, the others are just for the PSU failure LED on the front of the system.

 

[Chuntzu]

If using one of the opencompute windmill platforms that aren't running DC power that are for sale on ebay right now the power supply input voltages work from 200-270v so no need to muck around just plug it into 240. I would guess that it would be a whole lot easier to run a new outlet then reconstruct a powersupply. I run all my equipment off of 220-240 for the efficiency and amperage bump but I may be a weirdo!

 

[RobertFontaine]

Running a new 240 from the box in my basement seems to me a great deal more work than assembling a cable for a power supply and a lot less messy. I would have to buy my brother-in-law beer and pay for the cable, the breakers and listen to him for 2 hours. I can splice together power cables in 30 minutes with music in the background.

 

[smithse79]

For me it would require installing a new electrical panel (mine is full) Which means getting the power co. involved, then I would have to run cable and install an outlet. It'd be much simpler to build an adapter cable from ATX to this thing

 

[server_lol]

For me it would require installing a new electrical panel (mine is full) Which means getting the power co. involved, then I would have to run cable and install an outlet. It'd be much simpler to build an adapter cable from ATX to this thing

Could you share the adapter that can be used? The link you provided did not look the right adapter. It looked like an audio adapter.

Thanks

 

[server_lol]

Would running a step-up transformer 110v to 220v be horribly inefficient?