Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

server_lol

New Member
Jan 10, 2016
13
1
1
31
Are these of standard 1U height? Thanks...

By the way, I did buy one just to investigate. I have plenty of 208V power available, and serial port servers to manage them. Patrick, would be glad to write up something for the site.

Also: Looks 19" to me. The triple-wide version is definitely 21" though.

UPDATE: 19" wide but won't fit in a 19" rack, which have 17 3/4" openings between the rails. See later post for more details about these quirky servers.
 

dba

Moderator
Feb 20, 2012
1,478
183
63
San Francisco Bay Area, California, USA
My OpenCompute debug kit arrived today. Finally.

Below are a few photos of the $460** "OCP Debug Kit" that I ordered from Penguin Computing in Fremont, CA. The kit includes the tiny and very expensive OpenCompute debug board plus two relatively inexpensive custom cables. The debug board plugs into the standard OpenCompute debug port, and displays boot codes on two small numeric displays, and also presents a USB serial port that will get you console access.

A spend of $460 isn't much if you have a ton of these servers, but it's way too much if you are buying just one or two. Intel does publish the pin assignments for the debug port, so you can make your own serial cable.

Here is a photo of the full kit. Does it look like $460 worth of kit? No, but I'm just glad that this extremely specific gear is available, regardless of price.
No documentation is included. The tiny board plugs into the OCP debug port. What are the other two cables for? I have no idea.
IMG_1337.JPG
Here is a rear view of one sled in one of the eBay OCP servers for reference. Note that there is no rear plate and no top cover, and that the disk cables pretty much just dangle there. Extreme minimalism.
IMG_1338.JPG
Here is the debug board (outlined in blue) plugged into the debug port on the right side of the motherboard sled. Note the two single-digit displays, and the tiny USB port. There is also a small button that does - I don't know yet.
IMG_1339.JPG

**
Debug Board, Quanta OCP 1 $349.99 $349.99
Cable, Serial, USB-mini-B/6-pin, Passive OCP Debug 1 $12.00 $12.00
Cable, Serial, USB-A/6x1 header, 3.3V, FTDI 1 $31.00 $31.00
Cable, Extender, 2x7/2x7, 20mm, OCP Debug 1 $4.00 $4.00
Subtotal $396.99
Est. Sales Tax $34.74
S & H (3rd Day) $28.54
QUOTE TOTAL $460.27
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,214
5,301
113
@dba That is pretty crazy! Why do I feel like that is a $5-10 board to assemble? Such is life I guess.

I guess what it also means is that your OOB management is limited to when you can have someone plug those little boards in?
 

xnoodle

Active Member
Jan 4, 2011
259
48
28
Looks DIYable.

Of the 14 pins,
8 provide post codes that eventually get displayed on the two 7-segment displays,
2 are for a RS232 shifter.
1 is for a reset button.
1 to toggle between serial over lan or serial over debug port.
2 for gnd/5v

Button should be reset.

The cables are converting usb to serial. The left one has a FTDI chip it looks like, going by the model # on it.

See page 30 of http://www.opencompute.org/assets/download/Open-Compute-Project-Intel-Motherboard-v2.0.pdf
 
Last edited:

server_lol

New Member
Jan 10, 2016
13
1
1
31
@dba, nice write-up. The taller and wider than 1U put a damper on my plans to grab a dozen and co-locate these servers. Even if the data center could accommodate the width, we would still have to pay for 2U rather than 1U.
 

Rain

Active Member
May 13, 2013
249
91
28
OpenCompute is all about open standards. While that little debug board is $350, it will (likely) work on many (all?) other OpenCompute boards, even future boards, right? So, if you intend on fully investing in this stuff as it shows up on eBay, spending the money up front will likely pay off down the road. That's only speculation though; hopefully someone more well versed in this stuff can shed some more light.
 

dba

Moderator
Feb 20, 2012
1,478
183
63
San Francisco Bay Area, California, USA
@dba That is pretty crazy! Why do I feel like that is a $5-10 board to assemble? Such is life I guess.

I guess what it also means is that your OOB management is limited to when you can have someone plug those little boards in?
I agree that I paid $10 for the board and $450 for the privilege of buying Enterprise stuff. But I need only one kit to make my entire OpenCompute farm happy, so it's fine with it.

OOB management is serial over LAN (SOL) when the nodes are booted and healthy, and local serial port for local debugging. So I should only need the little board if things have gone haywire, not for day-to-day use. I also plan to get IPMI through the OS for properly booted nodes.
 
  • Like
Reactions: crotaphiticus

smithse79

Active Member
Sep 17, 2014
197
34
28
41
@dba Can you take a shot of the power ports on this? I think I'm gonna get one and try to configure it with a standard power supply
 

dba

Moderator
Feb 20, 2012
1,478
183
63
San Francisco Bay Area, California, USA
@dba Can you take a shot of the power ports on this? I think I'm gonna get one and try to configure it with a standard power supply
The photo is uninteresting - it just shows a port on the side of the power supply that mates to the midplane. Luckily for you, Intel publishes the specifications for the PSU port. Look on page 19 of this document:
http://www.opencompute.org/assets/download/Open-Compute-Project-Intel-Motherboard-v2.0.pdf

Quoting:
The midplane has one FCI 51939-582 male right angle header, which is mated directly with the PSU for 12.5VDC input.

Datasheet: http://www.datasheet.support/datasheets/fci/51939-582.pdf
 

smithse79

Active Member
Sep 17, 2014
197
34
28
41
I've seen those documents. The issue is that that connector is not available from any electronics supply house that I've seen without having to order thousands. I feel that I'm gonna have to splice in behind the connector in the chassis where the PSU connects directly to the cables going to the motherboard.
 

jap

Member
Feb 13, 2016
27
36
13
49
I agree that I paid $10 for the board and $450 for the privilege of buying Enterprise stuff. But I need only one kit to make my entire OpenCompute farm happy, so it's fine with it.

OOB management is serial over LAN (SOL) when the nodes are booted and healthy, and local serial port for local debugging. So I should only need the little board if things have gone haywire, not for day-to-day use. I also plan to get IPMI through the OS for properly booted nodes.
hi,

what about using simple usb ttl serial converter to catch the redirected serial port?
for example $1.35 here: USB To RS232 TTL UART PL2303HX Auto Converter USB to COM Cable Adapter Module

Jan
 

NorCalm

New Member
Feb 4, 2016
14
17
3
Ok, I'm going to take @Patrick up on his offer if someone else hasn't already! :)

This is great.
[...]
How about this. First person to buy one, get it up and running, and document it for main site content, I will pay you the $199 cost of the unit. This could be really interesting for folks.
Background: Long-time STH lurker who jumped at the opportunity to grab $65 E5-2670's. When I looked at what to put them in though, it all seemed so pricey. So I thought "well if these E5-2670's are coming from Facebook, what did Facebook put them in...?" and found this thread. I am not an expert at Servers and can truly speak from total inexperience in all this. I'm sure I missed a lot but I am completely impressed at the firepower that can be purchased with not much money.

I've been spending a month trying different components the hard way and getting to know the OCP Windmill platform. Yes, some RMAs and returns later.. I can present to you:

WINDMILL FOR DUMMIES

No experience required!

Bill of Materials

4 x E5-2670 ($65 ea, free shipping) - Source from eBay, see the STH thread for details

1 x Windmill OCP2 Node Server ($199 free pick up) - I actually went local for this and picked them up from Michael at Sonitek Computer: Wiwynn SV7210 2 Node Open Compute Server Base Unit 2x LGA2011

NB: This is a brand new unit. It includes all four heat sinks, with corresponding thermal paste for each one. It's a very nice unit for a newbie like me - no sharp edges, well machined, everything slides out including the hard plastic cowling over the CPUs. Very little noise. Well suited even if you don't have a standard rack - right now it's just on a chest of drawers in my home. There is no need to purchase a single fan, heat sink, or other cooling solution with this one-stop-shop.

32 x 8GB Nanya 8GB 2Rx4 DDR3L PC3L-10600 1333MHz ECC RAM ($12ea, free shipping) - Source from eBay, see multiple STH threads. I picked this up in a lot of 50 from eBay's KingMemoryUSA2 because I was actually outfitting two of these servers.

2 x AMD Radeon HD6350 512 MB DDR3 PCIe DMS59 (w/ dual VGA cable) ($10ea, free shipping) - Source from eBay's RogerKy: AMD Radeon HD6350 512MB DDR3 Full Size PCIeX16 DMS59 VGA CN-01CX3M ATI-102-C0903

1 x ELC T-1000 Toroidal Transformer ($48, Prime shipping) - Source from Amazon Prime
Amazon.com: ELC T-1000 1000-Watt Voltage Converter Transformer - Step Up/Down - 110V/220V - Circuit Breaker Protection: Electronics

2 x Crucial BX100 250 GB SATA SSDs - on sale on Amazon for $69ea. You can use any SATA drive you like.

Total Costs: $460 per node, for a two-node "1.5U" format box. Each node is a dual E5-2670 with 128GB RAM (so in total 4x E5-2670 and 256 GB RAM)

Optional: Make use of the PCIe x8 OCP Mezzanine Slot!

I also found OCP Mezzanine cards on eBay. Although there is a catalog out there with VGA ports and the like, the only real ones I could find either had single or dual 10Gbe ports. Here were inexpensive ones at $37 ea (now sold out, sorry):

Intel Ethernet Server Adapter X520-DA2 for Open Compute Project (OCP) X520DA2OCP

Optional: Make use of the other PCIe Slot!

With one slot used up by the VGA card, I purchased a Gigabyte USB 3.1 10Gbps card for $20 just to get some extra USB 3.0 ports on a USB 3.0 hub. These ports cannot be booted from, FYI.

Gigabyte I/O Card GA-USB3.1 USB 3.1 x 2 PCI-Express Interface Retail

I'm sure some STH brethren can tell me where I could alternately get a HBA/RAID/SATA/SAS card to expand on the 2 built-in SATA ports. This would make a pretty amazing Xpenology box....


----------------------

Bring Up

Make it easy on yourself

Learn from my mistakes. After futzing around both on the power, and on the serial console (the header, without the $459 OCP Debug Kit that @dba purchased, is very fine and requires a light touch. I used a Bus Pirate and extracted two pin receptacles from a 3.5mm floppy drive connector, discovered the hard way that it was 57600/8/n/1... but I wouldn't recommend it), I learned the easiest way to resolve things were:

1. Use the ELC toroidal transformer. Very efficient, quiet, and it's a one-stop solution to the 700W 240V supply. You plug in the transformer to the wall, flip the breaker on the unit, and plug in the 700W supply. Done.

2. Use a $10 video card, and forget about the serial console. I purchased a broken HD3450 and a new GT710, and neither worked. I started despairing (and RMAing)... until I embraced my inner miser and just bought cheap. The Radeon HD6450 and HD6350 have both worked well. For Linux, I recommend the $10 HD 6350 - it's ultralow power consumption - just make sure you plug the VGA1 monitor cable (not the VGA2). For Windows, the HD6450 is great and includes an HDMI/VGA/DVI trio, so you can get audio from the HDMI.

Debug

Installation of the CPUs was straightforward, and the heatsinks appear effective, especially when used with the quiet fans on the unit.

I had a few sticks of bad RAM at the start, so got to know Windmill very well. The first thing to note is that there is a "speaker/beep" yellow LED to watch for closely for error beep codes, etc. The general sequence is:

a. Blue LED (power) goes on
b. One beep on yellow LED
c. Green LED indicating hard drive access
d. Two beeps on yellow LED

And then you can access the BIOS.

Other than that, this is really a server for the masses. Besides bad or incompatible components (video cards, RAM), there wasn't anything tricky in bringup and I could boot from USB to install either Windows 7 x64 Pro or Linux. I tried Windows 10 and it hung on an AHCI screen. So Windows 7 x64 Pro it is!

What took a month was finding the compatible components and weathering some bad luck at the start. It's a tremendous system for $460.

What's additional info is helpful?

Please let me know what is helpful to document for you all with regards to this system - it's not for everyone (especially if you want it to fit in a standard rack) but great for a beginner. I can take photos of everything or elaborate about anything.
 
Last edited:

svt3391

Member
Feb 11, 2016
119
17
18
Ok, I'm going to take @Patrick up on his offer if someone else hasn't already! :)

Long quote omitted ...
Awesome write up! Especially about the voltage converter, which seems to be a very easy to follow approach.

Questions:
  1. are you able to find any BIOS update? At Wiwynn's web site I could not even find this SV7210 listed.
  2. in the BIOS, are you able to enable the VT-d and it keeps the settings even after reboot? one of my friend said his company used some of those OpenCompute unit and he was not able to keep the VT-d, even using the E5-2670 SR0KX which has the VT-d support from the chip
  3. is there a power button on each of the unit? From the photo it's hard to see how to power up individual unit. Or did you just use the transformer's power switch to either power up or down both units?
Thanks for the great work!
 
  • Like
Reactions: NorCalm

NorCalm

New Member
Feb 4, 2016
14
17
3
Answers:

1. Although Michael (the eBay seller for the SV7210) suggests there are BIOS updates from Wiwynn, when I emailed Wiwynn, their project director just asked "where did you find one of those old things!?" and did not further reply when I volunteered to be a maintainer of their last set of BIOS updates and drivers. I will follow up with Michael.

I will say that I chased down and found each and every Windows 7 driver for the C600 chipset and there are no yellow exclamation marks in my Device Manager. Is there something in the BIOS in particular you are looking for? It seems comprehensive to a newbie like me.

2. I will check this when I get back to that. Can you tell me how I can verify this at the OS level (either Windows 7 or Ubuntu)

3. There is a power button (red button) and reset button (black button) with each node, near the front USB port. So the power sequence is:

a. Plug everything in.
b. Turn on transformer's power switch on.
c. Power supply will go from green (starting) to yellow (ready)
d. I think in normal operation the nodes may autostart at staggered intervals.. but I found it a little inconsistent, possibly because I've been swapping in and out components. But one can hit the red button on either of the nodes and the blue LED will indicate power is now on.

NB: When swapping components out, there is a nice black-levered catch that allows each node to slide out (and away from the power bus). Keeps everything well grounded yet de-powered.

Also, I'm going to update my original post with something I forgot to mention re: the PCIe x8 mezzanine.

---

Awesome write up! Especially about the voltage converter, which seems to be a very easy to follow approach.

Questions:
  1. are you able to find any BIOS update? At Wiwynn's web site I could not even find this SV7210 listed.
  2. in the BIOS, are you able to enable the VT-d and it keeps the settings even after reboot? one of my friend said his company used some of those OpenCompute unit and he was not able to keep the VT-d, even using the E5-2670 SR0KX which has the VT-d support from the chip
  3. is there a power button on each of the unit? From the photo it's hard to see how to power up individual unit. Or did you just use the transformer's power switch to either power up or down both units?
Thanks for the great work!
 
  • Like
Reactions: svt3391

NorCalm

New Member
Feb 4, 2016
14
17
3
I too thought it would be much more elegant taking a single $99 node and powering it from ATX. Once I saw what you get "for free" ($199 includes the two $99 nodes, plus the chassis, plus the power supply, plus the power bus board) I got lazy and just paid $48.

Now I feel stupid, I have started building an adapter to convert from an ATX psu to serve this. Now I'm going to do it your way.