Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack
- Thread starter abstractalgebra
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
They are not 1U height. These are OpenCompute and use OpenCompute sizes, which are decidedly different than standard rack units.
My OpenCompute debug kit arrived today. Finally.
Below are a few photos of the $460** "OCP Debug Kit" that I ordered from Penguin Computing in Fremont, CA. The kit includes the tiny and very expensive OpenCompute debug board plus two relatively inexpensive custom cables. The debug board plugs into the standard OpenCompute debug port, and displays boot codes on two small numeric displays, and also presents a USB serial port that will get you console access.
A spend of $460 isn't much if you have a ton of these servers, but it's way too much if you are buying just one or two. Intel does publish the pin assignments for the debug port, so you can make your own serial cable.
Here is a photo of the full kit. Does it look like $460 worth of kit? No, but I'm just glad that this extremely specific gear is available, regardless of price.
No documentation is included. The tiny board plugs into the OCP debug port. What are the other two cables for? I have no idea.
Here is a rear view of one sled in one of the eBay OCP servers for reference. Note that there is no rear plate and no top cover, and that the disk cables pretty much just dangle there. Extreme minimalism.
Here is the debug board (outlined in blue) plugged into the debug port on the right side of the motherboard sled. Note the two single-digit displays, and the tiny USB port. There is also a small button that does - I don't know yet.
**
Debug Board, Quanta OCP 1 $349.99 $349.99
Cable, Serial, USB-mini-B/6-pin, Passive OCP Debug 1 $12.00 $12.00
Cable, Serial, USB-A/6x1 header, 3.3V, FTDI 1 $31.00 $31.00
Cable, Extender, 2x7/2x7, 20mm, OCP Debug 1 $4.00 $4.00
Subtotal $396.99
Est. Sales Tax $34.74
S & H (3rd Day) $28.54
QUOTE TOTAL $460.27
Below are a few photos of the $460** "OCP Debug Kit" that I ordered from Penguin Computing in Fremont, CA. The kit includes the tiny and very expensive OpenCompute debug board plus two relatively inexpensive custom cables. The debug board plugs into the standard OpenCompute debug port, and displays boot codes on two small numeric displays, and also presents a USB serial port that will get you console access.
A spend of $460 isn't much if you have a ton of these servers, but it's way too much if you are buying just one or two. Intel does publish the pin assignments for the debug port, so you can make your own serial cable.
Here is a photo of the full kit. Does it look like $460 worth of kit? No, but I'm just glad that this extremely specific gear is available, regardless of price.
No documentation is included. The tiny board plugs into the OCP debug port. What are the other two cables for? I have no idea.
Here is a rear view of one sled in one of the eBay OCP servers for reference. Note that there is no rear plate and no top cover, and that the disk cables pretty much just dangle there. Extreme minimalism.
Here is the debug board (outlined in blue) plugged into the debug port on the right side of the motherboard sled. Note the two single-digit displays, and the tiny USB port. There is also a small button that does - I don't know yet.
**
Debug Board, Quanta OCP 1 $349.99 $349.99
Cable, Serial, USB-mini-B/6-pin, Passive OCP Debug 1 $12.00 $12.00
Cable, Serial, USB-A/6x1 header, 3.3V, FTDI 1 $31.00 $31.00
Cable, Extender, 2x7/2x7, 20mm, OCP Debug 1 $4.00 $4.00
Subtotal $396.99
Est. Sales Tax $34.74
S & H (3rd Day) $28.54
QUOTE TOTAL $460.27
Looks DIYable.
Of the 14 pins,
8 provide post codes that eventually get displayed on the two 7-segment displays,
2 are for a RS232 shifter.
1 is for a reset button.
1 to toggle between serial over lan or serial over debug port.
2 for gnd/5v
Button should be reset.
The cables are converting usb to serial. The left one has a FTDI chip it looks like, going by the model # on it.
See page 30 of http://www.opencompute.org/assets/download/Open-Compute-Project-Intel-Motherboard-v2.0.pdf
Of the 14 pins,
8 provide post codes that eventually get displayed on the two 7-segment displays,
2 are for a RS232 shifter.
1 is for a reset button.
1 to toggle between serial over lan or serial over debug port.
2 for gnd/5v
Button should be reset.
The cables are converting usb to serial. The left one has a FTDI chip it looks like, going by the model # on it.
See page 30 of http://www.opencompute.org/assets/download/Open-Compute-Project-Intel-Motherboard-v2.0.pdf
Last edited:
OpenCompute is all about open standards. While that little debug board is $350, it will (likely) work on many (all?) other OpenCompute boards, even future boards, right? So, if you intend on fully investing in this stuff as it shows up on eBay, spending the money up front will likely pay off down the road. That's only speculation though; hopefully someone more well versed in this stuff can shed some more light.
I agree that I paid $10 for the board and $450 for the privilege of buying Enterprise stuff. But I need only one kit to make my entire OpenCompute farm happy, so it's fine with it.
OOB management is serial over LAN (SOL) when the nodes are booted and healthy, and local serial port for local debugging. So I should only need the little board if things have gone haywire, not for day-to-day use. I also plan to get IPMI through the OS for properly booted nodes.
I was googling mezzanine and I stumbled upon this http://www.qct.io/Product/Server/Ac...ezzanine/ON-10GbE-CX3-LACP-p194c77c70c149c210 and it looks like it might fit in the openrack server and it might be a good way to get 10gig plus a vga connection and it looks like it might also provide ikvm which might be nice for remote management.
The photo is uninteresting - it just shows a port on the side of the power supply that mates to the midplane. Luckily for you, Intel publishes the specifications for the PSU port. Look on page 19 of this document:
http://www.opencompute.org/assets/download/Open-Compute-Project-Intel-Motherboard-v2.0.pdf
Quoting:
The midplane has one FCI 51939-582 male right angle header, which is mated directly with the PSU for 12.5VDC input.
Datasheet: http://www.datasheet.support/datasheets/fci/51939-582.pdf
I've seen those documents. The issue is that that connector is not available from any electronics supply house that I've seen without having to order thousands. I feel that I'm gonna have to splice in behind the connector in the chassis where the PSU connects directly to the cables going to the motherboard.
hi,
what about using simple usb ttl serial converter to catch the redirected serial port?
for example $1.35 here: USB To RS232 TTL UART PL2303HX Auto Converter USB to COM Cable Adapter Module
Jan
Ok, I'm going to take @Patrick up on his offer if someone else hasn't already! 
I've been spending a month trying different components the hard way and getting to know the OCP Windmill platform. Yes, some RMAs and returns later.. I can present to you:
WINDMILL FOR DUMMIES
No experience required!
Bill of Materials
4 x E5-2670 ($65 ea, free shipping) - Source from eBay, see the STH thread for details
1 x Windmill OCP2 Node Server ($199 free pick up) - I actually went local for this and picked them up from Michael at Sonitek Computer: Wiwynn SV7210 2 Node Open Compute Server Base Unit 2x LGA2011
NB: This is a brand new unit. It includes all four heat sinks, with corresponding thermal paste for each one. It's a very nice unit for a newbie like me - no sharp edges, well machined, everything slides out including the hard plastic cowling over the CPUs. Very little noise. Well suited even if you don't have a standard rack - right now it's just on a chest of drawers in my home. There is no need to purchase a single fan, heat sink, or other cooling solution with this one-stop-shop.
32 x 8GB Nanya 8GB 2Rx4 DDR3L PC3L-10600 1333MHz ECC RAM ($12ea, free shipping) - Source from eBay, see multiple STH threads. I picked this up in a lot of 50 from eBay's KingMemoryUSA2 because I was actually outfitting two of these servers.
2 x AMD Radeon HD6350 512 MB DDR3 PCIe DMS59 (w/ dual VGA cable) ($10ea, free shipping) - Source from eBay's RogerKy: AMD Radeon HD6350 512MB DDR3 Full Size PCIeX16 DMS59 VGA CN-01CX3M ATI-102-C0903
1 x ELC T-1000 Toroidal Transformer ($48, Prime shipping) - Source from Amazon Prime
Amazon.com: ELC T-1000 1000-Watt Voltage Converter Transformer - Step Up/Down - 110V/220V - Circuit Breaker Protection: Electronics
2 x Crucial BX100 250 GB SATA SSDs - on sale on Amazon for $69ea. You can use any SATA drive you like.
Total Costs: $460 per node, for a two-node "1.5U" format box. Each node is a dual E5-2670 with 128GB RAM (so in total 4x E5-2670 and 256 GB RAM)
Optional: Make use of the PCIe x8 OCP Mezzanine Slot!
I also found OCP Mezzanine cards on eBay. Although there is a catalog out there with VGA ports and the like, the only real ones I could find either had single or dual 10Gbe ports. Here were inexpensive ones at $37 ea (now sold out, sorry):
Intel Ethernet Server Adapter X520-DA2 for Open Compute Project (OCP) X520DA2OCP
Optional: Make use of the other PCIe Slot!
With one slot used up by the VGA card, I purchased a Gigabyte USB 3.1 10Gbps card for $20 just to get some extra USB 3.0 ports on a USB 3.0 hub. These ports cannot be booted from, FYI.
Gigabyte I/O Card GA-USB3.1 USB 3.1 x 2 PCI-Express Interface Retail
I'm sure some STH brethren can tell me where I could alternately get a HBA/RAID/SATA/SAS card to expand on the 2 built-in SATA ports. This would make a pretty amazing Xpenology box....
----------------------
Bring Up
Make it easy on yourself
Learn from my mistakes. After futzing around both on the power, and on the serial console (the header, without the $459 OCP Debug Kit that @dba purchased, is very fine and requires a light touch. I used a Bus Pirate and extracted two pin receptacles from a 3.5mm floppy drive connector, discovered the hard way that it was 57600/8/n/1... but I wouldn't recommend it), I learned the easiest way to resolve things were:
1. Use the ELC toroidal transformer. Very efficient, quiet, and it's a one-stop solution to the 700W 240V supply. You plug in the transformer to the wall, flip the breaker on the unit, and plug in the 700W supply. Done.
2. Use a $10 video card, and forget about the serial console. I purchased a broken HD3450 and a new GT710, and neither worked. I started despairing (and RMAing)... until I embraced my inner miser and just bought cheap. The Radeon HD6450 and HD6350 have both worked well. For Linux, I recommend the $10 HD 6350 - it's ultralow power consumption - just make sure you plug the VGA1 monitor cable (not the VGA2). For Windows, the HD6450 is great and includes an HDMI/VGA/DVI trio, so you can get audio from the HDMI.
Debug
Installation of the CPUs was straightforward, and the heatsinks appear effective, especially when used with the quiet fans on the unit.
I had a few sticks of bad RAM at the start, so got to know Windmill very well. The first thing to note is that there is a "speaker/beep" yellow LED to watch for closely for error beep codes, etc. The general sequence is:
a. Blue LED (power) goes on
b. One beep on yellow LED
c. Green LED indicating hard drive access
d. Two beeps on yellow LED
And then you can access the BIOS.
Other than that, this is really a server for the masses. Besides bad or incompatible components (video cards, RAM), there wasn't anything tricky in bringup and I could boot from USB to install either Windows 7 x64 Pro or Linux. I tried Windows 10 and it hung on an AHCI screen. So Windows 7 x64 Pro it is!
What took a month was finding the compatible components and weathering some bad luck at the start. It's a tremendous system for $460.
What's additional info is helpful?
Please let me know what is helpful to document for you all with regards to this system - it's not for everyone (especially if you want it to fit in a standard rack) but great for a beginner. I can take photos of everything or elaborate about anything.
Background: Long-time STH lurker who jumped at the opportunity to grab $65 E5-2670's. When I looked at what to put them in though, it all seemed so pricey. So I thought "well if these E5-2670's are coming from Facebook, what did Facebook put them in...?" and found this thread. I am not an expert at Servers and can truly speak from total inexperience in all this. I'm sure I missed a lot but I am completely impressed at the firepower that can be purchased with not much money.
I've been spending a month trying different components the hard way and getting to know the OCP Windmill platform. Yes, some RMAs and returns later.. I can present to you:
WINDMILL FOR DUMMIES
No experience required!
Bill of Materials
4 x E5-2670 ($65 ea, free shipping) - Source from eBay, see the STH thread for details
1 x Windmill OCP2 Node Server ($199 free pick up) - I actually went local for this and picked them up from Michael at Sonitek Computer: Wiwynn SV7210 2 Node Open Compute Server Base Unit 2x LGA2011
NB: This is a brand new unit. It includes all four heat sinks, with corresponding thermal paste for each one. It's a very nice unit for a newbie like me - no sharp edges, well machined, everything slides out including the hard plastic cowling over the CPUs. Very little noise. Well suited even if you don't have a standard rack - right now it's just on a chest of drawers in my home. There is no need to purchase a single fan, heat sink, or other cooling solution with this one-stop-shop.
32 x 8GB Nanya 8GB 2Rx4 DDR3L PC3L-10600 1333MHz ECC RAM ($12ea, free shipping) - Source from eBay, see multiple STH threads. I picked this up in a lot of 50 from eBay's KingMemoryUSA2 because I was actually outfitting two of these servers.
2 x AMD Radeon HD6350 512 MB DDR3 PCIe DMS59 (w/ dual VGA cable) ($10ea, free shipping) - Source from eBay's RogerKy: AMD Radeon HD6350 512MB DDR3 Full Size PCIeX16 DMS59 VGA CN-01CX3M ATI-102-C0903
1 x ELC T-1000 Toroidal Transformer ($48, Prime shipping) - Source from Amazon Prime
Amazon.com: ELC T-1000 1000-Watt Voltage Converter Transformer - Step Up/Down - 110V/220V - Circuit Breaker Protection: Electronics
2 x Crucial BX100 250 GB SATA SSDs - on sale on Amazon for $69ea. You can use any SATA drive you like.
Total Costs: $460 per node, for a two-node "1.5U" format box. Each node is a dual E5-2670 with 128GB RAM (so in total 4x E5-2670 and 256 GB RAM)
Optional: Make use of the PCIe x8 OCP Mezzanine Slot!
I also found OCP Mezzanine cards on eBay. Although there is a catalog out there with VGA ports and the like, the only real ones I could find either had single or dual 10Gbe ports. Here were inexpensive ones at $37 ea (now sold out, sorry):
Intel Ethernet Server Adapter X520-DA2 for Open Compute Project (OCP) X520DA2OCP
Optional: Make use of the other PCIe Slot!
With one slot used up by the VGA card, I purchased a Gigabyte USB 3.1 10Gbps card for $20 just to get some extra USB 3.0 ports on a USB 3.0 hub. These ports cannot be booted from, FYI.
Gigabyte I/O Card GA-USB3.1 USB 3.1 x 2 PCI-Express Interface Retail
I'm sure some STH brethren can tell me where I could alternately get a HBA/RAID/SATA/SAS card to expand on the 2 built-in SATA ports. This would make a pretty amazing Xpenology box....
----------------------
Bring Up
Make it easy on yourself
Learn from my mistakes. After futzing around both on the power, and on the serial console (the header, without the $459 OCP Debug Kit that @dba purchased, is very fine and requires a light touch. I used a Bus Pirate and extracted two pin receptacles from a 3.5mm floppy drive connector, discovered the hard way that it was 57600/8/n/1... but I wouldn't recommend it), I learned the easiest way to resolve things were:
1. Use the ELC toroidal transformer. Very efficient, quiet, and it's a one-stop solution to the 700W 240V supply. You plug in the transformer to the wall, flip the breaker on the unit, and plug in the 700W supply. Done.
2. Use a $10 video card, and forget about the serial console. I purchased a broken HD3450 and a new GT710, and neither worked. I started despairing (and RMAing)... until I embraced my inner miser and just bought cheap. The Radeon HD6450 and HD6350 have both worked well. For Linux, I recommend the $10 HD 6350 - it's ultralow power consumption - just make sure you plug the VGA1 monitor cable (not the VGA2). For Windows, the HD6450 is great and includes an HDMI/VGA/DVI trio, so you can get audio from the HDMI.
Debug
Installation of the CPUs was straightforward, and the heatsinks appear effective, especially when used with the quiet fans on the unit.
I had a few sticks of bad RAM at the start, so got to know Windmill very well. The first thing to note is that there is a "speaker/beep" yellow LED to watch for closely for error beep codes, etc. The general sequence is:
a. Blue LED (power) goes on
b. One beep on yellow LED
c. Green LED indicating hard drive access
d. Two beeps on yellow LED
And then you can access the BIOS.
Other than that, this is really a server for the masses. Besides bad or incompatible components (video cards, RAM), there wasn't anything tricky in bringup and I could boot from USB to install either Windows 7 x64 Pro or Linux. I tried Windows 10 and it hung on an AHCI screen. So Windows 7 x64 Pro it is!
What took a month was finding the compatible components and weathering some bad luck at the start. It's a tremendous system for $460.
What's additional info is helpful?
Please let me know what is helpful to document for you all with regards to this system - it's not for everyone (especially if you want it to fit in a standard rack) but great for a beginner. I can take photos of everything or elaborate about anything.
Last edited:
Awesome write up! Especially about the voltage converter, which seems to be a very easy to follow approach.Ok, I'm going to take @Patrick up on his offer if someone else hasn't already!
Long quote omitted ...
Questions:
- are you able to find any BIOS update? At Wiwynn's web site I could not even find this SV7210 listed.
- in the BIOS, are you able to enable the VT-d and it keeps the settings even after reboot? one of my friend said his company used some of those OpenCompute unit and he was not able to keep the VT-d, even using the E5-2670 SR0KX which has the VT-d support from the chip
- is there a power button on each of the unit? From the photo it's hard to see how to power up individual unit. Or did you just use the transformer's power switch to either power up or down both units?
Answers:
1. Although Michael (the eBay seller for the SV7210) suggests there are BIOS updates from Wiwynn, when I emailed Wiwynn, their project director just asked "where did you find one of those old things!?" and did not further reply when I volunteered to be a maintainer of their last set of BIOS updates and drivers. I will follow up with Michael.
I will say that I chased down and found each and every Windows 7 driver for the C600 chipset and there are no yellow exclamation marks in my Device Manager. Is there something in the BIOS in particular you are looking for? It seems comprehensive to a newbie like me.
2. I will check this when I get back to that. Can you tell me how I can verify this at the OS level (either Windows 7 or Ubuntu)
3. There is a power button (red button) and reset button (black button) with each node, near the front USB port. So the power sequence is:
a. Plug everything in.
b. Turn on transformer's power switch on.
c. Power supply will go from green (starting) to yellow (ready)
d. I think in normal operation the nodes may autostart at staggered intervals.. but I found it a little inconsistent, possibly because I've been swapping in and out components. But one can hit the red button on either of the nodes and the blue LED will indicate power is now on.
NB: When swapping components out, there is a nice black-levered catch that allows each node to slide out (and away from the power bus). Keeps everything well grounded yet de-powered.
Also, I'm going to update my original post with something I forgot to mention re: the PCIe x8 mezzanine.
---
1. Although Michael (the eBay seller for the SV7210) suggests there are BIOS updates from Wiwynn, when I emailed Wiwynn, their project director just asked "where did you find one of those old things!?" and did not further reply when I volunteered to be a maintainer of their last set of BIOS updates and drivers. I will follow up with Michael.
I will say that I chased down and found each and every Windows 7 driver for the C600 chipset and there are no yellow exclamation marks in my Device Manager. Is there something in the BIOS in particular you are looking for? It seems comprehensive to a newbie like me.
2. I will check this when I get back to that. Can you tell me how I can verify this at the OS level (either Windows 7 or Ubuntu)
3. There is a power button (red button) and reset button (black button) with each node, near the front USB port. So the power sequence is:
a. Plug everything in.
b. Turn on transformer's power switch on.
c. Power supply will go from green (starting) to yellow (ready)
d. I think in normal operation the nodes may autostart at staggered intervals.. but I found it a little inconsistent, possibly because I've been swapping in and out components. But one can hit the red button on either of the nodes and the blue LED will indicate power is now on.
NB: When swapping components out, there is a nice black-levered catch that allows each node to slide out (and away from the power bus). Keeps everything well grounded yet de-powered.
Also, I'm going to update my original post with something I forgot to mention re: the PCIe x8 mezzanine.
---
Now I feel stupid, I have started building an adapter to convert from an ATX psu to serve this. Now I'm going to do it your way.
I too thought it would be much more elegant taking a single $99 node and powering it from ATX. Once I saw what you get "for free" ($199 includes the two $99 nodes, plus the chassis, plus the power supply, plus the power bus board) I got lazy and just paid $48.