Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack
- Thread starter abstractalgebra
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Looks DIYable.
Of the 14 pins,
8 provide post codes that eventually get displayed on the two 7-segment displays,
2 are for a RS232 shifter.
1 is for a reset button.
1 to toggle between serial over lan or serial over debug port.
2 for gnd/5v
Button should be reset.
The cables are converting usb to serial. The left one has a FTDI chip it looks like, going by the model # on it.
See page 30 of http://www.opencompute.org/assets/download/Open-Compute-Project-Intel-Motherboard-v2.0.pdf
Of the 14 pins,
8 provide post codes that eventually get displayed on the two 7-segment displays,
2 are for a RS232 shifter.
1 is for a reset button.
1 to toggle between serial over lan or serial over debug port.
2 for gnd/5v
Button should be reset.
The cables are converting usb to serial. The left one has a FTDI chip it looks like, going by the model # on it.
See page 30 of http://www.opencompute.org/assets/download/Open-Compute-Project-Intel-Motherboard-v2.0.pdf
Last edited:
OpenCompute is all about open standards. While that little debug board is $350, it will (likely) work on many (all?) other OpenCompute boards, even future boards, right? So, if you intend on fully investing in this stuff as it shows up on eBay, spending the money up front will likely pay off down the road. That's only speculation though; hopefully someone more well versed in this stuff can shed some more light.
I agree that I paid $10 for the board and $450 for the privilege of buying Enterprise stuff. But I need only one kit to make my entire OpenCompute farm happy, so it's fine with it.
OOB management is serial over LAN (SOL) when the nodes are booted and healthy, and local serial port for local debugging. So I should only need the little board if things have gone haywire, not for day-to-day use. I also plan to get IPMI through the OS for properly booted nodes.
I was googling mezzanine and I stumbled upon this http://www.qct.io/Product/Server/Ac...ezzanine/ON-10GbE-CX3-LACP-p194c77c70c149c210 and it looks like it might fit in the openrack server and it might be a good way to get 10gig plus a vga connection and it looks like it might also provide ikvm which might be nice for remote management.
The photo is uninteresting - it just shows a port on the side of the power supply that mates to the midplane. Luckily for you, Intel publishes the specifications for the PSU port. Look on page 19 of this document:
http://www.opencompute.org/assets/download/Open-Compute-Project-Intel-Motherboard-v2.0.pdf
Quoting:
The midplane has one FCI 51939-582 male right angle header, which is mated directly with the PSU for 12.5VDC input.
Datasheet: http://www.datasheet.support/datasheets/fci/51939-582.pdf
I've seen those documents. The issue is that that connector is not available from any electronics supply house that I've seen without having to order thousands. I feel that I'm gonna have to splice in behind the connector in the chassis where the PSU connects directly to the cables going to the motherboard.
hi,
what about using simple usb ttl serial converter to catch the redirected serial port?
for example $1.35 here: USB To RS232 TTL UART PL2303HX Auto Converter USB to COM Cable Adapter Module
Jan
Ok, I'm going to take @Patrick up on his offer if someone else hasn't already! 
I've been spending a month trying different components the hard way and getting to know the OCP Windmill platform. Yes, some RMAs and returns later.. I can present to you:
WINDMILL FOR DUMMIES
No experience required!
Bill of Materials
4 x E5-2670 ($65 ea, free shipping) - Source from eBay, see the STH thread for details
1 x Windmill OCP2 Node Server ($199 free pick up) - I actually went local for this and picked them up from Michael at Sonitek Computer: Wiwynn SV7210 2 Node Open Compute Server Base Unit 2x LGA2011
NB: This is a brand new unit. It includes all four heat sinks, with corresponding thermal paste for each one. It's a very nice unit for a newbie like me - no sharp edges, well machined, everything slides out including the hard plastic cowling over the CPUs. Very little noise. Well suited even if you don't have a standard rack - right now it's just on a chest of drawers in my home. There is no need to purchase a single fan, heat sink, or other cooling solution with this one-stop-shop.
32 x 8GB Nanya 8GB 2Rx4 DDR3L PC3L-10600 1333MHz ECC RAM ($12ea, free shipping) - Source from eBay, see multiple STH threads. I picked this up in a lot of 50 from eBay's KingMemoryUSA2 because I was actually outfitting two of these servers.
2 x AMD Radeon HD6350 512 MB DDR3 PCIe DMS59 (w/ dual VGA cable) ($10ea, free shipping) - Source from eBay's RogerKy: AMD Radeon HD6350 512MB DDR3 Full Size PCIeX16 DMS59 VGA CN-01CX3M ATI-102-C0903
1 x ELC T-1000 Toroidal Transformer ($48, Prime shipping) - Source from Amazon Prime
Amazon.com: ELC T-1000 1000-Watt Voltage Converter Transformer - Step Up/Down - 110V/220V - Circuit Breaker Protection: Electronics
2 x Crucial BX100 250 GB SATA SSDs - on sale on Amazon for $69ea. You can use any SATA drive you like.
Total Costs: $460 per node, for a two-node "1.5U" format box. Each node is a dual E5-2670 with 128GB RAM (so in total 4x E5-2670 and 256 GB RAM)
Optional: Make use of the PCIe x8 OCP Mezzanine Slot!
I also found OCP Mezzanine cards on eBay. Although there is a catalog out there with VGA ports and the like, the only real ones I could find either had single or dual 10Gbe ports. Here were inexpensive ones at $37 ea (now sold out, sorry):
Intel Ethernet Server Adapter X520-DA2 for Open Compute Project (OCP) X520DA2OCP
Optional: Make use of the other PCIe Slot!
With one slot used up by the VGA card, I purchased a Gigabyte USB 3.1 10Gbps card for $20 just to get some extra USB 3.0 ports on a USB 3.0 hub. These ports cannot be booted from, FYI.
Gigabyte I/O Card GA-USB3.1 USB 3.1 x 2 PCI-Express Interface Retail
I'm sure some STH brethren can tell me where I could alternately get a HBA/RAID/SATA/SAS card to expand on the 2 built-in SATA ports. This would make a pretty amazing Xpenology box....
----------------------
Bring Up
Make it easy on yourself
Learn from my mistakes. After futzing around both on the power, and on the serial console (the header, without the $459 OCP Debug Kit that @dba purchased, is very fine and requires a light touch. I used a Bus Pirate and extracted two pin receptacles from a 3.5mm floppy drive connector, discovered the hard way that it was 57600/8/n/1... but I wouldn't recommend it), I learned the easiest way to resolve things were:
1. Use the ELC toroidal transformer. Very efficient, quiet, and it's a one-stop solution to the 700W 240V supply. You plug in the transformer to the wall, flip the breaker on the unit, and plug in the 700W supply. Done.
2. Use a $10 video card, and forget about the serial console. I purchased a broken HD3450 and a new GT710, and neither worked. I started despairing (and RMAing)... until I embraced my inner miser and just bought cheap. The Radeon HD6450 and HD6350 have both worked well. For Linux, I recommend the $10 HD 6350 - it's ultralow power consumption - just make sure you plug the VGA1 monitor cable (not the VGA2). For Windows, the HD6450 is great and includes an HDMI/VGA/DVI trio, so you can get audio from the HDMI.
Debug
Installation of the CPUs was straightforward, and the heatsinks appear effective, especially when used with the quiet fans on the unit.
I had a few sticks of bad RAM at the start, so got to know Windmill very well. The first thing to note is that there is a "speaker/beep" yellow LED to watch for closely for error beep codes, etc. The general sequence is:
a. Blue LED (power) goes on
b. One beep on yellow LED
c. Green LED indicating hard drive access
d. Two beeps on yellow LED
And then you can access the BIOS.
Other than that, this is really a server for the masses. Besides bad or incompatible components (video cards, RAM), there wasn't anything tricky in bringup and I could boot from USB to install either Windows 7 x64 Pro or Linux. I tried Windows 10 and it hung on an AHCI screen. So Windows 7 x64 Pro it is!
What took a month was finding the compatible components and weathering some bad luck at the start. It's a tremendous system for $460.
What's additional info is helpful?
Please let me know what is helpful to document for you all with regards to this system - it's not for everyone (especially if you want it to fit in a standard rack) but great for a beginner. I can take photos of everything or elaborate about anything.
Background: Long-time STH lurker who jumped at the opportunity to grab $65 E5-2670's. When I looked at what to put them in though, it all seemed so pricey. So I thought "well if these E5-2670's are coming from Facebook, what did Facebook put them in...?" and found this thread. I am not an expert at Servers and can truly speak from total inexperience in all this. I'm sure I missed a lot but I am completely impressed at the firepower that can be purchased with not much money.
I've been spending a month trying different components the hard way and getting to know the OCP Windmill platform. Yes, some RMAs and returns later.. I can present to you:
WINDMILL FOR DUMMIES
No experience required!
Bill of Materials
4 x E5-2670 ($65 ea, free shipping) - Source from eBay, see the STH thread for details
1 x Windmill OCP2 Node Server ($199 free pick up) - I actually went local for this and picked them up from Michael at Sonitek Computer: Wiwynn SV7210 2 Node Open Compute Server Base Unit 2x LGA2011
NB: This is a brand new unit. It includes all four heat sinks, with corresponding thermal paste for each one. It's a very nice unit for a newbie like me - no sharp edges, well machined, everything slides out including the hard plastic cowling over the CPUs. Very little noise. Well suited even if you don't have a standard rack - right now it's just on a chest of drawers in my home. There is no need to purchase a single fan, heat sink, or other cooling solution with this one-stop-shop.
32 x 8GB Nanya 8GB 2Rx4 DDR3L PC3L-10600 1333MHz ECC RAM ($12ea, free shipping) - Source from eBay, see multiple STH threads. I picked this up in a lot of 50 from eBay's KingMemoryUSA2 because I was actually outfitting two of these servers.
2 x AMD Radeon HD6350 512 MB DDR3 PCIe DMS59 (w/ dual VGA cable) ($10ea, free shipping) - Source from eBay's RogerKy: AMD Radeon HD6350 512MB DDR3 Full Size PCIeX16 DMS59 VGA CN-01CX3M ATI-102-C0903
1 x ELC T-1000 Toroidal Transformer ($48, Prime shipping) - Source from Amazon Prime
Amazon.com: ELC T-1000 1000-Watt Voltage Converter Transformer - Step Up/Down - 110V/220V - Circuit Breaker Protection: Electronics
2 x Crucial BX100 250 GB SATA SSDs - on sale on Amazon for $69ea. You can use any SATA drive you like.
Total Costs: $460 per node, for a two-node "1.5U" format box. Each node is a dual E5-2670 with 128GB RAM (so in total 4x E5-2670 and 256 GB RAM)
Optional: Make use of the PCIe x8 OCP Mezzanine Slot!
I also found OCP Mezzanine cards on eBay. Although there is a catalog out there with VGA ports and the like, the only real ones I could find either had single or dual 10Gbe ports. Here were inexpensive ones at $37 ea (now sold out, sorry):
Intel Ethernet Server Adapter X520-DA2 for Open Compute Project (OCP) X520DA2OCP
Optional: Make use of the other PCIe Slot!
With one slot used up by the VGA card, I purchased a Gigabyte USB 3.1 10Gbps card for $20 just to get some extra USB 3.0 ports on a USB 3.0 hub. These ports cannot be booted from, FYI.
Gigabyte I/O Card GA-USB3.1 USB 3.1 x 2 PCI-Express Interface Retail
I'm sure some STH brethren can tell me where I could alternately get a HBA/RAID/SATA/SAS card to expand on the 2 built-in SATA ports. This would make a pretty amazing Xpenology box....
----------------------
Bring Up
Make it easy on yourself
Learn from my mistakes. After futzing around both on the power, and on the serial console (the header, without the $459 OCP Debug Kit that @dba purchased, is very fine and requires a light touch. I used a Bus Pirate and extracted two pin receptacles from a 3.5mm floppy drive connector, discovered the hard way that it was 57600/8/n/1... but I wouldn't recommend it), I learned the easiest way to resolve things were:
1. Use the ELC toroidal transformer. Very efficient, quiet, and it's a one-stop solution to the 700W 240V supply. You plug in the transformer to the wall, flip the breaker on the unit, and plug in the 700W supply. Done.
2. Use a $10 video card, and forget about the serial console. I purchased a broken HD3450 and a new GT710, and neither worked. I started despairing (and RMAing)... until I embraced my inner miser and just bought cheap. The Radeon HD6450 and HD6350 have both worked well. For Linux, I recommend the $10 HD 6350 - it's ultralow power consumption - just make sure you plug the VGA1 monitor cable (not the VGA2). For Windows, the HD6450 is great and includes an HDMI/VGA/DVI trio, so you can get audio from the HDMI.
Debug
Installation of the CPUs was straightforward, and the heatsinks appear effective, especially when used with the quiet fans on the unit.
I had a few sticks of bad RAM at the start, so got to know Windmill very well. The first thing to note is that there is a "speaker/beep" yellow LED to watch for closely for error beep codes, etc. The general sequence is:
a. Blue LED (power) goes on
b. One beep on yellow LED
c. Green LED indicating hard drive access
d. Two beeps on yellow LED
And then you can access the BIOS.
Other than that, this is really a server for the masses. Besides bad or incompatible components (video cards, RAM), there wasn't anything tricky in bringup and I could boot from USB to install either Windows 7 x64 Pro or Linux. I tried Windows 10 and it hung on an AHCI screen. So Windows 7 x64 Pro it is!
What took a month was finding the compatible components and weathering some bad luck at the start. It's a tremendous system for $460.
What's additional info is helpful?
Please let me know what is helpful to document for you all with regards to this system - it's not for everyone (especially if you want it to fit in a standard rack) but great for a beginner. I can take photos of everything or elaborate about anything.
Last edited:
Awesome write up! Especially about the voltage converter, which seems to be a very easy to follow approach.Ok, I'm going to take @Patrick up on his offer if someone else hasn't already!
Long quote omitted ...
Questions:
- are you able to find any BIOS update? At Wiwynn's web site I could not even find this SV7210 listed.
- in the BIOS, are you able to enable the VT-d and it keeps the settings even after reboot? one of my friend said his company used some of those OpenCompute unit and he was not able to keep the VT-d, even using the E5-2670 SR0KX which has the VT-d support from the chip
- is there a power button on each of the unit? From the photo it's hard to see how to power up individual unit. Or did you just use the transformer's power switch to either power up or down both units?
Answers:
1. Although Michael (the eBay seller for the SV7210) suggests there are BIOS updates from Wiwynn, when I emailed Wiwynn, their project director just asked "where did you find one of those old things!?" and did not further reply when I volunteered to be a maintainer of their last set of BIOS updates and drivers. I will follow up with Michael.
I will say that I chased down and found each and every Windows 7 driver for the C600 chipset and there are no yellow exclamation marks in my Device Manager. Is there something in the BIOS in particular you are looking for? It seems comprehensive to a newbie like me.
2. I will check this when I get back to that. Can you tell me how I can verify this at the OS level (either Windows 7 or Ubuntu)
3. There is a power button (red button) and reset button (black button) with each node, near the front USB port. So the power sequence is:
a. Plug everything in.
b. Turn on transformer's power switch on.
c. Power supply will go from green (starting) to yellow (ready)
d. I think in normal operation the nodes may autostart at staggered intervals.. but I found it a little inconsistent, possibly because I've been swapping in and out components. But one can hit the red button on either of the nodes and the blue LED will indicate power is now on.
NB: When swapping components out, there is a nice black-levered catch that allows each node to slide out (and away from the power bus). Keeps everything well grounded yet de-powered.
Also, I'm going to update my original post with something I forgot to mention re: the PCIe x8 mezzanine.
---
1. Although Michael (the eBay seller for the SV7210) suggests there are BIOS updates from Wiwynn, when I emailed Wiwynn, their project director just asked "where did you find one of those old things!?" and did not further reply when I volunteered to be a maintainer of their last set of BIOS updates and drivers. I will follow up with Michael.
I will say that I chased down and found each and every Windows 7 driver for the C600 chipset and there are no yellow exclamation marks in my Device Manager. Is there something in the BIOS in particular you are looking for? It seems comprehensive to a newbie like me.
2. I will check this when I get back to that. Can you tell me how I can verify this at the OS level (either Windows 7 or Ubuntu)
3. There is a power button (red button) and reset button (black button) with each node, near the front USB port. So the power sequence is:
a. Plug everything in.
b. Turn on transformer's power switch on.
c. Power supply will go from green (starting) to yellow (ready)
d. I think in normal operation the nodes may autostart at staggered intervals.. but I found it a little inconsistent, possibly because I've been swapping in and out components. But one can hit the red button on either of the nodes and the blue LED will indicate power is now on.
NB: When swapping components out, there is a nice black-levered catch that allows each node to slide out (and away from the power bus). Keeps everything well grounded yet de-powered.
Also, I'm going to update my original post with something I forgot to mention re: the PCIe x8 mezzanine.
---
Now I feel stupid, I have started building an adapter to convert from an ATX psu to serve this. Now I'm going to do it your way.
I too thought it would be much more elegant taking a single $99 node and powering it from ATX. Once I saw what you get "for free" ($199 includes the two $99 nodes, plus the chassis, plus the power supply, plus the power bus board) I got lazy and just paid $48.
I'm glad you confirmed that you can use a video card for this. Will make it much easier to get configured. Can it boot from USB?
For the VT-d question, I don't know if this answers your question, but on a lark I installed the Xen Hypervisor on my Ubuntu 16.04 LTS beta.
This is the output of the initial dom0 dmesg upon boot:
This is the output of the initial dom0 dmesg upon boot:
(XEN) Processor #45 6:13 APIC version 21
(XEN) Processor #47 6:13 APIC version 21
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 2, version 32, address 0xfec01000, GSI 24-47
(XEN) IOAPIC[2]: apic_id 3, version 32, address 0xfec40000, GSI 48-71
(XEN) Enabling APIC mode: Phys. Using 3 I/O APICs
(XEN) Switched to APIC driver x2apic_cluster.
(XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2600.028 MHz processor.
(XEN) Initing memory sharing.
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables enabled.
(XEN) Processor #47 6:13 APIC version 21
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 2, version 32, address 0xfec01000, GSI 24-47
(XEN) IOAPIC[2]: apic_id 3, version 32, address 0xfec40000, GSI 48-71
(XEN) Enabling APIC mode: Phys. Using 3 I/O APICs
(XEN) Switched to APIC driver x2apic_cluster.
(XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2600.028 MHz processor.
(XEN) Initing memory sharing.
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables enabled.
Yes. There are two USB 2.0 ports, one in the front, and one on the motherboard (they call it "internal").
They don't seem to be identical; the internal one sometimes works for boot.. and sometimes only works for a keyboard.
It's foolproof to use the front USB port, if your USB drive fits in the port. When you boot hit <DEL> to go to the BIOS and enable the USB key for boot (seems to support UEFI but I haven't tested it).
Also, note that there are some quirky things about more modern video cards - the GeForce GT 710 2GB didn't work at all and I had to return that. The $10 special I mentioned in the original article works great. The Windmill BIOS is dated Apr 2013.
Last edited: