Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

Discussion in 'Great Deals' started by abstractalgebra, Sep 15, 2015.

  1. server_lol

    server_lol New Member

    Joined:
    Jan 10, 2016
    Messages:
    13
    Likes Received:
    1
    Are these of standard 1U height? Thanks...

     
    #81
  2. dba

    dba Moderator

    Joined:
    Feb 20, 2012
    Messages:
    1,478
    Likes Received:
    180
    They are not 1U height. These are OpenCompute and use OpenCompute sizes, which are decidedly different than standard rack units.
     
    #82
  3. dba

    dba Moderator

    Joined:
    Feb 20, 2012
    Messages:
    1,478
    Likes Received:
    180
    My OpenCompute debug kit arrived today. Finally.

    Below are a few photos of the $460** "OCP Debug Kit" that I ordered from Penguin Computing in Fremont, CA. The kit includes the tiny and very expensive OpenCompute debug board plus two relatively inexpensive custom cables. The debug board plugs into the standard OpenCompute debug port, and displays boot codes on two small numeric displays, and also presents a USB serial port that will get you console access.

    A spend of $460 isn't much if you have a ton of these servers, but it's way too much if you are buying just one or two. Intel does publish the pin assignments for the debug port, so you can make your own serial cable.

    Here is a photo of the full kit. Does it look like $460 worth of kit? No, but I'm just glad that this extremely specific gear is available, regardless of price.
    No documentation is included. The tiny board plugs into the OCP debug port. What are the other two cables for? I have no idea.
    IMG_1337.JPG
    Here is a rear view of one sled in one of the eBay OCP servers for reference. Note that there is no rear plate and no top cover, and that the disk cables pretty much just dangle there. Extreme minimalism.
    IMG_1338.JPG
    Here is the debug board (outlined in blue) plugged into the debug port on the right side of the motherboard sled. Note the two single-digit displays, and the tiny USB port. There is also a small button that does - I don't know yet.
    IMG_1339.JPG

    **
    Debug Board, Quanta OCP 1 $349.99 $349.99
    Cable, Serial, USB-mini-B/6-pin, Passive OCP Debug 1 $12.00 $12.00
    Cable, Serial, USB-A/6x1 header, 3.3V, FTDI 1 $31.00 $31.00
    Cable, Extender, 2x7/2x7, 20mm, OCP Debug 1 $4.00 $4.00
    Subtotal $396.99
    Est. Sales Tax $34.74
    S & H (3rd Day) $28.54
    QUOTE TOTAL $460.27
     
    #83
    crotaphiticus, Chuckleb and Tobias like this.
  4. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,439
    Likes Received:
    4,379
    @dba That is pretty crazy! Why do I feel like that is a $5-10 board to assemble? Such is life I guess.

    I guess what it also means is that your OOB management is limited to when you can have someone plug those little boards in?
     
    #84
  5. xnoodle

    xnoodle Active Member

    Joined:
    Jan 4, 2011
    Messages:
    259
    Likes Received:
    48
    Looks DIYable.

    Of the 14 pins,
    8 provide post codes that eventually get displayed on the two 7-segment displays,
    2 are for a RS232 shifter.
    1 is for a reset button.
    1 to toggle between serial over lan or serial over debug port.
    2 for gnd/5v

    Button should be reset.

    The cables are converting usb to serial. The left one has a FTDI chip it looks like, going by the model # on it.

    See page 30 of http://www.opencompute.org/assets/download/Open-Compute-Project-Intel-Motherboard-v2.0.pdf
     
    #85
    Last edited: Jan 22, 2016
  6. server_lol

    server_lol New Member

    Joined:
    Jan 10, 2016
    Messages:
    13
    Likes Received:
    1
    @dba, nice write-up. The taller and wider than 1U put a damper on my plans to grab a dozen and co-locate these servers. Even if the data center could accommodate the width, we would still have to pay for 2U rather than 1U.
     
    #86
  7. Rain

    Rain Active Member

    Joined:
    May 13, 2013
    Messages:
    224
    Likes Received:
    71
    OpenCompute is all about open standards. While that little debug board is $350, it will (likely) work on many (all?) other OpenCompute boards, even future boards, right? So, if you intend on fully investing in this stuff as it shows up on eBay, spending the money up front will likely pay off down the road. That's only speculation though; hopefully someone more well versed in this stuff can shed some more light.
     
    #87
  8. dba

    dba Moderator

    Joined:
    Feb 20, 2012
    Messages:
    1,478
    Likes Received:
    180
    I agree that I paid $10 for the board and $450 for the privilege of buying Enterprise stuff. But I need only one kit to make my entire OpenCompute farm happy, so it's fine with it.

    OOB management is serial over LAN (SOL) when the nodes are booted and healthy, and local serial port for local debugging. So I should only need the little board if things have gone haywire, not for day-to-day use. I also plan to get IPMI through the OS for properly booted nodes.
     
    #88
    crotaphiticus likes this.
  9. legopc

    legopc Active Member

    Joined:
    Nov 2, 2014
    Messages:
    202
    Likes Received:
    30
    #89
    Patrick likes this.
  10. smithse79

    smithse79 Active Member

    Joined:
    Sep 17, 2014
    Messages:
    196
    Likes Received:
    33
    @dba Can you take a shot of the power ports on this? I think I'm gonna get one and try to configure it with a standard power supply
     
    #90
  11. dba

    dba Moderator

    Joined:
    Feb 20, 2012
    Messages:
    1,478
    Likes Received:
    180
    I should be able to take some snaps of one of the sleds later today.
     
    #91
  12. dba

    dba Moderator

    Joined:
    Feb 20, 2012
    Messages:
    1,478
    Likes Received:
    180
    The photo is uninteresting - it just shows a port on the side of the power supply that mates to the midplane. Luckily for you, Intel publishes the specifications for the PSU port. Look on page 19 of this document:
    http://www.opencompute.org/assets/download/Open-Compute-Project-Intel-Motherboard-v2.0.pdf

    Quoting:
    The midplane has one FCI 51939-582 male right angle header, which is mated directly with the PSU for 12.5VDC input.

    Datasheet: http://www.datasheet.support/datasheets/fci/51939-582.pdf
     
    #92
  13. smithse79

    smithse79 Active Member

    Joined:
    Sep 17, 2014
    Messages:
    196
    Likes Received:
    33
    I've seen those documents. The issue is that that connector is not available from any electronics supply house that I've seen without having to order thousands. I feel that I'm gonna have to splice in behind the connector in the chassis where the PSU connects directly to the cables going to the motherboard.
     
    #93
  14. jap

    jap Member

    Joined:
    Feb 13, 2016
    Messages:
    24
    Likes Received:
    36
    hi,

    what about using simple usb ttl serial converter to catch the redirected serial port?
    for example $1.35 here: USB To RS232 TTL UART PL2303HX Auto Converter USB to COM Cable Adapter Module

    Jan
     
    #94
  15. smithse79

    smithse79 Active Member

    Joined:
    Sep 17, 2014
    Messages:
    196
    Likes Received:
    33
    Well, I went ahead and ordered one. Here's hoping that I can get it working
     
    #95
    NorCalm likes this.
  16. NorCalm

    NorCalm New Member

    Joined:
    Feb 4, 2016
    Messages:
    14
    Likes Received:
    17
    Ok, I'm going to take @Patrick up on his offer if someone else hasn't already! :)

    Background: Long-time STH lurker who jumped at the opportunity to grab $65 E5-2670's. When I looked at what to put them in though, it all seemed so pricey. So I thought "well if these E5-2670's are coming from Facebook, what did Facebook put them in...?" and found this thread. I am not an expert at Servers and can truly speak from total inexperience in all this. I'm sure I missed a lot but I am completely impressed at the firepower that can be purchased with not much money.

    I've been spending a month trying different components the hard way and getting to know the OCP Windmill platform. Yes, some RMAs and returns later.. I can present to you:

    WINDMILL FOR DUMMIES

    No experience required!

    Bill of Materials

    4 x E5-2670 ($65 ea, free shipping) - Source from eBay, see the STH thread for details

    1 x Windmill OCP2 Node Server ($199 free pick up) - I actually went local for this and picked them up from Michael at Sonitek Computer: Wiwynn SV7210 2 Node Open Compute Server Base Unit 2x LGA2011

    NB: This is a brand new unit. It includes all four heat sinks, with corresponding thermal paste for each one. It's a very nice unit for a newbie like me - no sharp edges, well machined, everything slides out including the hard plastic cowling over the CPUs. Very little noise. Well suited even if you don't have a standard rack - right now it's just on a chest of drawers in my home. There is no need to purchase a single fan, heat sink, or other cooling solution with this one-stop-shop.

    32 x 8GB Nanya 8GB 2Rx4 DDR3L PC3L-10600 1333MHz ECC RAM ($12ea, free shipping) - Source from eBay, see multiple STH threads. I picked this up in a lot of 50 from eBay's KingMemoryUSA2 because I was actually outfitting two of these servers.

    2 x AMD Radeon HD6350 512 MB DDR3 PCIe DMS59 (w/ dual VGA cable) ($10ea, free shipping) - Source from eBay's RogerKy: AMD Radeon HD6350 512MB DDR3 Full Size PCIeX16 DMS59 VGA CN-01CX3M ATI-102-C0903

    1 x ELC T-1000 Toroidal Transformer ($48, Prime shipping) - Source from Amazon Prime
    Amazon.com: ELC T-1000 1000-Watt Voltage Converter Transformer - Step Up/Down - 110V/220V - Circuit Breaker Protection: Electronics

    2 x Crucial BX100 250 GB SATA SSDs - on sale on Amazon for $69ea. You can use any SATA drive you like.

    Total Costs: $460 per node, for a two-node "1.5U" format box. Each node is a dual E5-2670 with 128GB RAM (so in total 4x E5-2670 and 256 GB RAM)

    Optional: Make use of the PCIe x8 OCP Mezzanine Slot!

    I also found OCP Mezzanine cards on eBay. Although there is a catalog out there with VGA ports and the like, the only real ones I could find either had single or dual 10Gbe ports. Here were inexpensive ones at $37 ea (now sold out, sorry):

    Intel Ethernet Server Adapter X520-DA2 for Open Compute Project (OCP) X520DA2OCP

    Optional: Make use of the other PCIe Slot!

    With one slot used up by the VGA card, I purchased a Gigabyte USB 3.1 10Gbps card for $20 just to get some extra USB 3.0 ports on a USB 3.0 hub. These ports cannot be booted from, FYI.

    Gigabyte I/O Card GA-USB3.1 USB 3.1 x 2 PCI-Express Interface Retail

    I'm sure some STH brethren can tell me where I could alternately get a HBA/RAID/SATA/SAS card to expand on the 2 built-in SATA ports. This would make a pretty amazing Xpenology box....


    ----------------------

    Bring Up

    Make it easy on yourself

    Learn from my mistakes. After futzing around both on the power, and on the serial console (the header, without the $459 OCP Debug Kit that @dba purchased, is very fine and requires a light touch. I used a Bus Pirate and extracted two pin receptacles from a 3.5mm floppy drive connector, discovered the hard way that it was 57600/8/n/1... but I wouldn't recommend it), I learned the easiest way to resolve things were:

    1. Use the ELC toroidal transformer. Very efficient, quiet, and it's a one-stop solution to the 700W 240V supply. You plug in the transformer to the wall, flip the breaker on the unit, and plug in the 700W supply. Done.

    2. Use a $10 video card, and forget about the serial console. I purchased a broken HD3450 and a new GT710, and neither worked. I started despairing (and RMAing)... until I embraced my inner miser and just bought cheap. The Radeon HD6450 and HD6350 have both worked well. For Linux, I recommend the $10 HD 6350 - it's ultralow power consumption - just make sure you plug the VGA1 monitor cable (not the VGA2). For Windows, the HD6450 is great and includes an HDMI/VGA/DVI trio, so you can get audio from the HDMI.

    Debug

    Installation of the CPUs was straightforward, and the heatsinks appear effective, especially when used with the quiet fans on the unit.

    I had a few sticks of bad RAM at the start, so got to know Windmill very well. The first thing to note is that there is a "speaker/beep" yellow LED to watch for closely for error beep codes, etc. The general sequence is:

    a. Blue LED (power) goes on
    b. One beep on yellow LED
    c. Green LED indicating hard drive access
    d. Two beeps on yellow LED

    And then you can access the BIOS.

    Other than that, this is really a server for the masses. Besides bad or incompatible components (video cards, RAM), there wasn't anything tricky in bringup and I could boot from USB to install either Windows 7 x64 Pro or Linux. I tried Windows 10 and it hung on an AHCI screen. So Windows 7 x64 Pro it is!

    What took a month was finding the compatible components and weathering some bad luck at the start. It's a tremendous system for $460.

    What's additional info is helpful?

    Please let me know what is helpful to document for you all with regards to this system - it's not for everyone (especially if you want it to fit in a standard rack) but great for a beginner. I can take photos of everything or elaborate about anything.
     
    #96
    Last edited: Feb 27, 2016
  17. svt3391

    svt3391 Member

    Joined:
    Feb 11, 2016
    Messages:
    119
    Likes Received:
    17
    Awesome write up! Especially about the voltage converter, which seems to be a very easy to follow approach.

    Questions:
    1. are you able to find any BIOS update? At Wiwynn's web site I could not even find this SV7210 listed.
    2. in the BIOS, are you able to enable the VT-d and it keeps the settings even after reboot? one of my friend said his company used some of those OpenCompute unit and he was not able to keep the VT-d, even using the E5-2670 SR0KX which has the VT-d support from the chip
    3. is there a power button on each of the unit? From the photo it's hard to see how to power up individual unit. Or did you just use the transformer's power switch to either power up or down both units?
    Thanks for the great work!
     
    #97
    NorCalm likes this.
  18. NorCalm

    NorCalm New Member

    Joined:
    Feb 4, 2016
    Messages:
    14
    Likes Received:
    17
    Answers:

    1. Although Michael (the eBay seller for the SV7210) suggests there are BIOS updates from Wiwynn, when I emailed Wiwynn, their project director just asked "where did you find one of those old things!?" and did not further reply when I volunteered to be a maintainer of their last set of BIOS updates and drivers. I will follow up with Michael.

    I will say that I chased down and found each and every Windows 7 driver for the C600 chipset and there are no yellow exclamation marks in my Device Manager. Is there something in the BIOS in particular you are looking for? It seems comprehensive to a newbie like me.

    2. I will check this when I get back to that. Can you tell me how I can verify this at the OS level (either Windows 7 or Ubuntu)

    3. There is a power button (red button) and reset button (black button) with each node, near the front USB port. So the power sequence is:

    a. Plug everything in.
    b. Turn on transformer's power switch on.
    c. Power supply will go from green (starting) to yellow (ready)
    d. I think in normal operation the nodes may autostart at staggered intervals.. but I found it a little inconsistent, possibly because I've been swapping in and out components. But one can hit the red button on either of the nodes and the blue LED will indicate power is now on.

    NB: When swapping components out, there is a nice black-levered catch that allows each node to slide out (and away from the power bus). Keeps everything well grounded yet de-powered.

    Also, I'm going to update my original post with something I forgot to mention re: the PCIe x8 mezzanine.

    ---

     
    #98
    svt3391 likes this.
  19. smithse79

    smithse79 Active Member

    Joined:
    Sep 17, 2014
    Messages:
    196
    Likes Received:
    33
    #99
    NorCalm likes this.
  20. NorCalm

    NorCalm New Member

    Joined:
    Feb 4, 2016
    Messages:
    14
    Likes Received:
    17
    I too thought it would be much more elegant taking a single $99 node and powering it from ATX. Once I saw what you get "for free" ($199 includes the two $99 nodes, plus the chassis, plus the power supply, plus the power bus board) I got lazy and just paid $48.

     
    #100

Share This Page