Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

[hmartin]

I assume you mean that the quanta is close to $160 .

What I don't get is :
Why do you mention it does not include enclosure nor PSU ?
Also : say I start with something that's cheap like v1's. What do I have to make to make V2's work (i'd throw some 2660's v2)?


Thanks for all the input

Yes I mean the Quanta. From all the issues I've heard about the Wywinn and lack of firmware updates, I don't know why you'd want one over the Quanta unless it was very cheap.

There was some discussion earlier about board revisions, I think some of the earliest boards are not compatible with v2 CPUs. Assuming you have a compatible board, just replace the v1 CPUs with v2 and you're done.

I just bought one motherboard. I don't live in the US so shipping overseas is prohibitive, and plus I don't need more than one. You can run it without the enclosure and it's trivial to power them from 12V. I mocked a case out of cardboard which uses a 120mm fan at the front which is much quieter than any 60mm fan you'll buy. I'm redesigning it in CAD to be laser cut. The plan is to have it cut from 3mm plywood with space for 8 hard drives beneath.

 

[crotaphiticus]

I'm from Latam but do have plans to get big amount if I can be sure that I can :
put some v2's and make something like this ( who someone posted before):
https://i.imgur.com/iqPtsdk.png


Need it since I would like to put them into DC/warehouse in a rack.

EDIT: how are you planning to put 8 drives ? controller ?

 

[hmartin]

I'm from Latam but do have plans to get big amount if I can be sure that I can :
put some v2's and make something like this ( who someone posted before):
https://i.imgur.com/iqPtsdk.png


Need it since I would like to put them into DC/warehouse in a rack.

EDIT: how are you planning to put 8 drives ? controller ?

Hopefully with the on board SATA/SAS controller. The signals are routed from the chipset to the headers, the board just seems to be missing decoupling capacitors and the SAS connectors.

I ordered some 10nF 0402 capacitors which I soldered. Now I'm just waiting for the SAS connectors to arrive and I'll solder them on and hope for the best. Around $15 in parts ($14 of that being the SAS connectors), so if it doesn't work it's not the end of the world.

MicroSD card for scale. These things are a f---ing nightmare to solder.

 

[crotaphiticus]

Hopefully with the on board SATA/SAS controller. The signals are routed from the chipset to the headers, the board just seems to be missing decoupling capacitors and the SAS connectors.

I ordered some 10nF 0402 capacitors which I soldered. Now I'm just waiting for the SAS connectors to arrive and I'll solder them on and hope for the best. Around $15 in parts ($14 of that being the SAS connectors), so if it doesn't work it's not the end of the world.

MicroSD card for scale. These things are a f---ing nightmare to solder.

Dear lord your are out of you f mind hahah.
Can you please take some extra pics ?

I'll re read all the thread to see the way to plunge those bloody e5 v2 .

As for the cpu blades : once I'm 100% sure the cpus work as I intend to I'll see what I can do with the enclosures ( made 'À la carte' ) .

 

[hmartin]

Can you please take some extra pics ?

What specifically do you want pictures of?

As for the cpu blades : once I'm 100% sure the cpus work as I intend to I'll see what I can do with the enclosures ( made 'À la carte' ) .

Depends how professional you want it to be, and if you want it to fit in a rack. I wouldn't worry about the tray and power supply to be honest. These things take 12V and 5VSB on the PS_PG pin and they're quite happy.

You can easily get very high wattage server PSUs which output 12V for very little money. Something like the HP DPS-600PB can output 600W @ 12V and can be purchased used for around $15. My node draws 150W at full load, but that's with 60W CPUs. You could easily run 2 nodes off one PSU, and maybe 3 nodes if you're careful about peak consumption.

Just look at what the RC guys are doing to power their chargers and you'll get a good idea of which server PSUs are good and which ones aren't.

 

[crotaphiticus]

@hmartin of your general setup .

In detail the storage route you made use of .
I’m willing to grab as many as possible as I need to :
Parse html +lzma a consideradle amount of data which comes to me (from a redundant set of server and my clients give back processed tasks ).
If one of my boxes gets down that work gets say re assigned .
I can tell you I’ve tried a 10gbit link and given how data is being handled max I could do with a pair of 2670 (dmi decode says winter fell ) and frequency and provider say it’s a v1.
I want a ghetto yet easy to swap blades in case of failure .
As mentioned earlier on servers would be in a warehouse /dc in Europe .
Happpened to make use of a to some single cpu setups (>15 I 7 3770s and they performed poorly )

 

[hmartin]

In detail the storage route you made use of .

Right now my node is booting from a SATA SSD connected to one of the 2 onboard SATA 6G ports. I think the SATA/SAS signals from the PCH are routed to the headers, because Quanta sells a very similar motherboard as a storage node, but I haven't tested this yet. I will report my findings when the SAS headers arrive.

Overall, I'm probably planning to add 2-4 disks to use in RAID-Z.

I want a ghetto yet easy to swap blades in case of failure .

Then buy the sleds. There is no cheaper or easier way to swap blades. People have looked into it, but the connectors are difficult to obtain in small quantities and are relatively expensive.

The amount of time and effort you invest in designing and manufacturing your own hot swap chassis isn't worthwhile. Do what the Chinese guys did and vertically mount the chassis so you can fit them into a 19" rack. Any shop with metal working experience should be able to come up with suitable brackets for mounting the chassis vertically and to a 19" rack. Or see if you can find someone in China willing to sell you some for a reasonable price.

As mentioned earlier on servers would be in a warehouse /dc in Europe .

You're probably better off looking at something like the Dell C8220/C6220 system if you plan to put this in a data center:

There's another thread on STH about these.

Personally, I wouldn't bother trying to put the Windmill/Wywinn in a data center. You're either going to need OCP compatible racks, which means your own cage area ($$$), or you're going to put a lot of effort into designing an enclosure to rack these in a 19" rack for 5 year old hardware.

The HP c7000 can do 10GBit with virtual connect. The enclosure with PSUs and fans can be had used for around $500. 10GBit VC modules are around $700 each. The BL460c G8 takes E5 CPUs, and you can get barebone blades for around $180.

My point is if you're looking for cheap E5v1/v2 capable hardware and you plan to put it in a DC, there are better options than the Windmill.

 

[crotaphiticus]

Actually what I'm seeking to achieve is to cram as many e5's as possible.
I can make use of a single drive if needed but a big amount of cpus is my goal ( as rock bottom as possible) .

Is there any enclosure that would allow me to place more powerful cpus / mobos/ram when mine get so deprecated it would be pointless to keep them running ?


I'm aiming at something like the micro cloud but its way too expensive for what I can pay

 

[hmartin]

Is there any enclosure that would allow me to place more powerful cpus / mobos/ram when mine get so deprecated it would be pointless to keep them running ?

Yeah that's kind of the whole point of a blade enclosure. The HP c7000 is compatible with Gen 1 all the way up to Gen 9, though firmware updates are necessary and older onboard administrators (OA) are stuck on iLo 2 which is quite outdated by this point.

So if you want to be forward compatible, get a blade enclosure. They're really just a back plane to provide power, cooling, and signal paths to whatever I/O devices you install in the back. The back plane is usually rated for a very high throughput since it's just a passive component. For example the first c7000 enclosure, which is now over a decade old, is rated for 5Tbit throughput on the back plane. The newest revision (c7000 Platinum) is rated for 7Tbit.

Sure, they're not as cheap as a Windmill, and obviously they carry more vendor "value add" but overall they're fit for the purpose: cramming a lot of compute power into a small amount of rack Us and being very future proof. As a plus, they're fully redundant with N+N or N+1 PSUs and cooling. You can hot swap anything in these chassis (PSUs, OAs, blades, I/O modules) and they'll keep running, assuming you've configured everything correctly.

Actually blade enclosures allow you to crank the density so high most DCs won't allow you to put a fully populated blade enclosure in a rack since the energy consumption exceeds their standard cooling budget. To give you some idea of what I mean, EACH power supply in a c7000 is rated for 2.4kW, and you can populate 6 PSUs per chassis. Each fan is rated for something like 100W, and fully populated the chassis has 10 fans. A fully populated, empty chassis (meaning no blades installed) will draw something around 600W idle.

Anyway, tl;dr if you want forward compatible but still on the cheap, pick up a used blade enclosure and some last-gen blades.

• HP makes the c3000 (8 blades) and c7000 (16 blades). c7000 is way more common and quite cheap used
• Dell makes a blade chassis, but I don't have any experience
• Fujitsu makes a blade chassis, no experience
• Cisco makes a blade chassis, no experience
• Supermicro makes a blade chassis, no experience
• Lenovo makes a blade chassis, no experience

I think blade chassis are well beyond the scope of STH though

One final word: don't buy any blades thinking you can get them running outside the chassis. You can't. Don't waste your time trying. You absolutely need the chassis.

 

[crotaphiticus]

Appreciate you input. I've discarded such approach since I've read about someone in WHT or LET to put into use his HP blade and if my memory serves me right he was sort of question about what you mention : power consumption .

I'd go into a warehouse in .ro . Concerning the a+b : I doubt the ghetto DC even can supply that ( and I don't need it either ) .

TLDR: I get your point and I think it's the best choice if I want to continue plunging it more and more modules and if I want to upgrade, right ?

 

[hmartin]

Concerning the a+b : I doubt the ghetto DC even can supply that ( and I don't need it either ) .

Not sure what you mean here? You can buy a c7000 PSU for around $80 used and hot-wire it. You'll get a 12V power supply capable of providing 2.4kWh with 240V 3 phase input. There are YouTube videos of Russian guys doing this for crypto currency mining. Though IMHO 2.4kWh @ 12V through anything but a blade enclosure back plane is asking for fire.

TLDR: I get your point and I think it's the best choice if I want to continue plunging it more and more modules and if I want to upgrade, right ?

It depends on your use case. If you want something that provides power, cooling, remote management, lots of I/O options, and the ability to slap in newer hardware later with a minimal capital investment, then a blade enclosure is probably right for you.

If you don't have redundancy requirements, or you need only a few nodes, or power is very dear to you, or you don't like working around something that sounds like a jet taking off, then a blade enclosure is probably not the right choice.

 

[crotaphiticus]

A+B : != sources of energy ( different sources of energy ).

"power, cooling, remote management, lots of I/O options, and the ability to slap in newer hardware later with a minimal capital investment, then a blade enclosure is probably right for you"

I want all of those except lot of IO options
No need for redundancy

I'll inspect that HP better

Appreciate it

 

[kfriis]

Hi -

Quick question to all of you with a Quanta - are you able to boot from a SATA hdd connected to the onboard port?

I have installed several different OSes including Windows Server 2016 and Ubuntu Server but the server will not boot from the hdd after the installation. No matter what I have done in the BIOS, it always wants to boot PXE - and after timeout of that, I get "Operating System not found" even though there is clearly a OS on the harddrive.

The workaround for Windows Server is to leave a USB stick plugged in with the installation media on it. The server boots fine from the USB. The windows installer waits for keyboard input for a little bit and if it does not receive that, it will then automatically boot from the harddrive instead of the USB. Kind of an weird way to force boot from the hdd, but the only way I have found so far.

Could you please let me know if any of you are booting directly from the onboard SATA port - and if yes, what your BIOS setting to enable that is? For example, even if I disable PXE boot from anywhere I can find in the BIOS, it still reverts back to that method of booting.

Thanks!

 

[hmartin]

Could you please let me know if any of you are booting directly from the onboard SATA port - and if yes, what your BIOS setting to enable that is? For example, even if I disable PXE boot from anywhere I can find in the BIOS, it still reverts back to that method of booting.

Yes, I had endless problems booting from onboard SATA. Exactly the symptoms you describe, could boot from USB fine and access the hard drive, but the default boot sequence skipped the HDD entirely and went directly to PXE boot.

The solution for me was to update the firmware to the latest version. After the update, I can UEFI boot from a SATA device without any issues. The stock firmware is... quirky.

 

[kfriis]

Phew! Glad I am not the only one with this problem!

My BIOS says: F03_3B08 (dated 3/4/2014)

Where would I find the correct update? If anybody has a direct link I would appreciate it. I tried the Quanta web-site, but it is not clear to me which BIOS update to get.

 

[hmartin]

Phew! Glad I am not the only one with this problem!

My BIOS says: F03_3B08 (dated 3/4/2014)

Where would I find the correct update? If anybody has a direct link I would appreciate it. I tried the Quanta web-site, but it is not clear to me which BIOS update to get.

See: Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

Interesting though, your BIOS version date seems to be newer than what's on Quanta's website. After flashing my board I have BIOS F03C3A07 11/03/2014

I don't know, is this March or November? I don't know if the date style is American or European...

As always when updating firmware, try to have a backup of the original firmware off the BIOS chip before you do anything potentially destructive...

 

[kfriis]

Yes - I am not sure about what month it is either.

One thing I should have mentioned is that I have the v2 board (I think) - since it supports (and I have) E-26xx V2 CPUs. Maybe that explains the date difference?

Also note, that the link to the Quanta site you pointed to (thanks!), now includes a BIOS file dated in 2015 which is newer than both you and I have. Have you updated yours to this version yet?

I thought there was a way to "reset" the BIOS in case something went wrong (using a on-board jumper). Maybe that is not correct? Would the BIOS update utility even attempt to update the BIOS if it isn't supported?

 

[Syndroma]

I had the same issue due to UEFI/Legacy boot mismatch. Switching boot mode to Legacy solved the problem in my case.

I have 3B08 BIOS too.

 

[hmartin]

Also note, that the link to the Quanta site you pointed to (thanks!), now includes a BIOS file dated in 2015 which is newer than both you and I have. Have you updated yours to this version yet?

The 2015 update contains a BIOS dated 11/03/2014.

Would the BIOS update utility even attempt to update the BIOS if it isn't supported?

I don't know what kind of checks Quanta built into the update utility. The 128MBit (16MByte) SOIC16 flash chip is quite cheap online, and if you buy a SOIC16 socket you can clone the chip with a Raspberry Pi or ch341a programmer. The parts to create a byte-for-byte copy of your chip can be had for under $10.

One thing I should have mentioned is that I have the v2 board (I think)

Interesting. Is there any silk screening on the board about v2? I'm curious if there's any way to identify it as a v2 board physically.

 

[hmartin]

Hopefully with the on board SATA/SAS controller. The signals are routed from the chipset to the headers, the board just seems to be missing decoupling capacitors and the SAS connectors.

Just to update everyone on this: if you add 10nF capacitors and the SAS header, you can add additional drives using the controller on the chipset.

So far I just soldered one SAS connector, because they're quite frustrating to work with, but I can see the two additional SATA drives attached. Very happy to see my soldering was not wasted.

Imgur gallery