Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

[tbkdan]

So, as an update to the fan issue, it appears to be related to the Intel Management Engine stuff. I updated the BIOS (Quanta) which gave me some more features in the BIOS - that was relatively painless. However, I cannot get the ME to update. I've tried from DOS (ME_DOS.bat), WinPE 10 x64 (ME_win64.cmd), and Ubuntu 16.04 x64 (ME_lnx64.sh). DOS & WinPE both give an error of "[ME Update] Read Data from ff000000 Not Match !! Retry No. 0" through 4, then after the fifth try says "This update process failed because of 5 times retry for the same block. c2 - Error: Region is write-protected".

So, I tried disabling ME in the BIOS. After doing so (and issuing a reset through '# ipmitool -I lanplus -U USERID -P PASSW0RD -H 192.168.0.91 mc reset cold' my fans are nice and quiet. Turn ME back on and they slowly ramp back up. ME on or off doesn't matter - still can't update ME with the version from Quanta. ME off kills Serial over LAN which was nice for Ubuntu install while I waited for my GPU adapter Any ideas on how to get ME updated?

 

[aquarapid]

I note that the price of the chassis sold by pdneiman on the 'bay has dropped to $189 each. Shipping is still high (especially if you only order one unit), but if you are in CA, at least no tax.

I ordered a bunch from him last week (at the old price, grrrr), and received everything in good order as advertised, plus air ducts. No risers.

 

[ax206geek]

I was looking for OOB management from a remote system. The idea is the same. But just need the right settings in the BIOS and the right client commands to use. BTW these systems have the DCMI which is a stripped version of IPMI.

ipmitool 1.8.16 from Ubuntu 16.04 has dcmi option which works for me against the Quanta:

Data Center Management Interface commands
    discover               Used to discover supported DCMI capabilities
    power                  Platform power limit command options
    sensors                Prints the available DCMI sensors
    asset_tag              Prints the platform's asset tag
    set_asset_tag          Sets the platform's asset tag
    get_mc_id_string       Get management controller ID string
    set_mc_id_string       Set management controller ID string
    thermalpolicy          Thermal policy get/set
    get_temp_reading       Get Temperature Readings
    get_conf_param         Get DCMI Config Parameters
    set_conf_param         Set DCMI Config Parameters
    oob_discover           Ping/Pong Message for DCMI Discovery

BTW does anybody know what these codes mean? My node 2 of Quanta is throwing out these nonstop:

ipmitool> sol activate
[SOL Session operational.  Use ~? for help]
[17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01]

 

[grfxlab]

ipmitool 1.8.16 from Ubuntu 16.04 has dcmi option which works for me against the Quanta:

Do you know of any free windows based DCMI tools?

 

[Phildesign]

Put my Wywmnn up for sale on ebay.

Running well, but need a system that can accept new high powered graphic cards like the quadros.

4x XEON 8 CORE E5-2670 2.6GHz 128GB RAM 2x 1TB in total

windows 7 pro on each.

If anyone is interested search on ebay wywmnn

 

[smithse79]

Put my Wywmnn up for sale on ebay.View attachment 2539

Running well, but need a system that can accept new high powered graphic cards like the quadros.

4x XEON 8 CORE E5-2670 2.6GHz 128GB RAM 2x 1TB in total

windows 7 pro on each.

If anyone is interested search on ebay wywmnn

Can I buy the risers?

 

[tbkdan]

Do you know of any free windows based DCMI tools

I've had the best overall luck with ipmiutil which is cross-platform. It detects a lot of the specific error codes. Here's a good comparison of the various utilities out there:
Comparison of Linux IPMI Software

 

[tbkdan]

I'll write up more details later, but I wanted to post a few quick notes for those who may be interested.

• For those who are stuck with full-speed fan issues, try different RAM. I purchased supposed "Samsung" RAM from a seller in China. This reported rock solid DIMM temps of 80C - after a cold boot. A buddy's Quanta unit reported actual temps that fluctuated with load as one might expect. Additionally, when viewing "sdr list" output via IPMI, "TSOD SMBus Sts" (TSOD = Thermal Sensor On-Die) would report a warning which is what would eventually ramp the fans up to full speed. I swapped out for a stick of desktop non-ECC DDR3 as a quick test and the system booted with zero warnings and the fans stayed silent. A workaround for silent fans is to disable the ME components in BIOS which keeps the fans from ramping up to 17kRPM, but it also kills your Serial over LAN (SOL) capabilities.
• If you can't get your board to boot, and the orange/blue light is briefly flashing on with the fans followed by silence for 2-3 seconds, rinse->repeat, then you may have a BIOS issue. I was getting adventurous with my BIOS flashing to try and silence my fan issues and ended up with a bad flash that left me in this bricked state. I used my RPi with a 3D printed SOP/SIOC socket (had to reference the pinout for the Winbond BIOS chip) to flash the F03C3A05 BIN file from the Quanta site. I first tried the F03C3A07 version but that one didn't have any impact; 05 worked like a charm. I take no responsibility for what you choose to do, but if you suspect it could be the BIOS you could always purchase a spare Winbond 25Q128BVFG off eBay with a more user-friendly SOIC socket and use the spare with the experiments - it's really easy to pop the chip out and flash with flashrom on the RPi.
• Do yourself a favor and get familiar with the IPMI commands. You can get a lot of good info out of your board and control power/serial console remotely which can be amazing for ease of use (AKA, I'm lazy ).

Also, anybody have a part number for the plastic standoffs that hold up the mezzanine boards?

 

[tbkdan]

Good suggestion on using a spare eeprom. Does it need to be cross-flashed or can it be flashed right from the windmill motherboard? I guess by installing an un-programmed rom we loose booting to a disk to perform the flashing? Getting a flash programmer would be a project by itself. Would you mind flashing a few chips for us for those who are interested to pay for your time?

Correct, they need to be flashed outside of the system. The chips themselves have an SPI interface so nothing too fancy is needed - a Raspberry Pi with some very small clips would do the trick, but the SIOC adapter I linked to would make life easier. I did order a few extra BIOS chips with exactly that in mind It's gonna be a bit before they arrive though. I at least wanted to post the info for anybody else who may run into the same issues I did. I also want to point out there's nothing stopping you from updating the BIOS through normal means provided by Quanta (afudos, etc), but having a backup mechanism isn't a bad idea

 

[tbkdan]

Oh, also, my board shipped with F03_3A04A compared to the F03_3A07 that a lot of people have had with theirs. I may try to work with someone to pull the full original F03_3A07 image compared to the F03C3A07 that I've been using from the website...

 

[ax206geek]

• Do yourself a favor and get familiar with the IPMI commands. You can get a lot of good info out of your board and control power/serial console remotely which can be amazing for ease of use (AKA, I'm lazy ).

I found this site: Computer Cheese useful for some very powerful IPMI options, especially raw IPMI.

 

[xalyy]

Any similiar european deal? I just asked the seller and for me it would be 350$ shipping that is:

 

[Krunnal]

Any similiar european deal? I just asked the seller and for me it would be 350$ shipping that is:

Hi Fellows,

Just purchased a couple of these systems to work and learn about server setups. Hope to try and get them up and running for windows. Had a few questions regarding stacking. I realize they wont fit into the standard racks, any suggestion for an alternative.

Also The system i didnt recieve any information about the rail kit. Any recommendation as to how do i place them in the rack.
I had read previously in the discussion that the systems dont have top covers as they are meant to be placed on top of one another. Does this mean they donot require any rail kits and are just to be placed on top of one another? Wouldnt stacking say for example 26 systems on top of one another put the bottom system under substantial load.?

What if i have to remove one of the chassis? The chassis allows me to pull out one of the nodes, but if i do stack them as mentioned above how can i pull out a chassis if required?

Any recommendations for pdu ratings assuming we are able to put 25 such machines in a rack.

Please excuse us if some of these questions are too novice but any information would really help us learn a few things without making silly mistakes.

Thanks

 

[StammesOpfer]

Stacking 26 of these would be a bad idea. These require a special rack (wider) if you are going to put rails on them and rack them. If you are going to use a standard 19" rack then you have 2 options from my point of view.
1st if you have enough space to load them in from the side of the rack they should clear the rails. You could use a 1u shelf and stack maybe 4-5 per shelf.
2nd If option one wont work you could again use a full length 1u shelf and turn them on the side almost like oversized blades just have to make sure they aren't going to fall over or anything.

As far as power the PSU on these is 700w at 200-280v (no 110v) so make sure you have a 208v, 240v, or 277v power drop and a matching pdu. 700w at 200v is 3.5amp per chassis multiplied by 26 chassis is 91amps. So something like 3 phase 208 30amps per phase would work but then you have to allow for no more than 80% constant load so 50a per phase. So basically get an APC AP7998 or two since you will need the extra plugs.

 

[StammesOpfer]

Didn't quiet get that. But sounds like an idea.

Basically you should be able to stand these on the short side so that they are really tall. You should be able to fit 6 wide (13u tall +shelf) in a standard rack that way.

 

[tbkdan]

For those who are stuck with full-speed fan issues, try different RAM. I purchased supposed "Samsung" RAM from a seller in China. This reported rock solid DIMM temps of 80C - after a cold boot. A buddy's Quanta unit reported actual temps that fluctuated with load as one might expect. Additionally, when viewing "sdr list" output via IPMI, "TSOD SMBus Sts" (TSOD = Thermal Sensor On-Die) would report a warning which is what would eventually ramp the fans up to full speed. I swapped out for a stick of desktop non-ECC DDR3 as a quick test and the system booted with zero warnings and the fans stayed silent. A workaround for silent fans is to disable the ME components in BIOS which keeps the fans from ramping up to 17kRPM, but it also kills your Serial over LAN (SOL) capabilities.

Quick followup on the RAM: I ordered replacement Kingston RAM from Natex and now my DIMM temps report properly, the "TSOD SMBus Sts" sensor reports OK, and the fans behave normally and not in jet engine mode :-D

 

[Krunnal]

Thanks fellows for all the information. Managed to run windows on the machine 7 ...not much issues luckily.

Have decided to work with a local rack manufacturer to create something customized and cost effective. Will probably create a standard open rack to stack the machines.
A few queries..
the board i received has a 10g card on it. But i cannot see this on the device manager. Any ideas? I have installed windows 7 ultimate on the machine. Do i need to install the drivers separately for this.

A query regarding sleep mode. For some reason i am unable to make the machine to go into sleep mode. I tried using the commands but still nothing happens. I am have a requirement where i want my server to be able to go in sleep mode and also use WOL. Any inputs on this would be helpful Thanks

 

[aquarapid]

Here is a picture of what I did for standard rackmount. I basically cut them off using a Dremel Ultra-Saw (awesome piece of hardware, BTW). I am stacking them 6 high on a heavy-duty full-depth shelf, with some side-bracing. Power-conversion to external (non-original) power supplies has been done, as elsewhere in this thread.

Also, for those looking for an aftermarket riser, I have found that the "Star Empery PT319" (search eBay) fits just fine, and lines up just about perfectly with the lower PCIe retaining bracket position. This only allows one card, of course, but that is all I need.

 

[homestudio4k]

anyone has luck with a generic riser card? i bought one from eBay and seems not working. will try more.

 

[StammesOpfer]

I've got two on the way (from china) will let you know when I get to test them out.

also:

Also, for those looking for an aftermarket riser, I have found that the "Star Empery PT319" (search eBay) fits just fine, and lines up just about perfectly with the lower PCIe retaining bracket position. This only allows one card, of course, but that is all I need.