Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

smithse79

Active Member
Sep 17, 2014
196
33
28
40
My Quanta Windmill and 2x E5-2670 and 64GB memory finally came. But I have 2 problems. Can't seem to get USB booting from either ports. And the fan will go into "Jet Engine" mode and make enough noise to drive me crazy after a few minutes. There's nothing in the BIOS to check temperatures and I cannot boot into any OS to check temperatures!

What could have gone wrong?
Go into the BIOS, you have to enable booting from the USB. Mine came with that option turned off. If you have trouble, let me know, I can look at where it is tonight maybe.
 

tbkdan

New Member
May 2, 2016
19
2
3
32
that's probably why your fan was going nuts. you need to shroud to keep both cpu cool
I have mine fully covered by a box to ensure good airflow. Had it off for many hours, turned it on and it immediately goes into full speed fans (just shy of 17,000RPM) and IPMI doesn't indicate any issues (see below, the CPU0 TjMax is at 90C as soon as I hit power when cold, so I think that is literally just the max it's allowed to be..??). I only have one CPU installed right now, so I'm not sure if that's related. Installing Ubuntu 16.04 via SoL (ttyS4) as I type this (with earmuffs on...) and hoping the OS has some sane fan management or additional info I can glean to figure out what's up. Full BIOS reset (via jumper) brought it down to maybe 50% fan for a few minutes, then it ramped back up to 100%; messing with thermal and power management settings in BIOS also has no impact.

# ipmitool -I lanplus -U USERID -P PASSW0RD -H 192.168.0.91 sdr
Outlet Cntr Temp | 17 degrees C | ok
Inlet Temp | 17 degrees C | ok
PCH Temp | 43 % degrees C | ok
P0 Therm Margin | -50 degrees C | ok
P1 Therm Margin | no reading | ns
P0 DIMM Temp | no reading | ns
P1 DIMM Temp | no reading | ns
HSC0 Input Power | 68 Watts | ok
HSC0 Input Volt | 12.28 Volts | ok
CPU0 Tjmax | 90 degrees C | ok
CPU1 Tjmax | no reading | ns
SYS_Fan0 | 17040 RPM | ok
SYS_Fan1 | 16756 RPM | ok
TSOD SMBus Sts | 0xe8 | ok
CPU Therm Trip | 0x36 | ok
Pwr Thresh Evt | 0xfb | ok
Battery Mon | 0xe1 | ok
SEL Status | 0xb7 | ok
DCMI Watchdog | 0xa6 | ok
Processor Fail | 0x91 | ok
Chassis Pwr Sts | 0x76 | ok
Thermal Limit 1 | Not Readable | ns


Go into the BIOS, you have to enable booting from the USB. Mine came with that option turned off. If you have trouble, let me know, I can look at where it is tonight maybe.
Mine also had the internal USB port completely disabled by default.
 

tbkdan

New Member
May 2, 2016
19
2
3
32
So, as an update to the fan issue, it appears to be related to the Intel Management Engine stuff. I updated the BIOS (Quanta) which gave me some more features in the BIOS - that was relatively painless. However, I cannot get the ME to update. I've tried from DOS (ME_DOS.bat), WinPE 10 x64 (ME_win64.cmd), and Ubuntu 16.04 x64 (ME_lnx64.sh). DOS & WinPE both give an error of "[ME Update] Read Data from ff000000 Not Match !! Retry No. 0" through 4, then after the fifth try says "This update process failed because of 5 times retry for the same block. c2 - Error: Region is write-protected".

So, I tried disabling ME in the BIOS. After doing so (and issuing a reset through '# ipmitool -I lanplus -U USERID -P PASSW0RD -H 192.168.0.91 mc reset cold' my fans are nice and quiet. Turn ME back on and they slowly ramp back up. ME on or off doesn't matter - still can't update ME with the version from Quanta. ME off kills Serial over LAN which was nice for Ubuntu install while I waited for my GPU adapter :) Any ideas on how to get ME updated?
 

aquarapid

New Member
Apr 20, 2016
13
0
1
43
I note that the price of the chassis sold by pdneiman on the 'bay has dropped to $189 each. Shipping is still high (especially if you only order one unit), but if you are in CA, at least no tax.

I ordered a bunch from him last week (at the old price, grrrr), and received everything in good order as advertised, plus air ducts. No risers.
 

ax206geek

New Member
May 11, 2016
7
3
3
46
I was looking for OOB management from a remote system. The idea is the same. But just need the right settings in the BIOS and the right client commands to use. BTW these systems have the DCMI which is a stripped version of IPMI.
ipmitool 1.8.16 from Ubuntu 16.04 has dcmi option which works for me against the Quanta:

Code:
Data Center Management Interface commands
    discover               Used to discover supported DCMI capabilities
    power                  Platform power limit command options
    sensors                Prints the available DCMI sensors
    asset_tag              Prints the platform's asset tag
    set_asset_tag          Sets the platform's asset tag
    get_mc_id_string       Get management controller ID string
    set_mc_id_string       Set management controller ID string
    thermalpolicy          Thermal policy get/set
    get_temp_reading       Get Temperature Readings
    get_conf_param         Get DCMI Config Parameters
    set_conf_param         Set DCMI Config Parameters
    oob_discover           Ping/Pong Message for DCMI Discovery
BTW does anybody know what these codes mean? My node 2 of Quanta is throwing out these nonstop:

Code:
ipmitool> sol activate
[SOL Session operational.  Use ~? for help]
[17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01]
 
  • Like
Reactions: DonJon and grfxlab

Phildesign

New Member
Apr 24, 2016
3
0
1
Canada
Put my Wywmnn up for sale on ebay.WywServBoth.jpg

Running well, but need a system that can accept new high powered graphic cards like the quadros.

4x XEON 8 CORE E5-2670 2.6GHz 128GB RAM 2x 1TB in total

windows 7 pro on each.

If anyone is interested search on ebay wywmnn
 

tbkdan

New Member
May 2, 2016
19
2
3
32
I'll write up more details later, but I wanted to post a few quick notes for those who may be interested.
  • For those who are stuck with full-speed fan issues, try different RAM. I purchased supposed "Samsung" RAM from a seller in China. This reported rock solid DIMM temps of 80C - after a cold boot. A buddy's Quanta unit reported actual temps that fluctuated with load as one might expect. Additionally, when viewing "sdr list" output via IPMI, "TSOD SMBus Sts" (TSOD = Thermal Sensor On-Die) would report a warning which is what would eventually ramp the fans up to full speed. I swapped out for a stick of desktop non-ECC DDR3 as a quick test and the system booted with zero warnings and the fans stayed silent. A workaround for silent fans is to disable the ME components in BIOS which keeps the fans from ramping up to 17kRPM, but it also kills your Serial over LAN (SOL) capabilities.
  • If you can't get your board to boot, and the orange/blue light is briefly flashing on with the fans followed by silence for 2-3 seconds, rinse->repeat, then you may have a BIOS issue. I was getting adventurous with my BIOS flashing to try and silence my fan issues and ended up with a bad flash that left me in this bricked state. I used my RPi with a 3D printed SOP/SIOC socket (had to reference the pinout for the Winbond BIOS chip) to flash the F03C3A05 BIN file from the Quanta site. I first tried the F03C3A07 version but that one didn't have any impact; 05 worked like a charm. I take no responsibility for what you choose to do, but if you suspect it could be the BIOS you could always purchase a spare Winbond 25Q128BVFG off eBay with a more user-friendly SOIC socket and use the spare with the experiments - it's really easy to pop the chip out and flash with flashrom on the RPi.
  • Do yourself a favor and get familiar with the IPMI commands. You can get a lot of good info out of your board and control power/serial console remotely which can be amazing for ease of use (AKA, I'm lazy :) ).
Also, anybody have a part number for the plastic standoffs that hold up the mezzanine boards? :)
 

Attachments

tbkdan

New Member
May 2, 2016
19
2
3
32
Good suggestion on using a spare eeprom. Does it need to be cross-flashed or can it be flashed right from the windmill motherboard? I guess by installing an un-programmed rom we loose booting to a disk to perform the flashing? Getting a flash programmer would be a project by itself. Would you mind flashing a few chips for us for those who are interested to pay for your time?
Correct, they need to be flashed outside of the system. The chips themselves have an SPI interface so nothing too fancy is needed - a Raspberry Pi with some very small clips would do the trick, but the SIOC adapter I linked to would make life easier. I did order a few extra BIOS chips with exactly that in mind :) It's gonna be a bit before they arrive though. I at least wanted to post the info for anybody else who may run into the same issues I did. I also want to point out there's nothing stopping you from updating the BIOS through normal means provided by Quanta (afudos, etc), but having a backup mechanism isn't a bad idea :)
 

tbkdan

New Member
May 2, 2016
19
2
3
32
Oh, also, my board shipped with F03_3A04A compared to the F03_3A07 that a lot of people have had with theirs. I may try to work with someone to pull the full original F03_3A07 image compared to the F03C3A07 that I've been using from the website...
 

ax206geek

New Member
May 11, 2016
7
3
3
46
  • Do yourself a favor and get familiar with the IPMI commands. You can get a lot of good info out of your board and control power/serial console remotely which can be amazing for ease of use (AKA, I'm lazy :) ).
I found this site: Computer Cheese useful for some very powerful IPMI options, especially raw IPMI.
 

xalyy

New Member
Feb 25, 2016
3
0
1
30
Any similiar european deal? I just asked the seller and for me it would be 350$ shipping that is: :eek::eek:
 

Krunnal

New Member
May 26, 2016
3
0
1
37
Any similiar european deal? I just asked the seller and for me it would be 350$ shipping that is: :eek::eek:
Hi Fellows,

Just purchased a couple of these systems to work and learn about server setups. Hope to try and get them up and running for windows. Had a few questions regarding stacking. I realize they wont fit into the standard racks, any suggestion for an alternative.

Also The system i didnt recieve any information about the rail kit. Any recommendation as to how do i place them in the rack.
I had read previously in the discussion that the systems dont have top covers as they are meant to be placed on top of one another. Does this mean they donot require any rail kits and are just to be placed on top of one another? Wouldnt stacking say for example 26 systems on top of one another put the bottom system under substantial load.?

What if i have to remove one of the chassis? The chassis allows me to pull out one of the nodes, but if i do stack them as mentioned above how can i pull out a chassis if required?

Any recommendations for pdu ratings assuming we are able to put 25 such machines in a rack.

Please excuse us if some of these questions are too novice but any information would really help us learn a few things without making silly mistakes.

Thanks
 

StammesOpfer

Active Member
Mar 15, 2016
382
126
43
Stacking 26 of these would be a bad idea. These require a special rack (wider) if you are going to put rails on them and rack them. If you are going to use a standard 19" rack then you have 2 options from my point of view.
1st if you have enough space to load them in from the side of the rack they should clear the rails. You could use a 1u shelf and stack maybe 4-5 per shelf.
2nd If option one wont work you could again use a full length 1u shelf and turn them on the side almost like oversized blades just have to make sure they aren't going to fall over or anything.

As far as power the PSU on these is 700w at 200-280v (no 110v) so make sure you have a 208v, 240v, or 277v power drop and a matching pdu. 700w at 200v is 3.5amp per chassis multiplied by 26 chassis is 91amps. So something like 3 phase 208 30amps per phase would work but then you have to allow for no more than 80% constant load so 50a per phase. So basically get an APC AP7998 or two since you will need the extra plugs.
 
Last edited:

tbkdan

New Member
May 2, 2016
19
2
3
32
For those who are stuck with full-speed fan issues, try different RAM. I purchased supposed "Samsung" RAM from a seller in China. This reported rock solid DIMM temps of 80C - after a cold boot. A buddy's Quanta unit reported actual temps that fluctuated with load as one might expect. Additionally, when viewing "sdr list" output via IPMI, "TSOD SMBus Sts" (TSOD = Thermal Sensor On-Die) would report a warning which is what would eventually ramp the fans up to full speed. I swapped out for a stick of desktop non-ECC DDR3 as a quick test and the system booted with zero warnings and the fans stayed silent. A workaround for silent fans is to disable the ME components in BIOS which keeps the fans from ramping up to 17kRPM, but it also kills your Serial over LAN (SOL) capabilities.
Quick followup on the RAM: I ordered replacement Kingston RAM from Natex and now my DIMM temps report properly, the "TSOD SMBus Sts" sensor reports OK, and the fans behave normally and not in jet engine mode :-D