that's probably why your fan was going nuts. you need to shroud to keep both cpu cool
Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack
- Thread starter abstractalgebra
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Follow the link but only works on centos 6.7
Bug #1156667 “Need driver support for IPMI device on OCPv2 Windm...” : Bugs : The Open Compute Project
Tried but could not succeed on centos 7.
I think it will work some other debians or ubuntus
No idea about windows
Go into the BIOS, you have to enable booting from the USB. Mine came with that option turned off. If you have trouble, let me know, I can look at where it is tonight maybe.
I have mine fully covered by a box to ensure good airflow. Had it off for many hours, turned it on and it immediately goes into full speed fans (just shy of 17,000RPM) and IPMI doesn't indicate any issues (see below, the CPU0 TjMax is at 90C as soon as I hit power when cold, so I think that is literally just the max it's allowed to be..??). I only have one CPU installed right now, so I'm not sure if that's related. Installing Ubuntu 16.04 via SoL (ttyS4) as I type this (with earmuffs on...) and hoping the OS has some sane fan management or additional info I can glean to figure out what's up. Full BIOS reset (via jumper) brought it down to maybe 50% fan for a few minutes, then it ramped back up to 100%; messing with thermal and power management settings in BIOS also has no impact.
# ipmitool -I lanplus -U USERID -P PASSW0RD -H 192.168.0.91 sdr
Outlet Cntr Temp | 17 degrees C | ok
Inlet Temp | 17 degrees C | ok
PCH Temp | 43 % degrees C | ok
P0 Therm Margin | -50 degrees C | ok
P1 Therm Margin | no reading | ns
P0 DIMM Temp | no reading | ns
P1 DIMM Temp | no reading | ns
HSC0 Input Power | 68 Watts | ok
HSC0 Input Volt | 12.28 Volts | ok
CPU0 Tjmax | 90 degrees C | ok
CPU1 Tjmax | no reading | ns
SYS_Fan0 | 17040 RPM | ok
SYS_Fan1 | 16756 RPM | ok
TSOD SMBus Sts | 0xe8 | ok
CPU Therm Trip | 0x36 | ok
Pwr Thresh Evt | 0xfb | ok
Battery Mon | 0xe1 | ok
SEL Status | 0xb7 | ok
DCMI Watchdog | 0xa6 | ok
Processor Fail | 0x91 | ok
Chassis Pwr Sts | 0x76 | ok
Thermal Limit 1 | Not Readable | ns
Mine also had the internal USB port completely disabled by default.
So, as an update to the fan issue, it appears to be related to the Intel Management Engine stuff. I updated the BIOS (Quanta) which gave me some more features in the BIOS - that was relatively painless. However, I cannot get the ME to update. I've tried from DOS (ME_DOS.bat), WinPE 10 x64 (ME_win64.cmd), and Ubuntu 16.04 x64 (ME_lnx64.sh). DOS & WinPE both give an error of "[ME Update] Read Data from ff000000 Not Match !! Retry No. 0" through 4, then after the fifth try says "This update process failed because of 5 times retry for the same block. c2 - Error: Region is write-protected".
So, I tried disabling ME in the BIOS. After doing so (and issuing a reset through '# ipmitool -I lanplus -U USERID -P PASSW0RD -H 192.168.0.91 mc reset cold' my fans are nice and quiet. Turn ME back on and they slowly ramp back up. ME on or off doesn't matter - still can't update ME with the version from Quanta. ME off kills Serial over LAN which was nice for Ubuntu install while I waited for my GPU adapter
Any ideas on how to get ME updated?
So, I tried disabling ME in the BIOS. After doing so (and issuing a reset through '# ipmitool -I lanplus -U USERID -P PASSW0RD -H 192.168.0.91 mc reset cold' my fans are nice and quiet. Turn ME back on and they slowly ramp back up. ME on or off doesn't matter - still can't update ME with the version from Quanta. ME off kills Serial over LAN which was nice for Ubuntu install while I waited for my GPU adapter
Any ideas on how to get ME updated?I note that the price of the chassis sold by pdneiman on the 'bay has dropped to $189 each. Shipping is still high (especially if you only order one unit), but if you are in CA, at least no tax.
I ordered a bunch from him last week (at the old price, grrrr), and received everything in good order as advertised, plus air ducts. No risers.
I ordered a bunch from him last week (at the old price, grrrr), and received everything in good order as advertised, plus air ducts. No risers.
ipmitool 1.8.16 from Ubuntu 16.04 has dcmi option which works for me against the Quanta:
Code:
Data Center Management Interface commands
discover Used to discover supported DCMI capabilities
power Platform power limit command options
sensors Prints the available DCMI sensors
asset_tag Prints the platform's asset tag
set_asset_tag Sets the platform's asset tag
get_mc_id_string Get management controller ID string
set_mc_id_string Set management controller ID string
thermalpolicy Thermal policy get/set
get_temp_reading Get Temperature Readings
get_conf_param Get DCMI Config Parameters
set_conf_param Set DCMI Config Parameters
oob_discover Ping/Pong Message for DCMI Discovery
Code:
ipmitool> sol activate
[SOL Session operational. Use ~? for help]
[17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01][17][01]Put my Wywmnn up for sale on ebay.
Running well, but need a system that can accept new high powered graphic cards like the quadros.
4x XEON 8 CORE E5-2670 2.6GHz 128GB RAM 2x 1TB in total
windows 7 pro on each.
If anyone is interested search on ebay wywmnn
Running well, but need a system that can accept new high powered graphic cards like the quadros.
4x XEON 8 CORE E5-2670 2.6GHz 128GB RAM 2x 1TB in total
windows 7 pro on each.
If anyone is interested search on ebay wywmnn
I've had the best overall luck with ipmiutil which is cross-platform. It detects a lot of the specific error codes. Here's a good comparison of the various utilities out there:
Comparison of Linux IPMI Software
I'll write up more details later, but I wanted to post a few quick notes for those who may be interested.
- For those who are stuck with full-speed fan issues, try different RAM. I purchased supposed "Samsung" RAM from a seller in China. This reported rock solid DIMM temps of 80C - after a cold boot. A buddy's Quanta unit reported actual temps that fluctuated with load as one might expect. Additionally, when viewing "sdr list" output via IPMI, "TSOD SMBus Sts" (TSOD = Thermal Sensor On-Die) would report a warning which is what would eventually ramp the fans up to full speed. I swapped out for a stick of desktop non-ECC DDR3 as a quick test and the system booted with zero warnings and the fans stayed silent. A workaround for silent fans is to disable the ME components in BIOS which keeps the fans from ramping up to 17kRPM, but it also kills your Serial over LAN (SOL) capabilities.
- If you can't get your board to boot, and the orange/blue light is briefly flashing on with the fans followed by silence for 2-3 seconds, rinse->repeat, then you may have a BIOS issue. I was getting adventurous with my BIOS flashing to try and silence my fan issues and ended up with a bad flash that left me in this bricked state. I used my RPi with a 3D printed SOP/SIOC socket (had to reference the pinout for the Winbond BIOS chip) to flash the F03C3A05 BIN file from the Quanta site. I first tried the F03C3A07 version but that one didn't have any impact; 05 worked like a charm. I take no responsibility for what you choose to do, but if you suspect it could be the BIOS you could always purchase a spare Winbond 25Q128BVFG off eBay with a more user-friendly SOIC socket and use the spare with the experiments - it's really easy to pop the chip out and flash with flashrom on the RPi.
- Do yourself a favor and get familiar with the IPMI commands. You can get a lot of good info out of your board and control power/serial console remotely which can be amazing for ease of use (AKA, I'm lazy ).
Attachments
Correct, they need to be flashed outside of the system. The chips themselves have an SPI interface so nothing too fancy is needed - a Raspberry Pi with some very small clips would do the trick, but the SIOC adapter I linked to would make life easier. I did order a few extra BIOS chips with exactly that in mind
It's gonna be a bit before they arrive though. I at least wanted to post the info for anybody else who may run into the same issues I did. I also want to point out there's nothing stopping you from updating the BIOS through normal means provided by Quanta (afudos, etc), but having a backup mechanism isn't a bad idea I found this site: Computer Cheese useful for some very powerful IPMI options, especially raw IPMI.
- Do yourself a favor and get familiar with the IPMI commands. You can get a lot of good info out of your board and control power/serial console remotely which can be amazing for ease of use (AKA, I'm lazy
Hi Fellows,Any similiar european deal? I just asked the seller and for me it would be 350$ shipping that is:
Just purchased a couple of these systems to work and learn about server setups. Hope to try and get them up and running for windows. Had a few questions regarding stacking. I realize they wont fit into the standard racks, any suggestion for an alternative.
Also The system i didnt recieve any information about the rail kit. Any recommendation as to how do i place them in the rack.
I had read previously in the discussion that the systems dont have top covers as they are meant to be placed on top of one another. Does this mean they donot require any rail kits and are just to be placed on top of one another? Wouldnt stacking say for example 26 systems on top of one another put the bottom system under substantial load.?
What if i have to remove one of the chassis? The chassis allows me to pull out one of the nodes, but if i do stack them as mentioned above how can i pull out a chassis if required?
Any recommendations for pdu ratings assuming we are able to put 25 such machines in a rack.
Please excuse us if some of these questions are too novice but any information would really help us learn a few things without making silly mistakes.
Thanks
Stacking 26 of these would be a bad idea. These require a special rack (wider) if you are going to put rails on them and rack them. If you are going to use a standard 19" rack then you have 2 options from my point of view.
1st if you have enough space to load them in from the side of the rack they should clear the rails. You could use a 1u shelf and stack maybe 4-5 per shelf.
2nd If option one wont work you could again use a full length 1u shelf and turn them on the side almost like oversized blades just have to make sure they aren't going to fall over or anything.
As far as power the PSU on these is 700w at 200-280v (no 110v) so make sure you have a 208v, 240v, or 277v power drop and a matching pdu. 700w at 200v is 3.5amp per chassis multiplied by 26 chassis is 91amps. So something like 3 phase 208 30amps per phase would work but then you have to allow for no more than 80% constant load so 50a per phase. So basically get an APC AP7998 or two since you will need the extra plugs.
1st if you have enough space to load them in from the side of the rack they should clear the rails. You could use a 1u shelf and stack maybe 4-5 per shelf.
2nd If option one wont work you could again use a full length 1u shelf and turn them on the side almost like oversized blades just have to make sure they aren't going to fall over or anything.
As far as power the PSU on these is 700w at 200-280v (no 110v) so make sure you have a 208v, 240v, or 277v power drop and a matching pdu. 700w at 200v is 3.5amp per chassis multiplied by 26 chassis is 91amps. So something like 3 phase 208 30amps per phase would work but then you have to allow for no more than 80% constant load so 50a per phase. So basically get an APC AP7998 or two since you will need the extra plugs.
Last edited:
Basically you should be able to stand these on the short side so that they are really tall. You should be able to fit 6 wide (13u tall +shelf) in a standard rack that way.
Quick followup on the RAM: I ordered replacement Kingston RAM from Natex and now my DIMM temps report properly, the "TSOD SMBus Sts" sensor reports OK, and the fans behave normally and not in jet engine mode :-D