Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

[Monoman]

bubble burst! Thanks for the heads up!

 

[Indecided]

Has anybody figured out to get SOL (Serial Over LAN) or any form of IPMI working on these guys (specifically the Quanta ones) ? If so, it would be great if you could share the steps cos while I can see the IPMI IPs, i'm not really quite sure where to start to connect to them.

 

[Moff Tigriss]

Has anybody figured out to get SOL (Serial Over LAN) or any form of IPMI working on these guys (specifically the Quanta ones) ? If so, it would be great if you could share the steps cos while I can see the IPMI IPs, i'm not really quite sure where to start to connect to them.

Yes, you have a solution here : post #152

It work, but only with ipmitool, AFAIK.
You can't use it with the cluster tools of vSphere (and if someone know how to make it work, my day will be faaaar better ^^), unless you have a relay (and i don't know if an IPMI relay is a thing or even possible).

 

[staph]

I've actually found that the IPMI on these is pretty reliable all things considered, including SoL, wake on LAN, and your other most common IPMI uses.

I've got roughly 30 nodes managed by MAAS where IPMI is used in the process of basically every management function.

The trick for OOB is watching your DHCP logs for what address gets assigned --- assuming you are lucky enough to have gotten a board with IPMI stock/default settings. Look for a dynamic hostname "dcmi0123456789" etc where 0123456789 is your IPMI MAC address, which should be one variant hex digit from the MAC address of the NIC port you're using.

Unless, that is, your non-volatile IPMI wasn't reset by your reseller, since these are all used, and the previous owner/datacenter may have turned off DHCP / changed the IP address scheme / changed admin user / altered things up more severely.

One way to solve this is that if you have one of the older BIOS revisions, and if you can guess a username in the admin group (USERID, admin, root, etc) then you can just use the Cipher Suite 0 exploit to take care of business (pass -C 0 and you're in).

Another option is installing CentOS 6 (the latest builds work fine) and using the custom DCMI tool ---OR--- a recent-ish FreeIPMI (compiling your own RPM from latest source is pretty simple) for access the IPMI in-band. I think it was around 2014/2015 the Windmill workarounds/drivers were integrated into FreeIPMI's tools.

Otherwise you might have to reflash BIOS and MEI to reset the IPMI non-volatile memory.

As a bonus --- on a similarly related note --- I've found that even boards with F03A_1A01 BIOS will happily take the latest available BIOS (F03C_3A07) --- YMMV, proceed at your own risk, don't hold me liable for bricking your board, etc. But I've done this on roughly five 1A01 BIOS boards without incident, although there's always a chance my tests haven't covered everything and this introduces some pain I'm not aware of. If you want to be cautious, just reflash with your current BIOS and MEI version and IPMI will be reset to factory settings. Just pulling out the battery and resetting via jumper isn't enough.

 

[hmartin]

I don't have VGA working because the Quanta doesn't seem to like my GTX 650 and I don't have another card to test with. The uart console works though, and this leads me to two questions:

1. Every time when the board first powers up, I see the following error in the serial console: "ERROR: Type:2; Severity:40; Class:0; Subclass:6; Operation: A002"

According to Google, this is a UEFI error of some sort, but I don't have enough UEFI knowledge or specific knowledge of this board to know what it might refer to. Is this something other people are seeing on power-up? I am powering it with an ATX power supply for 12V and 5V on pin B4. No 10G mezz/PCIe cards installed.

2. When I try to boot with the GTX 650 installed, I always get the following error (repeating, after error in #1) and the board doesn't get to POST: "ERROR: Type:2; Severity:80; Class:3; Subclass: D; Operation: 3"

I've tried with the GTX 650 installed in the PCIe slot directly, on the riser card, and with a PCIe x1 riser designed for bitcoin mining that I bought. The GTX 650 works in other computers, and I've confirmed that the PCIe x1 riser works too. I thought initially it might be a power issue, so I bought the riser which injects power instead of drawing it from the slot, but it didn't resolve the problem. Then I thought maybe it was because all the PCIe ports were configured as GEN3, and the GTX 650 is only Gen2, but even forcing all the PCIe ports to GEN1 in UEFI hasn't gotten rid of the above error.

I'm fine to buy a different graphics card (e.g. GT 710) but it would be nice to know that this error won't occur again on whatever other card I buy. If anyone has recommendations for a reasonably priced used card (<$50) with DisplayPort outputs that you know works, I'd be keen to hear of it.

I'm still running the original 03 BIOS since without VGA I don't feel too confident in trying to upgrade it to 07. Did anyone else have more luck with VGA compatibility after updating the BIOS?

I don't need VGA in the long run since I plan to run this headless with Linux, but it would be really helpful for the initial setup since the uart console formatting is a bit eff'd.

 

[Klee]

I don't have VGA working because the Quanta doesn't seem to like my GTX 650 and I don't have another card to test with. The uart console works though, and this leads me to two questions:

1. Every time when the board first powers up, I see the following error in the serial console: "ERROR: Type:2; Severity:40; Class:0; Subclass:6; Operation: A002"

According to Google, this is a UEFI error of some sort, but I don't have enough UEFI knowledge or specific knowledge of this board to know what it might refer to. Is this something other people are seeing on power-up? I am powering it with an ATX power supply for 12V and 5V on pin B4. No 10G mezz/PCIe cards installed.

2. When I try to boot with the GTX 650 installed, I always get the following error (repeating, after error in #1) and the board doesn't get to POST: "ERROR: Type:2; Severity:80; Class:3; Subclass: D; Operation: 3"

I've tried with the GTX 650 installed in the PCIe slot directly, on the riser card, and with a PCIe x1 riser designed for bitcoin mining that I bought. The GTX 650 works in other computers, and I've confirmed that the PCIe x1 riser works too. I thought initially it might be a power issue, so I bought the riser which injects power instead of drawing it from the slot, but it didn't resolve the problem. Then I thought maybe it was because all the PCIe ports were configured as GEN3, and the GTX 650 is only Gen2, but even forcing all the PCIe ports to GEN1 in UEFI hasn't gotten rid of the above error.

I'm fine to buy a different graphics card (e.g. GT 710) but it would be nice to know that this error won't occur again on whatever other card I buy. If anyone has recommendations for a reasonably priced used card (<$50) with DisplayPort outputs that you know works, I'd be keen to hear of it.

I'm still running the original 03 BIOS since without VGA I don't feel too confident in trying to upgrade it to 07. Did anyone else have more luck with VGA compatibility after updating the BIOS?

I don't need VGA in the long run since I plan to run this headless with Linux, but it would be really helpful for the initial setup since the uart console formatting is a bit eff'd.

I just use an old oem Dell Nvidia pci-e card that I received with a used pc years ago.

Are there any computer shops near you that sell used parts?

 

[hmartin]

I just use an old oem Dell Nvidia pci-e card that I received with a used pc years ago.

Are there any computer shops near you that sell used parts?

I can definitely buy cheap used video cards on fleabay. I was just looking for specific model recommendations that people have confirmed to work. I find that usually used computer stores are final sale and the selection isn't as good as what you can find on fleabay.

I'd prefer to pick up something newer, since it will consume less power and support my monitor which only has digital inputs (HDMI/DisplayPort).

 

[Klee]

ATI Firepro 2260

$7.15 with free shipping.

Dell ATI FireMV 2260 256Mb PCIe x16 PCI Express 2 Display Port Video Card 7CJHP | eBay


Looks like the one sold with this Quanta Open Compute server so its good to go.

QUANTA OPEN COMPUTE WINDMILL SERVER 2 NODE 4x E5-2630L 64GB 2x 250GB ATI 2260 | eBay

 

[Barnett8]

I don't have VGA working because the Quanta doesn't seem to like my GTX 650 and I don't have another card to test with. The uart console works though, and this leads me to two questions:

1. Every time when the board first powers up, I see the following error in the serial console: "ERROR: Type:2; Severity:40; Class:0; Subclass:6; Operation: A002"

According to Google, this is a UEFI error of some sort, but I don't have enough UEFI knowledge or specific knowledge of this board to know what it might refer to. Is this something other people are seeing on power-up? I am powering it with an ATX power supply for 12V and 5V on pin B4. No 10G mezz/PCIe cards installed.

2. When I try to boot with the GTX 650 installed, I always get the following error (repeating, after error in #1) and the board doesn't get to POST: "ERROR: Type:2; Severity:80; Class:3; Subclass: D; Operation: 3"

I've tried with the GTX 650 installed in the PCIe slot directly, on the riser card, and with a PCIe x1 riser designed for bitcoin mining that I bought. The GTX 650 works in other computers, and I've confirmed that the PCIe x1 riser works too. I thought initially it might be a power issue, so I bought the riser which injects power instead of drawing it from the slot, but it didn't resolve the problem. Then I thought maybe it was because all the PCIe ports were configured as GEN3, and the GTX 650 is only Gen2, but even forcing all the PCIe ports to GEN1 in UEFI hasn't gotten rid of the above error.

I'm fine to buy a different graphics card (e.g. GT 710) but it would be nice to know that this error won't occur again on whatever other card I buy. If anyone has recommendations for a reasonably priced used card (<$50) with DisplayPort outputs that you know works, I'd be keen to hear of it.

I'm still running the original 03 BIOS since without VGA I don't feel too confident in trying to upgrade it to 07. Did anyone else have more luck with VGA compatibility after updating the BIOS?

I don't need VGA in the long run since I plan to run this headless with Linux, but it would be really helpful for the initial setup since the uart console formatting is a bit eff'd.

Unfortunately I don't have an answer, but I do have a similar issue on a Wywinn. I'm trying to install a GTX 660Ti on one of the boards and it worked *once*. Ubuntu booted no problem and the monitor showed my desktop. However, on every boot since the very first with the card the monitor shows "No Signal" I have tried everything including:

- Updating Nvidia drivers
- Uninstalling Nvidia drivers
- Trying a different graphics card (which worked with one of the ones recommended on this thread)
- Making sure the 660 worked on another computer (it did)
- Trying it without the pcie power connections plugged in (GPU output a prompt to plug them in)

So I'm running out of options here with how to attack this problem. Anyone have any ideas?

 

[Klee]

Unfortunately I don't have an answer, but I do have a similar issue on a Wywinn. I'm trying to install a GTX 660Ti on one of the boards and it worked *once*. Ubuntu booted no problem and the monitor showed my desktop. However, on every boot since the very first with the card the monitor shows "No Signal" I have tried everything including:

- Updating Nvidia drivers
- Uninstalling Nvidia drivers
- Trying a different graphics card (which worked with one of the ones recommended on this thread)
- Making sure the 660 worked on another computer (it did)
- Trying it without the pcie power connections plugged in (GPU output a prompt to plug them in)

So I'm running out of options here with how to attack this problem. Anyone have any ideas?

Nope... I only have used old PCI-e video cards just to install Ubuntu 17.04 server then I remove them and go headless.

 

[Klee]

I just remembered something I posted way back in this thread, some video cards like the bottom slot in the riser and some like the upper.

Try it with the riser and a GOOD quality powered pci-e ribbon extender if the card is more than one slot.

I tried a regular mining pci-e riser and it worked on an old card once and it worked using a second power supply to give the room to try to hot swap the bios chip.

Of course that was just a temp setup.

 

[hmartin]

Unfortunately I don't have an answer, but I do have a similar issue on a Wywinn. I'm trying to install a GTX 660Ti on one of the boards and it worked *once*. Ubuntu booted no problem and the monitor showed my desktop. However, on every boot since the very first with the card the monitor shows "No Signal" I have tried everything including:

- Updating Nvidia drivers
- Uninstalling Nvidia drivers
- Trying a different graphics card (which worked with one of the ones recommended on this thread)
- Making sure the 660 worked on another computer (it did)
- Trying it without the pcie power connections plugged in (GPU output a prompt to plug them in)

So I'm running out of options here with how to attack this problem. Anyone have any ideas?

I have a bit more information from my testing, but unfortunately still not a working solution for VGA output.

The following combination works repeatedly:

• BIOS configured for UEFI option ROM on PCIe devices
• GTX 650 plugged into the motherboard x16 slot, or riser (either slot), and using an additional mining riser and external power supply
• No monitor plugged in (however this means you won't get any VGA, but VT-d passthrough should be possible)

The following combinations DO NOT work for me:

• BIOS configured for Legacy option ROM on PCIe devices
• Monitor plugged into graphics card

I think I am closer to understanding the issue. For one thing, my GTX 650 is flashed with a UEFI firmware (this was not the factory state, I requested the UEFI firmware from EVGA) as I'm normally using it in a UEFI PC and want the advantages of UEFI firmware on the card. I think the issue is that the UEFI firmware on the Quanta cannot initialize a UEFI graphics stack, and this is the cause of the errors.

If I boot the board without a display plugged into the video card, it boots fine, with all output being to serial. I'm guessing, since I don't have any other card, that people are reporting older cards to work because they are legacy firmware cards only, and thus are initialized via the CSM module in UEFI to give VGA/BIOS graphics output.

Supposedly UEFI cards are supposed to be backward compatible to BIOS with a legacy ROM, but this setting isn't working on my card. I can try to flash it back to the original BIOS firmware and see if that resolves the issue, but it isn't high on my priority list since I managed to get Linux booting through the serial console (pro-tip: GRUB default is 38400, but Quanta default is 57600, change your serial console baud rate if you want to see GRUB at all).

The serial console is dead easy. Just use any old $2 UART to serial adapter and 2mm pitch pins. Connect to pins 9/10 on the debug header (Motherboard Tx/Rx) and ground on pin 13. Default settings are 57600 8n1, but you can change this in BIOS to 115200 and VT100+ which supports colours. Note that it's UART (3.3V) not RS-232. Any kind of UART to Arduino adapter should work.

 

[Barnett8]

• No monitor plugged in (however this means you won't get any VGA, but VT-d passthrough should be possible).

That's too weird! This worked for me! Thanks a ton for the tip, I only wanted the monitor to see if the card worked anyway, and it didn't occur to me that it could be the issue.

 

[hmartin]

Does anyone have the Quanta and are you using PWM fans for cooling?

I bought some 60mm PWM fans, but I'm not seeing any output on the FAN1_PWM (C2) and FAN2_PWM (C4) pins. This causes the fans to run at 100% all the time, which since they're HP server fans, means I'm wasting 50W blowing air on something drawing 30W idle...

Edit: I just checked with an oscilloscope and there is no output on FAN1_PWM or FAN2_PWM on my board. I checked BIOS and there's nothing obvious about enabling PWM fan control. I'm wondering if this might be because I was feeding 5V on pin C4 by mistake when I first got it, thinking that was the PSU_PG pin. Maybe I fried the PWM control on my board?

 

[hmartin]

I reflashed my GTX 650 with the original non-UEFI vBIOS and now I have working VGA!

BIOS configuration and POST messages are still only output via the onboard uart/serial console, but once the board has passed POST, VGA output is enabled. I have memtest running now, something which I could never get working via serial redirection.

So my conclusion is that newer video cards ship with UEFI vBIOS, and the Quanta firmware does not have the appropriate firmware components to initialize these cards in UEFI mode. Hence people are reporting older cards work, because they have a BIOS based vBIOS instead of UEFI capable vBIOS. I believe most cards started shipping with a UEFI vBIOS around 2013.

Easy way to tell if a card is supported or not: check in GPU-Z if the card has UEFI checked. If it does, not likely to work with the Quanta for VGA output:

 

[Klee]

I reflashed my GTX 650 with the original non-UEFI vBIOS and now I have working VGA!

BIOS configuration and POST messages are still only output via the onboard uart/serial console, but once the board has passed POST, VGA output is enabled. I have memtest running now, something which I could never get working via serial redirection.

So my conclusion is that newer video cards ship with UEFI vBIOS, and the Quanta firmware does not have the appropriate firmware components to initialize these cards in UEFI mode. Hence people are reporting older cards work, because they have a BIOS based vBIOS instead of UEFI capable vBIOS. I believe most cards started shipping with a UEFI vBIOS around 2013.

Easy way to tell if a card is supported or not: check in GPU-Z if the card has UEFI checked. If it does, not likely to work with the Quanta:

GTX 1070 showing in GPU-Z ?

 

[hmartin]

GTX 1070 showing in GPU-Z ?

I just pulled a random screenshot of GPU-Z showing the UEFI flag from Google. Not from my card.

 

[voxadam]

I wish I knew more about system firmware/BIOS modding.

Does the system refuse to boot at all when a modern UEFI GPU is installed? Are any errors logged in the PCH event log? When a UEFI GPU is installed is there any activity on Serial-over-LAN (SOL), IPMI, AMT, ME, SPS or other OOB management? What about the system's RS232 serial or debug header?

Has anyone tried installing an both a older card that's known to work and a modern UEFI card at the same time (in an attempt to boot using the old card and let the OS initialize the newer card).

 

[hmartin]

• Does the system refuse to boot at all when a modern UEFI GPU is installed?

If there is a monitor plugged in, yes. It won't pass POST. However you can boot without a monitor plugged in and the card should be available for VT-d passthrough.

• Are any errors logged in the PCH event log?

I'll have to check. Where do you view the PCH event log? I don't have IPMI working yet. There are a LOT of errors logged to the onboard serial console.

• When a UEFI GPU is installed is there any activity on Serial-over-LAN (SOL), IPMI, AMT, ME, SPS or other OOB management?

I'll have to check once I get IPMI working.

• What about the system's RS232 serial or debug header?

Yes, there is a repeating error on the debug header when you try to boot with a UEFI graphics card and a monitor plugged in.

The errors look similar to this one (but not exactly the same):
ERROR: Type:2; Severity:80; Class:3; Subclass: D; Operation: D

From my short research on UEFI errors, they seem to be vendor specific (e.g. referring to a specific module in the UEFI firmware) so it's only through observation that I personally believe it is the UEFI video stack causing issues.

• Has anyone tried installing an both a older card that's known to work and a modern UEFI card at the same time (in an attempt to boot using the old card and let the OS initialize the newer card).

I have only one card, so I cannot test this. From the behaviour I have witnessed, as long as there is no monitor plugged into the UEFI card, it will boot.

In my testing I have found:

1. BIOS flashed card: works
2. UEFI flashed card: works only if no monitor is connected (so there is no VGA output from UEFI or the main OS)

Since I got VGA working on my board, I have updated the firmware to the latest version (2014_WW45.1_F03C3A07.zip) which is much newer than the original firmware that shipped on my board:
Version 2.14.1219. Copyright (C) 2011 American Megatrends, Inc.
BIOS Date: 03/02/2012 13:08:57 Ver: F03_3A07

The newer firmware even has explicit options in setup to boot in UEFI mode, something I was never able to get working on the original firmware. I'm going to try and flash my GTX 650 back to UEFI firmware and see if it's possible to boot with VGA with the latest Quanta firmware from 2014.

 

[voxadam]

@hmartin, Thanks for the detailed reply. You're research and troubleshooting is truly top notch.

Have you tried connecting to the system using VNC when booted using a UEFI mode card (with or without a monitor attached)?

What happens when you attach a monitor after you've successfully booted using a UEFI mode graphics card? I assume your GTX 650 has multiple display interfaces (VGA, HDMI, etc.), have you tried booting with monitors connected each of those different interfaces? Have you tried connecting a monitor, post boot, to each of those interfaces?

I wonder if it might be possible to initialize the display output, post boot, in Linux. The Nouveau[1][2][3] and Linux DRM[4][5] developers would also likely be good resources for this kind of thing.

Has anyone tried reaching out to the OpenCompute-Server mailing list[6] to see if anyone might have some undocumented or inside knowledge, or other tips about these machines? It's probably a long shot since these machines are so old (from their perspective) but there's no harm in asking. The project also maintains a few IRC channels[7] on the Freenode network that might be worth a shot.

Again, your work is highly appreciated, thanks!


[1] nouveau
[2] Nouveau Info Page
[3] irc://chat.freenode.org/#nouveau
[4] DRM
[5] MailingLists
[6] Server - OpenCompute
[7] IRC Channels - OpenCompute