Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

[hmartin]

• Have you tried connecting to the system using VNC when booted using a UEFI mode card (with or without a monitor attached)?

I'm only interested in Linux (sorry Windows users). Since I haven't been able to get the board to boot via UEFI, I've just been stuck in the Arch Linux installation environment from the USB stick. This doesn't include X by default, so I haven't tested VNC. I don't see any reason why it wouldn't work, since with Linux you can run X11 in a VNC server without a graphics card installed. But I understand this isn't the intent of your question

• What happens when you attach a monitor after you've successfully booted using a UEFI mode graphics card?

Nothing. The video card is never initialized by the UEFI firmware, and thus the VGA console is not passed to the OS. So, you can boot without a monitor plugged in, but it means the monitor won't work as an output for the OS. As I said before though, the video card does exist on the PCIe bus, so VT-d passthrough is possible in theory, although I've been stuck in a minimalist installer environment so haven't tested this myself.

• I assume your GTX 650 has multiple display interfaces (VGA, HDMI, etc.), have you tried booting with monitors connected each of those different interfaces?

Yes, my card has DVI-I, DVI-D, and mini HDMI. If you have a monitor plugged into any of those and the card has a UEFI firmware, my board will not pass POST.

• Have you tried connecting a monitor, post boot, to each of those interfaces?

Yes. At least for Linux console, it doesn't work. To my understanding, this is because the kernel uses the framebuffer (for lack of a better word) passed from UEFI/BIOS, and without that there is no video output.

It's possible that starting X with the official NVidia driver would initialize the video card and you would get GUI output. I'm hoping that with the newer firmware I'll be able to boot Linux in UEFI mode, so I'll add this to my list of things to test.

Current list:

1. Do UEFI graphics cards work with the latest firmware from 2014? -> Will reflash GTX 650 with UEFI firmware and test.
2. Figure out UEFI OS booting.

If I have time, I can also test some combinations with Windows. Maybe someone can suggest if they'd prefer I test Windows 7 or Windows 10? Please keep in mind I only have the GTX 650 and no other cards, but I can try it in BIOS or UEFI mode since I can reflash it easily.

 

[voxadam]

I'm with you when it comes to Linux; I don't have much use for Windows and hope to fulfill what needs I do have (primarily proprietary mechanical and electrical CAD software) through virtualization.

I have to say, you've been incredibly thorough in your testing.

I should have been more explicit when I asked about VNC. The BIOS/firmware/BMC/vPro/ME/AMT on this board provides a system level VNC server that can be used even when the system has crashed, is an otherwise sub-optimal state, and even before it has completed booting.

• https://ami.com/ami_downloads/AMI_Debug_Remote_Users_Guide.pdf
  
• BIOS and console access via VNC | Loïc Dachary
  
• How to Remotely Control Your PC (Even When it Crashes)

I was thinking, there are a few kernel parameters that might impact your booting.

video=vesaff vga=normal
nomodeset nouveau.modeset=0
module_blacklist=nouveau
vga=NORMAL_VGA
intel_iommu=off
acpi=off
noapic
efi=debug​

Here are a few more kernel parameters that jumped out at me while scanning the kernel docs but seem less likely.

acpi=noirq
acpi=strict
intel_iommu=igfx_off
intel_iommu=strict
lapic
pci=bios
pci=nobios
pci=noacpi — Do not use ACPI for IRQ routing for PCI scanning
pci=noearly
pnpacpi=off
pnpbios=off
powersave=off
agp=off
debug
no_console_suspend
nowatchdog

source: The kernel’s command-line parameters — The Linux Kernel documentation​

 

[hmartin]

I should have been more explicit when I asked about VNC. The BIOS/firmware/BMC/vPro/ME/AMT on this board provides a system level VNC server that can be used even when the system has crashed, is an otherwise sub-optimal state, and even before it has completed booting.

• https://ami.com/ami_downloads/AMI_Debug_Remote_Users_Guide.pdf
  
• BIOS and console access via VNC | Loïc Dachary
  
• How to Remotely Control Your PC (Even When it Crashes)

Ah ha, I didn't think of the AMT VNC console. Everyone was talking about serial over LAN so I assumed that was the best remote console supported.

I read earlier that flashing UEFI and ME firmware should be enough to reset the LOM username and password. I guess I need to dig deeper to figure out how to configure that so remote VNC is working. The ME is enabled in setup, and I can see it requesting an IP address from my DHCP server, but the standard ports for AMT configuration (16992/3) don't seem to work at all.

Edit: I'm not convinced the ME includes a VNC server, unless someone has it working and can correct me. From the documentation:
Depending on system configuration and SPS-DM configuration, only one NIC is enabled for ME shared NIC out-of-band access. Remote access features include serial over LAN, remote power control (power on, power off, power cycle), and a system event log.

These all sound like basic IPMI functions, not a full fledged BMC similar to what you'd find on HP/Dell/etc.

Edit 2: it's working with ipmitool. I don't think VNC or web configuration is supported on this board?

$ ipmitool -I lan -H 192.168.1.X -U admin -a channel info
Password:
Get HPM.x Capabilities request failed, compcode = d4
Channel 0x1 info:
  Channel Medium Type   : 802.3 LAN
  Channel Protocol Type : IPMB-1.0
  Session Support       : multi-session
  Active Session Count  : 1
  Protocol Vendor ID    : 7154
  Volatile(active) Settings
    Alerting            : disabled
    Per-message Auth    : enabled
    User Level Auth     : enabled
    Access Mode         : always available
  Non-Volatile Settings
    Alerting            : disabled
    Per-message Auth    : enabled
    User Level Auth     : enabled
    Access Mode         : always available

 

[Klee]

• Does the system refuse to boot at all when a modern UEFI GPU is installed?

If there is a monitor plugged in, yes. It won't pass POST. However you can boot without a monitor plugged in and the card should be available for VT-d passthrough.

• Are any errors logged in the PCH event log?

I'll have to check. Where do you view the PCH event log? I don't have IPMI working yet. There are a LOT of errors logged to the onboard serial console.

• When a UEFI GPU is installed is there any activity on Serial-over-LAN (SOL), IPMI, AMT, ME, SPS or other OOB management?

I'll have to check once I get IPMI working.

• What about the system's RS232 serial or debug header?

Yes, there is a repeating error on the debug header when you try to boot with a UEFI graphics card and a monitor plugged in.

The errors look similar to this one (but not exactly the same):
ERROR: Type:2; Severity:80; Class:3; Subclass: D; Operation: D

From my short research on UEFI errors, they seem to be vendor specific (e.g. referring to a specific module in the UEFI firmware) so it's only through observation that I personally believe it is the UEFI video stack causing issues.

• Has anyone tried installing an both a older card that's known to work and a modern UEFI card at the same time (in an attempt to boot using the old card and let the OS initialize the newer card).

I have only one card, so I cannot test this. From the behaviour I have witnessed, as long as there is no monitor plugged into the UEFI card, it will boot.

In my testing I have found:

1. BIOS flashed card: works
2. UEFI flashed card: works only if no monitor is connected (so there is no VGA output from UEFI or the main OS)

Since I got VGA working on my board, I have updated the firmware to the latest version (2014_WW45.1_F03C3A07.zip) which is much newer than the original firmware that shipped on my board:
Version 2.14.1219. Copyright (C) 2011 American Megatrends, Inc.
BIOS Date: 03/02/2012 13:08:57 Ver: F03_3A07

The newer firmware even has explicit options in setup to boot in UEFI mode, something I was never able to get working on the original firmware. I'm going to try and flash my GTX 650 back to UEFI firmware and see if it's possible to boot with VGA with the latest Quanta firmware from 2014.

2014_WW45.1_F03C3A07.zip , Where did you find this? Tried google but no luck.

 

[voxadam]

Oddly, Bing seems to find it.

http://www.bing.com/search?q="2014_WW45.1_F03C3A07"

 

[hmartin]

2014_WW45.1_F03C3A07.zip , Where did you find this? Tried google but no luck.

Quanta website: Download Center | QCT- Quanta Cloud Technology

I have no idea if this board is really "F03C" but that's what was closest. F03A doesn't have any BIOS updates available. I guessed F03C based on the BIOS version present on my board originally: "Ver: F03_3A07"

It flashed successfully, and boots successfully. The firmware is a huge update from the previous version, UEFI booting is working perfectly for me now.

Edit: you will need working VGA if you plan to run the update using DOS. I tried my Google-Fu but couldn't figure out how to get DOS to redirect output to the serial console. Just run BIOS_dos.bat and ME_dos.bat and reboot. The included README has these instructions too.

Same deal as mentioned here: Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

 

[hmartin]

Do UEFI graphics cards work with the latest firmware from 2014? -> Will reflash GTX 650 with UEFI firmware and test.

I reflashed my GTX 650 with the UEFI firmware, and I can now confirm that I have working VGA with the UEFI vBIOS. This was never possible before on the old firmware from 2012.

So if you are having problems with a newer video card, which likely has a UEFI vBIOS, try updating your firmware to the latest version (link above).
BIOS F03C3A07 11/03/2014

With the latest firmware I now having working UEFI boot and UEFI video. The firmware even has options to configure Secure Boot, if that's your thing.

 

[Klee]

Thanks, now i'm tempted to flash my spare bios chip, and run it in one of my Wiwynn's just to see.

 

[voxadam]

Stellar work hmartin!

 

[hmartin]

My board came without the 10Gb mezz card. I'm looking at maybe buying one since they're pretty inexpensive, but I'm wondering if it is possible to run a 1Gb SFP in the SFP+ slot? I have no 10Gb equipment, but it would be nice to have an extra network interface without sacrificing a normal PCIe slot.

I've looked around and I can find Mellanox tables of SFP compatibility for their other ConnectX-3 cards, but not the OCP model. Curious if anyone is using a 1Gb SFP in their CX341/342.

 

[kfriis]

Hi Guys -

Thinking about one of these Quanta Windmill V2 systems. With the riser, do you think one of the new, low-profile Geforce GT 1030 cards would work - like this one?

https://www.amazon.com/GIGABYTE-GeF...id=1508869819&sr=8-5&keywords=geforce+gt+1030

If it fits, it would be a lot of performance (TDP 30W) in a small form-factor.

 

[hmartin]

Hi Guys -

Thinking about one of these Quanta Windmill V2 systems. With the riser, do you think one of the new, low-profile Geforce GT 1030 cards would work - like this one?

https://www.amazon.com/GIGABYTE-GeF...id=1508869819&sr=8-5&keywords=geforce+gt+1030

If it fits, it would be a lot of performance (TDP 30W) in a small form-factor.

Physically the dimensions are fine, the card will fit.

Whether it works with your board or not will probably depend on your firmware version. I'd recommend having an older card so you can update the firmware on your board. My GTX 650 with the UEFI firmware did not work until I installed the latest firmware available from Quanta (see previous posts).

Regarding the power, I'm not sure if the Windmill conforms to the PCI-E specification for power. In theory the x16 slot should provide 75W (or roughly 35W split between cards in the riser), but my GTX 650 requires 6 pin power to operate, so I can't say from my own experience that a 30W card would work problem free.

 

[kfriis]

Thanks.

I wonder what the most powerful graphics card somebody has made work in a Open Compute server is. Does anybody know?

There are several threads about these servers and I believe a lot of people on STH have them - I would think a least a few people have tried a modern card in the last year or so.

 

[Klee]

I only have used a old pci-e card when installing the OS or running Windows 7 to flash the bios.

But I did try to "hot flash" the bios , did not work as the server shuts down as soon as you try to remove it, and to have room I had to use a mining extender that uses a usb cable as a data cable and and a secondary power supply to power the adapter and the video card did work.

That does prove a pci-e extender will work on these servers, may come in handy for some one to know this.

 

[crotaphiticus]

Have to admit I've beeen quite lazy and have not read it all in detail.

EDIT:


Guys been reading this thread but have not seen anyone making use oE5V2 CPU's.
Has someone been able to make use of them ? Ebay seller pdeinman or something like that told me it would work but I find it odd not having seen it here.

Seen someone already asked about this. Have to continue reading this thread !



EDIT 2: 2X 2TB SATA ( 1 PER NODE ) . We have 2 nodes crammed in this setup. Does this imply I cannot add a second ( would rather make use of some ssds as mentioned before ) on each node?

/////////////////////////////////////////////////////


EDIT 3: I've seen the pics of the blade chinedse make and I am more interested now LOL.

Seems like esiso does include 10gb eth ( from the link someone else posted... odd that certain things would not show up with a normal ebay query nor googles ).

-IPMI ( found someone made a way to make it work)


Note to myself : read the whole f thread

 

[crotaphiticus]

Guys. Read most of the thread but I'm unsure if the same cheap $170 quanta chassis and mobo is the one being used here . I understand that in my sample seller makes use of one of the best V2's available but :
say 1k for cpus
500-600 for ram
200 for drives

so why does that config come so expensive ?

 

[Klee]

Guys. Read most of the thread but I'm unsure if the same cheap $170 quanta chassis and mobo is the one being used here . I understand that in my sample seller makes use of one of the best V2's available but :
say 1k for cpus
500-600 for ram
200 for drives

so why does that config come so expensive ?

Your link does not work.

 

[crotaphiticus]

QUANTA WINDMILL v2 SYSTEM NODES 4x 12 CORE E5-2695v2 2.4GHz 128GB 2x 2TB SATA | eBay

I should put the price of 4x cpus of that type closer to the 1.5k range, but still.

 

[hmartin]

QUANTA WINDMILL v2 SYSTEM NODES 4x 12 CORE E5-2695v2 2.4GHz 128GB 2x 2TB SATA | eBay

I should put the price of 4x cpus of that type closer to the 1.5k range, but still.

Let's break it down:
~$80 for each board w/heat sinks
~$250 for 128GB of RAM (8GB RDIMMs)
~$50 per 2TB HDD

So the total (minus enclosure and PSU) would be around $900.

Those CPUs are going for around $300 each, so that's another $1200:

Yeah, they're charging a premium, but it's a one-stop shop.

If you don't need the v2 CPU, v1 CPUs are quite a bit cheaper. I picked up 2xE5-2630L for $15 each from South Korea. You can buy the same CPU from a US seller for $21.

I bought just the board and populated it myself (also using an ATX power supply for 12V power). They've been pulled from eBay, but you can try to email/call the seller and see if they have any left? Maybe they all got sent to the scrap yard?

 

[crotaphiticus]

~$80 for each board w/heat sinks

I assume you mean that the quanta is close to $160 .

What I don't get is :
Why do you mention it does not include enclosure nor PSU ?
I've seen this one :
QUANTA WINDMILL SYSTEM 2 NODES W/ SYSTEM BOARD HEATSINK LGA2011 | eBay

I assume it's not V2 and it wont have e5 v2 capabilities, right ?

Also : say I start with something that's cheap like v1's. What do I have to make to make V2's work (i'd throw some 2660's v2)?


BTW I loved the home made CPU blade to make these babies easily rackable. Does someone know if there is some sort of CAD file ?

Thanks for all the input