Potential Deal: 2 x Dual 2011 nodes @$199, Quanta Openrack

[iguy]

The ASUS one you posted is some serious stuff!! But the idea is correct, and the nodes you posted are the right ones as well. I have nodes A07, B08, and B10( sticker by the 1gb ethernet )

This is the one I use: New M.2 NVMe SSD NGFF TO PCIE 3.0 X16 Adapter M Key Interface Card 6941701009999 | eBay

You don't NEED the SPI programmer, but having one around would ensure success. This is the one I have: 1PCS USB Programmer CH341A Series Burner Chip 24 EEPROM BIOS Writer 25 SPI Flash 806998424473 | eBay
Is cheap enough to have one just in case something happens and you need to flash the chip itself without relying on the motherboard.

Let me know if you need any instructions.

Thanks

 

[kfriis]

Thanks for all your information and for the BIOS mod files.

I understand the desire for the SPI as a backup. I am willing to take a chance and do the flash using just the motherboard. Is this done just as you would for a consumer board using some flash SW? If so, which SW would I use? I have flashed motherboards before, but not a server and I also do not know which software to use.

Thanks for all your help.

 

[ArcyneTheFirst]

Is there an alternate hosting site for the newest firmware? I don't have a Filedropper account and the side seems to require sign-up to download the B10/B11 firmware. Note, I have the Quanta board.

 

[kfriis]

@ArcyneTheFirst - it works for me. Maybe try another browser? (I am using Firefox).

 

[Mehmet]

View attachment 10630
The ASUS one you posted is some serious stuff!! But the idea is correct, and the nodes you posted are the right ones as well. I have nodes A07, B08, and B10( sticker by the 1gb ethernet )
Thanks

Asus one needs pcie bifurcation which requires motherboard support.

Does A07 bios support m.2 cards or do I need to upgrade B10 or B11 firmware?

Is there anybody who uses intel p3700 pci version or something like it?



Thank you

 

[hmartin]

Can someone with a dual node chassis tell me the weight?

I'll be in the US soon for an industry conference. Since shipping to the EU is quite dear (and most sellers refuse) it would be nice to throw one in the checked luggage. But I've only got 20kg, and while the dimensions of the chassis are available, I haven't found the weight published anywhere.

I'm looking at the CTO offer: 2 nodes with heatsinks, backplane, and PSU.

 

[Syndroma]

16-18kg

 

[kfriis]

... and you could also just "split up" the server by taking out the boards from the chassis. That will give you flexibility to use several pieces of luggage.

 

[kfriis]

@iguy

Thanks for the modded BIOS files. I was able to flash your modded B10 file w/o problems. It did not like the B11 due to size mismatch as you mentioned.

After flashing to the modded B10 BIOS, I booted up both Windows Server 2019 AND VMWare ESXi 6.7 U1 (latest version). Neither Windows nor VMWare is showing anything remotely related to NVMe anywhere.

Are there any other steps other than flashing with the modded BIOS file I should do? At this point, I don't even care about booting the NVMe drive - I just want to be able to use it as a storage device.

If it makes a difference, I have the Intel 660p 2TB NVMe drive. It is mounted in a generic eBay MVMe-to-PCIe adapter.

Thanks!

 

[iguy]

Form I understand you have 2 nodes, one with Windows and the other with ESXi, and you tried booting/viewing from OS, the drive(same drive?) from both nodes and nothing would show up in BIOS or the OS.

Also, Did you check this(from the previous post):
To get BOOT working just disable everything else, set it UEFI and it'll boot. To verify it's recognizing the storage look in the bios boot menu, you'll see PATA3 in BSS menu or something like that listed. Don't boot from it, instead, look for [Windows Boot Manager](in main bios boot window) if using windows.(after having it installed)...

Questions:

Have you tried booting/checking the NVMe drive using another machine, capable of the technology from the factory? (Trying to rule out the drive itself)

What's your UEFI boot settings like?

BONUS =]
#### Tools ####
To help with the task I have included a tool capable of that in the shared file below. Look into SCEWIN(AMISCE) folder, read the "Command to Run.txt" file. Rename "nvram.txt" to something else prior to running the commands found in the text file if you want to compare your settings with mine.(I use winmerge to compare text files, feel free to use anything you feel comfortable with)

Be careful if you plan to re-import the modified dump from your system, changing BIOS settings might crash your system during next boot. If you have a dump will make it easier to replicate the settings after clearing the BIOS though.

As you guessed, most of these tools are not for the unadvised-everyday-bios-configuration-tasks, they are intended for mass-deployments, custom manufacture modifications, etc... use with caution. Let me know if you have questions.

Some of the utilities found in the shared zip are described here:
https://ami.com/ami_downloads/Aptio_Utilities_Data_Sheet.pdf

and here:
Server Platform Services SPS Manageability Engine ME Firmware & Tools

The ZIP file also contains previously shared BIOS images and tools needed to flash them aside form some other interesting stuff.

Have fun:
Gofile - File sharing platform, anonymous and free

 

[iguy]

@Mehmet It doesn't support tf from out of the box, no. However, with the modified BIOS you will be able to do it since the NVMe driver is embedded in the BIOS.

 

[iguy]

@ArcyneTheFirst Re-posted the BIOS rom files. Look inside shared file few posts above.

 

[iguy]

@kfriis - Did you install these? Download Client NVMe* Microsoft Windows* Drivers for Intel® SSDs

 

[kfriis]

@iguy

Thanks again for your helpful and detailed response. I have now worked through you suggestions – unfortunately it did not work.
But let me explain and clarify further what I have done and also answer the questions you posed.

======

First, as you suggested, I needed to verify that I did not have defective hardware. This was an excellent suggestion that I had somewhat ignored because doing so involved considerable work. In short, here is what I did:

I booted up a newer MSI consumer motherboard with an Intel i5-7600K CPU. This motherboard has a built-in MVMe slot, so I know that it supports NVMe drives. I then installed the NVMe-PCIe adapter card with the Intel 660p drive in an expansion slot and booted into Windows 10 Pro. I did not touch the BIOS – I just booted straight into W10. Short story - W10 immediately recognized the Intel 660p drive as verified in both Device Manager (drive shows up under Disk drives and NVMe controller under Storage controllers) and in Disk Management (for good measure I initialized the disk here with a GPT partition table).

I did not even have to install the Intel NVMe drivers as you suggested in your last post. The drive was immediately recognized by W10 using the built-in Microsoft NVMe driver. For good measure, I also installed the Intel driver which just replaced the Microsoft storage controller with the Intel one. In both cases, the drive worked fine.

Now I knew the hardware worked.

=====

Back to the Quanta Winterfell node.

Next, I wanted to make sure I did not have a software/OS issue. That is why, as I explained in a prior post here, I tried several different operating systems on the Quanta node.

To recap and further clarify:

With your modded B10 BIOS successfully flashed, I have successfully booted and run the following operating systems:

1) Windows Server 2019 (share code-base with recent version of Windows 10)

This OS runs great on the Quanta node and installs w/o any issues. However, it does not recognize the NVMe drive OR controller in Device Manager or Disk Management.


2) VMWare VSphere ESXi v. 6.7 U1 – build 11675023 (latest version)

ESXi also runs great on the Quanta node and fully supports NVMe. This is my daily driver and I have run this OS for some time without any problems. ESXi also fails to recognize the NVMe drive even though the other controllers like SATA and SAS are fully recognized.


3) Intel Clear Linux

This is Intel’s version of Linux which is optimized to run on Intel hardware. As expected, this OS also runs great on the Quanta node since the motherboard is based on the Intel C602 chipset.

I have also been running this OS for a while with Linux kernels 4.19.x, 4.20.x and the latest available 5.03.x kernel. As a side note, this is my recommended Linux distro for the Quanta Winterfell nodes. Everything works great and it is a very speedy Linux distro.

As you might have guessed, this OS also fails to recognize the NVMe drive. Among other things, no NVMe devices show up in /dev, nothing about nvme when running lspci, lsmod or lsblk.

I believe that running three operating systems as different as the above, proves that there are no software issues that would prevent the NVMe drive being recognized in the Quanta node. Even if one of the above OS would fail to recognize the NVMe for some reason, it is unlikely that they all three would.

=====

I have now ruled out anything wrong with the NVMe hardware or the software configuration.

As I see it, the only thing left is the combination of this particular Intel 660p NVMe drive running in the Quanta node. My node is the same hardware version as yours – it has the A07 sticker next to the Ethernet port.

I have also ruled out a problem with the PCIe riser card as I have been using this riser for a while with an AMD HD 3450 graphics card. I have also tried booting the Quanta with or without the graphics card just to see if it would make a difference.

I have verified the BIOS boot settings as you suggested. I disabled all the boot devices and just enabled UEFI boot. This works as expected as I can press F11 during POST and select the boot device. To boot windows, I select the Windows Boot Loarder. To boot Intel Clear Linux, I select the internal (UEFI) SATA harddrive directly (Clear Linux and Windows Server are installed on two different primary partitions on the SATA drive). To boot VMWare ESXi, I select the USB port that has a USB stick attached where ESXi is installed. Everything works great that way and enables me to boot three different operating systems on the same node.

I have also looked at all the settings in the BIOS. I do not claim to understand all of them, but I have looked numerous times (more than I care to admit) for anything that would indicate the NVMe drive being recognized in the PCIe slot. In fact, I do not see a difference in any of the BIOS settings whether the NVMe card/drive is mounted in the PCIe slot or not.

Specifically, I do not see, anywhere in the BIOS, the PATA3 boot device option, as you mentioned in your original response. I have the exact same boot device options whether the NVMe card/drive is mounted in the PCIe slot or not. If seeing PATA3 is supposed to be the “proof” that the NVMe drive is recognized by the BIOS, I can say that it isn’t.

Nothing in my BIOS settings indicate that the NVMe drive is recognized. I am not sure if any settings in the BIOS needs to be changed for the BIOS itself to recognize the drive. In my experience with other motherboards, the BIOS has always recognized the attached hardware directly, but sometimes you need to change BIOS settings for the OS to recognize the hardware.

I should also mention again, that I do not even care if I can boot from the NVMe drive or not. I just want the Quanta to recognize it as any other storage device, as I plan to boot from either the SATA drive or USB stick as I have been doing so far. I have noticed many people having problem booting from NVMe drives on older hardware, but this is not what I am trying to do.

Any suggestions on how to proceed now? I believe I am now at a point where the problem has been narrowed down specifically to be centered around the BIOS itself recognizing the Intel 660p drive.

Thanks for all your help!

 

[iguy]

@iguy

Thanks again for your helpful and detailed response. I have now worked through you suggestions – unfortunately it did not work.
But let me explain and clarify further what I have done and also answer the questions you posed.

======

First, as you suggested, I needed to verify that I did not have defective hardware. This was an excellent suggestion that I had somewhat ignored because doing so involved considerable work. In short, here is what I did:

I booted up a newer MSI consumer motherboard with an Intel i5-7600K CPU. This motherboard has a built-in MVMe slot, so I know that it supports NVMe drives. I then installed the NVMe-PCIe adapter card with the Intel 660p drive in an expansion slot and booted into Windows 10 Pro. I did not touch the BIOS – I just booted straight into W10. Short story - W10 immediately recognized the Intel 660p drive as verified in both Device Manager (drive shows up under Disk drives and NVMe controller under Storage controllers) and in Disk Management (for good measure I initialized the disk here with a GPT partition table).

I did not even have to install the Intel NVMe drivers as you suggested in your last post. The drive was immediately recognized by W10 using the built-in Microsoft NVMe driver. For good measure, I also installed the Intel driver which just replaced the Microsoft storage controller with the Intel one. In both cases, the drive worked fine.

Now I knew the hardware worked.

=====

Back to the Quanta Winterfell node.

Next, I wanted to make sure I did not have a software/OS issue. That is why, as I explained in a prior post here, I tried several different operating systems on the Quanta node.

To recap and further clarify:

With your modded B10 BIOS successfully flashed, I have successfully booted and run the following operating systems:

1) Windows Server 2019 (share code-base with recent version of Windows 10)

This OS runs great on the Quanta node and installs w/o any issues. However, it does not recognize the NVMe drive OR controller in Device Manager or Disk Management.


2) VMWare VSphere ESXi v. 6.7 U1 – build 11675023 (latest version)

ESXi also runs great on the Quanta node and fully supports NVMe. This is my daily driver and I have run this OS for some time without any problems. ESXi also fails to recognize the NVMe drive even though the other controllers like SATA and SAS are fully recognized.


3) Intel Clear Linux

This is Intel’s version of Linux which is optimized to run on Intel hardware. As expected, this OS also runs great on the Quanta node since the motherboard is based on the Intel C602 chipset.

I have also been running this OS for a while with Linux kernels 4.19.x, 4.20.x and the latest available 5.03.x kernel. As a side note, this is my recommended Linux distro for the Quanta Winterfell nodes. Everything works great and it is a very speedy Linux distro.

As you might have guessed, this OS also fails to recognize the NVMe drive. Among other things, no NVMe devices show up in /dev, nothing about nvme when running lspci, lsmod or lsblk.

I believe that running three operating systems as different as the above, proves that there are no software issues that would prevent the NVMe drive being recognized in the Quanta node. Even if one of the above OS would fail to recognize the NVMe for some reason, it is unlikely that they all three would.

=====

I have now ruled out anything wrong with the NVMe hardware or the software configuration.

As I see it, the only thing left is the combination of this particular Intel 660p NVMe drive running in the Quanta node. My node is the same hardware version as yours – it has the A07 sticker next to the Ethernet port.

I have also ruled out a problem with the PCIe riser card as I have been using this riser for a while with an AMD HD 3450 graphics card. I have also tried booting the Quanta with or without the graphics card just to see if it would make a difference.

I have verified the BIOS boot settings as you suggested. I disabled all the boot devices and just enabled UEFI boot. This works as expected as I can press F11 during POST and select the boot device. To boot windows, I select the Windows Boot Loarder. To boot Intel Clear Linux, I select the internal (UEFI) SATA harddrive directly (Clear Linux and Windows Server are installed on two different primary partitions on the SATA drive). To boot VMWare ESXi, I select the USB port that has a USB stick attached where ESXi is installed. Everything works great that way and enables me to boot three different operating systems on the same node.

I have also looked at all the settings in the BIOS. I do not claim to understand all of them, but I have looked numerous times (more than I care to admit) for anything that would indicate the NVMe drive being recognized in the PCIe slot. In fact, I do not see a difference in any of the BIOS settings whether the NVMe card/drive is mounted in the PCIe slot or not.

Specifically, I do not see, anywhere in the BIOS, the PATA3 boot device option, as you mentioned in your original response. I have the exact same boot device options whether the NVMe card/drive is mounted in the PCIe slot or not. If seeing PATA3 is supposed to be the “proof” that the NVMe drive is recognized by the BIOS, I can say that it isn’t.

Nothing in my BIOS settings indicate that the NVMe drive is recognized. I am not sure if any settings in the BIOS needs to be changed for the BIOS itself to recognize the drive. In my experience with other motherboards, the BIOS has always recognized the attached hardware directly, but sometimes you need to change BIOS settings for the OS to recognize the hardware.

I should also mention again, that I do not even care if I can boot from the NVMe drive or not. I just want the Quanta to recognize it as any other storage device, as I plan to boot from either the SATA drive or USB stick as I have been doing so far. I have noticed many people having problem booting from NVMe drives on older hardware, but this is not what I am trying to do.

Any suggestions on how to proceed now? I believe I am now at a point where the problem has been narrowed down specifically to be centered around the BIOS itself recognizing the Intel 660p drive.

Thanks for all your help!

Hi there, great steps you took troubleshooting this. Thanks for being clear with the write up as well.

Ok, now I see where you stand.. I think some setting in your bios might be different than mine. I can't think of a setting that might be causing this though.. Oddly enough not even the OS is able to read the drive.

It might not matter much but what's the pinout configuration you have on your raiser?
X8 and X8 ? That would be advisable.

Please download the file I shared a few posts back. You will find a folder named SCEWIN(AMISCE tool), that's an application to dump the BIOS settings to a plain text file. Check "Command to run.txt" file.
There is a "nvram.txt" file in the folder, before dumping your configuration, rename this file so you can compare the settings later.

If you are running a 64-bit system(Win 2019 sure is)..
Open a command prompt window(cmd) as administrator, navigate to the folder..
The command is: SCEWIN_64 /o /s nvram.txt /h Hii.db /v /q
This will dump your current settings to a text file(nvram.txt)
Compare this file with the one you previously renamed.
I use winmerge( WinMerge - You will see the difference… ) to compare the files.

Please let me know what you find.
Share your config file. I'm curious to see what might be causing this.

Thanks

 

[kfriis]

@iguy

See attached BIOS dump from my Quanta node flashed with your modded B10 BIOS file.

I am also very curious to see where your settings differ from mine. Would you mind comparing my file with a file from one of your B10 BIOS nodes?

 

[iguy]

@iguy

See attached BIOS dump from my Quanta node flashed with your modded B10 BIOS file.

I am also very curious to see where your settings differ from mine. Would you mind comparing my file with a file from one of your B10 BIOS nodes?

Sorry about the delay. Busy with work and some home chores. Finally, the snow melted over here. Anyways, I've looked at your bios settings and would change these lines..

Lines:
526 =1 - Show hidden settings**
4752= *[02]Auto - PCIe lane port config***
4779= *[02]Auto - PCIe lane port config***
4889= *[00]Disabled - Disable 4G decoding ( Might be interfering with PCIe storage drive )*

*Do you need 4G decoding in order to use your graphics card ? Do you use multiple GPUs? Coin mining?
I have 2 GTX 1060's(same node) without this setting turned on and they are working great.
If you would like to read about it check this out:
What is "Above 4G decoding"?

**Will enable you to view all adjustable settings.

***You might be assigning more PCIe lanes than needed. Leaving at "Auto" might fix that.
OBS: The setting above is for a port that is not implemented physically on this node, while the BIOS/CPU/South Bridge can assign lanes to the non-existent port, it'll be waste nonetheless.

Please let me know how it goes.

Thanks

 

[Dmytro Burianov]

The bios, drivers (windows and Linux) can be downloaded for the Quanta system here. Please note this is not the exact system but is a compatible motherboard. (download the bios and read the release notes to verify if you want.) I wonder if it would work on a Wiwynn? The mobo bios chip is supposed to be replaceable...might be worth trying for someone who needs windows.

Download Center

The system is considered an F03B motherboard.

A better manual with bios settings and descriptions can be found here...

QUANTA RACKGO X SERIES F03A TECHNICAL MANUAL Pdf Download.

This manual is for a different model so although the bios looks the same the rest of the manual is partially usable. i.e. the info about IPMI and remote KVM do not apply to the windmill boards we have. Other info may or may not apply directly but is very similar.

Hope it helps.

Are work on it (F03A)MB KVM
Ipmi via ipmitool work but GUI do not work

 

[Dmytro Burianov]

I don't know if that's new or not, but IPMI for the server is working from Supermicro's IPMIView

Just scan the network for IPMI 2.0, it will auto find IPMI of your nodes.
Log in with default admin:admin credentials.
It was working to Power Up/Power Down/Reset for me. Other functions did not work, though may need additional testing.

nmap say: available only ipmi port(623/udp) and and I do not found any answer for it

 

[kfriis]

Sorry about the delay. Busy with work and some home chores. Finally, the snow melted over here. Anyways, I've looked at your bios settings and would change these lines..

Lines:
526 =1 - Show hidden settings**
4752= *[02]Auto - PCIe lane port config***
4779= *[02]Auto - PCIe lane port config***
4889= *[00]Disabled - Disable 4G decoding ( Might be interfering with PCIe storage drive )*

*Do you need 4G decoding in order to use your graphics card ? Do you use multiple GPUs? Coin mining?
I have 2 GTX 1060's(same node) without this setting turned on and they are working great.
If you would like to read about it check this out:
What is "Above 4G decoding"?

**Will enable you to view all adjustable settings.

***You might be assigning more PCIe lanes than needed. Leaving at "Auto" might fix that.
OBS: The setting above is for a port that is not implemented physically on this node, while the BIOS/CPU/South Bridge can assign lanes to the non-existent port, it'll be waste nonetheless.

Please let me know how it goes.

Thanks

@iguy

Thank you for all your help - it is very much appreciated.

I am sorry to say that none of the suggestions worked.

I changed the BIOS settings as you suggested and even some others to make my settings as close as possible to yours - including turning off above 4G decoding. No matter what, the BIOS is clearly not recognizing the NVMe drive.

And - as before - neither Windows, Linux or VMWare ESXi is recognizing the drive either. There are no PATA or similar drives in the Boot devices area of the BIOS. Nothing anywhere indicates anything related to NVMe or an extra storage device.

I know the hardware is working as I have tested the drive/PCIe adapter in a consumer motherboard - as previously mentioned.

More and more, I am thinking that the particular BIOS mod you provided is not working in my case - for some reason.

I am reaching a point of desparation where I am considering trying another BIOS mod even though it doesn't make much sense as you have yours working on the exact same board as mine.

Do you have the UNmodified B10 and B11 files? Or are they already included in the ZIP file you posted above?

I am considering following this procedure - what do you think about it? They claim it should work in ANY Intel board:
[Guide] How to get full NVMe support for all Systems with an AMI UEFI BIOS

Thanks.