Disappearing M.2 NVMe on Supermicro X11SSH-CTF

[Setarcos]

Hi folks,

I have run into an odd issue with a new Supermicro X11SSH-CTF based server that I have started putting together that I could use some suggestions on how to resolve. Any NVMe SSD that I connect to the M.2 connector on the motherboard is only visible (in the BIOS boot menu and after the OS loads) if I change any setting in the BIOS and then save and apply. The NVMe SSD will stay visible (across reboots, etc) until I power down the server. If I power it up again, the NVMe device will disappear.

I have a second NVMe SSD connected to a PCIe x4 adapter card (like this: https://www.amazon.com/gp/product/B01LY9HRDB) that is installed in the motherboard's x8 slot that is detected reliably on every startup.

It's only an NVMe connected to the M.2 slot that is disappearing (I have tried multiple different M.2 NVMes and U.2 NVMes connected via M.2 adapters like this https://www.amazon.com/gp/product/B073WGN61Y and they all behave the same)

I am running the latest BIOS v2.6.

The only difference I can see is that the x8 PCIe slot is directly connected to the CPU, while the M.2 slot is connected to the PCH C236:


Any thoughts?

 

[Antonio]

Please check if you have the following changes in the BIOS settings:
1. Advanced >> PCI/PCIe/PnP Configuration >> NVMe firmware source >> AMI Native support

 

[Setarcos]

Yes, this is the current setting I am using. I have tried with both the default of vendor defined and AMI native with the same results.

 

[Setarcos]

I reached out to Supermicro and unfortunately they haven't been of much help on this (only Supermicro qualified M.2 2260 NVMes are supported, yada yada). I ordered an x8 PCIe switch card and will give it a try next week when it arrives.

 

[Setarcos]

The PCIe switch card worked a treat for both of my NVMes, so I was able to work around the issue with the motherboard's M.2 slot. I reported this to Supermicro, but it is unlikely that they will pursue a fix as this use case is 'out of spec'.

 

[RolloZ170]

thought about a faulty M.2 ?
ask supermicro to swap the board.

 

[Setarcos]

Nope, the M.2 slot works fine, but I need to hit Delete during POST to open the BIOS settings, change something (anything) and then F4 to Save and Exit after powering the server up to get the NVMe attached to the M.2 slot to show up. Once I have done so, the M.2 connected NVMe remains visible across reboots and otherwise it works fine. It seems like there is a race condition during POST that is causing this, and it only happens to the NVMe connected to the M.2 slot.

 

[diogin]

My X11SRM-F has a problem of losing NVMe SSDs after reboot, for a long time.
After a lot of testing, I have found the cause of the problem: Fans.

To resolve it, I took the following steps:
First, connect at least 2 Fans, one to FAN1/2/3/4, one to FANA/B.
Second, ensure there are no fan errors found in BMC log. If you found any fan errors, the NVMe SSDs are considered to be unusable by BMC/UEFI and might not be found by UEFI.

Fans are not widely compatible under Supermicro mainboards. If fans runs slow RPM while system is idle, fan errors occur. If fan errors are found, BMC/UEFI won't detect NVMe SSDs.

To solve fan errors, use "ipmiutil" to reduce the threshold of all fans connected to the board:

$ sudo ipmiutil sensor -g fan -q
$ sudo ipmiutil sensor -n 41 -u 300:200:100:25300:25400:25500

Change -n 41 to any numbers of your board. Notice: After upgrading BMC firmware, these settings might be overwritten.

Just for your information.

 

[Setarcos]

That is interesting. I had adjusted the fan speed thresholds for my four fans with ipmiutil and no longer get any errors (BMC logs were cleared after tuning), but don't currently have a fan attached to FANA. I'll have to give that a try and see if it behaves any differently with the M.2 slot on my X11SSH-CTF.