I am wondering if BIOS settings on my Supermicro X11-SCA-F motherboard (BIOS 1.2), are causing problems with my AOC-SHG3-4M2P (4x NVMe M.2 PLX switch). Perhaps power saving settings?
I previously installed an AOC-SHG3-4M2P into PCIe slot 6 of my Supermicro X11-SCA-F.
I fitted 2x M.2 NVMe drives (Corsair 2TB MP510) and using ESXi 6.7U3, I passed the drives through to a VM (OmniOS/napp-it for ZFS).
The drives worked fine for a few days, even with big file transfers. But they developed problems when I pushed other resources on my system, such as concurrently running virtual desktop gaming on pass-through GPUs.
After 15mins of pushing the other resources, the OmniOS/napp-it VM would then report multiple errors on the drives, which would increase into the 100'000s and slow to a halt until I removed the drives and rebuilt the pool. There were no excessive heat issues I could see.
I repeated this issue with both drives in the AOC-SHG3-4M2P, then each drive separately.
With the NVMe drives in a standard 'PCIe to single M.2' adaptor in the same PCIe slot 6, I have had no problems (over 8 weeks of use now).
I had also tested the ram for over 24 hours without any reported problems.
After speaking with Supermicro, I then tried supplying the adaptor with power directly from the PSU as there is a socket on the adaptor, but no instruction in the manual. This - maybe coincidentally - resulted in me being able to push the system for about 6 hours without issue, but then the errors came back.
Supermicro advised me to return the AOC-SHG3-4M2P and I have since received a replacement, but not yet fitted it.
For other reasons, I also have a new X11-SCA-F motherboard, so the 2 have never met. Both previous and new motherboards work fine with the standard 'PCIe to single M.2' adaptor.
Before I just try the same tests again, I wanted to know if I should look into changing any BIOS settings?
The BIOS settings I have adjusted from default are to allow concurrent use of: GPUs, onboard iGPU and iKVM ie:
- Load Optimised Defaults
- Primary Display = PCI
- Primary PEG = Slot 4 (GPU)
- Primary PCI = Onboard
- Internal Graphics = Enabled
- Option ROM, Video = UEFI
CPU is a Xeon E2278G, 128GB memory, 1000W PSU
Would it matter if I used PCIe slot 4 or 6 on this mother board? My GPU (Quadro P4000) doesn't really fit in slot 6 due to an internal USB header.
I previously installed an AOC-SHG3-4M2P into PCIe slot 6 of my Supermicro X11-SCA-F.
I fitted 2x M.2 NVMe drives (Corsair 2TB MP510) and using ESXi 6.7U3, I passed the drives through to a VM (OmniOS/napp-it for ZFS).
The drives worked fine for a few days, even with big file transfers. But they developed problems when I pushed other resources on my system, such as concurrently running virtual desktop gaming on pass-through GPUs.
After 15mins of pushing the other resources, the OmniOS/napp-it VM would then report multiple errors on the drives, which would increase into the 100'000s and slow to a halt until I removed the drives and rebuilt the pool. There were no excessive heat issues I could see.
I repeated this issue with both drives in the AOC-SHG3-4M2P, then each drive separately.
With the NVMe drives in a standard 'PCIe to single M.2' adaptor in the same PCIe slot 6, I have had no problems (over 8 weeks of use now).
I had also tested the ram for over 24 hours without any reported problems.
After speaking with Supermicro, I then tried supplying the adaptor with power directly from the PSU as there is a socket on the adaptor, but no instruction in the manual. This - maybe coincidentally - resulted in me being able to push the system for about 6 hours without issue, but then the errors came back.
Supermicro advised me to return the AOC-SHG3-4M2P and I have since received a replacement, but not yet fitted it.
For other reasons, I also have a new X11-SCA-F motherboard, so the 2 have never met. Both previous and new motherboards work fine with the standard 'PCIe to single M.2' adaptor.
Before I just try the same tests again, I wanted to know if I should look into changing any BIOS settings?
The BIOS settings I have adjusted from default are to allow concurrent use of: GPUs, onboard iGPU and iKVM ie:
- Load Optimised Defaults
- Primary Display = PCI
- Primary PEG = Slot 4 (GPU)
- Primary PCI = Onboard
- Internal Graphics = Enabled
- Option ROM, Video = UEFI
CPU is a Xeon E2278G, 128GB memory, 1000W PSU
Would it matter if I used PCIe slot 4 or 6 on this mother board? My GPU (Quadro P4000) doesn't really fit in slot 6 due to an internal USB header.