LSI 9300-8i only works in PCH PCI-E slot

[alphasierra]

Specs
Motherboard: Supermicro X11SPL-F
CPU: Xeon Gold 6128 Processor
Card: LSI 9300-8i
Firmware Version: 16.00.10.00
BIOS Version: 08.37.00.00
UEFI BSD Version 18.00.00.00

I'm having a weird issue where my HBA only works in the PCH PCI-E slot (Slot 1). It doesn't appear under any of the CPU PCI-E slots. The slots are all set to have EFI OPROM. I updated the firmware on the card using sas3flash running the command:

sudo sas3flash -o -f SAS9300-8i_IT.bin

I'm not sure what else to do at this point. What else should I try to make this card work in the CPU slots?

Thanks!

EDIT: I tried manually setting all the PCI-E slots to 8x8 bifurcation, and put the card in Slot 5. After a long time got back into the BIOS, the status says the link did not train.

 

[Moopere]

Mmm I don't know the board nor the card. Have had problems with 92xx cards which are PCIe v2 in v3 slots though. In these cases if I force PCIe to be gen2 then the card is always detected and works fine ... If you're seeing training errors perhaps change the PCIe 'auto' setting to a setting that matches the cards PCIe gen?

 

[alphasierra]

An update to this. I was told by an experienced person who is retired from the IT space that many HBAs by design will only work when connected to the chipset, and not function in CPU connected PCI-E slots. They didn't have an explanation for this, but apparently it this was a common issue they observed across many years with different systems. Not a great end to the story, but I'm out of ideas.

 

[Moopere]

I've heard this a lot .. its almost an urban legend, that some PCIe slots are designed to only be GPU slots ... despite hearing this for many years I've been exposed to I don't know how many hundreds of machines and have never yet bumped into one of these unicorns.

What I _have_ seen though are some motherboards that won't respect a PCIe cards OPROM - usually manifesting as simply ignoring the ROM altogether which can mean an inoperable device depending on the card. On boards like this the OPROM is ignored no matter where you put the card, the first or Nth PCIe slot ... none work.

I'm not sure if most HBA's would even be smart enough or care which slot they are in. I can't imagine why a HBA manufacturer would specifically code in a check/fail for this. IME its the motherboards fault, the manufacturer either too lazy or cheap to use a BIOS that supports OPROMs or trying to protect their server hardware sales by not allowing desktop boards to be used in a server type way.

Dell Optiplex desktop PC's are particularly prone to not respecting OPROM on PCIe cards IME.

 

[alphasierra]

I've heard this a lot .. its almost an urban legend, that some PCIe slots are designed to only be GPU slots ... despite hearing this for many years I've been exposed to I don't know how many hundreds of machines and have never yet bumped into one of these unicorns.

What I _have_ seen though are some motherboards that won't respect a PCIe cards OPROM - usually manifesting as simply ignoring the ROM altogether which can mean an inoperable device depending on the card. On boards like this the OPROM is ignored no matter where you put the card, the first or Nth PCIe slot ... none work.

I'm not sure if most HBA's would even be smart enough or care which slot they are in. I can't imagine why a HBA manufacturer would specifically code in a check/fail for this. IME its the motherboards fault, the manufacturer either too lazy or cheap to use a BIOS that supports OPROMs or trying to protect their server hardware sales by not allowing desktop boards to be used in a server type way.

Dell Optiplex desktop PC's are particularly prone to not respecting OPROM on PCIe cards IME.

I agree it doesn't make sense the card would be designed like that, it is probably something to do with the motherboard that's a widespread issue.

 

[Whaaat]

Check the physical integrity of the pcie lines on the LSI pcb. I've had similar issue with 430-8i card. It has tiniest SMD elements onboard and very thin traces. One of the traces got accidentally damaged and the card could only be detected in a 4x slot. Being installed in a x8 slot it never showed up in the system.

 

[Moopere]

I agree it doesn't make sense the card would be designed like that, it is probably something to do with the motherboard that's a widespread issue.

Its not widespread IME - I've never seen a card not able to be seen in an otherwise working PCIe slot. What I'm talking about is lack of OPROM support. This manifests as a HBA's inability to boot from a card that requires its own OPROM in order to boot. If you boot your system using other methods the card can still be seen in the PCIe slot.

I've never (yet) seen a system that is PCIe slot dependent, though such cases a widely reported anecdotally.

Does _anything_ work in the CPU pcie slot?

 

[alphasierra]

Check the physical integrity of the pcie lines on the LSI pcb. I've had similar issue with 430-8i card. It has tiniest SMD elements onboard and very thin traces. One of the traces got accidentally damaged and the card could only be detected in a 4x slot. Being installed in a x8 slot it never showed up in the system.

The PCB seems fine.

Its not widespread IME - I've never seen a card not able to be seen in an otherwise working PCIe slot. What I'm talking about is lack of OPROM support. This manifests as a HBA's inability to boot from a card that requires its own OPROM in order to boot. If you boot your system using other methods the card can still be seen in the PCIe slot.

I've never (yet) seen a system that is PCIe slot dependent, though such cases a widely reported anecdotally.

Does _anything_ work in the CPU pcie slot?

Yes, I tested a NIC I had laying around and it works in all the slots.

 

[Moopere]

I said this in a post above:

> I've never seen a card not able to be seen in an otherwise working PCIe slot.

But thats not right. As I said earlier in the thread I've seen this behaviour with PCIe v2 cards in PCIe v3 slots. The CPU slot is likely the highest revision PCIe with the southbridge very often the previous generation - are you sure its not this? Does the motherboard allow you to cap the PCIe generation via a BIOS setting?

 

[alphasierra]

I said this in a post above:

> I've never seen a card not able to be seen in an otherwise working PCIe slot.

But thats not right. As I said earlier in the thread I've seen this behaviour with PCIe v2 cards in PCIe v3 slots. The CPU slot is likely the highest revision PCIe with the southbridge very often the previous generation - are you sure its not this? Does the motherboard allow you to cap the PCIe generation via a BIOS setting?

Yes, and I did try locking the slots at PCIe 3.0. I guess I could give it a try at 2.0.

 

[alphasierra]

I said this in a post above:

> I've never seen a card not able to be seen in an otherwise working PCIe slot.

But thats not right. As I said earlier in the thread I've seen this behaviour with PCIe v2 cards in PCIe v3 slots. The CPU slot is likely the highest revision PCIe with the southbridge very often the previous generation - are you sure its not this? Does the motherboard allow you to cap the PCIe generation via a BIOS setting?

I was finally able to give this a try. Capping at PCIE gen 2 did not work.