One bad PCIe slot on Dell R740xd? (riser 2A, slot 6)

[mostlycorn]

Anyone ever experience a bad PCIe slot?

I've got a Dell R740xd. I can't seem to get any cards in slot 6 on riser 2A (LP, the bottom most of the three on this riser) to be recognized. By iDRAC or by OS.

I've tried a few different cards. I've removed and reseated the riser itself multiple times. No signs of damage or debris.

iDRAC recognizes that I've got the machine in an 8 slot configuration because it lists slot 6, but it always shows it as empty.

I have also verified that the cards I'm testing with work fine in other slots (and NVME AIC and a SAS HBA).

Since risers are cheap, I've got one on the way, but wanted to see if anyone has had this happen to them.

Thanks!

 

[sam55todd]

Just a random thought based on my experience with 4th Gen Scalable CPUs (your server supports 1st/2nd gen): as alternative hypothesis vs defect with that server - some lower-grade Intel Scalable CPUs can have fewer PCIe lanes.

added/p.s. but based on Gen1/Gen2 specifications from Intel - practically all models do support 48 PCIe lanes.
also - slot 6 on this diagram is marked for PERC or BOSS purposes, could be one of those Dell white-listing limitation tricks.

Optionally - who has installed CPUs there? Prior mounting those - it's always recommended to clean CPU contact surface with isopropyl to avoid potential issues.

 

[RolloZ170]

iDRAC recognizes that I've got the machine in an 8 slot configuration because it lists slot 6, but it always shows it as empty.

maybe slot is diabled in BIOS > Integrated Devices.

 

[mostlycorn]

Just a random thought based on my experience with 4th Gen Scalable CPUs (your server supports 1st/2nd gen): as alternative hypothesis vs defect with that server - some lower-grade Intel Scalable CPUs can have fewer PCIe lanes.

added/p.s. but based on Gen1/Gen2 specifications from Intel - practically all models do support 48 PCIe lanes.
also - slot 6 on this diagram is marked for PERC or BOSS purposes, could be one of those Dell white-listing limitation tricks.

Optionally - who has installed CPUs there? Prior mounting those - it's always recommended to clean CPU contact surface with isopropyl to avoid potential issues.

Thanks! Yeah, these CPUs are are 8280s (platinum, 48 lanes each).

Good idea about who installed the CPUs. It was me I checked them both again this morning. Pulled both CPUs, examined for any dust or oil. Didn't see any, but gave them a light cleaning anyway with 90% isopropyl alcohol. No change. Nothing in slot 6 is recognized.

Regarding that diagram. Interesting. I'm sort of assuming the labeling of slot 6 as BOSS/PERC is more of a suggestion than a "this is the only thing that works in slot 6". From my research, I'm fairly sure my HBA or my NVME AIC should work in slot 6. That is, I'm fairly confident that Dell doesn't restricted that slot (beyond its normal physical and electrical parameters).

maybe slot is diabled in BIOS > Integrated Devices.

Good idea. I checked enablement and bifurcation. All slots enabled and bifurcation is set to "platform default".

I'm out of ideas. When the new riser board arrives, I'll see if that changes anything and post back.

Thank you both for the suggestions!

 

[mostlycorn]

Huh. I installed the replacement riser 2A and got the same result. As far as I can tell, the riser socket on the motherboard looks fine.

I also tried out a few more cards to make sure. Not even my Dell BOSS card is detected. Very odd.

I guess my next step is to swap the CPUs and see if that changes anything.

 

[nexox]

Did you inspect the CPU socket pins, and did you torque it down properly? I have seen many users on here with missing or flaky PCIe or RAM slots from incorrect (mostly too loose) installation of the CPU/cooler on LGA3647.

 

[mostlycorn]

Did you inspect the CPU socket pins, and did you torque it down properly? I have seen many users on here with missing or flaky PCIe or RAM slots from incorrect (mostly too loose) installation of the CPU/cooler on LGA3647.

Thanks, a good idea to check that! Yes, before I bought the replacement riser card, I inspected the CPU socket pins and the bottom of each CPU. I didn't see any bent pins, debris, or fingerprints/smudges. I believe I have torqued it down properly. Two bolts for each socket. One turn to each, alternating until they "stop". I didn't use a torque wrench so I don't know how much torque I actually applied.

I just now swapped CPU1 and CPU2, and reinspected the sockets and CPU surfaces. All seemed cleaned and in good shape. Still no slot 6 The cards in the other slots (all seven of them) continue to work fine.

 

[mostlycorn]

So the slot isn't completely dead! I noticed in iDRAC's inventory that some cards are visible (to BIOS/iDRAC). For example, an x8 Samsung NVME AIC is identified as an NVME, but its size shows as 0. Interestingly, iDRAC says it only links up at x1. Given this new information, I pulled CPU-1 again and took some photos.

This one area of the socket looks suspect. Looks like two or possibly three bent pins to me, but I'd appreciate a second opinion. Think this might be the cause of slot 6 not functioning normally?

Worth trying to bend them back into alignment?

 

[nexox]

I have never tried to fix that kind of issue with a socket, but I would probably try. You can look up the pinout to see if those are related to the PCIe slot or not.

 

[nexox]

Also there are marks/scratches on the plastic socket bed next to the row of pins to the left of those, I wonder if someone has already repaired worse damage.

 

[mostlycorn]

Good eye! I didn't even notice that.

I've been looking for a reference that would help me understand if these pins are related to pcie or not, but I haven't yet found anything reliable. The LLMs seem to think they are, but without a primary source I don't trust em.

 

[nexox]

I believe Intel publishes the pinout and much more details in probably a 900 page PDF that I can't find at the moment, because web search is broken now.

 

[RolloZ170]

Worth trying to bend them back into alignment?

db1 = PE1_TX_DP[1]
cy1 = vss
cv1 = vss

 

[mostlycorn]

db1 = PE1_TX_DP[1]
cy1 = vss
cv1 = vss
View attachment 50109

You rock!

Searching for "PE1_TX_DP", led me to
Second Generation Intel® Xeon® Scalable Processors Datasheet, Volume One: Electrical.

Seems like PE1_TX_DP is something like... PCI Express Controller, Interface block 1, Transmit (output from the CPU perspective), differential signal, positive polarity.

So yeah, seems likely to be a cause of my slot 6 problem.

 

[ketiljo]

They are quite easy to bend back but make sure you see what you're doing! Use a loupe and some fine tip tweezers. And don't move them to much they will break easily.