Why would M.2 to U.2 adapters only work on slots that are mapped to the CPU on AMD's AM5 platform?

[solomonshv]

hello,

i recently "upgraded" my unraid box from 11th gen intel to AMD AM5 (wanted more cores for docker). my SSD cache drives are enterprise U.2 drives that I connected to the M.2 slots using some gigabyte M.2 to U.2 adapters. on intel platforms these work in every M.2 slot, regardless of whether it's mapped to the CPU, the chipset or split with PEX switch. on AM5 platforms, these adapters only work when plugged into M.2 slots that connect directly to the CPU (see attachment so you know what i mean). i tried gigabyte, asus and MSI motherboards. 8 different models in total. same result every time.

the drives won't show up in the BIOS when connected to a slot going to the chipset

i also have a PCI-E to U.2 adapter, and same thing happens. works in PCI-E slots mapped to the CPU, will not work in PCI-E slots mapped to chipset. and again, on intel it worked when plugged in anywhere. even if the slot only had 1 gen 2 PCI-E lane, it still worked.

i can't figure out a reason for it, or how i'd go about fixing it. if anyone can clue me in, i'd appreciate it.

thanks

 

[nabsltd]

Check your motherboard docs to see if the M.2 sockets and PCIe slots share PCIe lanes. It's possible that by plugging something into a PCIe slot, you disable the M.2 socket. Or two PCIe slots could share lanes, and you have something plugged into another slot taking all the lanes.

Generally, you test these sorts of things by removing every card but those absolutely required to boot, and you can remove every card but video if you have a USB boot environment with the correct testing tools.

 

[Tech Junky]

https://pg.asrock.com/mb/AMD/X670E%20PG%20Lightning/index.asp#Specification

I have an M2 card adapter on the 64gbps off the chipset and it works at full speed.
https://www.amazon.com/dp/B09HTPWLXM

 

[Magnet]

I have an Asus x670E-I with 7800X3D and 96GB DDR5 and can confirm cheap AliExpress m.2 to u.2 adapter works on the m.2 that is provided via chipset. Drive just shows up like normal.

 

[Tech Junky]

A couple of things come to mind....

Chipset based lanes are bottlenecked to X4 total
Using a B650 might be an issue with less lanes compared to the X670/E
Lower end CPU could also give less lanes

Some more info would be helpful in figuring out the issue @solomonshv

A complete overview of the system / cards would help in solving the issue.

 

[solomonshv]

https://pg.asrock.com/mb/AMD/X670E%20PG%20Lightning/index.asp#Specification

I have an M2 card adapter on the 64gbps off the chipset and it works at full speed.
https://www.amazon.com/dp/B09HTPWLXM

I have an Asus x670E-I with 7800X3D and 96GB DDR5 and can confirm cheap AliExpress m.2 to u.2 adapter works on the m.2 that is provided via chipset. Drive just shows up like normal.

well, poop. maybe i'm not packing enough horse shoes. which U.2 drives do you guys use?

A couple of things come to mind....

Chipset based lanes are bottlenecked to X4 total
Using a B650 might be an issue with less lanes compared to the X670/E
Lower end CPU could also give less lanes

Some more info would be helpful in figuring out the issue @solomonshv

A complete overview of the system / cards would help in solving the issue.

all the motherboards i used were X670E chipset motherboards. the board currently in the system is the Asus ProArt X670E-CREATOR. i tried multiple proart boards. i have 3 of them right now. but i also tried an Asus Strix X670E-E Gaming. MSI X670E Gaming Plus, one gigabyte board and one asrock board. forgot the model numbers of the later two.

i'm only putting 1 SSD on every 4x lane/M.2 slot. and even if the bandwidth was throttled down to say 2x, it's not a problem because the SSDs aren't that fast anyway.

system specs:
Ryzen 7900X (also tried the 7950X3D from my editing rig, just to make sure the CPU is not a dud)
Gkill 2x32GB 6000Mhz, CL30 memory
Sandisk Skyhawk 3.84TB U.2 SSDs (P/N: SDLC2LLR-038T-3NA1)
gigabyte U.2 adapters

i don't know what else would be relevant

Check your motherboard docs to see if the M.2 sockets and PCIe slots share PCIe lanes. It's possible that by plugging something into a PCIe slot, you disable the M.2 socket. Or two PCIe slots could share lanes, and you have something plugged into another slot taking all the lanes.

Generally, you test these sorts of things by removing every card but those absolutely required to boot, and you can remove every card but video if you have a USB boot environment with the correct testing tools.

that's not an issue here. i'm not using the M.2 slot that shares lanes with the PCI-E slot. i'm currently only running 1. and even if i was, i don't think that should be a problem as the M.2 slot would just slow down to 2x rather than 4x. but it should still work

i tried all the typical troubleshooting still 800 million times. the U.2 won't show up in the BIOS even if it's the only thing in the computer when plugged into a chipset slot.

 

[alaricljs]

You keep talking about the BIOS... what OS are you using? Are you certain that it doesn't see the drives? I've had systems where the BIOS doesn't provide info about connected drives because it's just too stupid to look. The OS still finds them and they work fine, you just can't make them boot devices.

 

[Magnet]

I was testing an Optane 905P.

 

[solomonshv]

You keep talking about the BIOS... what OS are you using? Are you certain that it doesn't see the drives? I've had systems where the BIOS doesn't provide info about connected drives because it's just too stupid to look. The OS still finds them and they work fine, you just can't make them boot devices.

in the very first sentence of the post i mention that this was an unraid computer. it's running unraid, which is Linux. and no, they don't show up in unraid either when plugged into slots mapped to the chipset

 

[Tech Junky]

Here's my current setup.... As for drives I'm running a Kioxia w/ the adapter I mentioned before and a 50cm cable. When I first started playing with these drives though I had an issue with longer cables not getting a signal and thus no drive showing up.

sudo inxi -F

System:
  Host: server Kernel: 6.8.0-060800-generic arch: x86_64 bits: 32 Console: pty pts/9 Distro: Ubuntu
    23.10 (Mantic Minotaur)
Machine:
  Type: Desktop Mobo: ASRock model: X670E PG Lightning serial:  UEFI: American
    Megatrends LLC. v: 2.00 date: 11/06/2023
CPU:
  Info: 12-core model: AMD Ryzen 9 7900X bits: 64 type: MT MCP cache: L2: 12 MiB
  Speed (MHz): avg: 1718 min/max: 400/5733 cores: 1: 400 2: 5436 3: 400 4: 400 5: 400 6: 400
    7: 400 8: 4718 9: 3109 10: 4479 11: 5439 12: 5440 13: 400 14: 400 15: 3106 16: 400 17: 400
    18: 400 19: 3107 20: 400 21: 400 22: 400 23: 400 24: 400
Graphics:
  Device-1: Intel DG2 [Arc A380] driver: i915 v: kernel
  Device-2: Conexant Systems CX23887/8 PCIe Broadcast Audio and Video Decoder with 3D Comb
    driver: cx23885 v: 0.0.4
  Device-3: Conexant Systems CX23887/8 PCIe Broadcast Audio and Video Decoder with 3D Comb
    driver: cx23885 v: 0.0.4
  Device-4: AMD Raphael driver: amdgpu v: kernel
  Display: x11 server: X.org v: 1.21.1.7 with: Xwayland v: 23.1.1 driver: X:
    loaded: amdgpu,modesetting unloaded: fbdev,vesa dri: iris,radeonsi gpu: i915 tty: 210x78
    resolution: 3840x2160
  API: OpenGL Message: GL data unavailable in console and glxinfo missing.
Audio:
  Device-1: Intel DG2 Audio driver: snd_hda_intel
  Device-2: Conexant Systems CX23887/8 PCIe Broadcast Audio and Video Decoder with 3D Comb
    driver: cx23885
  Device-3: Conexant Systems CX23887/8 PCIe Broadcast Audio and Video Decoder with 3D Comb
    driver: cx23885
  Device-4: AMD Rembrandt Radeon High Definition Audio driver: snd_hda_intel
  Device-5: AMD Family 17h/19h HD Audio driver: snd_hda_intel
  API: ALSA v: k6.8.0-060800-generic status: kernel-api
Network:
  Device-1: Aquantia AQC111 NBase-T/IEEE 802.3bz Ethernet [AQtion] driver: atlantic
  IF: eth1 state: down mac: 24:5e:be:4d:c4:53
  Device-2: Aquantia AQC111 NBase-T/IEEE 802.3bz Ethernet [AQtion] driver: atlantic
  IF: eth2 state: down mac: 24:5e:be:4d:c4:54
  Device-3: Aquantia AQC111 NBase-T/IEEE 802.3bz Ethernet [AQtion] driver: atlantic
  IF: eth3 state: down mac: 24:5e:be:4d:c4:52
  Device-4: Aquantia AQC111 NBase-T/IEEE 802.3bz Ethernet [AQtion] driver: atlantic
  IF: eth4 state: down mac: 24:5e:be:4d:c4:51
  Device-5: Realtek RTL8125 2.5GbE driver: r8169
  IF: eth0 state: up speed: 2500 Mbps duplex: full mac: 9c:6b:00:14:04:34
  Device-6: Qualcomm driver: ath12k_pci
  IF: wlan0 state: down mac: bc:f4:d4:a1:fc:15
  IF-ID-1: br0 state: up speed: 2500 Mbps duplex: unknown mac: b2:07:ae:14:97:7e
  IF-ID-2: nordlynx state: unknown speed: N/A duplex: N/A mac: N/A
  IF-ID-3: wwan0 state: unknown mac: N/A
Drives:
  Local Storage: total: 15.79 TiB used: 2.29 TiB (14.5%)
  ID-1: /dev/nvme0n1 vendor: KIOXIA model: KCD8XRUG15T3 size: 13.97 TiB
  ID-2: /dev/nvme1n1 vendor: Western Digital model: WD BLACK SN770 1TB size: 931.51 GiB
  ID-3: /dev/nvme2n1 vendor: Western Digital model: WDS100T1X0E-00AFY0 size: 931.51 GiB
Partition:
  ID-1: / size: 915.76 GiB used: 29.87 GiB (3.3%) fs: ext4 dev: /dev/nvme2n1p2
  ID-2: /boot/efi size: 54.1 MiB used: 6.1 MiB (11.2%) fs: vfat dev: /dev/nvme2n1p1
Swap:
  Alert: No swap data was found.
Sensors:
  System Temperatures: cpu: 52.5 C mobo: N/A gpu: amdgpu temp: 46.0 C
  Fan Speeds (RPM): N/A
Info:
  Processes: 550 Uptime: 1d 58m Memory: total: 32 GiB note: est. available: 30.47 GiB
  used: 4.31 GiB (14.1%) Init: systemd target: graphical (5) Shell: Sudo inxi: 3.3.28