Anyone used a U.2 SSD to PCIe X16 Adapter PCIE 4.0 to 4 Disk U.2 SFF-8639 SSD Riser Card

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
T

TomSwift

Member
May 14, 2020
37
13
8
I have 4 NVMe U.2 SSDs in one of these cards

I have 2 Kioxia CD6-R 7.68TB and 2 Kingston DC1500M 960GB (they seem to be on the newest FW), this is on a ROMED8-2T and the PCIe slot is set to 4x4x4x4

In linux I was getting a ton of errors about the NVMe drives, and I currently have one drive with Server 2022 installed on it and the server is giving a ton of this error:

A corrected hardware error has occurred.

Component: PCI Express Root Port
Error Source: Generic

Primary Bus:Device:Function: 0x80:0x1:0x2
Secondary Bus:Device:Function: 0x82:0x0:0x0
Primary Device Name:pCI\VEN_1022&DEV_1483&SUBSYS_14531022&REV_00
Secondary Device Name:pCI\VEN_1E0F&DEV_0007&SUBSYS_00211E0F&REV_01

in IPMI the error is:

PCI PERR - Asserted
Event Data1 PCI PERR
Event Data2 PCI Bus number : 128
Event Data3 PCI Device number : 1, PCI Function number : 2


I tried changing the location of the NVMe drives in the card, and no real change.

The system works and I'm currently throttling the CPU and memory mining with the server.
I would love a fool proof way to test out the drives to make sure they are good, or a better solution connecting these drives to the motherboard.
 
M

mrpasc

Well-Known Member
Jan 8, 2022
581
324
63
Munich, Germany
Try to disable all PCIE power saving options in your Bios. Had similar issues with a Xeon-D board which I solved this way. Enterprise NVME are not made for power savings and may show some weird quirks if enabled.
 
  • Like
Reactions: TomSwift
T

TomSwift

Member
May 14, 2020
37
13
8
mrpasc said:
Try to disable all PCIE power saving options in your Bios. Had similar issues with a Xeon-D board which I solved this way. Enterprise NVME are not made for power savings and may show some weird quirks if enabled.
Click to expand...
Did you do c-state, anything else like PCIe slot power saving settings?
 
J

jei

Active Member
Aug 8, 2021
193
113
43
Finland
Is the adapter card inserted in the closest PCI-E slot in relation to the CPU? AFAIK signal quality is often a problem with these.
 
T

TomSwift

Member
May 14, 2020
37
13
8
I just finished playing the move everything around game, and looks like my two Kioxia CD6-R drives (KCD6XLUL7T68) are causing all the errors.

Need to figure out how to find an updated firmware or get some tech support from Kioxia.
 
M

mpepper

New Member
Nov 22, 2020
14
16
3
This is certainly going to be related to the quality of that unbranded AIC. Test each drive individually with a cable to your MB (use the two Oculink ports). Confirm no PCIE signal errors. Then toss that adapter and buy from Startech or Linkreal.
 
M

mpepper

New Member
Nov 22, 2020
14
16
3
Anecdotal but I've been using multiple of their AICs and cables for about three years now with no issues on two different Romed8-2Ts.
 
T

TomSwift

Member
May 14, 2020
37
13
8
I ordered some generic single AIC to rule out everything (checked all the slots and all my nvme drives around). The Kingston drives are solid, and only the Kioxia CD6-R drives have any issue (still looking for a FW update). The drives can be formatted and store data without an issue but they throw so many pages errors, as you can see by the time stamps (c1 way less than 46):

21:56:43 kernel: {51610}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:56:38 kernel: nvme 0000:c1:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:56:38 kernel: {51609}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:56:33 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:56:33 kernel: {51608}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:56:27 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:56:27 kernel: {51607}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:56:22 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:56:22 kernel: {51606}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:56:16 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:56:16 kernel: {51605}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:56:11 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:56:11 kernel: {51604}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:56:05 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:56:05 kernel: {51603}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:56:00 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:56:00 kernel: {51602}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:55:54 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:55:54 kernel: {51601}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:55:49 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:55:49 kernel: {51600}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:55:43 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:55:43 kernel: {51599}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:55:38 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:55:38 kernel: {51598}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:55:32 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:55:32 kernel: {51597}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:55:27 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:55:27 kernel: {51596}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:55:21 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:55:21 kernel: {51595}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:55:16 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:55:16 kernel: {51594}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:55:10 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:55:10 kernel: {51593}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:55:05 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:55:05 kernel: {51592}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:54:59 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:54:54 kernel: {51591}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:54:48 kernel: nvme 0000:c1:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:54:48 kernel: {51590}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:54:43 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:54:43 kernel: {51589}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:54:37 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:54:37 kernel: {51588}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:54:32 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:54:32 kernel: {51587}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:54:26 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:54:26 kernel: {51586}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:54:21 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:54:21 kernel: {51585}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:54:15 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:54:15 kernel: {51584}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:54:10 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:54:10 kernel: {51583}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:54:04 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:54:04 kernel: {51582}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:53:59 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:53:59 kernel: {51581}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:53:53 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:53:53 kernel: {51580}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:53:48 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:53:48 kernel: {51579}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:53:42 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
21:53:42 kernel: {51578}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0000
21:53:37 kernel: nvme 0000:46:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
 
You must log in or register to reply here.
Top Bottom