CWWK/Topton/... Nxxx quad NIC router

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
M

mack

New Member
Jul 17, 2022
20
6
3
I'm thinking, Odroid h4+ or h4 ultra are practically the same constructions as cwwk, do you think that the bios from h4 will work on cwwk boards? :)

Look at the power consumption, Odroid h4 ultra with n305 with ASPM enabled. With CWWK boards it is unattainable
1731865791526.png
 
D

Drabon

New Member
May 25, 2024
18
14
3
Not sure which CWWK product is supposed to be similar to the Odruid h4, but no. It will have a lower power consumption though... being bricked and all.
 
M

mack

New Member
Jul 17, 2022
20
6
3
Drabon said:
Not sure which CWWK product is supposed to be similar to the Odruid h4, but no. It will have a lower power consumption though... being bricked and all.
Click to expand...
Is same as Pionieer Edition or x86-p5, but probably you're right, besides I think it's not a BIOS problem but a construction problem, it's broken.
 
Last edited:
D

Drabon

New Member
May 25, 2024
18
14
3
That is a definite no to similarity, too. The ports and layout are different. Cross flashing a BIOS is always risky and only done with almost identical hardware in order to keep close to 100% functionality. So having a different product (ports, layout, vendor) will almost certainly result in a bricked non booting device (insert explained joke about reduced power consumption here).
 
S

slybunda

Active Member
Jan 30, 2023
170
99
28
DRW said:
Just fyi, if anyone is thinking of picking up one of these with active cooling, the fan on mine lasted about 9 months of continuous use before the bearings started dying, causing it to overheat and services to crash. $30 for them to send me a replacement fan (though I care more about the waste of my time to tear it down and replace).

Kind of annoying for a router. I don't think I've ever seen a fan die that fast. I won't buy one of these with active cooling again unless the fan is a standard size I can easily replace.
Click to expand...
thought it was a given that people should use their own higher quality fans. iv got a old scythe fdb fan on my one doing very well.
if i didnt have a spare fan lying around and needed to buy a new one id have gone with the arctic cooling 12cm slim fan which is fdb and keeps overall profile low.
 
D

DRW

Member
May 1, 2021
38
16
8
I think we're talking about different things. I'm talking about the Topton with the 8505 chip. It has a laptop-style blower fan with what appears to be proprietary mounting holes attached to a copper heatsink.
Edit: this thing S63aa21101ddc4b3a971bfbe08db2ea46j.png
 
Last edited:
E

eldee

New Member
Nov 25, 2024
4
2
3
Hi all, I have four of the Topton N305 variation A, and I am struggling to figure out how to enable the use of a SATA 2.5 SSD.
I have the right SATA data cable paired with the 4pin connector for power, but the bios does not seem to recognize the SSDs out of the box. Also I don’t see the he when I try to install Ubuntu.

Are there specific bios settings I would need to enable? Anyone with the same router that successfully had it running with an attached SSD?

Thank you in advance.
 
A

audit13

Member
Jun 26, 2024
49
18
8
eldee said:
Hi all, I have four of the Topton N305 variation A, and I am struggling to figure out how to enable the use of a SATA 2.5 SSD.
I have the right SATA data cable paired with the 4pin connector for power, but the bios does not seem to recognize the SSDs out of the box. Also I don’t see the he when I try to install Ubuntu.

Are there specific bios settings I would need to enable? Anyone with the same router that successfully had it running with an attached SSD?

Thank you in advance.
Click to expand...
I do not have the same Topton unit but my Topton came with the cable as show in the picture. It worked with an old sata3 OCZ Vertex SSD without any changes to the bios for a bare metal install of opnsense.
 

Attachments

E

eldee

New Member
Nov 25, 2024
4
2
3
Thank you for the answer!
Interesting, in my case I had to buy cables and found them in Ali express , but now I a doubting my purchase. Is there anyone on this long thread (which I tried to parse all) that has success with a cable they bought and could recommend a link?
 
Last edited:
A

audit13

Member
Jun 26, 2024
49
18
8
eldee said:
Thank you for the answer!
Interesting, in my case I had to buy cables and found them in Ali express , but now I a doubting my purchase. Is there anyone on this king thread (which I tried to parse all) that has success with a cable they bought and could recommend a link?
Click to expand...
If you can connect a mechanical notebook drive and feel it spinning, at least you know the power part of the cable is probably good and it's the SATA part that doesn't seem to be working.

Have you emailed Topton to see if they can provide a suitable cable?
 
E

eldee

New Member
Nov 25, 2024
4
2
3
Not sure I have one of those drives around, but I might! I’ll look around.
i did not reach out to Topton after I read terrible experiences on this forum from people trying to connect with them.
 
C

Chunks

New Member
Jul 20, 2018
3
0
1
I just ordered a CWWK (no SSD no RAM, N100, type C I think) minipc for use as a proxmox/Opnsense machine.
I am wondering if, from a heat perspective, would an SSD make more sense than a NVME? Since I imagine the HDD usage would be pretty low, maybe I'm just over thinking things. But my googling hasn't yielded any answers to this..
 
R

RuckusVol

New Member
Jul 2, 2024
11
8
3
I'm looking to upgrade my router/firewall hardware and after hours and hours I'm still not sure what to do. I'm about to have access to fiber at up to 8GbE symmetrical, although I'll probably start at 3 or 5GbE.

I plan to run Opnsense but I've had a Firewalla Gold SE so I'm not very familiar with hardware requirements to do recommended security stuff like Zenarmor, Suricata, etc.

The things I'm most unclear about are CPU and SFP. It seems clear to want the i226-v Ethernet and DDR5 RAM, probably 16GB. But I could probably max out an 8GbE connection and would like to not upgrade for a few years.

Is the U300E or 13th gen i5 a worthwhile upgrade over the n305? I'm guessing the improved single core performance would be worth it. The x520 SFP nic is ancient, but I would prefer 2 SFP and 4 eth, since my switch is 10G/1G. But I don't see options with newer SFP.

I've sorted through the current options from CWWK, Topton, Qotom, etc but don't see much difference once you make sure you're getting the components you want. Is there anything I'm easily missing there?

Mainly, if anyone has any recommendations I'd be very appreciative. I was hoping to order during Cyber Monday deals if I can decide on a plan. Or would it be smart and cost effective to build my own?
 
A

audit13

Member
Jun 26, 2024
49
18
8
Chunks said:
I just ordered a CWWK (no SSD no RAM, N100, type C I think) minipc for use as a proxmox/Opnsense machine.
I am wondering if, from a heat perspective, would an SSD make more sense than a NVME? Since I imagine the HDD usage would be pretty low, maybe I'm just over thinking things. But my googling hasn't yielded any answers to this..
Click to expand...
If you have a spare SATA3 SSD, use that to determine if the performance is suitable. If it isn't, then look into getting a NVMe drive.

I have a Topton n100 and wanted to keep power usage and heat to a minimum so I never considered using a NVMe drive for bare metal opnsense as I had a spare 60 GB SATA3 SSD that was just sitting in a drawer.

Without intensive read/writes to the SSD, I'm not sure there would be much difference in terms of heat and/or power usage.
 
  • Like
Reactions: Chunks
E

eldee

New Member
Nov 25, 2024
4
2
3
eldee said:
Not sure I have one of those drives around, but I might! I’ll look around.
i did not reach out to Topton after I read terrible experiences on this forum from people trying to connect with them.
Click to expand...
Posting this here for future reference, it turns out that, 1/ my cable for the SATA data+pwr was the right one, but 2/ the SATA port on this model (Topton N305 variation A) only activates if I remove the NVME 2230 to 2280 adapter. This means that I can only have a total of either 2 NVME 2280 ssd drives or 1 NVME SSD and 1 SATA drive. I was hoping to be able to use all the 3 of them. Also there seems to be no option in the BIOS to enable all of them at the same time.
 
  • Like
Reactions: audit13 and Chunks
M

Malkaf

New Member
Mar 18, 2021
2
0
1
I have a CWWK CW-AD4L-N V1 and I want to update the BIOS. Does anybody know what file to use? I'm not sure going from the naming on the website.
 
M

mikronassi

New Member
Dec 9, 2024
4
3
3
I have this N100 X86 P5 development board with 4*M2 NVMe expansion board: Link to CWWK product page

I'm having issues with the NVME expansion board. Sometimes 2-3 out of 4 attached NVME drives are not be detected. Reboot / attaching drives (same positions) and expansion board again helped for a while, but now only one of expansion board nvme drives is recognized. At one point I was able to create pool from 4 x NVMe drives in truenas.

BIOS:
  • project version: cw-adlntbx-1c2l
  • build date: 06/26/2024
RAM:
NVME drives:
  • Kingston KC3000 2TB x 4
  • Kingston KC3000 512GB ( in 2230 A+E M.2 slot with adapter board )

I have thought this as a software issue, but now I am starting to think that this might be hardware related. Maybe the expansion board's 2-pin power cable could be the issue? @rogervn did you find solution to your nvme drops?

dmesg from truenas:
Code: 
truenas_admin@truenas[~]$ sudo dmesg | grep nvme
[    0.000000] Command line: BOOT_IMAGE=/ROOT/24.10.0.2@/boot/vmlinuz-6.6.44-production+truenas root=ZFS=boot-pool/ROOT/24.10.0.2 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
[    0.020309] Kernel command line: BOOT_IMAGE=/ROOT/24.10.0.2@/boot/vmlinuz-6.6.44-production+truenas root=ZFS=boot-pool/ROOT/24.10.0.2 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
[    3.596977] nvme 0000:04:00.0: platform quirk: setting simple suspend
[    3.596979] nvme 0000:05:00.0: platform quirk: setting simple suspend
[    3.597152] nvme nvme0: pci function 0000:04:00.0
[    3.597581] nvme nvme1: pci function 0000:05:00.0
[    3.602281] nvme nvme0: Shutdown timeout set to 10 seconds
[    3.602328] nvme nvme1: Shutdown timeout set to 10 seconds
[    3.605981] nvme nvme1: 4/0/0 default/read/poll queues
[    3.606233] nvme nvme0: 4/0/0 default/read/poll queues
[    3.608396]  nvme1n1: p1 p2 p3
journalctl -k
Code: 
truenas_admin@truenas[~]$ sudo journalctl -k
Jun 16 02:44:31 truenas kernel: microcode: updated early: 0x11 -> 0x17, date = 2023-12-07
Jun 16 02:44:31 truenas kernel: Linux version 6.6.44-production+truenas (root@tnsbuilds01.tn.ixsystems.net) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Fri Nov  8 18:37:36 UTC 2024
Jun 16 02:44:31 truenas kernel: Command line: BOOT_IMAGE=/ROOT/24.10.0.2@/boot/vmlinuz-6.6.44-production+truenas root=ZFS=boot-pool/ROOT/24.10.0.2 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
Jun 16 02:44:31 truenas kernel: x86/split lock detection: #AC: crashing the kernel on kernel split_locks and warning on user-space split_locks
Jun 16 02:44:31 truenas kernel: BIOS-provided physical RAM map:
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x000000000009e000-0x000000000009efff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] usable
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000000100000-0x000000006fe52fff] usable
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x000000006fe53000-0x0000000072f52fff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000072f53000-0x0000000073034fff] ACPI data
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000073035000-0x00000000730f4fff] ACPI NVS
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000730f5000-0x0000000073e65fff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000073e66000-0x0000000073efefff] type 20
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000073eff000-0x0000000073efffff] usable
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000073f00000-0x0000000079ffffff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x000000007aa00000-0x000000007abfffff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x000000007b000000-0x00000000803fffff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000c0000000-0x00000000cfffffff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000fed00000-0x00000000fed00fff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000fed20000-0x00000000fed7ffff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000100000000-0x000000087fbfffff] usable
Jun 16 02:44:31 truenas kernel: NX (Execute Disable) protection: active
Jun 16 02:44:31 truenas kernel: APIC: Static calls initialized
Jun 16 02:44:31 truenas kernel: efi: EFI v2.8 by American Megatrends
Jun 16 02:44:31 truenas kernel: efi: ACPI=0x730d1000 ACPI 2.0=0x730d1014 TPMFinalLog=0x730a0000 SMBIOS=0x73bb0000 SMBIOS 3.0=0x73baf000 MEMATTR=0x6cdc1018 ESRT=0x6ded4898 MOKvar=0x73b83000
Jun 16 02:44:31 truenas kernel: efi: Remove mem78: MMIO range=[0xc0000000-0xcfffffff] (256MB) from e820 map
Jun 16 02:44:31 truenas kernel: e820: remove [mem 0xc0000000-0xcfffffff] reserved
Jun 16 02:44:31 truenas kernel: efi: Not removing mem79: MMIO range=[0xfe000000-0xfe010fff] (68KB) from e820 map
Jun 16 02:44:31 truenas kernel: efi: Not removing mem80: MMIO range=[0xfec00000-0xfec00fff] (4KB) from e820 map
Jun 16 02:44:31 truenas kernel: efi: Not removing mem81: MMIO range=[0xfed00000-0xfed00fff] (4KB) from e820 map
Jun 16 02:44:31 truenas kernel: efi: Not removing mem83: MMIO range=[0xfee00000-0xfee00fff] (4KB) from e820 map
Jun 16 02:44:31 truenas kernel: efi: Remove mem84: MMIO range=[0xff000000-0xffffffff] (16MB) from e820 map
Jun 16 02:44:31 truenas kernel: e820: remove [mem 0xff000000-0xffffffff] reserved
Jun 16 02:44:31 truenas kernel: SMBIOS 3.6.0 present.
Jun 16 02:44:31 truenas kernel: DMI: Default string Default string/Default string, BIOS 5.27 06/26/2024
Jun 16 02:44:31 truenas kernel: tsc: Detected 800.000 MHz processor
Jun 16 02:44:31 truenas kernel: tsc: Detected 806.400 MHz TSC
Jun 16 02:44:31 truenas kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
Jun 16 02:44:31 truenas kernel: e820: remove [mem 0x000a0000-0x000fffff] usable
Jun 16 02:44:31 truenas kernel: last_pfn = 0x87fc00 max_arch_pfn = 0x400000000
Jun 16 02:44:31 truenas kernel: MTRR map: 6 entries (3 fixed + 3 variable; max 23), built from 10 variable MTRRs
Jun 16 02:44:31 truenas kernel: x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT
Jun 16 02:44:31 truenas kernel: last_pfn = 0x73f00 max_arch_pfn = 0x400000000
Jun 16 02:44:31 truenas kernel: esrt: Reserving ESRT space from 0x000000006ded4898 to 0x000000006ded4948.
Jun 16 02:44:31 truenas kernel: e820: update [mem 0x6ded4000-0x6ded4fff] usable ==> reserved
Jun 16 02:44:31 truenas kernel: Using GB pages for direct mapping
Jun 16 02:44:31 truenas kernel: Secure boot disabled
Jun 16 02:44:31 truenas kernel: RAMDISK: [mem 0x2f1bb000-0x338d4fff]
Jun 16 02:44:31 truenas kernel: ACPI: Early table checksum verification disabled
Jun 16 02:44:31 truenas kernel: ACPI: RSDP 0x00000000730D1014 000024 (v02 ALASKA)
Jun 16 02:44:31 truenas kernel: ACPI: XSDT 0x00000000730D0728 00010C (v01 ALASKA A M I    01072009 AMI  01000013)
Jun 16 02:44:31 truenas kernel: ACPI: FACP 0x0000000073032000 000114 (v06 ALASKA A M I    01072009 AMI  01000013)
Jun 16 02:44:31 truenas kernel: ACPI: DSDT 0x0000000072FBC000 0752B7 (v02 ALASKA A M I    01072009 INTL 20200717)
Jun 16 02:44:31 truenas kernel: ACPI: FACS 0x00000000730F4000 000040
Jun 16 02:44:31 truenas kernel: ACPI: FIDT 0x0000000072FBB000 00009C (v01 ALASKA A M I    01072009 AMI  00010013)
Jun 16 02:44:31 truenas kernel: ACPI: SSDT 0x0000000073034000 00038C (v02 PmaxDv Pmax_Dev 00000001 INTL 20200717)
Jun 16 02:44:31 truenas kernel: ACPI: SSDT 0x0000000072FB5000 005D0B (v02 CpuRef CpuSsdt  00003000 INTL 20200717)
Jun 16 02:44:31 truenas kernel: ACPI: SSDT 0x0000000072FB2000 002935 (v02 SaSsdt SaSsdt   00003000 INTL 20200717)
Jun 16 02:44:31 truenas kernel: ACPI: SSDT 0x0000000072FAE000 0033D3 (v02 INTEL  IgfxSsdt 00003000 INTL 20200717)
Jun 16 02:44:31 truenas kernel: ACPI: SSDT 0x0000000072FA0000 00D487 (v02 INTEL  TcssSsdt 00001000 INTL 20200717)
Jun 16 02:44:31 truenas kernel: ACPI: HPET 0x0000000073033000 000038 (v01 ALASKA A M I    01072009 AMI  01000013)
Jun 16 02:44:31 truenas kernel: ACPI: APIC 0x0000000072F9F000 0001DC (v05 ALASKA A M I    01072009 AMI  01000013)
Jun 16 02:44:31 truenas kernel: ACPI: MCFG 0x0000000072F9E000 00003C (v01 ALASKA A M I    01072009 AMI  01000013)
Jun 16 02:44:31 truenas kernel: ACPI: SSDT 0x0000000072F9C000 001F1A (v02 ALASKA Ther_Rvp 00001000 INTL 20200717)
Jun 16 02:44:31 truenas kernel: ACPI: UEFI 0x0000000073087000 000048 (v01 ALASKA A M I    01072009 AMI  01000013)
UPDATE: went through syslog, had the same suggestion as @rogervn
("Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off"") :

Code: 
Dec  8 10:03:44 truenas kernel: Command line: BOOT_IMAGE=/ROOT/24.10.0.2@/boot/vmlinuz-6.6.44-production+truenas root=ZFS=boot-pool/ROOT/24.10.0.2 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
Dec  8 10:03:44 truenas kernel: Kernel command line: BOOT_IMAGE=/ROOT/24.10.0.2@/boot/vmlinuz-6.6.44-production+truenas root=ZFS=boot-pool/ROOT/24.10.0.2 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
Dec  8 10:03:44 truenas kernel: nvme 0000:02:00.0: platform quirk: setting simple suspend
Dec  8 10:03:44 truenas kernel: nvme 0000:01:00.0: platform quirk: setting simple suspend
Dec  8 10:03:44 truenas kernel: nvme 0000:03:00.0: platform quirk: setting simple suspend
Dec  8 10:03:44 truenas kernel: nvme 0000:04:00.0: platform quirk: setting simple suspend
Dec  8 10:03:44 truenas kernel: nvme nvme1: pci function 0000:02:00.0
Dec  8 10:03:44 truenas kernel: nvme 0000:05:00.0: platform quirk: setting simple suspend
Dec  8 10:03:44 truenas kernel: nvme nvme3: pci function 0000:04:00.0
Dec  8 10:03:44 truenas kernel: nvme nvme0: pci function 0000:01:00.0
Dec  8 10:03:44 truenas kernel: nvme nvme4: pci function 0000:05:00.0
Dec  8 10:03:44 truenas kernel: nvme nvme2: pci function 0000:03:00.0
Dec  8 10:03:44 truenas kernel: nvme nvme2: Shutdown timeout set to 10 seconds
Dec  8 10:03:44 truenas kernel: nvme nvme0: Shutdown timeout set to 10 seconds
Dec  8 10:03:44 truenas kernel: nvme nvme3: Shutdown timeout set to 10 seconds
Dec  8 10:03:44 truenas kernel: nvme nvme1: Shutdown timeout set to 10 seconds
Dec  8 10:03:44 truenas kernel: nvme nvme4: Shutdown timeout set to 10 seconds
Dec  8 10:03:44 truenas kernel: nvme nvme3: 4/0/0 default/read/poll queues
Dec  8 10:03:44 truenas kernel: nvme nvme1: 4/0/0 default/read/poll queues
Dec  8 10:03:44 truenas kernel: nvme nvme2: 4/0/0 default/read/poll queues
Dec  8 10:03:44 truenas kernel: nvme nvme4: 4/0/0 default/read/poll queues
Dec  8 10:03:44 truenas kernel: nvme nvme0: 4/0/0 default/read/poll queues
Dec  8 10:03:44 truenas kernel:  nvme3n1: p1
Dec  8 10:03:44 truenas kernel:  nvme2n1: p1
Dec  8 10:03:44 truenas kernel:  nvme1n1: p1
Dec  8 10:03:44 truenas kernel:  nvme4n1: p1 p2 p3
Dec  8 10:03:44 truenas kernel:  nvme0n1: p1
Dec  8 10:07:32 truenas systemd[1]: nvmefc-boot-connections.service - Auto-connect to subsystems on FC-NVME devices found during boot was skipped because of an unmet condition check (ConditionPathExists=/sys/class/fc/fc_udev_device/nvme_discovery).
Dec  8 10:07:32 truenas systemd[1]: Starting nvmf-autoconnect.service - Connect NVMe-oF subsystems automatically during boot...
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme0n1, opened
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme0n1, KINGSTON SKC3000D2048G, S/N:50026B7686EB36EF, FW:EIFK31.7, NSID:1, 2.04 TB
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme0n1, is SMART capable. Adding to "monitor" list.
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme1n1, opened
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme1n1, KINGSTON SKC3000D2048G, S/N:50026B7686FBD3BB, FW:EIFK51.2, NSID:1, 2.04 TB
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme1n1, is SMART capable. Adding to "monitor" list.
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme2n1, opened
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme2n1, KINGSTON SKC3000D2048G, S/N:50026B7686EB2B94, FW:EIFK31.7, NSID:1, 2.04 TB
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme2n1, is SMART capable. Adding to "monitor" list.
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme3n1, opened
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme3n1, KINGSTON SKC3000D2048G, S/N:50026B7686FBD264, FW:EIFK51.2, NSID:1, 2.04 TB
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme3n1, is SMART capable. Adding to "monitor" list.
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme4n1, opened
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme4n1, KINGSTON SKC3000S512G, S/N:50026B7686DFB7FF, FW:EIFK51.2, NSID:1, 512 GB
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme4n1, is SMART capable. Adding to "monitor" list.
Dec  8 10:07:32 truenas smartd[2842]: Monitoring 0 ATA/SATA, 0 SCSI/SAS and 5 NVMe devices
Dec  8 10:07:32 truenas systemd[1]: Finished nvmf-autoconnect.service - Connect NVMe-oF subsystems automatically during boot.
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme0n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686EB36EF-n1.nvme.state
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme1n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686FBD3BB-n1.nvme.state
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme2n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686EB2B94-n1.nvme.state
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme3n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686FBD264-n1.nvme.state
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme4n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000S512G-50026B7686DFB7FF-n1.nvme.state
Dec  8 10:07:32 truenas sensors[2861]: nvme-pci-0100
Dec  8 10:07:32 truenas sensors[2861]: nvme-pci-0200
Dec  8 10:07:32 truenas sensors[2861]: nvme-pci-0300
Dec  8 10:07:32 truenas sensors[2861]: nvme-pci-0500
Dec  8 10:07:32 truenas sensors[2861]: nvme-pci-0400
Dec  8 10:08:53 truenas kernel: nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
Dec  8 10:08:53 truenas kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
Dec  8 10:08:53 truenas kernel: nvme nvme0: Does your device have a faulty power saving mode enabled?
Dec  8 10:08:53 truenas kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
Dec  8 10:08:53 truenas kernel: nvme nvme1: Does your device have a faulty power saving mode enabled?
Dec  8 10:08:53 truenas kernel: nvme nvme1: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
Dec  8 10:08:53 truenas kernel: nvme nvme2: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
Dec  8 10:08:53 truenas kernel: nvme nvme2: Does your device have a faulty power saving mode enabled?
Dec  8 10:08:53 truenas kernel: nvme nvme2: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
Dec  8 10:08:53 truenas kernel: nvme 0000:02:00.0: Unable to change power state from D3cold to D0, device inaccessible
Dec  8 10:08:53 truenas kernel: nvme 0000:03:00.0: Unable to change power state from D3cold to D0, device inaccessible
Dec  8 10:08:53 truenas kernel: nvme nvme1: Disabling device after reset failure: -19
Dec  8 10:08:53 truenas kernel: nvme nvme2: Disabling device after reset failure: -19
Dec  8 10:08:53 truenas kernel: nvme 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
Dec  8 10:08:53 truenas kernel: nvme nvme0: Disabling device after reset failure: -19
Dec  8 10:09:44 truenas smartd[2842]: Device: /dev/nvme0n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686EB36EF-n1.nvme.state
Dec  8 10:09:44 truenas smartd[2842]: Device: /dev/nvme1n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686FBD3BB-n1.nvme.state
Dec  8 10:09:44 truenas smartd[2842]: Device: /dev/nvme2n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686EB2B94-n1.nvme.state
Dec  8 10:09:44 truenas smartd[2842]: Device: /dev/nvme3n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686FBD264-n1.nvme.state
Dec  8 10:09:44 truenas smartd[2842]: Device: /dev/nvme4n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000S512G-50026B7686DFB7FF-n1.nvme.state
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme0n1, opened
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme0n1, NVMe Identify Controller failed
Dec  8 10:09:44 truenas smartd[6041]: Unable to register NVMe device /dev/nvme0n1 at line 1 of file /etc/smartd.conf
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme0n1, not available
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme1n1, opened
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme1n1, NVMe Identify Controller failed
Dec  8 10:09:44 truenas smartd[6041]: Unable to register NVMe device /dev/nvme1n1 at line 2 of file /etc/smartd.conf
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme1n1, not available
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme2n1, opened
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme2n1, NVMe Identify Controller failed
Dec  8 10:09:44 truenas smartd[6041]: Unable to register NVMe device /dev/nvme2n1 at line 3 of file /etc/smartd.conf
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme2n1, not available
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme3n1, opened
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme3n1, KINGSTON SKC3000D2048G, S/N:50026B7686FBD264, FW:EIFK51.2, NSID:1, 2.04 TB
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme3n1, is SMART capable. Adding to "monitor" list.
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme3n1, state read from /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686FBD264-n1.nvme.state
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme4n1, opened
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme4n1, KINGSTON SKC3000S512G, S/N:50026B7686DFB7FF, FW:EIFK51.2, NSID:1, 512 GB
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme4n1, is SMART capable. Adding to "monitor" list.
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme4n1, state read from /var/lib/smartmontools/smartd.KINGSTON_SKC3000S512G-50026B7686DFB7FF-n1.nvme.state
Dec  8 10:09:44 truenas smartd[6041]: Monitoring 0 ATA/SATA, 0 SCSI/SAS and 2 NVMe devices
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme3n1, NVMe error count increased from 76 to 80 (0 new, 4 ignored, 0 unknown)
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme4n1, NVMe error count increased from 86 to 90 (0 new, 4 ignored, 0 unknown)
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme3n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686FBD264-n1.nvme.state
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme4n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000S512G-50026B7686DFB7FF-n1.nvme.state
Dec  8 10:10:33 truenas smartd[6041]: Device: /dev/nvme3n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686FBD264-n1.nvme.state
Dec  8 10:10:33 truenas smartd[6041]: Device: /dev/nvme4n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000S512G-50026B7686DFB7FF-n1.nvme.state
Tried to set :
Code: 
midclt call system.advanced.update '{"kernel_extra_options": "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off"}'
did not make a difference.
 
Last edited:
M

mikronassi

New Member
Dec 9, 2024
4
3
3
Managed to get all 4 nvme drives recognized again by adding them one by one to expansion board and booting after every added drive. Did not change order of the drives. What could cause this?
 
  • Like
Reactions: Stovar and KevinR
R

rogervn

New Member
Sep 22, 2024
17
5
3
mikronassi said:
I have this N100 X86 P5 development board with 4*M2 NVMe expansion board: Link to CWWK product page

I'm having issues with the NVME expansion board. Sometimes 2-3 out of 4 attached NVME drives are not be detected. Reboot / attaching drives (same positions) and expansion board again helped for a while, but now only one of expansion board nvme drives is recognized. At one point I was able to create pool from 4 x NVMe drives in truenas.

BIOS:
  • project version: cw-adlntbx-1c2l
  • build date: 06/26/2024
RAM:
NVME drives:
  • Kingston KC3000 2TB x 4
  • Kingston KC3000 512GB ( in 2230 A+E M.2 slot with adapter board )

I have thought this as a software issue, but now I am starting to think that this might be hardware related. Maybe the expansion board's 2-pin power cable could be the issue? @rogervn did you find solution to your nvme drops?

dmesg from truenas:
Code: 
truenas_admin@truenas[~]$ sudo dmesg | grep nvme
[    0.000000] Command line: BOOT_IMAGE=/ROOT/24.10.0.2@/boot/vmlinuz-6.6.44-production+truenas root=ZFS=boot-pool/ROOT/24.10.0.2 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
[    0.020309] Kernel command line: BOOT_IMAGE=/ROOT/24.10.0.2@/boot/vmlinuz-6.6.44-production+truenas root=ZFS=boot-pool/ROOT/24.10.0.2 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
[    3.596977] nvme 0000:04:00.0: platform quirk: setting simple suspend
[    3.596979] nvme 0000:05:00.0: platform quirk: setting simple suspend
[    3.597152] nvme nvme0: pci function 0000:04:00.0
[    3.597581] nvme nvme1: pci function 0000:05:00.0
[    3.602281] nvme nvme0: Shutdown timeout set to 10 seconds
[    3.602328] nvme nvme1: Shutdown timeout set to 10 seconds
[    3.605981] nvme nvme1: 4/0/0 default/read/poll queues
[    3.606233] nvme nvme0: 4/0/0 default/read/poll queues
[    3.608396]  nvme1n1: p1 p2 p3
journalctl -k
Code: 
truenas_admin@truenas[~]$ sudo journalctl -k
Jun 16 02:44:31 truenas kernel: microcode: updated early: 0x11 -> 0x17, date = 2023-12-07
Jun 16 02:44:31 truenas kernel: Linux version 6.6.44-production+truenas (root@tnsbuilds01.tn.ixsystems.net) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Fri Nov  8 18:37:36 UTC 2024
Jun 16 02:44:31 truenas kernel: Command line: BOOT_IMAGE=/ROOT/24.10.0.2@/boot/vmlinuz-6.6.44-production+truenas root=ZFS=boot-pool/ROOT/24.10.0.2 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
Jun 16 02:44:31 truenas kernel: x86/split lock detection: #AC: crashing the kernel on kernel split_locks and warning on user-space split_locks
Jun 16 02:44:31 truenas kernel: BIOS-provided physical RAM map:
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x000000000009e000-0x000000000009efff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] usable
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000000100000-0x000000006fe52fff] usable
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x000000006fe53000-0x0000000072f52fff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000072f53000-0x0000000073034fff] ACPI data
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000073035000-0x00000000730f4fff] ACPI NVS
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000730f5000-0x0000000073e65fff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000073e66000-0x0000000073efefff] type 20
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000073eff000-0x0000000073efffff] usable
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000073f00000-0x0000000079ffffff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x000000007aa00000-0x000000007abfffff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x000000007b000000-0x00000000803fffff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000c0000000-0x00000000cfffffff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000fed00000-0x00000000fed00fff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000fed20000-0x00000000fed7ffff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
Jun 16 02:44:31 truenas kernel: BIOS-e820: [mem 0x0000000100000000-0x000000087fbfffff] usable
Jun 16 02:44:31 truenas kernel: NX (Execute Disable) protection: active
Jun 16 02:44:31 truenas kernel: APIC: Static calls initialized
Jun 16 02:44:31 truenas kernel: efi: EFI v2.8 by American Megatrends
Jun 16 02:44:31 truenas kernel: efi: ACPI=0x730d1000 ACPI 2.0=0x730d1014 TPMFinalLog=0x730a0000 SMBIOS=0x73bb0000 SMBIOS 3.0=0x73baf000 MEMATTR=0x6cdc1018 ESRT=0x6ded4898 MOKvar=0x73b83000
Jun 16 02:44:31 truenas kernel: efi: Remove mem78: MMIO range=[0xc0000000-0xcfffffff] (256MB) from e820 map
Jun 16 02:44:31 truenas kernel: e820: remove [mem 0xc0000000-0xcfffffff] reserved
Jun 16 02:44:31 truenas kernel: efi: Not removing mem79: MMIO range=[0xfe000000-0xfe010fff] (68KB) from e820 map
Jun 16 02:44:31 truenas kernel: efi: Not removing mem80: MMIO range=[0xfec00000-0xfec00fff] (4KB) from e820 map
Jun 16 02:44:31 truenas kernel: efi: Not removing mem81: MMIO range=[0xfed00000-0xfed00fff] (4KB) from e820 map
Jun 16 02:44:31 truenas kernel: efi: Not removing mem83: MMIO range=[0xfee00000-0xfee00fff] (4KB) from e820 map
Jun 16 02:44:31 truenas kernel: efi: Remove mem84: MMIO range=[0xff000000-0xffffffff] (16MB) from e820 map
Jun 16 02:44:31 truenas kernel: e820: remove [mem 0xff000000-0xffffffff] reserved
Jun 16 02:44:31 truenas kernel: SMBIOS 3.6.0 present.
Jun 16 02:44:31 truenas kernel: DMI: Default string Default string/Default string, BIOS 5.27 06/26/2024
Jun 16 02:44:31 truenas kernel: tsc: Detected 800.000 MHz processor
Jun 16 02:44:31 truenas kernel: tsc: Detected 806.400 MHz TSC
Jun 16 02:44:31 truenas kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
Jun 16 02:44:31 truenas kernel: e820: remove [mem 0x000a0000-0x000fffff] usable
Jun 16 02:44:31 truenas kernel: last_pfn = 0x87fc00 max_arch_pfn = 0x400000000
Jun 16 02:44:31 truenas kernel: MTRR map: 6 entries (3 fixed + 3 variable; max 23), built from 10 variable MTRRs
Jun 16 02:44:31 truenas kernel: x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT
Jun 16 02:44:31 truenas kernel: last_pfn = 0x73f00 max_arch_pfn = 0x400000000
Jun 16 02:44:31 truenas kernel: esrt: Reserving ESRT space from 0x000000006ded4898 to 0x000000006ded4948.
Jun 16 02:44:31 truenas kernel: e820: update [mem 0x6ded4000-0x6ded4fff] usable ==> reserved
Jun 16 02:44:31 truenas kernel: Using GB pages for direct mapping
Jun 16 02:44:31 truenas kernel: Secure boot disabled
Jun 16 02:44:31 truenas kernel: RAMDISK: [mem 0x2f1bb000-0x338d4fff]
Jun 16 02:44:31 truenas kernel: ACPI: Early table checksum verification disabled
Jun 16 02:44:31 truenas kernel: ACPI: RSDP 0x00000000730D1014 000024 (v02 ALASKA)
Jun 16 02:44:31 truenas kernel: ACPI: XSDT 0x00000000730D0728 00010C (v01 ALASKA A M I    01072009 AMI  01000013)
Jun 16 02:44:31 truenas kernel: ACPI: FACP 0x0000000073032000 000114 (v06 ALASKA A M I    01072009 AMI  01000013)
Jun 16 02:44:31 truenas kernel: ACPI: DSDT 0x0000000072FBC000 0752B7 (v02 ALASKA A M I    01072009 INTL 20200717)
Jun 16 02:44:31 truenas kernel: ACPI: FACS 0x00000000730F4000 000040
Jun 16 02:44:31 truenas kernel: ACPI: FIDT 0x0000000072FBB000 00009C (v01 ALASKA A M I    01072009 AMI  00010013)
Jun 16 02:44:31 truenas kernel: ACPI: SSDT 0x0000000073034000 00038C (v02 PmaxDv Pmax_Dev 00000001 INTL 20200717)
Jun 16 02:44:31 truenas kernel: ACPI: SSDT 0x0000000072FB5000 005D0B (v02 CpuRef CpuSsdt  00003000 INTL 20200717)
Jun 16 02:44:31 truenas kernel: ACPI: SSDT 0x0000000072FB2000 002935 (v02 SaSsdt SaSsdt   00003000 INTL 20200717)
Jun 16 02:44:31 truenas kernel: ACPI: SSDT 0x0000000072FAE000 0033D3 (v02 INTEL  IgfxSsdt 00003000 INTL 20200717)
Jun 16 02:44:31 truenas kernel: ACPI: SSDT 0x0000000072FA0000 00D487 (v02 INTEL  TcssSsdt 00001000 INTL 20200717)
Jun 16 02:44:31 truenas kernel: ACPI: HPET 0x0000000073033000 000038 (v01 ALASKA A M I    01072009 AMI  01000013)
Jun 16 02:44:31 truenas kernel: ACPI: APIC 0x0000000072F9F000 0001DC (v05 ALASKA A M I    01072009 AMI  01000013)
Jun 16 02:44:31 truenas kernel: ACPI: MCFG 0x0000000072F9E000 00003C (v01 ALASKA A M I    01072009 AMI  01000013)
Jun 16 02:44:31 truenas kernel: ACPI: SSDT 0x0000000072F9C000 001F1A (v02 ALASKA Ther_Rvp 00001000 INTL 20200717)
Jun 16 02:44:31 truenas kernel: ACPI: UEFI 0x0000000073087000 000048 (v01 ALASKA A M I    01072009 AMI  01000013)
UPDATE: went through syslog, had the same suggestion as @rogervn
("Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off"") :

Code: 
Dec  8 10:03:44 truenas kernel: Command line: BOOT_IMAGE=/ROOT/24.10.0.2@/boot/vmlinuz-6.6.44-production+truenas root=ZFS=boot-pool/ROOT/24.10.0.2 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
Dec  8 10:03:44 truenas kernel: Kernel command line: BOOT_IMAGE=/ROOT/24.10.0.2@/boot/vmlinuz-6.6.44-production+truenas root=ZFS=boot-pool/ROOT/24.10.0.2 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
Dec  8 10:03:44 truenas kernel: nvme 0000:02:00.0: platform quirk: setting simple suspend
Dec  8 10:03:44 truenas kernel: nvme 0000:01:00.0: platform quirk: setting simple suspend
Dec  8 10:03:44 truenas kernel: nvme 0000:03:00.0: platform quirk: setting simple suspend
Dec  8 10:03:44 truenas kernel: nvme 0000:04:00.0: platform quirk: setting simple suspend
Dec  8 10:03:44 truenas kernel: nvme nvme1: pci function 0000:02:00.0
Dec  8 10:03:44 truenas kernel: nvme 0000:05:00.0: platform quirk: setting simple suspend
Dec  8 10:03:44 truenas kernel: nvme nvme3: pci function 0000:04:00.0
Dec  8 10:03:44 truenas kernel: nvme nvme0: pci function 0000:01:00.0
Dec  8 10:03:44 truenas kernel: nvme nvme4: pci function 0000:05:00.0
Dec  8 10:03:44 truenas kernel: nvme nvme2: pci function 0000:03:00.0
Dec  8 10:03:44 truenas kernel: nvme nvme2: Shutdown timeout set to 10 seconds
Dec  8 10:03:44 truenas kernel: nvme nvme0: Shutdown timeout set to 10 seconds
Dec  8 10:03:44 truenas kernel: nvme nvme3: Shutdown timeout set to 10 seconds
Dec  8 10:03:44 truenas kernel: nvme nvme1: Shutdown timeout set to 10 seconds
Dec  8 10:03:44 truenas kernel: nvme nvme4: Shutdown timeout set to 10 seconds
Dec  8 10:03:44 truenas kernel: nvme nvme3: 4/0/0 default/read/poll queues
Dec  8 10:03:44 truenas kernel: nvme nvme1: 4/0/0 default/read/poll queues
Dec  8 10:03:44 truenas kernel: nvme nvme2: 4/0/0 default/read/poll queues
Dec  8 10:03:44 truenas kernel: nvme nvme4: 4/0/0 default/read/poll queues
Dec  8 10:03:44 truenas kernel: nvme nvme0: 4/0/0 default/read/poll queues
Dec  8 10:03:44 truenas kernel:  nvme3n1: p1
Dec  8 10:03:44 truenas kernel:  nvme2n1: p1
Dec  8 10:03:44 truenas kernel:  nvme1n1: p1
Dec  8 10:03:44 truenas kernel:  nvme4n1: p1 p2 p3
Dec  8 10:03:44 truenas kernel:  nvme0n1: p1
Dec  8 10:07:32 truenas systemd[1]: nvmefc-boot-connections.service - Auto-connect to subsystems on FC-NVME devices found during boot was skipped because of an unmet condition check (ConditionPathExists=/sys/class/fc/fc_udev_device/nvme_discovery).
Dec  8 10:07:32 truenas systemd[1]: Starting nvmf-autoconnect.service - Connect NVMe-oF subsystems automatically during boot...
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme0n1, opened
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme0n1, KINGSTON SKC3000D2048G, S/N:50026B7686EB36EF, FW:EIFK31.7, NSID:1, 2.04 TB
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme0n1, is SMART capable. Adding to "monitor" list.
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme1n1, opened
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme1n1, KINGSTON SKC3000D2048G, S/N:50026B7686FBD3BB, FW:EIFK51.2, NSID:1, 2.04 TB
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme1n1, is SMART capable. Adding to "monitor" list.
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme2n1, opened
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme2n1, KINGSTON SKC3000D2048G, S/N:50026B7686EB2B94, FW:EIFK31.7, NSID:1, 2.04 TB
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme2n1, is SMART capable. Adding to "monitor" list.
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme3n1, opened
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme3n1, KINGSTON SKC3000D2048G, S/N:50026B7686FBD264, FW:EIFK51.2, NSID:1, 2.04 TB
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme3n1, is SMART capable. Adding to "monitor" list.
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme4n1, opened
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme4n1, KINGSTON SKC3000S512G, S/N:50026B7686DFB7FF, FW:EIFK51.2, NSID:1, 512 GB
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme4n1, is SMART capable. Adding to "monitor" list.
Dec  8 10:07:32 truenas smartd[2842]: Monitoring 0 ATA/SATA, 0 SCSI/SAS and 5 NVMe devices
Dec  8 10:07:32 truenas systemd[1]: Finished nvmf-autoconnect.service - Connect NVMe-oF subsystems automatically during boot.
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme0n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686EB36EF-n1.nvme.state
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme1n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686FBD3BB-n1.nvme.state
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme2n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686EB2B94-n1.nvme.state
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme3n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686FBD264-n1.nvme.state
Dec  8 10:07:32 truenas smartd[2842]: Device: /dev/nvme4n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000S512G-50026B7686DFB7FF-n1.nvme.state
Dec  8 10:07:32 truenas sensors[2861]: nvme-pci-0100
Dec  8 10:07:32 truenas sensors[2861]: nvme-pci-0200
Dec  8 10:07:32 truenas sensors[2861]: nvme-pci-0300
Dec  8 10:07:32 truenas sensors[2861]: nvme-pci-0500
Dec  8 10:07:32 truenas sensors[2861]: nvme-pci-0400
Dec  8 10:08:53 truenas kernel: nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
Dec  8 10:08:53 truenas kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
Dec  8 10:08:53 truenas kernel: nvme nvme0: Does your device have a faulty power saving mode enabled?
Dec  8 10:08:53 truenas kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
Dec  8 10:08:53 truenas kernel: nvme nvme1: Does your device have a faulty power saving mode enabled?
Dec  8 10:08:53 truenas kernel: nvme nvme1: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
Dec  8 10:08:53 truenas kernel: nvme nvme2: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
Dec  8 10:08:53 truenas kernel: nvme nvme2: Does your device have a faulty power saving mode enabled?
Dec  8 10:08:53 truenas kernel: nvme nvme2: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
Dec  8 10:08:53 truenas kernel: nvme 0000:02:00.0: Unable to change power state from D3cold to D0, device inaccessible
Dec  8 10:08:53 truenas kernel: nvme 0000:03:00.0: Unable to change power state from D3cold to D0, device inaccessible
Dec  8 10:08:53 truenas kernel: nvme nvme1: Disabling device after reset failure: -19
Dec  8 10:08:53 truenas kernel: nvme nvme2: Disabling device after reset failure: -19
Dec  8 10:08:53 truenas kernel: nvme 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
Dec  8 10:08:53 truenas kernel: nvme nvme0: Disabling device after reset failure: -19
Dec  8 10:09:44 truenas smartd[2842]: Device: /dev/nvme0n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686EB36EF-n1.nvme.state
Dec  8 10:09:44 truenas smartd[2842]: Device: /dev/nvme1n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686FBD3BB-n1.nvme.state
Dec  8 10:09:44 truenas smartd[2842]: Device: /dev/nvme2n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686EB2B94-n1.nvme.state
Dec  8 10:09:44 truenas smartd[2842]: Device: /dev/nvme3n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686FBD264-n1.nvme.state
Dec  8 10:09:44 truenas smartd[2842]: Device: /dev/nvme4n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000S512G-50026B7686DFB7FF-n1.nvme.state
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme0n1, opened
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme0n1, NVMe Identify Controller failed
Dec  8 10:09:44 truenas smartd[6041]: Unable to register NVMe device /dev/nvme0n1 at line 1 of file /etc/smartd.conf
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme0n1, not available
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme1n1, opened
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme1n1, NVMe Identify Controller failed
Dec  8 10:09:44 truenas smartd[6041]: Unable to register NVMe device /dev/nvme1n1 at line 2 of file /etc/smartd.conf
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme1n1, not available
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme2n1, opened
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme2n1, NVMe Identify Controller failed
Dec  8 10:09:44 truenas smartd[6041]: Unable to register NVMe device /dev/nvme2n1 at line 3 of file /etc/smartd.conf
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme2n1, not available
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme3n1, opened
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme3n1, KINGSTON SKC3000D2048G, S/N:50026B7686FBD264, FW:EIFK51.2, NSID:1, 2.04 TB
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme3n1, is SMART capable. Adding to "monitor" list.
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme3n1, state read from /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686FBD264-n1.nvme.state
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme4n1, opened
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme4n1, KINGSTON SKC3000S512G, S/N:50026B7686DFB7FF, FW:EIFK51.2, NSID:1, 512 GB
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme4n1, is SMART capable. Adding to "monitor" list.
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme4n1, state read from /var/lib/smartmontools/smartd.KINGSTON_SKC3000S512G-50026B7686DFB7FF-n1.nvme.state
Dec  8 10:09:44 truenas smartd[6041]: Monitoring 0 ATA/SATA, 0 SCSI/SAS and 2 NVMe devices
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme3n1, NVMe error count increased from 76 to 80 (0 new, 4 ignored, 0 unknown)
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme4n1, NVMe error count increased from 86 to 90 (0 new, 4 ignored, 0 unknown)
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme3n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686FBD264-n1.nvme.state
Dec  8 10:09:44 truenas smartd[6041]: Device: /dev/nvme4n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000S512G-50026B7686DFB7FF-n1.nvme.state
Dec  8 10:10:33 truenas smartd[6041]: Device: /dev/nvme3n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000D2048G-50026B7686FBD264-n1.nvme.state
Dec  8 10:10:33 truenas smartd[6041]: Device: /dev/nvme4n1, state written to /var/lib/smartmontools/smartd.KINGSTON_SKC3000S512G-50026B7686DFB7FF-n1.nvme.state
Tried to set :
Code: 
midclt call system.advanced.update '{"kernel_extra_options": "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off"}'
did not make a difference.
Click to expand...
In my case I believe that updating the firmware of each of my drives or trying to press the board enough so it fits correctly so far has solved for me. But I'm scared of opening the device again and the disconnections returning.
 
  • Like
Reactions: Stovar and mikronassi
M

mikronassi

New Member
Dec 9, 2024
4
3
3
rogervn said:
In my case I believe that updating the firmware of each of my drives or trying to press the board enough so it fits correctly so far has solved for me. But I'm scared of opening the device again and the disconnections returning.
Click to expand...
Same feeling, lost the pool after one day of uptime. Couple of drives unrecognized again. Next I'll try to replace the expansion board power cable. Updating drive firmware did not cross my mind, that might be the next step.

Overall experience: not the level of reliability I was expecting :D Shame as this board ticks all the other boxes for my use case (small, power efficient, silent, nvme).
 
  • Like
Reactions: Stovar
You must log in or register to reply here.
Top Bottom