CWWK/Topton/... Nxxx quad NIC router

igotserved · Aug 5, 2023

Can someone who bought from the CWWK.net store (not Aliexpress) confirm that the N100 that you got is the variant 'C' case (tall straight fins)?

MajorPayneDOF · Aug 5, 2023

Does anyone know what thermal compound they are using on these? What about room to add a heatsink to the m.2?

lucker · Aug 6, 2023

MajorPayneDOF said:
What about room to add a heatsink to the m.2?

There is some room for that, but adding a heatsink makes it impossible to install internal fan if you use both native NVMe slot and H-type PCB, which comes with the device (see my previous posts here). I chose the very thin, ca. 3mm copper heatsink and lack about 1-2 mm for the slimmest internal fan 80x80x10 available. So I added an external one.
If you opt in for 4xNMVe board the CWWK store on Ali offers, you have no room even without heatsink (see my posts below)

lucker · Aug 6, 2023

Continue reporting on some tests (see my previous posts: first, second)

Seems like the performance issues I have had were due to me stupidly forgot to align the partition and didn't pay attention to ashift while creating ZFS pools. With that corrected I also did some optimisation in regard of ZFS performance (see this guide the awesome guys posted and the link in there to their previous posts).

So, I made two mirrored pools (ashift 12 for 4k); didn't mirror swap though but added a reserve partition just in case:

zroot - the system partition 32GB with lz4 compression
zdata - the data for jails and othe stuff with geli encryption and lz4 compression

Code:

11:35 /root # gpart show
=>        40  2000409184  nda0  GPT  (954G)
          40     1048576     1  efi  (512M)
     1048616        2008        - free -  (1.0M)
     1050624    67108864     2  freebsd-zfs  (32G)
    68159488    10485760     3  freebsd-zfs  (5.0G)
    78645248  1916796928     4  freebsd-zfs  (914G)
  1995442176     4966400     5  freebsd-swap  (2.4G)
  2000408576         648        - free -  (324K)

=>        40  2000409184  nda1  GPT  (954G)
          40     1048576     1  efi  (512M)
     1048616        2008        - free -  (1.0M)
     1050624    67108864     2  freebsd-zfs  (32G)
    68159488    10485760     3  freebsd-zfs  (5.0G)
    78645248  1916796928     4  freebsd-zfs  (914G)
  1995442176     4966400     5  freebsd-swap  (2.4G)
  2000408576         648        - free -  (324K)

Below the outcomes of the fio test for both zpools:

fio 16 parallel 64KiB random write

Code:

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1

fio Single 4KiB random write

Code:

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1

FreeBSD basics over SSH, vertically positioned box with small external fan (fixed ca. 1500RPM); ambient t=26C

fio 16 parallel 64KiB random write processes zroot
At start
nvme0 Temperature: 42 Celsius
nvme1 Temperature: 37 Celsius
CPU38C
At end
nvme0 Temperature: 44 Celsius
nvme1 Temperature: 39 Celsius
CPU 39C
Results:
WRITE: bw=405MiB/s (425MB/s), 24.5MiB/s-26.0MiB/s (25.7MB/s-27.3MB/s), io=24.0GiB (25.8GB), run=60577-60584msec

fio 16 parallel 64KiB random write processes zdata
At start
nvme0 Temperature: 36 Celsius
nvme1 Temperature: 32 Celsius
CPU 36-37
At end
nvme0 Temperature: 42 Celsius
nvme1 Temperature: 37 Celsius
Results:
WRITE: bw=416MiB/s (436MB/s), 25.7MiB/s-26.7MiB/s (27.0MB/s-28.0MB/s), io=31.0GiB (33.3GB), run=76468-76474msec

fio Single 4KiB random write process zroot
At start
nvme0 Temperature: 42 Celsius
nvme1 Temperature: 36 Celsius
At end
nvme0 Temperature: 42 Celsius
nvme1 Temperature: 36 Celsius
CPU 39-40
Results
WRITE: bw=71.8MiB/s (75.3MB/s), 71.8MiB/s-71.8MiB/s (75.3MB/s-75.3MB/s), io=4313MiB (4523MB), run=60089-60089msec

fio Single 4KiB random write process zdata
At start
nvme0 Temperature: 39 Celsius
nvme1 Temperature: 31 Celsius
At end
nvme0 Temperature: 42 Celsius
nvme1 Temperature: 35 Celsius
CPU 40
Results
WRITE: bw=93.5MiB/s (98.0MB/s), 93.5MiB/s-93.5MiB/s (98.0MB/s-98.0MB/s), io=5721MiB (5999MB), run=61188-61188msec

	Before ZFS optimisation	After ZFS optimisation
fio 16 parallel 64KiB random write zroot	WRITE: bw=336MiB/s (353MB/s), 15.1MiB/s-27.0MiB/s (15.8MB/s-28.3MB/s), io=20.3GiB (21.8GB), run=61841-61852msec	WRITE: bw=405MiB/s (425MB/s), 24.5MiB/s-26.0MiB/s (25.7MB/s-27.3MB/s), io=24.0GiB (25.8GB), run=60577-60584msec
fio 16 parallel 64KiB random write zdata	WRITE: bw=278MiB/s (291MB/s), 16.4MiB/s-19.5MiB/s (17.2MB/s-20.5MB/s), io=19.8GiB (21.2GB), run=72860-72870msec	WRITE: bw=416MiB/s (436MB/s), 25.7MiB/s-26.7MiB/s (27.0MB/s-28.0MB/s), io=31.0GiB (33.3GB), run=76468-76474msec
fio Single 4KiB random write zroot	WRITE: bw=78.7MiB/s (82.5MB/s), 78.7MiB/s-78.7MiB/s (82.5MB/s-82.5MB/s), io=4724MiB (4954MB), run=60024-60024msec	WRITE: bw=71.8MiB/s (75.3MB/s), 71.8MiB/s-71.8MiB/s (75.3MB/s-75.3MB/s), io=4313MiB (4523MB), run=60089-60089msec
fio Single 4KiB random write zdata	WRITE: bw=72.0MiB/s (75.5MB/s), 72.0MiB/s-72.0MiB/s (75.5MB/s-75.5MB/s), io=4334MiB (4545MB), run=60169-60169msec	WRITE: bw=93.5MiB/s (98.0MB/s), 93.5MiB/s-93.5MiB/s (98.0MB/s-98.0MB/s), io=5721MiB (5999MB), run=61188-61188msec

I have to mention a couple of things:

The first pass of 4k tests (when no random-write files exist in the directory) show better performance, ca. 25-30%
I didn't have enough time to run the same 3-5 tests in a row, but I have noticed that same setup and condition may results in slightly (or more than slightly for 4k tests) outcomes; my take is that ZFS does not provide consistent performance

Let me know if you would like to take a look at the test details in verbose format, I edit this post

lucker · Aug 6, 2023

Continue reporting on some tests (see my previous posts: first, second, third)

Finally I took my chance on the 4xNVMe PCB the nice guys from CWWK at AliExpress have put in my box for free. I made a bootable CF-card with the Ventoy tool and put there BIOS for the PCB. I went smothly without any issue. The instructions strongly suggest that one should disconnect power cord when installing or removing this board. The docs also mentioned that I should not touch switches on the PCB because they meant for USB, which are not present. So I complied.

What I do not like:

connecting the two parts of the PCB is a hell of PITA; I was afraid to jam the pins, but eventually managed to couple two pieces
this huge (well, relatively) board doesn't have any way to fix it in place. It's just hanging there on this pins that connects two parts of the board
it covers most of the RAM module, so I do not do any RAM test or stress test that involving RAM anymore: glad I did it earlier
I have to remove the dampening rubber nails for the fan and bolt it to the box hard since the rubber nail bottom pushed on the SSD slightly, which I do not like at all. BTW, this ExeGate fan is quite good, it has 2-Ball bearing, good airflow 28.5CFM and causes no perceptible vibration at 1500RPM and makes no noise; and it's dirty cheap

Surprisingly, the BIOS doesn't have "optimized default" option to load the settings and I forgot to take photos of the previous BIOS screens. So, I have a choice to flash BIOS back and forth or find the "optimized" settings somewhere. If you, by any chance, playing with the default BIOS, I would appreciate if you load the "optimized default", take pictures of every screen and post it here. I'm sure, a lot of people down the road would appreciate that as well.

Another surprise: in HW monitoring menu the settings for the secondary fan appeared and they actually manage the attached fan. I still have no clue how to translate the setting into the actual behavior of the fan. I set

Start temp 42
Stop temp 40
Full seed temp 60
Starting PWM 5
Steep 2 PWM

which make the fan barely spin and then stop, and then again and again at the temp about 40C. I would like the fan slowly starts from ca. 40C and to ca. 60C has a full speed. Seems like the settings mean something else.
Box is also throttling in stress test at 60C, bringing frequencies down to ca. 2600 MHz, which may be due to the heat or the not "optimized settings"in BIOS. But I'll get there. Eventually.

The board run without any issue. I didn't test other NVMe slots since I have only two SSDs, and changing the randomly is tiresome, and I am lazy, and have no time for that, and... But I am pretty confident they work. Anyway, I decided to stay with the board. I even had a brief fantasy to install Intel ax210 in the freed slot, pass it through to bhyve with OpenWRT and finally run AP in the box as well. But then I discovered that it won't work while intel doesn't allow to use 5GHz & 6 GHz as an access point, at least on OpenWRT. Sad story.

So, finally the test outcomes. The setup is the same, I haven't even had to re-install FreeBSD after changing the board (see my previous post). So it's FreeBSD basics over SSH, vertically positioned box with small external fan, PCB 4xNVMe with two SSDs on slots 1 and 2; ambient t=28C.

fio 16 parallel 64KiB random write processes zroot
At start
nvme0 Temperature: 43 Celsius
nvme1 Temperature: 44 Celsius
CPU 42-43
At end
nvme0 Temperature: 48 Celsius
nvme1 Temperature: 49 Celsius
CPU 39
Results: WRITE: bw=419MiB/s (439MB/s), 24.6MiB/s-28.7MiB/s (25.8MB/s-30.1MB/s), io=24.8GiB (26.6GB), run=60494-60508msec

fio 16 parallel 64KiB random write processes zdata
At start
nvme0 Temperature: 44 Celsius
nvme1 Temperature: 49 Celsius
CPU 42-43
At end
nvme0 Temperature: 50 Celsius
nvme1 Temperature: 52 Celsius
CPU 44-45
Results: WRITE: bw=431MiB/s (452MB/s), 24.0MiB/s-30.2MiB/s (25.1MB/s-31.6MB/s), io=25.6GiB (27.5GB), run=60870-60878msec

fio Single 4KiB random write process zroot
At start
nvme0 Temperature: 51 Celsius
nvme1 Temperature: 54 Celsius
CPU 45-46
At end
nvme0 Temperature: 50 Celsius
nvme1 Temperature: 52 Celsius
CPU 47-48
Results: WRITE: bw=52.4MiB/s (54.9MB/s), 52.4MiB/s-52.4MiB/s (54.9MB/s-54.9MB/s), io=3145MiB (3298MB), run=60045-60045msec

fio Single 4KiB random write process zdata
At start
nvme0 Temperature: 50 Celsius
nvme1 Temperature: 52 Celsius
CPU 43-44
At end
nvme0 Temperature: 50 Celsius
nvme1 Temperature: 52 Celsius
CPU 44-45
Results
WRITE: bw=52.2MiB/s (54.8MB/s), 52.2MiB/s-52.2MiB/s (54.8MB/s-54.8MB/s), io=3135MiB (3287MB), run=60018-60018msec

Let me know if you would like to dive into details of the test outcomes, I edit the post

For me the test time is over. Hope to install an older version of OPPNsense today and then upgrade it in order to acquaint myself with the system, the procedure and to discover any possible pitfalls on the road.

Tortelli · Aug 6, 2023

Regarding cooling, i'm thinking about adding an 40mm or 80mm fan to my Topton N6005. But do normal fans work on the fan headers provided by topton? Read some comments that they were not normal?

numanumani · Aug 6, 2023

I got the barebones N100 from cwwk.net and have run into a problem that leaves me stumped.
Since I wanted to make the most out of Proxmox running on ZFS I connected two new WD WDS500G2B0C SN550 NVMe drives to the computer.

The problem is that one of the drives, specifically always the one connected with the H-adapter board, fails to get detected and initialised. When this happens the drive doesn't show up in BIOS and obviously not in Proxmox either. Typically a reboot is not enough to change anything but a shutdown and start often brings up the missing drive.

I have tried swapping places of the two NVMe's but the problem stays with whatever drive happens to be connected with the adapter board.

The PCIe settings for lane 1 (that I believe the working NVMe is in) are unchanged other than me trying to add a 3000 ms delay to see it it helps (it did not).
The uncooperative NVMe-slot appears to be lane 6, but that lane's PCIe settings aren't editable in the BIOS since it seems teamed with one of the other ones.

The RAM I use is Kingston FURY Impact SO DDR5-4800 C38, it passes Memtest86+.
I don't believe it's a temperature issue since it happens right at boot, besides SN550:s run cool, smartctl lists ~33C as a typical temperature.

I contacted CWWK directly the other day and am waiting for a reply.

Anyone here seen something similar?

MajorPayneDOF · Aug 7, 2023

I just purchased a HUNSN Amazon.com: HUNSN Micro Firewall Appliance, Mini PC, OPNsense, VPN, Router PC, Intel Alder Lake-N 12th Gen N100, RJ35, GPIO, TF Slot, HDMI, DP, 4 x 2.5GbE I226-V, 8G DDR5 RAM, 128G SSD : Tools & Home Improvement and will be getting it tomorrow supposedly. I will take pictures if anyone wants to see what is in the box and the insides of the device.

Donut7059 · Aug 7, 2023

Got my Amazon ordered cwwk n200 today , it’s the case featured in the latest sth video and came with the 4 nvme adapter board….guess I better order more drives now

waiting on ram to deliver before I can power it up

athurdent · Aug 8, 2023

The awesome people at HUNSN were kind enough to send me a fan kit.

Here we go...

There's a recess for the fan plug, never noticed that. Really good quality those things...

Fixing the fan

Went with auto mode for now. And I can only hear the fan if I put my ear next to the bottom of the device...

Has anyone found a way to read the fan speed in Debian? I was unsuccessful finding the sensor so far...

MajorPayneDOF · Aug 8, 2023

athurdent said:
The awesome people at HUNSN were kind enough to send me a fan kit.

Here we go...
View attachment 30794
There's a recess for the fan plug, never noticed that. Really good quality those things...
View attachment 30795
View attachment 30796
View attachment 30797
View attachment 30798
Fixing the fan
View attachment 30799
View attachment 30800
View attachment 30801
View attachment 30802
Went with auto mode for now. And I can only hear the fan if I put my ear next to the bottom of the device...
View attachment 30803
View attachment 30804

Has anyone found a way to read the fan speed in Debian? I was unsuccessful finding the sensor so far...

Is this your only nXXX device? I'm just curious on any differences you have seen

athurdent · Aug 8, 2023

MajorPayneDOF said:
Is this your only nXXX device? I'm just curious on any differences you have seen

Yep, so far only got that one.

NoNamesLeft · Aug 8, 2023

athurdent said:
Yep, so far only got that one.

Hi Athur, we have the same system except I purchased mine from Topton. I am using Opnsence on bare metal and I would like to check to see if my bios has the misconfigured port.

Besides doing iperf, what command did you use to show the pci-e lanes? Thanks for all of your help!

athurdent · Aug 8, 2023

NoNamesLeft said:
Hi Athur, we have the same system except I purchased mine from Topton. I am using Opnsence on bare metal and I would like to check to see if my bios has the misconfigured port.

Besides doing iperf, what command did you use to show the pci-e lanes? Thanks for all of your help!

Hi,

use dmesg | grep igc

If the first interface does not show the full 4.000 Gb/s available, then you need a newer BIOS I guess.

Code:

root@hunsn:~# dmesg | grep igc
[    1.016157] igc 0000:01:00.0: enabling device (0000 -> 0002)
[    1.016309] igc 0000:01:00.0: PTM enabled, 4ns granularity
[    1.064369] igc 0000:01:00.0 (unnamed net_device) (uninitialized): PHC added
[    1.091506] igc 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)
.
.
.

athurdent · Aug 8, 2023

Does automatic fan control work for anyone? I set the limit for full speed to 40 as a test, and while the system is at 44, the fan doesn’t speed up.

MajorPayneDOF · Aug 8, 2023

Is that fan 14mm? Is it a pwm?

athurdent · Aug 8, 2023

MajorPayneDOF said:
Is that fan 14mm? Is it a pwm?

80x80x10, it's a PWM. But it does not really work as configured in the BIOS. Just configured 40 - 50 - 60 as thresholds, and while it should not even be running, now it runs at 1200 instead of 800 with stamdard automatic settings. I'm at loss.

MajorPayneDOF · Aug 8, 2023

My HUNSN Device has arrived, I did get the 8GB ram and 128gb nvme. I will be swaping out the ram and drive later today but wanted to give you the screen shots now. Looks like the NVME comes with a heatsink.

MajorPayneDOF · Aug 8, 2023

athurdent said:
The awesome people at HUNSN were kind enough to send me a fan kit.

Here we go...
View attachment 30794
There's a recess for the fan plug, never noticed that. Really good quality those things...
View attachment 30795
View attachment 30796
View attachment 30797
View attachment 30798
Fixing the fan
View attachment 30799
View attachment 30800
View attachment 30801
View attachment 30802
Went with auto mode for now. And I can only hear the fan if I put my ear next to the bottom of the device...
View attachment 30803
View attachment 30804

Has anyone found a way to read the fan speed in Debian? I was unsuccessful finding the sensor so far...

My device looks very different from yours. Is yours a N200 or N305?

athurdent · Aug 8, 2023

MajorPayneDOF said:
My device looks very different from yours. Is yours a N200 or N305?

N100

CWWK/Topton/... Nxxx quad NIC router

New Member

Member

Member

Member

Member

New Member

New Member

Member

New Member

Member

Member

Member

New Member

Member

Member

Member

Member

Member

Attachments

Member

Member