CWWK/Topton/... Nxxx quad NIC router

[Becks0815]

Opnsense 23.1 is finally running after a longer fight:

Setup of 23.1 failed, I got constant errors about msdos_fs not accessible after the partition creation. According to google a common "feature" of FreeBSD13. So I started with 22.7 and went along the full upgrade road to 23.1_5

If you fail to install opnsense often enough, even 22.7 reacts with file copy problems/ partistion errors. Solution is to fire up a Gnome GPartd iso and remove all existing partitions and then start from scratch.

If you encounter read/write issues/crashes during installation, use default settings for PL1/PL2 CPU and package. No idea why this happens, maybe some heat spikes in the core leading to this. Have to monitor this a bit and see if it also happens during runtime.



Bios settings (at the moment):

PCIe Configuration: Port 1: ASPM auto L1 Substate: disabled L1 low disabled
PCIe Configuration: Port 7,9: ASPM auto, L1 Substate: disabled L1 low enabled
Port 10,11,12: disabled

-> don't enable L1 Sub or L1 low on port 1, it leads to drive read errors
-> don't enable L1 Substae on port 7,9 - the NIC become invisible/unusable
-> leave port 10,11 enabled if NIC #3,#4 are needed

SATA diasbled, USB HSII on xHCI disabled, HD audio disabled, Serial I2C5 controller disabled, eMMC 5.1 Controller disabled, Sensor Hub type None

CPU Config: Intel VMX disabled, PECI enabled
GT - Power Management: Maximum GT freq 100MHz

CPU Power management: Platform PL1: disable, PL2 disable
CPU Power management -> Turbo options -> Package Power PL1 = 6000, PL2 = 20000
-> need to run some more stability tests on the PL settings. They have no influence on the idle power consumption but define the upper limits of the CPU

unplug monitor after changing primary display in opnsense to serial. the system otherwise hangs during boot. Unplug the keyboard.



Tunables in opnsense:

hw.acpi.cpu.cx_lowest = C3
hw.ibrs_disable = 1
vm.pmap.pti = 0
dev.hwpstate_intel.0.epp = 99, dev.hwpstate_intel.1.epp =99, dev.hwpstate_intel.2.epp =99, dev.hwpstate_intel.3.epp = 99

-> hw.acpi.cpu.cx_lowest can be set down to C8, however I haven't seen any relevant power reduction below C3 and it takes longer to wake up a cpu from C8 than from C3, so setting C8 might result in some data transfer delays
-> dev.cpu.3.cx_lowest changes the same flag like hw.acpi.cpu.cx_lowest, so only one needs to be set


Power consumption, 1 NIC connected, opnsense idle and not filtering anything: 7W at the all

 

[fta]

If you encounter read/write issues/crashes during installation, use default settings for PL1/PL2 CPU and package.

It might be the power supply. They ship these things with junk power supplies, and I could reliably reproduce a crash at full power. With a quality power supply I have no issues.

 

[fagonzalezro]

It might be the power supply. They ship these things with junk power supplies, and I could reliably reproduce a crash at full power. With a quality power supply I have no issues.

Hello

Where i can find quality power supplys for the mini firewalls (with international shipping)?

Best Regards

 

[Becks0815]

Europe: Leicke 12V (on Amazon), or Meanwell GST90A12-P1M (digikey, Mouser).

My opnsense box died again. Looks like file corruption, and I fear FreeBSD has issues with the chipset, because Linux runs without issues. So now I am working on Proxmox and also had a problem - the X-server didn't start. Solution here is: Generic Solution when install gets FrameBuffer Mode Fails

- identify the VGA port ( 02:00.0 for me), then create /usr/share/X11/xorg.conf.d/ n100.conf:

Section "Device"
    Identifier "Card0"
    Driver "fbdev"
    BusID "pci0:02:0:0:"
EndSection

and start x-server again. Now I just have to learn how proxmox works

 

[Stovar]

Power consumption, 1 NIC connected, opnsense idle and not filtering anything: 7W at the all

Still impressive 7watt with that bios tweaking, with my cwwk 6 port nic and 5105 cpu still hit around 8.5-9watt idle with bare openwrt setup.

Would have liked to get the N100 but, don't think there will be much difference for my own user case as with many others but think many of us felt that way on the other big thread.

Still tempted to try an N300 to replace my pc, but will need to wait on them to ramp up production.

 

[alexandre.pitre]

Did anyone find a similar router with the latest CPUs with 6 NICs? Browsing aliexpress is total nightmare.

 

[TJFriday]

Watching where this goes... Many thanks for those that are sharing their experiences with these here.

Just ordered a N305 as I saw they're available.

 

[TXAG26]

I'd even be game for a 4 NIC mini with the latest CPU's.

 

[Becks0815]

Proxmox is running, Opnsense installed and also running, and I was able to use the 23.1 dvd iso for installation, something which didn't work with opnsense baremetal. And the whole system is stable, no kernel panics, no errors. So what I asume for now is that FreeBSD has issues with the new hardware used for the NVME drive part, resulting in this unstable/unusable situation, buzt I might be wrong.
It is not the power brick. I switched to a Leicke 65W 12V adapter for the tests today and still had problems under FreeBSD while a stress test under Linux with a much higher load and power consumption (up to 26W) didn't crash the system.

So for now I will look around for some more informtaion about Proxmox/Opnsense and also for some tunings, especially concerning power consumption.

 

[Stovar]

Did anyone find a similar router with the latest CPUs with 6 NICs? Browsing aliexpress is total nightmare.

Not seen any so far but yeah ali express is a nightmare as always though.

 

[Stovar]

Watching where this goes... Many thanks for those that are sharing their experiences with these here.

Just ordered a N305 as I saw they're available.

Which one did you go for?

I see topton, kingnovy and the usual suspects for around £268

nice to see there mentioning AV1 processing and also hdmi 2.1 in some units like this one

the beelink version is there also it mentions 9watts idle for the N305 24/7

See cwwk got the i305 unit also with nvme x4 and hdmi 2.1 also here

Id expect the N305 more then powerful enough to replace any desktop pc

 

[TJFriday]

Which one did you go for?

I went for a Topton from AliExpress. Was torn between that and the similar unit from CWWK. I chose AliExpress as it is likely to arrive sooner.

 

[Becks0815]

Opnsense is now running and used "in production" here at home. After the mess with bare metal I installed (multiple times, steep learning curve) Proxmox, pushed the kernel to 6.2.9 which also showed better power consumption on my J5040 and installed opnsens plus AdGuard home plugin as VM.

I decided to direct pass through two NICs and not use virtual ones and use a third one for the proxmox machine. Looks good



Power consumption: with around 1.5 MB/sec upload - around 8W. I have seen short spikes up to 11W, but the consumption is lower than during the first try with virtual NICs where I constantly had 11-13W.

CPU temp.:

root@proxmox:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +35.0°C  (high = +105.0°C, crit = +105.0°C)
Core 0:        +33.0°C  (high = +105.0°C, crit = +105.0°C)
Core 1:        +33.0°C  (high = +105.0°C, crit = +105.0°C)
Core 2:        +33.0°C  (high = +105.0°C, crit = +105.0°C)
Core 3:        +33.0°C  (high = +105.0°C, crit = +105.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1:        +27.8°C  (crit = +110.0°C)

nvme-pci-0100
Adapter: PCI adapter
Composite:    +41.9°C  (low  =  -5.2°C, high = +79.8°C)
                       (crit = +84.8°C)

A bit more data:
2MB/sec down, 13MB/sec up via torrent, 10W power consumption and 8% CPU load according to ProxMox. CPU 38°C all packages/cores looks like this is what the typical temp. looks like if the case is still cool and can store some energy.

9MB/sec down, 18 MB/sec up via torrent, 11W power consumption 8.2% CPU load.

 

[Becks0815]

Opnsense tunables which can be used if you use a processor type "host" and direct pass through NIC settings in Proxmox. I also have disabled all these Spctre etc. settings in the advanced section of the CPU on Proxmox.

hw.acpi.cpu.cx_lowest = C3 -> can be set, don't see any changes, but doesn't hurt
hw.ibrs_disable = 1
vm.pmap.pti = 0
dev.igc.0.fc =0, dev.igc.1.fc = 0, ....

New ones:
hw.igc.eee_setting = 0 - enables energy efficient NIC settings in the driver

Additional line in /etc/rc.conf
harvest_mask="351"

- The random data generator uses multiple sources as input, and two of them are interrupts and net data. The mask removes these two from the list. I haven't found details about security issues with a less good random seed, so...
more info: https://papers.freebsd.org/2018/asi...reebsd_for_routing_and_firewalling-slides.pdf

the latest one I am looking into is : Loader.conf.local tuning for modern hardware - especially hw.igc.rx_process_limit and hw.igc.tx_process_limit -set to 100 at the moment, but could be set higher without causing hick usps. As far as I understand, it defines how many packets per interrupt can be processed, and the low value was set to avoid the issues that an interrupt can be interrupted by another interrupt while the CPU is still processing the first data chunk. With fast CPU's this shouldn't be a real problem anymore.

And power stats. The high power consumption was caused by some stress tests with 30 torrents and a high connection limit. The system worked without problems and at the same time I also had better loading times while surfing on my notebook. The system can draw 11-12W easily under higher load. But please also note that I am down to a bit more than 5W while the system is idle, which is nice.

 

[alexw1982]

And power stats. The high power consumption was caused by some stress tests with 30 torrents and a high connection limit. The system worked without problems and at the same time I also had better loading times while surfing on my notebook. The system can draw 11-12W easily under higher load. But please also note that I am down to a bit more than 5W while the system is idle, which is nice.

View attachment 28516

Thanks for all the efforts you are putting into this! What software/hardware did you use to get those power graphs? I want that!

 

[Becks0815]

Thanks for all the efforts you are putting into this! What software/hardware did you use to get those power graphs? I want that!

I am monitoring the 12V power line of the machine using an ESP8266 and an INA219 sensor module. The ESP constantly collects the data and send the rolling average consumption, the maximum peak and the minimum every 10 seconds to IObroker via the MQTT protocol. Here the data is stored in a db. The rest is just a plugin in Iobroker called E-charts. It generates an automatically updated webpage with the data.

So you need to add maybe 25% to the power consumption when measuring at the wall and while the system runs below 10W and 15% above 10W. Here the power supplies have a higher efficiency, and this is btw. I am running two machines on the same 12V power supply. The combined consumption is higher than 10W. A simple way to reduce effective power consumption by 10%.

One of the power monitor boxes:

ESP8266, the INA219 in the upper left, a voltage converter 12V ->3.3V lower left - that's all. And the wiring is stupid simple. Just connect 3.3V plus and minus with both the ESP and the INA219, add two connecting cables between the INA and the ESP for the serial data (I2C bus) and write a small program using the Arduino IDE or Visual Studio Code. If you add the espota library, then you can update/reprogram the ESP over Wifi.

On top these things are cheap. A INA219 module is 1USD, the ESP starts at 3 USD and the bucket converter is at around 0.5 USD. Buy from the Samiore store on Ali, they are really good and ship fast.

 

[DustyExtender]

Can someone confirm whether the USB ports on these are 2.0 or 3.0?

The ports appear to be marked on the case with the ""SS" SuperSpeed logo but the specs I've seen on AliExpress contradict that and say they are only USB 2.0.

Although I'd be using it as a router/firewall, it's a deal breaker for me buying something in this day and age without USB 3.0 in case I use it in future for something else. (I'd also ideally like a box with a COM port and a BIOS that supports serial console redirection). So for that reason a N5105 still looks better than these for my use case.

If the USB ports are only 2.0 they shouldn't have that logo.....

 

[Becks0815]

Can someone confirm whether the USB ports on these are 2.0 or 3.0?

Both.

root@proxmox:~# lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 05e3:0751 Genesys Logic, Inc. microSD Card Reader
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

I have enabled hardware offload of TCP/UDP (settings in opnsense) now, something which is possible if you don't use Zenarmor or any other IDS. hw.igc.tx_process_limit =-1 is also set now, and after a reboot the system is not too bad, I must say.

5.3W power consumption idle, with proxmox and opnsense running and 3 NICs plugged in (2x for opnsense pass through and one for managing), and (*drum roll*): 8-9 W consumption while sending 15 MB/sec data across the router. That's 3-4W or almost 30-35% less than before the optimization, and the CPU core temps only reach 35-38°C under load while dropping to 28°C when idle:



red bar - before the last changes, running some torrents at around 8-12MB/sec
blue bar: changes applied. They don't have a dramatic influence while the system is idle.
green bar - checksum offload to the NIC and allowing an unlimited amount of packets per interrupt instead of 100 reduces the amount of interrupts, and the less interrupts we have, the more the CPU can spend on higher C-states. If you compare red and green, the difference is obvious.

 

[alexw1982]

8-9 W consumption while sending 15 MB/sec data across the router.

Just to clarify/ validate —> MB/s (Megabytes) or MBit/s (Megabits)

thanks again for all the testing & insights!

 

[Becks0815]

megabyte. 15mb/sec while taking 9w.