CWWK/Topton/... Nxxx quad NIC router

[athurdent]

Tried a different switch in between?

 

[stewart]

Tried a different switch in between?

I don't have a 2.5Gbe switch to test with. Connecting the 2 computers directly gives me roughly 8Gbps on iperf, though.

 

[stewart]

Resetting the BIOS didn't help.

 

[idle_user]

Has anyone mentioned or noticed that the m.2 slots don't all run at x4 lane speeds? (4GB/s = PCIe3.0x4)

I have 3 units that use have 2 NVMe each, and when checking `lspci -vvv` I notice that for each unit, one of the slots is downgraded to x1.

LnkSta: Speed 8GT/s (downgraded), Width x1 (downgraded)

This results in a max of about 1GB/s. And to add-on, the downgraded m.2's are the middle ones.


Can someone that has a single NVMe inserted post their `lspci -vvv` results for that slot?
You can limit results by specifying the address - mine were:

`lspci -vvv -s 01:00.0` and `lspci -vvv -s 06:00.0`

---

I wonder if there's BIOS settings available to configure it to max speeds


Edit: after researching more a bit, this might be due to power saving feature. Meaning that the downgrade is only temporary as the device doesn't need to run at full speed from my understanding. Can anyone confirm?

 

[idle_user]

Resetting the BIOS didn't help.

What pfsense version are you running? Last I heard, it didn't have i226 support, which is one of the reasons people use opnsense instead.

 

[stewart]

What pfsense version are you running? Last I heard, it didn't have i226 support, which is one of the reasons people use opnsense instead.

2.7.0CE. 2.6.0 had support for the older i225-v NICs. Support for the i226-v NICs came out with 2.7.0. But I don't think it's pfSense since Windows exhibits the same performance.

 

[xmancharl5]

New person here...

been using TOPTON N100 for a few months, works okay

Notice cisco umbrella has a thingy: change your DNS server address...

is this done with opnsense or what the heck?

Newbie here!!!

 

[Klez]

I purchased a HUNSN RJ38 and I can't seem to get full speed out of the NICs. Most every connection only gives me 1.6Gbps. I'm using 2 PCs with 10Gbe NICs to test with. The ports link at 2.5Gbps but I don't get those speeds. CPU usage is fine. I've tried running Windows on the HUNSN and pfSense to route the iperf between the 2 PCs. All the same. Anyone else having difficulties getting their units to hit the full 2.5Gbps? Any ideas on tweaks? I've contacted HUNSN and they instructed me to enable Turbo, but it comes on by default.

I've also done some performance testing to see the balance of power/speed of the CPU. If anyone wants to look it over I've posted my numbers here in the pfSense forums. I'll update it as I do more. The short of it is that ST maxes at 7W and MT maxes at 9W. 10W was slightly faster and anything higher slowly decreased performance (at least from a Passmark standpoint).

PL1- Default (6) = Passmark MT/ST = 4559/1854
PL1-7 = Passmark MT/ST = 5153/2095
PL1-8 = Passmark MT/ST = 5617/2084
PL1-9 = Passmark MT/ST = 6009/2083
PL1-10 = Passmark MT/ST = 6118/2081
PL2 was set to 0 and showed 25. That was default and I didn't change it.

I had similar issues under opnsense vm. Couldn't obtain more than 1,6g there. Tried different tuning settings posted here and there to make it multi threaded pf. Didn't help. Spawned vm with openwrt 23.05 and it was line rate from the get go

 

[idle_user]

Has anyone mentioned or noticed that the m.2 slots don't all run at x4 lane speeds? (4GB/s = PCIe3.0x4)

I have 3 units that use have 2 NVMe each, and when checking `lspci -vvv` I notice that for each unit, one of the slots is downgraded to x1.

LnkSta: Speed 8GT/s (downgraded), Width x1 (downgraded)

This results in a max of about 1GB/s. And to add-on, the downgraded m.2's are the middle ones.


Can someone that has a single NVMe inserted post their `lspci -vvv` results for that slot?
You can limit results by specifying the address - mine were:

`lspci -vvv -s 01:00.0` and `lspci -vvv -s 06:00.0`

---

I wonder if there's BIOS settings available to configure it to max speeds


Edit: after researching more a bit, this might be due to power saving feature. Meaning that the downgrade is only temporary as the device doesn't need to run at full speed from my understanding. Can anyone confirm?

After messing around a bit I think I can confirm (at least on my n305 units) that only one of the NVMe reads at x4 and the other is at x1.
So I advise those that are using a single NVMe to check your current link speeds. And if necessary swap that NVMe over to the x4 slot; its easy to think that the center/larger NVMe slot is the primary one to use. Gotta read the little print on slots to catch this.

Anyway, here's a quick image of my findings:




This is all due to the limitation of the PCIe Lanes from the CPU. There's a total 9 available.
i226: x4
m.2s: 4 + 1,

This is also the reason we don't have USB3.0 on these boards as well.


This also gets me wondering what speeds the new add-on boards (the one that you can add 4 NVMes to) will run at. There's definitely has to be a limitation there too right?

 

[Synthetickiller]

New to the forum.
I ordered the n100 version of the CWWK router (shipped from China directly, but bought through amazon). Neither OPNsense or PFsense see the wan.
My currently deployed i5 3570k pfsense rig sees my router/wan, so it's not a cable issue.

Are the i226-v nics still flaky despite the newly added support to pfsense 2.7.0 or the newest build of opnsense?
Any ideas to get this little guy up & running?

Thanks.

 

[sos_nz]

Has anyone bought this unit ?

I bought an older gen, but better-cased version recently: https://www.aliexpress.com/item/1005005593329224.html

It was cheaper than the newer versions, but the N100 works just fine, and on my bare-metal OPNsense box runs between 47 and 49C, and is brilliant.

My currently deployed i5 3570k pfsense rig sees my router/wan, so it's not a cable issue.

Are the i226-v nics still flaky

Snap. I had an i5-3470T before upgrading to the N100.

I run OPNsense 23.7 - and the i226's work just fine. No errors / dropped packets / crashes etc.

 

[moonman]

I bought an older gen, but better-cased version recently: https://www.aliexpress.com/item/1005005593329224.html

It was cheaper than the newer versions, but the N100 works just fine, and on my bare-metal OPNsense box runs between 47 and 49C, and is brilliant.

Thanks. It seems the Topton store is known to ship orders very slowly anyway so I decided to cancel that order and get this unit instead. 15 bucks more, but then I don't have to worry about cooling

 

[sos_nz]

It seems the Topton store is known to ship orders very slowly anyway so I decided to cancel that order and get this unit instead

Good luck. My order from the link I posted arrived 7 days from click to door. Of course YMMV!

 

[stewart]

I had similar issues under opnsense vm. Couldn't obtain more than 1,6g there. Tried different tuning settings posted here and there to make it multi threaded pf. Didn't help. Spawned vm with openwrt 23.05 and it was line rate from the get go

Interesting, but I've ruled out pfSense by installing Windows on a second SSD and testing. Same issue with both Operating systems.

 

[moonman]

Good luck. My order from the link I posted arrived 7 days from click to door. Of course YMMV!

Thanks. I suspect it will take substantially longer to deliver to Canada. Usually ~3 weeks. But I'm secretly hoping it will be here in 1 haha

 

[Devedse]

I just received mine . I was wondering how you guys all get these nice temperature graphs. It seems thatI can't even get the current temperature as the lm_sensors package seems to be unavailable to install on Proxmox 8.0.

 

[moonman]

I just received mine . I was wondering how you guys all get these nice temperature graphs. It seems thatI can't even get the current temperature as the lm_sensors package seems to be unavailable to install on Proxmox 8.0.

From my proxmox 8.0.4


Did you enable no-subscription repository for proxmox? It's disabled by default

Package Repositories - Proxmox VE

pve.proxmox.com

I believe the easiest way is in the GUI under <your instance name> -> Updates -> Repositories

 

[Devedse]

From my proxmox 8.0.4
View attachment 30974

Did you enable no-subscription repository for proxmox? It's disabled by default

Package Repositories - Proxmox VE

pve.proxmox.com

I believe the easiest way is in the GUI under <your instance name> -> Updates -> Repositories

I have these setup now:


This is the result of running a script a friend of mine gave me:

Proxmox VE Helper-Scripts | Scripts for Streamlining Your Homelab with Proxmox VE

Scripts for Streamlining Your Homelab with Proxmox VE

tteck.github.io

Proxmox VE Post Install Script

 

[moonman]

I have these setup now:

Welp then it should just be a matter of

apt update
apt install lm-sensors

 

[idle_user]

I just received mine . I was wondering how you guys all get these nice temperature graphs. It seems thatI can't even get the current temperature as the lm_sensors package seems to be unavailable to install on Proxmox 8.0.

As others said, just update and install lm-sensors. Proxmox is Debian for future reference.

As for the nice temp graphs, at least for me, is using prometheus/victoriametrics and grafana