Qotom Denverton fanless system with 4 SFP+

[Maunose]

Just as an FYI, I repasted the CPU and it seems the paste they use is garbage cause standard thermal paste from a noctua heatsink dropped temps by 10c.

Just repasted mine yesterday and got 5c off, mine is the 1U unit with Atom C3808 and it runs about 45 degrees on idle now.

Changing topic:

Does anyone runs these systems as 10Gbit firewall? I installed pfsense+ on mine, betting it would be enough for my 10Gbit WAN but I'm experiencing all kinds of issues, for example:
- My proxmox host when updating apt repositories or downloading updates sometimes doesn't go past 400KB/s, some times 900KB/s and some times full speed. I can't go past 6Gbps and to be honest, most of the time I'm only seeing 3Gbps on speedtests (tested with another router and it gets 8Gbps as expected from a XGSPON) but CPU doesn't go past 60%.
- I'm also experiencing a lot of IP input drops (net.inet.ip.intr_queue_drops), even though my queue is at 4000 already (It says I have 140791 drops now with an uptime of 17 hours).

I'm gonna give OPNSense a try, but I don't think it will make much diference. I was looking at VyOS but when I saw it doesn't support DHCP6-PD (RA) on WAN I gave up on it. Gonna take a look at TNSR 23.06 (found an ISO somewhere, don't ask) but I don't think it will support DHCP6-PD as well.

 

[VivienM]

Just repasted mine yesterday and got 5c off, mine is the 1U unit with Atom C3808 and it runs about 45 degrees on idle now.

Changing topic:

Does anyone runs these systems as 10Gbit firewall? I installed pfsense+ on mine, betting it would be enough for my 10Gbit WAN but I'm experiencing all kinds of issues, for example:
- My proxmox host when updating apt repositories or downloading updates sometimes doesn't go past 400KB/s, some times 900KB/s and some times full speed. I can't go past 6Gbps and to be honest, most of the time I'm only seeing 3Gbps on speedtests (tested with another router and it gets 8Gbps as expected from a XGSPON) but CPU doesn't go past 60%.
- I'm also experiencing a lot of IP input drops (net.inet.ip.intr_queue_drops), even though my queue is at 4000 already (It says I have 140791 drops now with an uptime of 17 hours).

I'm gonna give OPNSense a try, but I don't think it will make much diference. I was looking at VyOS but when I saw it doesn't support DHCP6-PD (RA) on WAN I gave up on it. Gonna take a look at TNSR 23.06 (found an ISO somewhere, don't ask) but I don't think it will support DHCP6-PD as well.

I'm running mine as a 10 gigabit firewall running opnsense. My WAN is 8 gigabits, no PPPoE. I think I've gotten 7.x gigabit speedtest.nets from my iMac with a 10 gigabit port. Rarely see speeds that high in practice, but it's hard to tell what the bottleneck might be - my proxmox hosts with crossflashed Intel 710 cards, for example, are... older...

But I have the C3758R.

I should note that I had very disappointing results with the first 10GBaseT SFP+ I tried (my ISP's ONT is 10G copper). Mad error rates. The second one has been much better.

 

[Barrette]

Just as an FYI, I repasted the CPU and it seems the paste they use is garbage cause standard thermal paste from a noctua heatsink dropped temps by 10c.

Thank you for this, went from 41C at idle to 27C by replacing the paste with some MX4 I had laying around.

 

[blunden]

Does anyone runs these systems as 10Gbit firewall? I installed pfsense+ on mine, betting it would be enough for my 10Gbit WAN but I'm experiencing all kinds of issues, for example:
- My proxmox host when updating apt repositories or downloading updates sometimes doesn't go past 400KB/s, some times 900KB/s and some times full speed. I can't go past 6Gbps and to be honest, most of the time I'm only seeing 3Gbps on speedtests (tested with another router and it gets 8Gbps as expected from a XGSPON) but CPU doesn't go past 60%.
- I'm also experiencing a lot of IP input drops (net.inet.ip.intr_queue_drops), even though my queue is at 4000 already (It says I have 140791 drops now with an uptime of 17 hours).

I'm gonna give OPNSense a try, but I don't think it will make much diference. I was looking at VyOS but when I saw it doesn't support DHCP6-PD (RA) on WAN I gave up on it. Gonna take a look at TNSR 23.06 (found an ISO somewhere, don't ask) but I don't think it will support DHCP6-PD as well.

I do, running VyOS on the C3758 (non-R model). I can routinely get over 9 Gbps in speed tests each way. Also, I got about 15-16 Gbps in a quick bidirectional test using iperf3 using multiple streams, if I recall correctly. That was just an internet facing test too, so it was possibly limited by the other host. These are all with software flow offloading enabled.

Maybe try a Linux based OS next, just to rule out BSD OS or drivers being an issue.

I have no issues receiving working IPv6 on VyOS with an ISP using DHCPv6-PD on VyOS, but perhaps you meant running a DHCPv6-PD server for a downstream device?

 

[lastditch]

Alright next thing. Ive got it setup and repasted but the temps now sit at the spot where the fan will ramp up or down every few minutes, anyone know of a way to adjust the fan curve on these? Or does anyone have info on how to hookup another 40mm fan off the board?

 

[lastditch]

Also fun fact, was replugging things in and one of the ports I was working with just went dark. I could not get it come back up with a reboot or anything other than pulling power and reinserting, then it came back online no problem. Not sure why.

 

[Maunose]

I do, running VyOS on the C3758 (non-R model). I can routinely get over 9 Gbps in speed tests each way. Also, I got about 15-16 Gbps in a quick bidirectional test using iperf3 using multiple streams, if I recall correctly. That was just an internet facing test too, so it was possibly limited by the other host. These are all with software flow offloading enabled.

Maybe try a Linux based OS next, just to rule out BSD OS or drivers being an issue.

I have no issues receiving working IPv6 on VyOS with an ISP using DHCPv6-PD on VyOS, but perhaps you meant running a DHCPv6-PD server for a downstream device?

There's definitely something wrong with pfSense's ix drivers. I installed VyOS as suggested and it pushes 8Gbps without breaking a sweat, CPU usage is less than 10% while speedtest runs full speed.
I'll have to spend some time to get to know how the firewall rules works but I'll keep VyOS as my router OS for sure!
Thanks for the tip.

 

[blunden]

There's definitely something wrong with pfSense's ix drivers. I installed VyOS as suggested and it pushes 8Gbps without breaking a sweat, CPU usage is less than 10% while speedtest runs full speed.
I'll have to spend some time to get to know how the firewall rules works but I'll keep VyOS as my router OS for sure!
Thanks for the tip.

That's great to hear! I wonder what the underlying issue is.

Make sure to set up software flow offloading to keep CPU usage low at high speeds after you implement your firewall rules.

 

[CaptainLegot]

I just tried the Nemix 64GB ECC memory kit (NEMIX RAM 64GB (2X32GB) DDR4 2666MHZ PC4-21300 2Rx8 1.2V 260-PIN ECC Unbuffered SODIMM Memory KIT - Newegg.com) on mine, as well as a mem-store 64gb kit (both 2x 32GB) and neither would POST in either of my two C3808 models 1u. They work with some random Micron 16GB ECC sodimms though. Also one of my fans died after a week so that's cool.

 

[mdedetrich]

So I am running bare metal OpnSense with ECC micron SO-DIMM memory and its working like a charm (detected as ECC memory both in Windows which I temporarily installed to update firmware and then FreeBSD)

Specifically I have MTA18ASF4G72HZ-3G2R

 

[TechUnsupport]

What exactly is the need that you guys are using 2x32GB on this unit for anyway? I personally feel that 64GB is a bit excessive for the CPU or type of work this CPU can do. Again, the SK Hynix 2x16GB ECC seem to be the best bang for the buck for this. It's only like $10 more than 1x16GB Cruicial/Micron ECC. That if you don't need 64GB RAM.

 

[VivienM]

(detected as ECC memory both in Windows which I temporarily installed to update firmware and then FreeBSD)

This caught my eye... which firmware were you updating?

 

[CaptainLegot]

What exactly is the need that you guys are using 2x32GB on this unit for anyway? I personally feel that 64GB is a bit excessive for the CPU or type of work this CPU can do. Again, the SK Hynix 2x16GB ECC seem to be the best bang for the buck for this. It's only like $10 more than 1x16GB Cruicial/Micron ECC. That if you don't need 64GB RAM.

ZFS ARC will use something like 50% of free RAM in the default configuration, and that is particularly useful when running decently sized pools of spinning disks. I wouldn't consider 64GB on a 12 core device particularly excessive even if they are slower cores.

 

[kng42]

Just repasted mine yesterday and got 5c off, mine is the 1U unit with Atom C3808 and it runs about 45 degrees on idle now.

Changing topic:

Does anyone runs these systems as 10Gbit firewall? I installed pfsense+ on mine, betting it would be enough for my 10Gbit WAN but I'm experiencing all kinds of issues, for example:
- My proxmox host when updating apt repositories or downloading updates sometimes doesn't go past 400KB/s, some times 900KB/s and some times full speed. I can't go past 6Gbps and to be honest, most of the time I'm only seeing 3Gbps on speedtests (tested with another router and it gets 8Gbps as expected from a XGSPON) but CPU doesn't go past 60%.
- I'm also experiencing a lot of IP input drops (net.inet.ip.intr_queue_drops), even though my queue is at 4000 already (It says I have 140791 drops now with an uptime of 17 hours).

I'm gonna give OPNSense a try, but I don't think it will make much diference. I was looking at VyOS but when I saw it doesn't support DHCP6-PD (RA) on WAN I gave up on it. Gonna take a look at TNSR 23.06 (found an ISO somewhere, don't ask) but I don't think it will support DHCP6-PD as well.

So I've tried bare metal OPNsense/pfSense on the C3808 and the best I could manage for moving data from my desktop to my server was ~250MB/s on the 10Gbps connection, I tried a DAC and optical... usually only ~100 errors per iperf3 test. If you do the -P 4 on iperf 3 or greater you can get a SUM of 9.5Gbps. The story I've pieced together is BSD is 1 session = 1 CPU core, and 2.5Gbps is what each denverton core can handle (at least under BSD).
Virtualization in Proxmox was worse.... I was checking if network card drivers were at fault.

Solution for now was OpenWRT x64. Transfers are the full 9.5Gbps between PC's, it has unbound/wire-guard/VLAN's etc.

 

[solawind]

I quite like this device , i am currently using a pair of them for my friend NPO they can handle hundred users office network with ease providing internet sharing, basic network services, vpn etc . pretty stable so far but i have one on-site as a replacement in case the first one fails. nas box connected via 10g port and another one for internet uplink

it also has full serial console redirection which is a huge plus when device is operating in a closed room

 

[blunden]

So I've tried bare metal OPNsense/pfSense on the C3808 and the best I could manage for moving data from my desktop to my server was ~250MB/s on the 10Gbps connection, I tried a DAC and optical... usually only ~100 errors per iperf3 test. If you do the -P 4 on iperf 3 or greater you can get a SUM of 9.5Gbps. The story I've pieced together is BSD is 1 session = 1 CPU core, and 2.5Gbps is what each denverton core can handle (at least under BSD).
Virtualization in Proxmox was worse.... I was checking if network card drivers were at fault.

Solution for now was OpenWRT x64. Transfers are the full 9.5Gbps between PC's, it has unbound/wire-guard/VLAN's etc.

Yes, performance on BSD seems much worse than on Linux. It performs well for me on VyOS as well as a 10 Gbps router.

 

[Pheckphul]

My CWWK-made 6-port one came with the same brand PSU as this STH review: "Dajing".
Probably "Dajing" brand is pretty popular in the multiple-port PC stores.

I just bought two Mean Well replacements, with one as a spare. In my experience with electronics, PSUs are one of the more failure-prone components, and sleep better when I use a decent brand.

 

[Tismo]

There's something really wrong with this "fix" they applied... I just got a xgs-pon stick today (WAS-110 based, to try to bypass my ISP ONU) and while I was testing it, I tried plugging it to the werid port and guess what? It's up... although it doesn't show likn speed, it's up and I can connect to it. If I connect it to another SFP+ port it shows as a 10GBase-LR media.

View attachment 39943

Tests before were done using 10Gtek 10GBase-SR and 10Gbase-T (80 meters version), which both worked on the other 3 ports, but not on ix3 (Port 9)

Thinking about getting the WAS-110 module from 8311 discord. I am planning of removing the BGW320-500 and plugging straight into the rack mount version of qotom. Will the sfp+ work ok on it fine from your experience?

 

[blunden]

I was browsing around the Qotom website checking if they have an SFP+ version of one of their other boxes that I was interested in when I found a different model based on the same motherboard as the one this thread is about.

https://www.qotom.com/products/show/MinI-PC-Q20300G9-S20-Series

Basically, it seems like same motherboard but a different case (S10 vs. S20). I have no idea whether it's meaningfully different, but just found it interesting.

Pricing on AliExpress seems to be very similar.

 

[Barrette]

It would be interesting to see how the heatsink differs from the previous model.