Qotom Denverton fanless system with 4 SFP+

[Zaf9670]

Before me and others started contacting Qotom about the 3758R, they had basically stopped producing them since Qotom didn't think the price difference between the C3758 and C3758R made sense. Apparently they could source the C3758 quite a bit cheaper.

That's what they told me at least. That's probably still true. They might make a batch of them once in while to satisfy demand if they deem it to be large enough.

That's a bit suprising since OPNsense used to be faster at porting to newer FreeBSD versions, especially for the Community Edition releases. What you say appears to be true though, based on some quick googling.

Besides 200 MHz extra clock speed, they are basically identical.

Yeah I follow Lawrence Systems since they deploy plenty of PFsense boxes and that was something he brought up recently. Not sure if it's due to the Plus vs Community divide making it harder for OPNsense to get ahold of PF's homework or if the new BSD just needs more greybeards due to BSD popularity slowly diminishing. Even TrueNAS moving to a Linux build feels like a forecast of what to expect in 5-10 years so who knows.

I just got my RAM so I'm about to fire mine up. I'm also moving in a few weeks so I can easily make the "oh the network must be because of the new install" excuse my way through tinkering.

 

[blunden]

Yeah I follow Lawrence Systems since they deploy plenty of PFsense boxes and that was something he brought up recently. Not sure if it's due to the Plus vs Community divide making it harder for OPNsense to get ahold of PF's homework or if the new BSD just needs more greybeards due to BSD popularity slowly diminishing. Even TrueNAS moving to a Linux build feels like a forecast of what to expect in 5-10 years so who knows.

I just got my RAM so I'm about to fire mine up. I'm also moving in a few weeks so I can easily make the "oh the network must be because of the new install" excuse my way through tinkering.

Yeah, something clearly changed. Netgate was always contributing much more to FreeBSD than the team behind OPNsense.

Yes, we might not see any major open source projects being based on BSD in a few years. It kind of makes sense too since practically all company resources are put into Linux.

 

[sko]

Besides 200 MHz extra clock speed, they are basically identical.

The R variants also have more PCIe lanes. The C3558R and C3578R both have 20 lanes while their non-R counterparts only have 12 and 16 lanes:

Intel Atom® Processor C3758R (16M Cache, 2.40 GHz) - Product Specifications | Intel

Intel Atom® Processor C3758R (16M Cache, 2.40 GHz) quick reference with specifications, features, and technologies.

ark.intel.com

Intel Atom® Processor C3558R (8M Cache, 2.40 GHz) - Product Specifications | Intel

Intel Atom® Processor C3558R (8M Cache, 2.40 GHz) quick reference with specifications, features, and technologies.

ark.intel.com

Intel Atom® Processor C3758 (16M Cache, up to 2.20 GHz) - Product Specifications | Intel

Intel Atom® Processor C3758 (16M Cache, up to 2.20 GHz) quick reference with specifications, features, and technologies.

ark.intel.com

Intel Atom® Processor C3558 (8M Cache, up to 2.20 GHz) - Product Specifications | Intel

Intel Atom® Processor C3558 (8M Cache, up to 2.20 GHz) quick reference with specifications, features, and technologies.

ark.intel.com

IIRC for the non-R versions quotom added a notice that the B-key M.2 slot shares its lanes with one of the M-key slots, i.e. if you use a Wifi card you can only use one of the other M.2 slots for storage.

 

[blunden]

The R variants also have more PCIe lanes. The C3558R and C3578R both have 20 lanes while their non-R counterparts only have 12 and 16 lanes:

Intel Atom® Processor C3758R (16M Cache, 2.40 GHz) - Product Specifications | Intel

Intel Atom® Processor C3758R (16M Cache, 2.40 GHz) quick reference with specifications, features, and technologies.

ark.intel.com

Intel Atom® Processor C3558R (8M Cache, 2.40 GHz) - Product Specifications | Intel

Intel Atom® Processor C3558R (8M Cache, 2.40 GHz) quick reference with specifications, features, and technologies.

ark.intel.com

Intel Atom® Processor C3758 (16M Cache, up to 2.20 GHz) - Product Specifications | Intel

Intel Atom® Processor C3758 (16M Cache, up to 2.20 GHz) quick reference with specifications, features, and technologies.

ark.intel.com

Intel Atom® Processor C3558 (8M Cache, up to 2.20 GHz) - Product Specifications | Intel

Intel Atom® Processor C3558 (8M Cache, up to 2.20 GHz) quick reference with specifications, features, and technologies.

ark.intel.com

IIRC for the non-R versions quotom added a notice that the B-key M.2 slot shares its lanes with one of the M-key slots, i.e. if you use a Wifi card you can only use one of the other M.2 slots for storage.

I knew that there was a small difference in the number of PCI-E lanes but I was under the impression that they weren't being used currently. I guess I missed that note from Qotom.

Still, if used as a wired router (which appears to be the intended primary use case) they are essentially identical besides the 200 MHz higher CPU clock speed, as SSD performance makes no difference in that use case. If you have another use case, you are right that it could matter.

 

[sko]

I was under the impression that they weren't being used currently. I guess I missed that note from Qotom.

I asked them to confirm if this limitation is for both CPU variants, because I only found that note in the description of an item that only has the non-R variants for selection.
Turns out this limitation is true for all variants:

 

[blunden]

I asked them to confirm if this limitation is for both CPU variants, because I only found that note in the description of an item that only has the non-R variants for selection.
Turns out this limitation is true for all variants:
View attachment 37192

Thanks for getting the clarification!

 

[antst]

Did someone tried to use C3758R version as edge/gateway router (routing to WAN with firewall rules, NAT etc involved) with 10g WAN?
My testing with netgate 6100 (C3558) shows that pfSense(bare metal) is maxed out around 3.5Gbps and VyOS(bare metal) is maxed out around 5.5Gbps.
So, I estimate that under VyOS C3758R can sustain 10Gbps and under pfSense it will cap around 7Gbps.
Anyone has different numbers?

 

[blunden]

Did someone tried to use C3578R version as edge/gateway router (routing to WAN with firewall rules, NAT etc involved) with 10g WAN?
My testing with netgate 6100 (C3558) shows that pfSense(bare metal) is maxed out around 3.5Gbps and VyOS(bare metal) is maxed out around 5.5Gbps.
So, I estimate that under VyOS C3578R can sustain 10Gbps and under pfSense it will cap around 7Gbps.
Anyone has different numbers?

I haven't see too many such benchmarks for it, no (assuming C3578R was a typo for C3758R). Netgate has official benchmarks for the Netgate 8200 with the same SoC for both pfSense and TNSR (VPP) that I posted much earlier in this thread.

I'm curious about any VyOS performance tuning you did though, if any? Also any additional details about that comparison testing as I'm going to run VyOS on mine. I already ported over the configuration.

 

[antst]

I am aware of benchmarks and that TNSR is waaaay more capable. If they'd only have TNSR available for homelab...
Actually, I think I did nothing on VyOS, out of the box rolling 1.5.

Took me whole day to convert config from pfSense, at the end, I just wrote small program for conversion.

 

[blunden]

I am aware of benchmarks and that TNSR is waaaay more capable. If they'd only have TNSR available for homelab...
Actually, I think I did nothing on VyOS, out of the box rolling 1.5.

Took me whole day to convert config from pfSense, at the end, I just wrote small program for conversion.

Ok, good. Not everyone reads entire forum threads so I figured I should mention the benchmarks in case you had missed those links.

Yeah, unless VyOS decides to make their final VPP inplementation available there won't be any supported router/firewall OSes at this time that 't aren't fairly expensive unless you roll your own.

Ok, good to know. I'm surprised that the difference was that big to be honest.

In my case I compared it to my EdgeRouter 6P config. Since EdgeOS is based on an old version of Vyatta there were many similarities, even though there were also quite a few of differences as well. Still took hours to do.

 

[ZPrime]

Is there a list of known-good ECC RAM for these boards? The STH search doesn't appear to give me any way to search only this thread for that info, and it's 18 pages long...

 

[sko]

Is there a list of known-good ECC RAM for these boards? The STH search doesn't appear to give me any way to search only this thread for that info, and it's 18 pages long...

the list is in every product description for those systems/boards...

 

[ZPrime]

the list is in every product description for those systems/boards...

Yeah, and that list has one single part for ECC memory and it is difficult to find (I'm not turning it up on eBay searches at least). I assume that "the official list" is really just a handful of parts that Qotom have tested/attempted.

Was hoping maybe people have found other alternatives that work.

 

[Brent Geery]

Yeah, and that list has one single part for ECC memory and it is difficult to find (I'm not turning it up on eBay searches at least). I assume that "the official list" is really just a handful of parts that Qotom have tested/attempted.

Was hoping maybe people have found other alternatives that work.

Read a few pages back. You'll find both a source (newegg) for the "official" tested ECC RAM and user-tested alternatives that seem to work fine as well.

 

[sko]

I assume that "the official list" is really just a handful of parts that Qotom have tested/attempted.

That's true for such lists from all vendors. They only test a small fraction of parts/variants from what's available on the market - so you either stick with that (very) limited selection or take your own guesses from it. If a specific vendors parts are all/mostly listed as compatible, usually other variants from that vendor will work as well (and vice versa).
Usually, sticking with overall 'known good' vendors like e.g. micron will 'just work'™ - while especially with budget labels (where you can't be sure if 2 batches even have memory chips from the same manufacturer) are usually a hit and miss. This isn't exclusive to server or embedded systems, but for desktop/consumer gear as well.

Given the available selection of SODIMM ECC modules isn't terribly large - at least for europe there's basically only offerings for kingston, micron and transcend available - and micron or sk hynix (->kingston usually uses sk hynix on their server modules) are all listed as working, just stick with those two.
Basically the only one listed as not working is teamgroup, which falls under the "uses whatever was lying around in the factory" budget labels.
That being said - Samsung usually also *was* rather reliable and without complications, but the few modules that failed or showed incompatibilities in the last ~1-2 years for me were from Samsung...

 

[ZPrime]

Read a few pages back. You'll find both a source (newegg) for the "official" tested ECC RAM and user-tested alternatives that seem to work fine as well.

Thanks, I poked around and realized the search is able to search within a thread too, just have to click some options.
I was looking at one of those "Nemix" kits on Newegg, figured that if they sell it for Denverton-based ASUSTOR NAS boxes, it would probably work...

 

[scyto]

When i populate a cage, the interface disappears completely on reboot.

I found when i have any one or more of 3 of the SFPs populated the unit boots fine, but if the 4 port has an SFP in it then it fails to boot - if i populate that sfp port after boot all is fine.... i tried 3 different brands of SFP and nothing fixed it, i have decided to keep it as i only need 2 SFPs ports for what i really intend to do with it

did you solve your SFP issue?

 

[Techmaniac]

Hi all,
I read all 18 pages... you are great but I didn't found tests about the 3558R version which I'm planning to buy to lower TDP while cores are only 4 then not really suitable for proxmox/vmware esxi 7.x.
Then I decided to go for the 3758R Q20331G9 (without RAMand disks)
I'll add only 2 SFP+ DAC 10Gb for now on (as WAN and LAN ports).
I'll got the second SDD adapter and I'll modify it (thanks SlowmoDK and ccelebiler for soting out) to use 2 ssd I have with SW mirror.
Then I have to buy RAM for 64GB ECC...
The native BUS for this CPU is 2400MHZ, and for this pc max 64GB RAM.
In this post I can see a list of various tested solutions from 2400 to 3200 MHz and I don't know what to get for 64 GB ECC without spending more than for the minipc... can you help me choose?
I think I'll go with 2 of this.

Thanks for any advice you may have.

 

[sko]

I found when i have any one or more of 3 of the SFPs populated the unit boots fine, but if the 4 port has an SFP in it then it fails to boot - if i populate that sfp port after boot all is fine.... i tried 3 different brands of SFP and nothing fixed it, i have decided to keep it as i only need 2 SFPs ports for what i really intend to do with it

did you solve your SFP issue?

can't confirm that issue:

# ifconfig -v | grep ix.: -A9
ix0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=48538b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
        ether 20:7c:14:f3:e3:17
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
        vendor: FS PN: SFP-10GSR-85 SN: F2210814302 DATE: 2022-07-29
        module temperature: 49.67 C voltage: 3.26 Volts
        lane 1: RX power: 0.58 mW (-2.36 dBm) TX bias: 6.86 mA
ix1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 20:7c:14:f3:e3:18
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
        vendor: FS PN: SFP-10GSR-85 SN: F2130315175 DATE: 2021-12-31
        module temperature: 47.43 C voltage: 3.33 Volts
        lane 1: RX power: 0.57 mW (-2.41 dBm) TX bias: 6.86 mA
ix2: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 20:7c:14:f3:e3:19
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
        vendor: FS PN: SFP-10GSR-85 SN: F2130315176 DATE: 2021-12-31
        module temperature: 49.91 C voltage: 3.37 Volts
        lane 1: RX power: 0.56 mW (-2.54 dBm) TX bias: 6.82 mA
ix3: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 20:7c:14:f3:e3:1a
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
        vendor: FS PN: SFP-10GSR-85 SN: F2130315178 DATE: 2021-12-31
        module temperature: 49.91 C voltage: 3.33 Volts
        lane 1: RX power: 0.64 mW (-1.96 dBm) TX bias: 6.91 mA

edit: this is on the 3758 variant:

# sysctl hw.model
hw.model: Intel(R) Atom(TM) CPU C3758 @ 2.20GHz

 

[voltagex]

Anyone know if these will go below 50W? It seems like none of the power management features work? What am I doing wrong?