Qotom Denverton fanless system with 4 SFP+

[John T Davis]

It's a 10nm part at least, But really, QuickAssist pushes the C3758R's performance to about ~20-25% higher than the N6005 in encryption and compression.

How CPU or RAM gated are those Proxmox backups? Sounds like an expensive thing to do.

It's a ZFS-based OS/storage install (the pool is a single mirror vdev), so it enjoys using as much RAM as it can get for ARC. That machine has 64 GiB of DDR4-SODIMM laptop RAM, and uses about half of that for cache at idle. I need to watch it when it's actually working and see how much more it uses to actually back up.

I suspect the bottleneck there that's keeping it from using more RAM is at least partly the 2.5 Gbps NIC. If I had a 10 GbE NIC on that machine, I suspect it'd cache harder.

I just ran a manual backup to the Proxmox Backup Server after I just rebooted PBS, and it barely used 1 GiB of RAM on PBS itself. But it also didn't really have to do much of anything since it backs up every day at midnight and those VMs store very little data that changes every 24 hours. I'll need to create a new VM or two and watch it while it's doing the first backup of those.

That said, PBS-based encryption is done client-side. Encrypted data is sent to the server, so the server itself never has to do encryption operations. Most of the CPU horsepower would be going to chunking the data and compression (since it's ZFS-based and compression is enabled).

 

[siematos]

Oh, let me clarify what i mean with RAM gated: It sounded like an in-memory copy-compare, you got to safe the blocks somewhere. That's not necessarily expensive capacity wise, but just an expensive thing to do from a compute/programming perspective, gating it latency wise.

That said, PBS-based encryption is done client-side. Encrypted data is sent to the server, so the server itself never has to do encryption operations.

I'd assume, that network traffic still is encrypted. That's not just for privacy but security, verifying the client/server authenticity. That part is accelerated, too.

 

[John T Davis]

Oh, let me clarify what i mean with RAM gated: It sounded like an in-memory copy-compare, you got to safe the blocks somewhere. That's not necessarily expensive capacity wise, but just an expensive thing to do from a compute/programming perspective, gating it latency wise.

Oh! That makes sense. Thanks for clarifying.
Yeah, that's probably not the fastest RAM ever. It's DDR4-2933 MT/s in a dual channel config.
I never really intended for this to be a backup server. It just happened to be the best usable hardware I had when I realized that I really wanted to deploy PBS when I started getting into Proxmox.

I'd assume, that network traffic still is encrypted. That's not just for privacy but security, verifying the client/server authenticity. That part is accelerated, too.

Ah. Yes, that's definitely encrypted.

And ZFS on PBS uses ZFS' default compression by default to store data.

 

[John T Davis]

Sorry to double-reply, @siematos , but I realized there was one more thing I wanted advice on to close the loop on this.

In that case, let me compare both the N6005 and the C3758R for you: from the pure core-per-core perspective, the C3758R should be ~half as powerful, but it's got twice the cores. What really moves the needle is hardware acceleration. Intel QuickAssist gives acceleration for encryption and compression tasks, so in these workload categories, the cores punch way above their weight compared to the N6005. I don't know that much about Proxmox, but from a pure backup perspective, you typically have some write cache in front of HDDs, and with a 7200RPM HDD, you can write with like 300-400+ MB/s to a ZRAID10. That's why I was surprised.

My current OPNSense box is an i5-6500. How does that CPU compare to the Atom C3758R?

I don't know if any of these X vs. Y CPU comparison sites are actually correct, but I did find this:

https://www.cpu-world.com/Compare/195/Intel_Atom_C3758R_vs_Intel_Core_i5_i5-6500.html

Looks like the C3758R wins on number of cores (8 vs 4), TDP (by 2.7x: 26w vs 65w), but not clock frequency. Then again, I suspect that the C3758R might have a higher IPC.

 

[mrpops2ko]

i cant wait until we get some QAT gen 3 for reasonable pricing so we can do wireguard acceleration

 

[John T Davis]

i cant wait until we get some QAT gen 3 for reasonable pricing so we can do wireguard acceleration

Which CPUs have gen 3 QAT?

 

[blunden]

i cant wait until we get some QAT gen 3 for reasonable pricing so we can do wireguard acceleration

Unfortunately the later Atom generations where QAT might be included are too power hungry to be useful in applications like this, last time I checked.

 

[siematos]

Unfortunately the later Atom generations where QAT might be included are too power hungry to be useful in applications like this, last time I checked.

Yessss, C5000 processors essentially run on 41W+ with the exception of a single 4-core at 32W and 1.6Ghz base freq. But you get QAT AND 8 ethernet ports with 100gbit routing capability in 32W, so I think it's fairly good tbh. Sub-50W for a full 8port 10gbit router? It's pretty good tbh. We're not talking about switches here.

Just make sure to get the C53xx series if you need the network ports. C51xx don't have the full capabilities

 

[blunden]

Yessss, C5000 processors essentially run on 41W+ with the exception of a single 4-core at 32W and 1.6Ghz base freq. But you get QAT AND 8 ethernet ports with 100gbit routing capability in 32W, so I think it's fairly good tbh. Sub-50W for a full 8port 10gbit router? It's pretty good tbh. We're not talking about switches here.

Just make sure to get the C53xx series if you need the network ports. C51xx don't have the full capabilities

Do you have a link to some benchmarks?

 

[siematos]

Do you have a link to some benchmarks?

No, but I vaguely remember the STH post a few years ago.

looked it up for you: STH C5000+P5000

QAT speed:

Atom C5000 SKUs integrate QAT Gen 2 capable of 10Gbps look-aside encryption throughput for the 4-core SKUs; 20Gbps for the 8-core SKUs.
All Atom P5300 and P5700 SKUs integrate QAT Gen 3 with 100Gbps crypto throughput; P5700 SKUs offer both inline and lookaside. (Source: Intel e-mail to STH)

Turns out i remembered it wrongly, switching capacity is 'only' 50gbit 'throughput' for the C5000 models. But that's still more than 2.5 times as fast as the C3758R The integrated switches are in the P series, and the P5742 would offer 200gbit switching at 67W

 

[Harddisk9144]

Bought 2 Q20331G9-S10 models 1 month apart from each other ~11 months ago.
This week, the box I bought first, experienced a random freeze and 2 shutdowns.
Today it wouldn't power on anymore, the ethernet interface still flickers when power is supplied.

Unplugging the CMOS battery as suggested in this thread got it powering on again, currently monitoring it to see if the random shutdowns still occur or not.
Going to be buying some CMOS batteries, I'll assume the same will happen to my second box sooner or later. I will check on whether I can actually measure CMOS battery drainage.

Welp... It's been a month and a bit now, yesterday I got the random shutdown and I can't boot it back up.
Gonna remove the CMOS battery again to see if it turns on again, but it seems the replacement wasn't the solution or only a temporary solution.

 

[blunden]

Welp... It's been a month and a bit now, yesterday I got the random shutdown and I can't boot it back up.
Gonna remove the CMOS battery again to see if it turns on again, but it seems the replacement wasn't the solution or only a temporary solution.

What power supply are you using for it? Maybe that is going bad?

 

[Harddisk9144]

What power supply are you using for it? Maybe that is going bad?

Good idea, I currently use the one that came with it (Dajing ADP-60e2).
Swapped it out with another adapter that has the same ratings, no reaction sadly.
1 month ago, removing the CMOS battery made it boot up again, but now it's not turning on with or without it.
I have zero experience/tools to troubleshoot what might be a dead motherboard. I think I have to buy a multimeter and learn how to use it.

 

[blunden]

Good idea, I currently use the one that came with it (Dajing ADP-60e2).
Swapped it out with another adapter that has the same ratings, no reaction sadly.
1 month ago, removing the CMOS battery made it boot up again, but now it's not turning on with or without it.
I have zero experience/tools to troubleshoot what might be a dead motherboard. I think I have to buy a multimeter and learn how to use it.

Ok. It was worth a try.

You could try contacting Qotom to see if they have any suggestions.

 

[Harddisk9144]

Ok. It was worth a try.

You could try contacting Qotom to see if they have any suggestions.

Got a friend of mine to take a look at it, the fuse near the power button was fried (he suspects low quality fuse as he didn't see obvious signs of a short circuit), he replaced it and we're up and running again.

 

[siematos]

Got a friend of mine to take a look at it, the fuse near the power button was fried (he suspects low quality fuse as he didn't see obvious signs of a short circuit), he replaced it and we're up and running again.

Can you tell, which revision the board is? Mine arrived and were manufactured mid april as a v1.3 - judging from the case rework, i thought there might be a board overhaul as well.

My current OPNSense box is an i5-6500. How does that CPU compare to the Atom C3758R?

I don't know if any of these X vs. Y CPU comparison sites are actually correct, but I did find this:

https://www.cpu-world.com/Compare/195/Intel_Atom_C3758R_vs_Intel_Core_i5_i5-6500.html

Looks like the C3758R wins on number of cores (8 vs 4), TDP (by 2.7x: 26w vs 65w), but not clock frequency. Then again, I suspect that the C3758R might have a higher IPC.

Sorry, i somehow missed this question. That one is highly workload dependent. In compression/encryption tasks, the C3758R should be slightly faster (maybe 10-15%) due to QAT+core count. Single core, the i5 should be absolutely slaughtering the C3758R with at least twice the performance, I'd say.

So it depends on what you actually do on the OPNSense box, but there's a good chance, that you get more performance out of it with a C3758R, because routing/firewall/vpn/haproxy/nginx-reverse-proxy workloads are accelerate-able, while workloads like serving dynamic web pages (NodeJS/PHP) should be like 2 times faster with the i5 due to the massive single core advantage for non-accelerated workloads. I hope that answers your questions.

 

[blunden]

Got a friend of mine to take a look at it, the fuse near the power button was fried (he suspects low quality fuse as he didn't see obvious signs of a short circuit), he replaced it and we're up and running again.

Interesting. Could the failure of the fuse be power supply related (improper filtering or something), or is the use of a low quality fuse the most likely scenario?

Was it easily replaceable or did you have to solder it?

 

[siematos]

Interesting. Could the failure of the fuse be power supply related (improper filtering or something), or is the use of a low quality fuse the most likely scenario?

Was it easily replaceable or did you have to solder it?

It has to be soldered. It's a 15A surface soldered fuse in board revision V1.3, doesn't look cheap to me. The 1u MeanWell PSU puts out 12v@4.2A, the external PSU produces 12V@5A.

 

[blunden]

It has to be soldered. It's a 15A surface soldered fuse in board revision V1.3, doesn't look cheap to me. The 1u MeanWell PSU puts out 12v@4.2A, the external PSU produces 12V@5A.

Ah, this was the rackmount version. That should have a pretty good PSU. I was thinking of the desktop version and its Dajing PSU.

 

[siematos]

Ah, this was the rackmount version. That should have a pretty good PSU. I was thinking of the desktop version and its Dajing PSU.

Yeah, I don't have the equipment to test that. I can ask in our local hacker space, maybe somebody's willing to look into the quality of that thing, but an answer might take 2-3 weeks.