Qotom Denverton fanless system with 4 SFP+

[foureight84]

pulled the nvme until i can restore some console
whats available in "legacy"? haven't seen a board not boot into bios before other than a usual bricking nightmare; i hadn't gotten that far in a mod, not certain if i deliberately disabled CSM...how would it have 'dropped' into UEFI?

for now,RTC battery yanked and chassis unpowered, but no idea if this'll reset anything

Just checking because I made that mistake after updating the bios. You would have had to look for it to change it, it's under the Advanced options I believe and wouldn't be easy to change by mistake.

The one thing I noticed about these unit, not sure if it would help, sometimes warm reboots will just hang and won't post. Then forcing shut off and cold booting will cause it to hang for a good 5 minutes before posting. But I don't think you have this problem.

 

[foureight84]

It's useful for IPsec and OpenVPN DCO as far as I know (faster than AES-NI), but the version of QAT in the Intel Atom C3000 series doesn't support the ciphers used by Wireguard.

The C3000 QAT seems very limited after looking more into it. Intel doesn't even support it in their newer drivers.

 

[blunden]

The C3000 QAT seems very limited after looking more into it. Intel doesn't even support it in their newer drivers.

It's a platform originally from 2017. It supports the most important ciphers/algorithms from that time and is really powerful if it matches your use case, but naturally lacks some modern features.

I would expect the drivers to be stable at this point so I don't see that as much of an issue. Not sure why you're having problems in Proxmox. Perhaps it doesn't include the QAT driver?

 

[foureight84]

It's a platform originally from 2017. It supports the most important ciphers/algorithms from that time and is really powerful if it matches your use case, but naturally lacks some modern features.

I would expect the drivers to be stable at this point so I don't see that as much of an issue. Not sure why you're having problems in Proxmox. Perhaps it doesn't include the QAT driver?

I am able to enable SR-IOV for it but it doesn't show up in the PCI list unlike the x553 NICs. I tried to compile drivers but ran into a compilation error that got me stuck. But I think the drivers should already be there.

root@pve-gateway:/usr/local# lsmod | grep qat
qat_c3xxxvf            12288  0
qat_c3xxx              12288  0
intel_qat             425984  2 qat_c3xxxvf,qat_c3xxx

root@pve-gateway:/usr/local# dmesg | grep qat
[   12.702202] c3xxx 0000:01:00.0: qat_dev0 started 6 acceleration engines
[   74.653342] c3xxx 0000:01:00.0: qat_dev0 stopped 6 acceleration engines
[   74.655143] c3xxx 0000:01:00.0: Resetting device qat_dev0
[   74.884148] c3xxx 0000:01:00.0: qat_dev0 started 6 acceleration engines

root@pve-gateway:/usr/local# grep qat /proc/crypto
driver       : qat_deflate
module       : intel_qat
driver       : qat-dh
module       : intel_qat
driver       : qat-rsa
module       : intel_qat
driver       : qat_aes_cbc_hmac_sha512
module       : intel_qat
driver       : qat_aes_cbc_hmac_sha256
module       : intel_qat
driver       : qat_aes_cbc_hmac_sha1
module       : intel_qat
driver       : qat_aes_xts
module       : intel_qat
driver       : qat_aes_ctr
module       : intel_qat
driver       : qat_aes_cbc
module       : intel_qat

 

[blunden]

Hmm, not sure then. I used mine bare metal and never bothered with QAT since I use Wireguard.

 

[Alex Rosenberg]

Looking at Intel's datasheet for the C3000-series chips, which doesn't have this mystery C3908 on it, it looks like this generation of Denverton is limited to 2133Mhz. Qotom's specs for these boxes claiming 2400 is just wrong.

I got a second box, a Q20351G9 with a C3958 in it, which can run RAM at 2400MHz while the C3908 is limited to 2133. There may be other differences, but the Qotom sales folks couldn't answer them.

 

[blunden]

I got a second box, a Q20351G9 with a C3958 in it, which can run RAM at 2400MHz while the C3908 is limited to 2133. There may be other differences, but the Qotom sales folks couldn't answer them.

My C3758 unit can also run them faster than 2133 MHz last time I checked.

 

[sko]

IO-SRV on those old X5** chipsets has always been rather buggy/broken. X7xx *should* work (I tried once and could create VEs, haven't tested any further).
The easiest ones to work with are usually mellanox and chelsio. I'm actively using SR-IOV on a bunch of Mellanox ConnectX4/5 cards passing VEs into jails.

Regarding QAT: I have it enabled and it is used by angie for TLS via KTLS. Essentially anything that uses KTLS or /dev/crypto automatically uses offloading via the qat driver. It can be easily observed by looking at the fw_counters:

# sysctl dev.qat.0.fw_counters
dev.qat.0.fw_counters:
+------------------------------------------------+
| FW Statistics for Qat Device                                     |
+------------------------------------------------+
AE  5
Firmware Responses:0
Firmware Requests:0
AE  4
Firmware Responses:0
Firmware Requests:0
AE  3
Firmware Responses:23011
Firmware Requests:23011
AE  2
Firmware Responses:0
Firmware Requests:0
AE  1
Firmware Responses:0
Firmware Requests:0
AE  0
Firmware Responses:23012
Firmware Requests:23012

I just rebooted the host after a kernel update, hence the counters aren't that high. Also KTLS seems to have quite a high affinity to the AEs it first attaches - I rarely see more than 2 or 3 engines being utilized... I'm not running any VPN on that host, so angie/TLS is pretty much the only thing that uses KTLS/QAT.

 

[foureight84]

The x553 NICs are quite annoying to work with in terms of 10G Base-T. You either get one that only does 10G Base-T or your make sure the terminating node is set to 10G only instead of allowing it to auto negotiate. The latter causes the NIC to go down and would need a restart (probably reinit from OS but I can't do that with ESXi).

 

[foureight84]

Really curious. Swapped out my old router for this device running Mikrotik CHR on ESXi with hardware passthrough for 4 of the 5 i226 and all 4 of the x553 ports. Plugged in 2 RJ45 to 226 and one 10GBase-T, everything looks good. I plugin a 3rd RJ45 to the i226, no traffic to this port. The router and traffic isn't forwarding to the new connected port. Reboot the entire system and no the SFP+ port isn't forwarding traffic.

3 seems to be the limit before some sort of hardware fault kicks in but I am not yet able to see what's the error. The Mikrotik CHR 60 day trial license doesn't mention limit to numbers of ports, just maximum speed. 10GBase-T doesn't seem to be the issue since the same thing occurs with an Intel SFP+ transceiver.

EDIT: more than 3 RJ45 in the i226 NICs same issue. Any combination of more than 3 including one with SPF+ LR results in the same error.

 

[sko]

as written in countless other posts: that SFP+ to copper crap is beyond any specification and draws 2-3x the power allowed for SFP+. Just use proper transceivers, they are dirt-cheap nowadays and so are fiber patch cables.

And why on earth would you put so many layers on top of each other? There *have* to be issues, especially with all that proprietary crap inbetween...
Others have reported some issues with port 4 not powering up correctly on some hardware revisions, just search in this thread for that.
OTOH I've been running 4 transcievers (huawei single-mode) pretty much from day 1 in that unit without any issues.

 

[blunden]

as written in countless other posts: that SFP+ to copper crap is beyond any specification and draws 2-3x the power allowed for SFP+. Just use proper transceivers, they are dirt-cheap nowadays and so are fiber patch cables.

And why on earth would you put so many layers on top of each other? There *have* to be issues, especially with all that proprietary crap inbetween.

Well, 2-3 times more power than allowed is an exaggeration. Modern 10GBASE-T transceivers with recent PHYs are at 1.5-1.8 W, which isn't too far from some other transceivers. They are still sort of hacky though and I still recommend against them when not absolutely necessary.

Yes, it's certainly possible that it's an issue with the virtualization layer or the OS.

 

[foureight84]

as written in countless other posts: that SFP+ to copper crap is beyond any specification and draws 2-3x the power allowed for SFP+. Just use proper transceivers, they are dirt-cheap nowadays and so are fiber patch cables.

And why on earth would you put so many layers on top of each other? There *have* to be issues, especially with all that proprietary crap inbetween...
Others have reported some issues with port 4 not powering up correctly on some hardware revisions, just search in this thread for that.
OTOH I've been running 4 transcievers (huawei single-mode) pretty much from day 1 in that unit without any issues.

It's not just 10GBase-T. I tested it out with just i226. As long as more than 3 are connected the 4th will not work (any combinations not just the mentioned port 4 issue). Also, the SFP+ 10GBase-T adapter I have draws 1.5W max.

As @bluden mentioned it's highly likely to be virtualization layering issues. I don't know if it's Mikrotik OS related. I looked through their documentation and there's no restriction on the number of ports for evaluation license.

Also, Mikrotik CHR is not made for bare metal. It's explicitly made for virtualization usage. I wanted to take this router first before just using a baremetal router OS so that I can actually use some of the left over resources for other VMs.

Lastly, I have to use 10GBase-T because my ISP only provides RJ45 termination for their 10GBit connection. Of course, I could go and buy a 10Gbit router with 10Gbit RJ45 WAN but I have this thing on-hand so might as well try to see if I can make it work before shelling out more money. The ISP won't allow me to use my own ONT either. I already talked to them about that to try and get around the need for using 10GBase-T.