Qotom Denverton fanless system with 4 SFP+

[foureight84]

It's a platform originally from 2017. It supports the most important ciphers/algorithms from that time and is really powerful if it matches your use case, but naturally lacks some modern features.

I would expect the drivers to be stable at this point so I don't see that as much of an issue. Not sure why you're having problems in Proxmox. Perhaps it doesn't include the QAT driver?

I am able to enable SR-IOV for it but it doesn't show up in the PCI list unlike the x553 NICs. I tried to compile drivers but ran into a compilation error that got me stuck. But I think the drivers should already be there.

root@pve-gateway:/usr/local# lsmod | grep qat
qat_c3xxxvf            12288  0
qat_c3xxx              12288  0
intel_qat             425984  2 qat_c3xxxvf,qat_c3xxx

root@pve-gateway:/usr/local# dmesg | grep qat
[   12.702202] c3xxx 0000:01:00.0: qat_dev0 started 6 acceleration engines
[   74.653342] c3xxx 0000:01:00.0: qat_dev0 stopped 6 acceleration engines
[   74.655143] c3xxx 0000:01:00.0: Resetting device qat_dev0
[   74.884148] c3xxx 0000:01:00.0: qat_dev0 started 6 acceleration engines

root@pve-gateway:/usr/local# grep qat /proc/crypto
driver       : qat_deflate
module       : intel_qat
driver       : qat-dh
module       : intel_qat
driver       : qat-rsa
module       : intel_qat
driver       : qat_aes_cbc_hmac_sha512
module       : intel_qat
driver       : qat_aes_cbc_hmac_sha256
module       : intel_qat
driver       : qat_aes_cbc_hmac_sha1
module       : intel_qat
driver       : qat_aes_xts
module       : intel_qat
driver       : qat_aes_ctr
module       : intel_qat
driver       : qat_aes_cbc
module       : intel_qat

 

[blunden]

Hmm, not sure then. I used mine bare metal and never bothered with QAT since I use Wireguard.

 

[Alex Rosenberg]

Looking at Intel's datasheet for the C3000-series chips, which doesn't have this mystery C3908 on it, it looks like this generation of Denverton is limited to 2133Mhz. Qotom's specs for these boxes claiming 2400 is just wrong.

I got a second box, a Q20351G9 with a C3958 in it, which can run RAM at 2400MHz while the C3908 is limited to 2133. There may be other differences, but the Qotom sales folks couldn't answer them.

 

[blunden]

I got a second box, a Q20351G9 with a C3958 in it, which can run RAM at 2400MHz while the C3908 is limited to 2133. There may be other differences, but the Qotom sales folks couldn't answer them.

My C3758 unit can also run them faster than 2133 MHz last time I checked.

 

[sko]

IO-SRV on those old X5** chipsets has always been rather buggy/broken. X7xx *should* work (I tried once and could create VEs, haven't tested any further).
The easiest ones to work with are usually mellanox and chelsio. I'm actively using SR-IOV on a bunch of Mellanox ConnectX4/5 cards passing VEs into jails.

Regarding QAT: I have it enabled and it is used by angie for TLS via KTLS. Essentially anything that uses KTLS or /dev/crypto automatically uses offloading via the qat driver. It can be easily observed by looking at the fw_counters:

# sysctl dev.qat.0.fw_counters
dev.qat.0.fw_counters:
+------------------------------------------------+
| FW Statistics for Qat Device                                     |
+------------------------------------------------+
AE  5
Firmware Responses:0
Firmware Requests:0
AE  4
Firmware Responses:0
Firmware Requests:0
AE  3
Firmware Responses:23011
Firmware Requests:23011
AE  2
Firmware Responses:0
Firmware Requests:0
AE  1
Firmware Responses:0
Firmware Requests:0
AE  0
Firmware Responses:23012
Firmware Requests:23012

I just rebooted the host after a kernel update, hence the counters aren't that high. Also KTLS seems to have quite a high affinity to the AEs it first attaches - I rarely see more than 2 or 3 engines being utilized... I'm not running any VPN on that host, so angie/TLS is pretty much the only thing that uses KTLS/QAT.

 

[foureight84]

The x553 NICs are quite annoying to work with in terms of 10G Base-T. You either get one that only does 10G Base-T or your make sure the terminating node is set to 10G only instead of allowing it to auto negotiate. The latter causes the NIC to go down and would need a restart (probably reinit from OS but I can't do that with ESXi).

 

[foureight84]

Really curious. Swapped out my old router for this device running Mikrotik CHR on ESXi with hardware passthrough for 4 of the 5 i226 and all 4 of the x553 ports. Plugged in 2 RJ45 to 226 and one 10GBase-T, everything looks good. I plugin a 3rd RJ45 to the i226, no traffic to this port. The router and traffic isn't forwarding to the new connected port. Reboot the entire system and no the SFP+ port isn't forwarding traffic.

3 seems to be the limit before some sort of hardware fault kicks in but I am not yet able to see what's the error. The Mikrotik CHR 60 day trial license doesn't mention limit to numbers of ports, just maximum speed. 10GBase-T doesn't seem to be the issue since the same thing occurs with an Intel SFP+ transceiver.

EDIT: more than 3 RJ45 in the i226 NICs same issue. Any combination of more than 3 including one with SPF+ LR results in the same error.

 

[sko]

as written in countless other posts: that SFP+ to copper crap is beyond any specification and draws 2-3x the power allowed for SFP+. Just use proper transceivers, they are dirt-cheap nowadays and so are fiber patch cables.

And why on earth would you put so many layers on top of each other? There *have* to be issues, especially with all that proprietary crap inbetween...
Others have reported some issues with port 4 not powering up correctly on some hardware revisions, just search in this thread for that.
OTOH I've been running 4 transcievers (huawei single-mode) pretty much from day 1 in that unit without any issues.

 

[blunden]

as written in countless other posts: that SFP+ to copper crap is beyond any specification and draws 2-3x the power allowed for SFP+. Just use proper transceivers, they are dirt-cheap nowadays and so are fiber patch cables.

And why on earth would you put so many layers on top of each other? There *have* to be issues, especially with all that proprietary crap inbetween.

Well, 2-3 times more power than allowed is an exaggeration. Modern 10GBASE-T transceivers with recent PHYs are at 1.5-1.8 W, which isn't too far from some other transceivers. They are still sort of hacky though and I still recommend against them when not absolutely necessary.

Yes, it's certainly possible that it's an issue with the virtualization layer or the OS.

 

[foureight84]

as written in countless other posts: that SFP+ to copper crap is beyond any specification and draws 2-3x the power allowed for SFP+. Just use proper transceivers, they are dirt-cheap nowadays and so are fiber patch cables.

And why on earth would you put so many layers on top of each other? There *have* to be issues, especially with all that proprietary crap inbetween...
Others have reported some issues with port 4 not powering up correctly on some hardware revisions, just search in this thread for that.
OTOH I've been running 4 transcievers (huawei single-mode) pretty much from day 1 in that unit without any issues.

It's not just 10GBase-T. I tested it out with just i226. As long as more than 3 are connected the 4th will not work (any combinations not just the mentioned port 4 issue). Also, the SFP+ 10GBase-T adapter I have draws 1.5W max.

As @bluden mentioned it's highly likely to be virtualization layering issues. I don't know if it's Mikrotik OS related. I looked through their documentation and there's no restriction on the number of ports for evaluation license.

Also, Mikrotik CHR is not made for bare metal. It's explicitly made for virtualization usage. I wanted to take this router first before just using a baremetal router OS so that I can actually use some of the left over resources for other VMs.

Lastly, I have to use 10GBase-T because my ISP only provides RJ45 termination for their 10GBit connection. Of course, I could go and buy a 10Gbit router with 10Gbit RJ45 WAN but I have this thing on-hand so might as well try to see if I can make it work before shelling out more money. The ISP won't allow me to use my own ONT either. I already talked to them about that to try and get around the need for using 10GBase-T.

 

[Pheckphul]

as written in countless other posts: that SFP+ to copper crap is beyond any specification and draws 2-3x the power allowed for SFP+. Just use proper transceivers, they are dirt-cheap nowadays and so are fiber patch cables.

<SNIP>

SFP+ to copper may be be "crap," but my ISP's Adtran 622v ONT only has copper 10GbE. I'd have to either buy a switch with copper and SFP+ ports or a media converter to put between my Qotom OPNsense box and my ISP's box. Unless you have a better solution?

Has anyone tried Proxmox v9 and the X553s? Am curious if the problems we were seeing with the 6.x kernels reared its ugly head with Proxmox's kernels in v9. Any issues I should be aware of with Proxmox v9 and this Qotom board?

Thank you for taking the time to read this.

 

[Caennanu]

SFP+ to copper may be be "crap," but my ISP's Adtran 622v ONT only has copper 10GbE. I'd have to either buy a switch with copper and SFP+ ports or a media converter to put between my Qotom OPNsense box and my ISP's box. Unless you have a better solution?

Has anyone tried Proxmox v9 and the X553s? Am curious if the problems we were seeing with the 6.x kernels reared its ugly head with Proxmox's kernels in v9. Any issues I should be aware of with Proxmox v9 and this Qotom board?

Thank you for taking the time to read this.

Just wanted to put my 2cts in.
You're using an SFP+ capable device, you're talking about an ONT, which makes me assume you're on the PON technology from the ISP.
Here in holland, when PON technology is used, the SFP requires activation by the ISP through either mac or serial number.
There are PON SFP+ transceivers available that you can use directly, bypassing the ONT all together. What would be the limiting factor for doing this?

 

[blunden]

SFP+ to copper may be be "crap," but my ISP's Adtran 622v ONT only has copper 10GbE. I'd have to either buy a switch with copper and SFP+ ports or a media converter to put between my Qotom OPNsense box and my ISP's box. Unless you have a better solution?

Has anyone tried Proxmox v9 and the X553s? Am curious if the problems we were seeing with the 6.x kernels reared its ugly head with Proxmox's kernels in v9. Any issues I should be aware of with Proxmox v9 and this Qotom board?

Thank you for taking the time to read this.

While I still recommend against 10GBASE-T transceivers when you don't have to use them as I said above, sometimes they are the best option available. I've personally used them for almost 2 years now and it has remained perfectly stable (knock on wood) so far. Over year of that time was with this Qotom box, and now with a BPI-R4.

You absolutely have to make sure to use one of the transceivers with the modern lower power draw PHYs though (rated for 80 och 100 meters) of you want it to work with the Intel X553 though, as others have tested earlier in this thread. It's also preferable that it spoofs being a 10GBASE-SR transceiver (most of them do) for compatibility reasons.

Regarding Proxmox, you should be able to find out in which kernel version the problematic driver commit was reverted and compare with the Proxmox kernel.

Just wanted to put my 2cts in.
You're using an SFP+ capable device, you're talking about an ONT, which makes me assume you're on the PON technology from the ISP.
Here in holland, when PON technology is used, the SFP requires activation by the ISP through either mac or serial number.
There are PON SFP+ transceivers available that you can use directly, bypassing the ONT all together. What would be the limiting factor for doing this?

While it's possible to bypass the PON ONT in many cases, it's not allowed by most ISPs. Those PON SFP+ transceivers also tend to draw more power, to the point where it might not actually work with the Intel X553 NIC. It's therefore kind of an expensive gamble. They also tend to require active cooling from what I've seen.

Still, they are a great option if your PON ISP's ONT is trash (which is certainly the case for some ISPs, from what I've read).

 

[mrpops2ko]

SFP+ to copper may be be "crap," but my ISP's Adtran 622v ONT only has copper 10GbE. I'd have to either buy a switch with copper and SFP+ ports or a media converter to put between my Qotom OPNsense box and my ISP's box. Unless you have a better solution?

Has anyone tried Proxmox v9 and the X553s? Am curious if the problems we were seeing with the 6.x kernels reared its ugly head with Proxmox's kernels in v9. Any issues I should be aware of with Proxmox v9 and this Qotom board?

Thank you for taking the time to read this.

depending on the ONT, i'd only recommend doing the ONT bypass methods if your ONT is junk and does funky stuff. if its just a transparent bridge / media converter then it'll be fine to do line rate without any issues and theres no real need to bypass it.

I do the router on a stick method (ONT plugs into managed switch) and you lock the ONT WAN behind a vlan and just access it across the network that way.

its a great way to create failover internet too using CARP / VRRP / keepalived too. thats what I recommend, its really easy to have a backup router VM / LXC which only kicks in once the main one is down (reboots / server changes) and it helps cut down on complaints when the internet is down.

I have the MikroTik CRS310-8G 2S IN.

 

[Caennanu]

While I still recommend against 10GBASE-T transceivers when you don't have to use them as I said above, sometimes they are the best option available. I've personally used them for almost 2 years now and it has remained perfectly stable (knock on wood) so far. Over year of that time was with this Qotom box, and now with a BPI-R4.

You absolutely have to make sure to use one of the transceivers with the modern lower power draw PHYs though (rated for 80 och 100 meters) of you want it to work with the Intel X553 though, as others have tested earlier in this thread. It's also preferable that it spoofs being a 10GBASE-SR transceiver (most of them do) for compatibility reasons.

Regarding Proxmox, you should be able to find out in which kernel version the problematic driver commit was reverted and compare with the Proxmox kernel.

While it's possible to bypass the PON ONT in many cases, it's not allowed by most ISPs. Those PON SFP+ transceivers also tend to draw more power, to the point where it might not actually work with the Intel X553 NIC. It's therefore kind of an expensive gamble. They also tend to require active cooling from what I've seen.

Still, they are a great option if your PON ISP's ONT is trash (which is certainly the case for some ISPs, from what I've read).

Yeah i figured, the only reason you wouldn't implement it is if the isp doesn't provide the option to use your own stuff. But if you're in the EU, its a non-issue and they should comply.
Not sure on the active cooling part tho. I see the zaram SFP's seem fine (also from what i read on tests done in the tweakers.net threads), the fs.com versions do seem to have additional fins specifically for this purpose. (and with a planned network upgrade for my area, i should probably look into this)

 

[blunden]

Yeah i figured, the only reason you wouldn't implement it is if the isp doesn't provide the option to use your own stuff. But if you're in the EU, its a non-issue and they should comply.
Not sure on the active cooling part tho. I see the zaram SFP's seem fine (also from what i read on tests done in the tweakers.net threads), the fs.com versions do seem to have additional fins specifically for this purpose. (and with a planned network upgrade for my area, i should probably look into this)

I haven't heard of a single PON ISP that allow you to use your own ONT, nor do I think you have any explicit legal right to do so. Your own router, sure. Your own ONT, no. Feel free to provide some proof for that assertion.

I thankfully don't have to deal with PON, but my understanding is that at least some of those PON transceivers tend to idle at close to their max operating temperature without active cooling. I have no personal experience with them though, so I would suggest researching that if you decide that you want to try using one of those.

 

[Caennanu]

I haven't heard of a single PON ISP that allow you to use your own ONT, nor do I think you have any explicit legal right to do so. Your own router, sure. Your own ONT, no. Feel free to provide some proof for that assertion.

I thankfully don't have to deal with PON, but my understanding is that at least some of those PON transceivers tend to idle at close to their max operating temperature without active cooling. I have no personal experience with them though, so I would suggest researching that if you decide that you want to try using one of those.

Ill supply some more details later. But this is atleast the Dutch ruling: Vrije modemkeuze | ACM

It is a bit vague, but what it comes down to is that anything after the fiber termination is part of our right for having your own router. And the isp is obligated to tell you what it should comply to.

 

[Caennanu]

To add to the above. Here is the very very official documentation for atleast the dutch residents:

Staatsblad 2016, 524 | Overheid.nl > Officiële bekendmakingen

zoek.officielebekendmakingen.nl

And this... should be the EU version:

Router Freedom - FSFE

It should go without saying that in our society we should be able to freely choose technical devices for use in our homes like we are free to choose what m...

fsfe.org

.

 

[blunden]

Ill supply some more details later. But this is atleast the Dutch ruling: Vrije modemkeuze | ACM

It is a bit vague, but what it comes down to is that anything after the fiber termination is part of our right for having your own router. And the isp is obligated to tell you what it should comply to.

Interesting. The Dutch ruling could be interpreted like that, as long as the transceiver behaves properly. That doesn't mean it applies to the entire EU though as far as I can tell.

To add to the above. Here is the very very official documentation for atleast the dutch residents:

Staatsblad 2016, 524 | Overheid.nl > Officiële bekendmakingen

zoek.officielebekendmakingen.nl

And this... should be the EU version:

Router Freedom - FSFE

It should go without saying that in our society we should be able to freely choose technical devices for use in our homes like we are free to choose what m...

fsfe.org

.

I read the EU version as referring to your right to you your own router, not PON ONT or media converter (in the case of AON). Sure, it talks about the Network Termination Point so there i still some room for interpretation, but all the arguments for why the law is needed doesn't apply nearly as much to ONTs.

 

[Caennanu]

Interesting. The Dutch ruling could be interpreted like that, as long as the transceiver behaves properly. That doesn't mean it applies to the entire EU though as far as I can tell.

I read the EU version as referring to your right to you your own router, not PON ONT or media converter (in the case of AON). Sure, it talks about the Network Termination Point so there i still some room for interpretation, but all the arguments for why the law is needed doesn't apply nearly as much to ONTs.

Yeah i am aware. Its legal Mambo jambo which always leaves stuff open for interpretation...
However without disclosing where i work. The ONT for the Dutch is indeed part of this right.