Qotom Denverton fanless system with 4 SFP+

[kr33]

Based on other people's testing in here and by STH, the SFP+ slots are not vendor locked.

The in-kernel driver for that NIC in Linux 6.x is broken, which probably what you are running into. Look a few pages back. Intel's out-of-tree driver works fine, which VyOS switched back to for that very reason.

OPNsense isn't affected because the BSD kernel driver doesn't have an equivalent broken change.

Thanks for the reply.

Seemed to work actually. Now it so happens that my unit just died after having it for 2 days. It was also very slow and then just stopped powering on. You can still see the green light through the microsd/sim slot but when you press the power button...sweet nothing so have to return it now.

I'm not sure if I should get a replacement or find another alternative with 10G support.

 

[blunden]

Thanks for the reply.

Seemed to work actually. Now it so happens that my unit just died after having it for 2 days. It was also very slow and then just stopped powering on. You can still see the green light through the microsd/sim slot but when you press the power button...sweet nothing so have to return it now.

I'm not sure if I should get a replacement or find another alternative with 10G support.

It's fairly slow to POST when you cold boot it. Have you tried just waiting longer?

 

[Brent Geery]

Has anyone installed the new BIOS update yet, beside me? It may have changed the BIOS setting State After G3 default to Power on, so no more hunting down jumpers or worrying about a BIOS battery dying and reverting the setting back to OFF.

I'd also like to see if "unlocking" the optical port speed fix might address the throughput issues some have reported having here.

BTW, there is an error in the flashing instructions (at least in QDNV0121V11.) To make things work for me, I configured the BIOS to boot into the built-in UEFI shell, in UEFI shell switched to the flash drive with "fs0:", flashed the optical controller with "fdesc.nsh", rebooted back into the UEFI shell, again switched back to the flash drive with "fs0:", flashed the BIOS with "f.nsh", rebooted again and reconfigured the new BIOS options.
Index of /download/Bios/Q20300G9_QDNV01
PS: The error is a typo in the "fm.nsh" file that refers to "QDNV0112.V11" instead of the correct file "QDNV0121.V11".

 

[Pheckphul]

Take another look at the BIOS settings. It's there. North Bridge chipset S3 power state or something like that. The BIOS option labels are all very technical and unhelpful.

It's a reference BIOS, not really meant for end-users such as us, but for motherboard manufacturers to base their BIOSs around. Thus the engineering-level technical terms instead of "clicky here to make it worky."

 

[xoofx]

Has anyone installed the new BIOS update yet, beside me? It may have changed the BIOS setting State After G3 default to Power on, so no more hunting down jumpers or worrying about a BIOS battery dying and reverting the setting back to OFF.

I'd also like to see if "unlocking" the optical port speed fix might address the throughput issues some have reported having here.

I did it a few days ago and it seems going ok. I didn't have any particular issues before with the throughput (on opnsense)

BTW, there is an error in the flashing instructions (at least in QDNV0121V11.) To make things work for me, I configured the BIOS to boot into the built-in UEFI shell, in UEFI shell switched to the flash drive with "fs0:", flashed the optical controller with "fdesc.nsh", rebooted back into the UEFI shell, again switched back to the flash drive with "fs0:", flashed the BIOS with "f.nsh", rebooted again and reconfigured the new BIOS options.
Index of /download/Bios/Q20300G9_QDNV01
PS: The error is a typo in the "fm.nsh" file that refers to "QDNV0112.V11" instead of the correct file "QDNV0121.V11".

Thanks for this, as it helped to not waste time in figuring out why the script would fail.

 

[SCS]

Seems to be a large discussion here on this unit and this page was brought to my attention while submitting a pfSense post on Netgate Fourms.

Long story short my 2G/2G connection was only ~2G/5M after moving to this unit from a HP T730. Come to find out it appears not to like a number of my SFP+ DAC cables from Cisco and 10Gtek. About half a dozen in total.

Thought I'd post a link as after we worked through there wasn't really a config issue, we went back to physical issues.



https://forum.pfsense.com/topic/186744/abysmal-performance-after-pfsense-hardware-upgrade

 

[Khadgar]

I'm having similar issues where normally I'd get around 1000/1000, but on one of my 10g SFP+ adapters I get 1000/25. The other one is fine. Both are ipolex. I think in my case it's the top left SFP+ port, not the adapter itself. When I play musical chairs with my adapters three of the ports are fine, the fourth not so much. Bummer. My BIOS was already updated to the latest when I got it, and the default wasn't to power on, sadly.

Aside from that one SFP+ adapter everything's been running smoothly. My old Qotom firewall ran without needing a fan, but this one understandably does need one as it was hot to the touch when testing and setting up OPNSense on it. I rigged a 120mm fan on the top that plugs into one of the USB ports. It's only getting 5V of power so it runs a bit slow, but it runs silently and basically keeps the top of the thing at room temperature. Looks like a mess, but hey it works haha.

 

[sko]

this looks more like a bad fiber on the TX side (e.g. insertion loss, dust/dirt etc..). Have you looked at the RX/TX power and dBm values? e.g. ifconfig -v <iface> if you run FreeBSD or something FreeBSD based like OPN-/pfSense and sho int <interface> tra det on cisco switches...

 

[SCS]

this looks more like a bad fiber on the TX side (e.g. insertion loss, dust/dirt etc..). Have you looked at the RX/TX power and dBm values? e.g. ifconfig -v <iface> if you run FreeBSD or something FreeBSD based like OPN-/pfSense and sho int <interface> tra det on cisco switches...

This is the current 1" DAC



[2.7.2-RELEASE][admin@pfSense-Edge01.scs.lan]/root: ifconfig -v ix1
ix1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: LAN_1
options=48138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,HWSTATS,MEXTPG>
ether 20:7c:14:f3:8b:42
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
inet 172.16.1.1 netmask 0xffffffff broadcast 172.16.1.1
inet6 fe80::227c:14ff:fef3:8b42%ix1 prefixlen 64 scopeid 0x7
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
plugged: SFP/SFP+/SFP28 Unknown (Copper pigtail)
vendor: OEM PN: SFP-H10GB-CU1M SN: CSC210701320268 DATE: 2021-07-09

This is one of the Cisco DAC's I was having an issue with



[2.7.2-RELEASE][admin@pfSense-Edge01.scs.lan]/root: ifconfig -v ix1
ix1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: LAN_1
options=48138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,HWSTATS,MEXTPG>
ether 20:7c:14:f3:8b:42
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
inet 172.16.1.1 netmask 0xffffffff broadcast 172.16.1.1
inet6 fe80::227c:14ff:fef3:8b42%ix1 prefixlen 64 scopeid 0x7
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
plugged: SFP/SFP+/SFP28 Unknown (Copper pigtail)
vendor: CISCO-MOLEX PN: 74752-9520 SN: MOC155107L8 DATE: 2011-12-23


Back on 1' DAC



ifconfig -v ix1
ix1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: LAN_1
options=48138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,HWSTATS,MEXTPG>
ether 20:7c:14:f3:8b:42
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
inet 172.16.1.1 netmask 0xffffffff broadcast 172.16.1.1
inet6 fe80::227c:14ff:fef3:8b42%ix1 prefixlen 64 scopeid 0x7
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
plugged: SFP/SFP+/SFP28 Unknown (Copper pigtail)
vendor: OEM PN: SFP-H10GB-CU1M SN: CSC210701320268 DATE: 2021-07-09

 

[blunden]

@SCS Is there anything obviously different between the working and non-working DACs? I assume you don't mean that it's 1 inch long? I've never seen one that short.

 

[SCS]

if I stated 1" (1 inch) I meant 1' (1 foot/0.3m)

Nothing obvious that I can tell as far as a difference that should make a difference.

Difference would be different length, all the ones that didn't work were about 8' ish, and a mix of Cisco and 10GTek. Different DOM's dating back to 2011 for the Cisco's and far newer for the 10GTek's

The one that worked was only 1' and made by 10GTek

 

[blunden]

if I stated 1" (1 inch) I meant 1' (1 foot/0.3m)

Nothing obvious that I can tell as far as a difference that should make a difference.

Difference would be different length, all the ones that didn't work were about 8' ish, and a mix of Cisco and 10GTek. Different DOM's dating back to 2011 for the Cisco's and far newer for the 10GTek's

The one that worked was only 1' and made by 10GTek

Looking at your post again, you did indeed write 1'. However, that's an absolutely meaningless notation for most of the world, so it didn't even register for me as "feet".

If it works with shorter passive DAC cables I suppose it could be a signal integrity issue. If so, it would presumably work with an active DAC, but that would obviously negate most of the cost and power savings.

 

[sko]

Looking at your post again, you did indeed write 1'. However, that's an absolutely meaningless notation for most of the world,

actually it's the standard notation for an arc minute, so except if you talk about apparent distances/sizes in the sky, it has nothing to do with an actual measurement for length...


regarding the problem:
I was initially referring to @Khadgars post as he is using optical transceivers.
DACs have always been fiddly, picky and massively annoying, especially copper ones and *especially* passive DACs (which are the worst junk on the planet...). As they have almost no advantage and cost the same (or even more) as proper transceivers and fiber patch cables, I never actually used them anywhere and haven't toched a DAC in almost 15 years... Best advice I could give: don't use DACs, just use proper transceivers or at least optical DACs if you absolutely *have* to restrict yourself with those things...

 

[blunden]

actually it's the standard notation for an arc minute, so except if you talk about apparent distances/sizes in the sky, it has nothing to do with an actual measurement for length...

Haha, true. Now that you mention it, I have seen that before years ago.

regarding the problem:
I was initially referring to @Khadgars post as he is using optical transceivers.
DACs have always been fiddly, picky and massively annoying, especially copper ones and *especially* passive DACs (which are the worst junk on the planet...). As they have almost no advantage and cost the same (or even more) as proper transceivers and fiber patch cables, I never actually used them anywhere and haven't toched a DAC in almost 15 years... Best advice I could give: don't use DACs, just use proper transceivers or at least optical DACs if you absolutely *have* to restrict yourself with those things...

While optical transceivers tend to be easier to get working, passive DACs do still objectively have benefits if they work in your particular situation. They do cost less. In my case, moving from DACs to cheap optical transceivers + multi-mode fiber cost roughly 3 times more per link compared to DACs. They also draw less power. Those are facts.

Can you get cheaper transceivers if you buy them used? Probably. Is that a reasonable option outside the US? No, most often not.

Optical "DACs" (i.e. AOC cables) make little sense to me as they cost about the same as transceivers + cable and provide none of the other DAC benefits.

 

[xoofx]

Hey, just wanted to share my experience with the Qotom Q20331G9S10 and OPNsense setup on it: Building a secure 10G Home Network | xoofx I'm not an expert in setuping such appliance, so the article will not be entertaining for the experts, but it might help the decision for some folks.

 

[SCS]

regarding the problem:
I was initially referring to @Khadgars post as he is using optical transceivers.
DACs have always been fiddly, picky and massively annoying, especially copper ones and *especially* passive DACs (which are the worst junk on the planet...). As they have almost no advantage and cost the same (or even more) as proper transceivers and fiber patch cables, I never actually used them anywhere and haven't toched a DAC in almost 15 years... Best advice I could give: don't use DACs, just use proper transceivers or at least optical DACs if you absolutely *have* to restrict yourself with those things...

by optical DAC do you mean the AOC's, 10G SFP+ Active Optical Cable?

 

[blunden]

by optical DAC do you mean the AOC's, 10G SFP+ Active Optical Cable?

I can't imagine there being anything else like that besides AOCs.

 

[wonder_rohmie]

I was wondering whether anybody has (successfully) used an M.2 WWAN card with the B-key SIM slot?

My experience so far has been that the M.2 card works as expected (USB passthrough from Proxmox 7.4 to an OPNsense 24 host) - when using a FreeBSD-compatible WWAN card, it is recognized in OPNsense and I can communicate successfully via cu. But SIM-related commands (e.g. AT+CPIN?) return errors, so I assume there is some problem with the B-key interface? I haven't thoroughly checked the myriad of BIOS settings yet, so there might be something I've missed...

Otherwise I'm pretty happy with the purchase so far (C3758R, non-rack version), but it's still not "in production" in my home network.

 

[azhagan]

I have one of these with an Atom C3758R, and put 2 SFP+ 10G adapters in there and everything slows to a crawl. The 2.5 GB ports seems to work fine without any issues. I've tried with OPNsense, VyOS and OpenWRT, and facing the same issues. Any thoughts?

 

[blunden]

I have one of these with an Atom C3758R, and put 2 SFP+ 10G adapters in there and everything slows to a crawl. The 2.5 GB ports seems to work fine without any issues. I've tried with OPNsense, VyOS and OpenWRT, and facing the same issues. Any thoughts?

What transceivers did you use? Do the ports you use make a difference?