Qotom Denverton fanless system with 4 SFP+

[kingneutron]

FYI, for Linux LMDE6 appears to have the best support for the NICs. 2.5Gbit working out of the box at proper speed with in-kernel module, MTU 1500 over vsftpd/lftp with multiple transfers enabled ( lftp -- set mirrorarallel-transfer-count 2 ) or higher number.

For 10Gbit SFP+ ports, I tested the two right-side ports with a 10Gbit DAC and MTU 9000 and got as-expected numbers with iperf3.

iperf3 -c 172.16.10.2

[ 5] local 172.16.10.2 port 37950 connected to 172.16.10.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.57 GBytes 13.5 Gbits/sec 0 1.19 MBytes
[ 5] 5.00-6.00 sec 1.67 GBytes 14.4 Gbits/sec 0 1.19 MBytes

$ uname -a
Linux qotom-lmde6 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux

As you can see this is a very recent kernel, hopefully they fixed the bugs reported earlier in this thread. I have a proper 2-port 10Gbit SFP+ pcie card on order and 2x 10Gbase-T transceivers so I can connect the Qotom to my ZFS server.

Also have 4x 10Gbit fiber transceivers and will try connecting my 2.5Gbit 4-port and 8-port switches with the SFP+ ports to see if it makes a difference.

---

PROTIP: You don't have to spend a lot of money if you want to use SAS/SATA enclosure with the Qotom.

https://www.amazon.com/gp/product/B0BV142WM5/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

^ 5-bay SAS/SATA hotswap enclosure, all you need is a standard PC power supply with 2xMOLEX.
Under $90 at time of posting, and you get a spare bay for drive swaps.

I paired the enclosure with a SAS internal 4-breakout cable to the Qotom and it works just fine:

https://www.amazon.com/gp/product/B018YHS8BS/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

Currently my pc power supply is also connected to a spare 5-bay hddrack for power-on, but you should be able to get things working with a power supply jump-cable if needed.

How to jump a psu?

I have a m12ii evo bronze 620w and im not sure how to jump it to see if its working

linustechtips.com

 

[blunden]

FYI, for Linux LMDE6 appears to have the best support for the NICs. 2.5Gbit working out of the box at proper speed with in-kernel module, MTU 1500 over vsftpd/lftp with multiple transfers enabled ( lftp -- set mirrorarallel-transfer-count 2 ) or higher number.

For 10Gbit SFP+ ports, I tested the two right-side ports with a 10Gbit DAC and MTU 9000 and got as-expected numbers with iperf3.

iperf3 -c 172.16.10.2

[ 5] local 172.16.10.2 port 37950 connected to 172.16.10.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.57 GBytes 13.5 Gbits/sec 0 1.19 MBytes
[ 5] 5.00-6.00 sec 1.67 GBytes 14.4 Gbits/sec 0 1.19 MBytes

$ uname -a
Linux qotom-lmde6 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux

As you can see this is a very recent kernel, hopefully they fixed the bugs reported earlier in this thread. I have a proper 2-port 10Gbit SFP+ pcie card on order and 2x 10Gbase-T transceivers so I can connect the Qotom to my ZFS server.

Also have 4x 10Gbit fiber transceivers and will try connecting my 2.5Gbit 4-port and 8-port switches with the SFP+ ports to see if it makes a difference.

---

No, I have seen no indication that it has been fixed upstream unfortunately. Either the broken commit only got merged in later kernel versions (most tests I've seen are with 6.5 and 6.6), or the devices you tested with just happened to be compatible with that link negotiation behavior change. After all, it doesn't break all devices and driver combinations, just most of them. Also, my understanding is that two X553 NICs both running the same "broken" driver will work fine.

VyOS switched back to the out-of-tree driver yesterday due to this very issue.

Kernel: T5887: T5619: T5995: Update to v6.6.15, add IXGBE out-of-tree drivers and Huawei NIC driver (backport #490) by mergify[bot] · Pull Request #491 · vyos/vyos-build

This is an automatic backport of pull request #490 done by Mergify. Mergify commands and options More conditions and actions can be found in the documentation. You can also trigger Mergify actio...

github.com

 

[SlowmoDK]

No, I have seen no indication that it has been fixed upstream unfortunately. Either the broken commit only got merged in later kernel versions (most tests I've seen are with 6.5 and 6.6), or the devices you tested with just happened to be compatible with that link negotiation behavior change. After all, it doesn't break all devices and driver combinations, just most of them. Also, my understanding is that two X553 NICs both running the same "broken" driver will work fine.

VyOS switched back to the out-of-tree driver yesterday due to this very issue.

Kernel: T5887: T5619: T5995: Update to v6.6.15, add IXGBE out-of-tree drivers and Huawei NIC driver (backport #490) by mergify[bot] · Pull Request #491 · vyos/vyos-build

This is an automatic backport of pull request #490 done by Mergify. Mergify commands and options More conditions and actions can be found in the documentation. You can also trigger Mergify actio...

github.com

There was also kernel update for proxmox 8 yesterday.. I suspect same issue. x553 might finally work out of box, without tinkering

But someone else have to test


EDIT : nope no dice

proxmox-kernel-6.5 (6.5.11-8) bookworm; urgency=medium

* fix #5077: cherry-pick revert for aacraid resets

* fix #5158: cherry-pick ext4 fix for high-CPU flush

 

[Cpgeek]

I have the latest UEFI QDNVO111.v10 of 08/29/2023 for the Q20321G9 C3558R device. However if I disconnect the power cable and reconnect the device is not booting.

Do I need to do some settings in the UEFI?

Jan. 5th: To answer myself: It's hidden in the UEFI setup: Intel RC Setup → South Bridge Chipset Configuration → State after G3 → Power auto

Thank you so much for this post! I've been looking for this information for the last couple of weeks! I got caught up with so many people kept saying that there was a jumper and then fuzzy instructions on how to make that work (not sure exactly which pins to jump if it even works at all because of the fuzzy image posted). I went out and found way higher quality pics of the board and couldn't exactly match them up... but then I saw your post and like you said, they burried and hid it on us. I would NOT have ever expected the setting to be in this menu AT ALL, nor would I expect it to be called "state after g3" whatever the heck that is... but thank you so much for the guidance!

 

[Cpgeek]

There's a video out now of the 1U verison

NOOOOOOO!!! THIS IS EXACTLY THE SOLUTION I WAS LOOKING FOR! (and i picked up a Q20331G9-S10 a couple weeks ago).
my plan is to model and 3d print a rack shelf for it (but it'll be 3u due to the limitations of the unit), and of course I'll need to put a usb fan on it. wish I'd waited but I doubt i'm going to wait for this new 1u unit to drop and go through the hassle of buying one and reselling the old unit at this point. *sigh*

 

[Cpgeek]

Is the ~30 seconds before the system POSTs normal for these units? I nearly threw out some RAM cause I thought it was bad, but no system just took forever before it would beep and present BIOS.

yes, it takes a good long time to do checks pre-post.

 

[Marjan]

In some of the previous posts, someone mentioned that this unit will not work with DDR4 3200MHz. Just to share my experience with this.
I got my unit couple of weeks ago, bought 2 sticks, 32GB 3200MHz Kingston Value RAM. Of course it didn't boot, even after waiting for few minutes. I hear beep from PC speaker buthing on screen.
Since I don't have any other RAM to try, I went back to IT shop where I bought RAM, and they were willing to help.
It did boot with with 2666 MHz memory, doesn't boot with 3200. Until it does for some reason.
Anyway, now it works without any issues, rebooting, shuting down and unplugging from power, staring up again, all good. No idea why.
And yes, it needs around 1 minute to start booting, after beep from PC speaker until there is something on the screen.

 

[blunden]

Thank you so much for this post! I've been looking for this information for the last couple of weeks! I got caught up with so many people kept saying that there was a jumper and then fuzzy instructions on how to make that work (not sure exactly which pins to jump if it even works at all because of the fuzzy image posted). I went out and found way higher quality pics of the board and couldn't exactly match them up... but then I saw your post and like you said, they burried and hid it on us. I would NOT have ever expected the setting to be in this menu AT ALL, nor would I expect it to be called "state after g3" whatever the heck that is... but thank you so much for the guidance!

Yeah, I also ran across it in that menu a couple of weeks ago when looking through the settings. I didn't expect to find it there, that's for sure.

NOOOOOOO!!! THIS IS EXACTLY THE SOLUTION I WAS LOOKING FOR! (and i picked up a Q20331G9-S10 a couple weeks ago).

my plan is to model and 3d print a rack shelf for it (but it'll be 3u due to the limitations of the unit), and of course I'll need to put a usb fan on it. wish I'd waited but I doubt i'm going to wait for this new 1u unit to drop and go through the hassle of buying one and reselling the old unit at this point. *sigh*

If both of you who mentioned missing out on this model contacted Qotom and asked about buying the 1U case, cooler and PSU, it's possible that they might sell those separately in the same way as they sell this motherboard separately. That's certainly what I would try before rigging up alternative solutions.

These units have already started shipping btw. See the post below if you want to see pictures of it:

https://forums.servethehome.com/index.php?threads/qotom-denverton-fanless-system-with-4-sfp.41331/post-410194

 

[MacGyver16502]

Hey guys !
I got the q20331g9 s10 model C3758R from Amazon, took 2 days to process shipping since I bought it over the weekend on Sep 16 and arrived Sep 21, pretty fast for actually starting shipping Monday and coming from hong honk,

link:
Qotom Q20332G9-S10 Computer 8... https://www.amazon.com/dp/B0CGM3C6D1?ref=ppx_pop_mob_ap_share

Running opnsense can answer any questions you like.

Operating temp is 50 ,

doesn’t come with a bios jump pin ( going to find the power jump pins and going to install a jumper I’ll update the location for who is interested)

The nic for the sfp is x553

there are more than one model see the following link, you can find that when looking for bios update .

http://www.qotom.us/download/Bios/Q20300G9_QDNV01/Q20300G9_Readme.txt

Looks like :
Q20311G9 4 x 1G SFP+ C3338R
Q20311G9 4 x 2.5G SFP+ C3338R
Q20321G9/Q20322G9 2 x 10G SFP+, 2 x 1G SFP+ C3558/C3558R
Q20321G9/Q20322G9 2 x 10G SFP+, 2 x 2.5G SFP+ C3558/C3558R
Q20331G9/Q20332G9 4 x 10G SFP+ x553, 4 x 2.5g + 2.5g wan port C3758R/C3758 (I have this one can confirm this is correct)

I bought the Q20331G9/Q20332G9 as well and am having issues finding transceivers that work... Or mine are dead... Although Proxmox recognizes that they exist. Do you have links to specific ones that are working in your unit?
The switch I'm attaching to is HP Aruba S2500 and it has three rj45 transceivers that have been tested and work. (Cable length is 2 meters)
Does the Q20331G9/Q20332G9 have issues with hot plugging SFP modules that you have noticed?

 

[blunden]

I bought the Q20331G9/Q20332G9 as well and am having issues finding transceivers that work... Or mine are dead... Although Proxmox recognizes that they exist. Do you have links to specific ones that are working in your unit?
The switch I'm attaching to is HP Aruba S2500 and it has three rj45 transceivers that have been tested and work. (Cable length is 2 meters)
Does the Q20331G9/Q20332G9 have issues with hot plugging SFP modules that you have noticed?

Read the last few pages and you'll realize that the ixgbe driver (for the Intel X553 NICs) in the Linux kernel is broken, preventing it from establishing a link in most cases. You'll need to either use Intel's out-of-tree driver (which VyOS now does) or revert the problematic git commit and recompile the driver/the kernel.

These units should already accept third-party transceivers just fine. I've had success with both DACs and Ipolex fiber transceivers (manufactured by 10Gtek). The fiber transceivers are the Intel programmed ones (they happened to be cheaper), but Patrick and others confirmed that there is no vendor lock. No, hot plugging works just fine.

This all assumes that you have a working driver though, which Proxmox doesn't at the moment unless you downgrade the kernel. Note that this hasn't been fixed in later versions of the upstream Linux kernel yet either.

 

[MacGyver16502]

Read the last few pages and you'll realize that the ixgbe driver (for the Intel X553 NICs) in the Linux kernel is broken, preventing it from establishing a link in most cases. You'll need to either use Intel's out-of-tree driver (which VyOS now does) or revert the problematic git commit and recompile the driver/the kernel.

These units should already accept third-party transceivers just fine. I've had success with both DACs and Ipolex fiber transceivers (manufactured by 10Gtek). The fiber transceivers are the Intel programmed ones (they happened to be cheaper), but Patrick and others confirmed that there is no vendor lock. No, hot plugging works just fine.

This all assumes that you have a working driver though, which Proxmox doesn't at the moment unless you downgrade the kernel. Note that this hasn't been fixed in later versions of the upstream Linux kernel yet either.

I did get them working by following this set of instructions:
(The only other thing was to be sure to switch to the non-enterprise repository)
Proxmox SFP+ Port Intel not working

1. Michal Rysanek January 13, 2024 At 1:45 pm
Solved – Build the latest ixgbe driver:
– Download the latest ixgbe source from Intel: Intel® Network Adapter Driver for PCIe* Intel® 10 Gigabit Ethernet Network Connections under Linux*
(pick the top choice – ixgbe-5.19.9.tar.gz)
– untar it to /usr/src/ixgbe-5.19.9
– follow How to download and install ixgbe driver on Ubuntu or Debian (but ignore the wget and tar -xvzf lines, you’ve already downloaded it)
– Don’t forget to add ixgbe to the end of /etc/modules
– Reboot.

Results - I have 1 Intel SFP+ to RJ45 Transceiver, 1 Cheap SFP+ to RJ45 Transceiver, & 1 Cheap SFP+ to Fiber Transceiver - all showing 9.2Gbps+ for transfer rate with iperf.
Current Config - 4 Cores on OpnSense, with 16GB Ram; 2 Cores on PiHole with 4GB Ram; so 2 Cores and 44GB Ram currently unused...

 

[Vesalius]

I did get them working by following this set of instructions:
(The only other thing was to be sure to switch to the non-enterprise repository)
Proxmox SFP+ Port Intel not working

1. Michal Rysanek January 13, 2024 At 1:45 pm
Solved – Build the latest ixgbe driver:
– Download the latest ixgbe source from Intel: Intel® Network Adapter Driver for PCIe* Intel® 10 Gigabit Ethernet Network Connections under Linux*
(pick the top choice – ixgbe-5.19.9.tar.gz)
– untar it to /usr/src/ixgbe-5.19.9
– follow How to download and install ixgbe driver on Ubuntu or Debian (but ignore the wget and tar -xvzf lines, you’ve already downloaded it)
– Don’t forget to add ixgbe to the end of /etc/modules
– Reboot.

Results - I have 1 Intel SFP+ to RJ45 Transceiver, 1 Cheap SFP+ to RJ45 Transceiver, & 1 Cheap SFP+ to Fiber Transceiver - all showing 9.2Gbps+ for transfer rate with iperf.
Current Config - 4 Cores on OpnSense, with 16GB Ram; 2 Cores on PiHole with 4GB Ram; so 2 Cores and 44GB Ram currently unused...

I would add for those doing this on proxmox to install pve-headers instead of linux-headers-$(uname -r).

intel x553 SFP+ ixgbe no go on PVE8

These quit working with the shift to linux kernel 6.*. I reported initially here, but no luck with any of the PVE8 kernels either. I am currently using the last 5.15.108-1-pve kernel successfully otherwise software is latest pve8. Bringing this up again because I have been following along with...

forum.proxmox.com

 

[TheGeekn°72]

NOOOOOOO!!! THIS IS EXACTLY THE SOLUTION I WAS LOOKING FOR! (and i picked up a Q20331G9-S10 a couple weeks ago).
my plan is to model and 3d print a rack shelf for it (but it'll be 3u due to the limitations of the unit), and of course I'll need to put a usb fan on it. wish I'd waited but I doubt i'm going to wait for this new 1u unit to drop and go through the hassle of buying one and reselling the old unit at this point. *sigh*

hahahaha I felt exactly the same when I saw it drop, I had a faint hope that they'd sell the case separately but nope
oh, well, maybe I'll go through the same maneuver some other time, I'm neck deep in network building so I don't have the leisure (nor budget for that matter) to deal with that
if I reach a point where everything is calm and I feel like making a fancy purchase for the sake of having everything rack mounted, then that'll definitely be a thing to do
in the mean time, the industrial box with its USB fan will be a perfectly fine solution !

 

[TheGeekn°72]

NOOOOOOO!!! THIS IS EXACTLY THE SOLUTION I WAS LOOKING FOR!

Good news everyone !

I contacted Qotom through the AliExpress chat, asked them if they would sell the case + PSU + fan/heatsink combo for me to transfer the motherboard of my fanless version into, including shipping, they quoted me 80$ !
Bit pricey but eh, I'm willing to spend the extra cost for properly rack mounting it, it will go very nicely with my brand new Zyxel switch !

 

[Pheckphul]

Got my 1U version in today. Order from the Qotom store on Aliexpress and it took 11 days to get here in the states from the day I ordered. It actually came decently packed. It has a heatsink on the bottom and exhaust out the side. Also added a pic of the power supply. It does come with rack ears and a power cord.

Yeah, that's the same cooling setup that came with the spare mobo I bought from Qotom, except I don't have power and ground wires soldered to mine. I really wish they'd put more thought into the rack-mount unit. As I said in the YouTube comments, this is the first rack-mounted bit of computer kit I've come across in more than 20 years of IT work that has air flow in the bottom of the case. Basically makes it a 2U server, and I can't spare that much real estate in my puny 12U rack at home. I'm gonna toss it on top with my KVM and the other miscellanea.

 

[scampower]

Hey all, did anyone managed to get the serial console port working with Opnsense? Is there a setting I need to enable it? I have enabled serial console in Opnsense but I ain't getting a output from the serial console port.

 

[Garth Shader]

Hey all, did anyone managed to get the serial console port working with Opnsense? Is there a setting I need to enable it? I have enabled serial console in Opnsense but I ain't getting a output from the serial console port.

Short story: I had to use a VGA to HDMI adapter...

Longer story: When I bought mine, I wanted to just use the serial port to load OPNSense. It did not work. I had to order a VGA to HDMI adapter. After that, I think I had to enable the Serial port in the BIOS, but even then, it did not display everything on boot up (IIRC). Once I had the VGA to HDMI adapter, I did not invest too much additional time on getting the serial port working perfectly, as OPNSense is a headless system, and once it is installed, I can just use SSH or the Web Interface..

Nothing beats a machine with IPMI and Full Serial Port support. My SuperMicro server has Serial Port over Ethernet built into the IPMI.. Takes some effort to get working, but it is pretty cool. Though even that does not display everything perfectly, but in a disaster, it would be enough.

 

[blunden]

Short story: I had to use a VGA to HDMI adapter...

Longer story: When I bought mine, I wanted to just use the serial port to load OPNSense. It did not work. I had to order a VGA to HDMI adapter. After that, I think I had to enable the Serial port in the BIOS, but even then, it did not display everything on boot up (IIRC). Once I had the VGA to HDMI adapter, I did not invest too much additional time on getting the serial port working perfectly, as OPNSense is a headless system, and once it is installed, I can just use SSH or the Web Interface...

Same here, but with VyOS.

Someone in the comments of the STH review claims to have gotten it working though. Never managed to get any further details unfortunately.

 

[Brent Geery]

New BIOS release, 2/22 Index of /download/Bios/Q20300G9_QDNV01

 

[blunden]

New BIOS release, 2/22 Index of /download/Bios/Q20300G9_QDNV01

I wonder what the changelog is. Did anyone see one?

Also, does anyone know what they mean by "update optical port speed" with the use of unlock.bin?