Qotom Denverton fanless system with 4 SFP+

[GuybrushThreepwood]

Oh this is an issue I'm facing on the Dell Edge 640 box as well. Anything with kernel 6.x is having this issue with link never showing up (Proxmox 8, Debian 12, Ubuntu 24.04 nightly build, VyOS 1.4/1.5 rolling).

The only time it could work is if the DAC cable is directly connected between 2 servers. But once through the switch, it never links up after OS fully loaded.

I have one of the Dell Edge 640 running now with Proxmox 8 by installing Proxmox 8 ISO, then uploading to it PVE 7's kernel deb files:
pve-headers-5.19.17-2-pve_5.19.17-2_amd64.deb
pve-kernel-5.19.17-2-pve_5.19.17-2_amd64.deb

And then pinning it to boot. Then it will work with the DAC and switch. Give it a try?

 

[SlowmoDK]

Oh this is an issue I'm facing on the Dell Edge 640 box as well. Anything with kernel 6.x is having this issue with link never showing up (Proxmox 8, Debian 12, Ubuntu 24.04 nightly build, VyOS 1.4/1.5 rolling).

The only time it could work is if the DAC cable is directly connected between 2 servers. But once through the switch, it never links up after OS fully loaded.

I have one of the Dell Edge 640 running now with Proxmox 8 by installing Proxmox 8 ISO, then uploading to it PVE 7's kernel deb files:
pve-headers-5.19.17-2-pve_5.19.17-2_amd64.deb
pve-kernel-5.19.17-2-pve_5.19.17-2_amd64.deb

And then pinning it to boot. Then it will work with the DAC and switch. Give it a try?

Thanks for chiming in

This is a known fix/workaround but doesn't solve the underlying problem in 6.xx
Proxmox 7.4 works out of the box with x553, so if new SDN features are not needed, u can just stick to 7.4, should work for years still
but again not a real fix

 

[TheGeekn°72]

Yes, the platform should support dual-channel memory configurations. However, multiple DIMMs will draw slightly more power and the Netgate router/firewall applicances built on the same platform don't bother with dual-channel, so presumably the performance difference is negligible when used as a router/firewall. This has been discussed earlier in the thread.

Here there, me again !

So we've established that dual channel isn't worth it at this power scale and the perf gap is negligible, OK

I've looked into my spare parts stock and found two sticks I could use :
- 4GB @ 2400 1Rx16
- 8GB @ 3200 1Rx8
Would the lower bandwidth of the 1Rx16 stick affect performance or should I go with the 8GB stick ? (I know this chip can only handle as high as 2400MHz in transfer speed anyways)

 

[blunden]

Here there, me again !

So we've established that dual channel isn't worth it at this power scale and the perf gap is negligible, OK

I've looked into my spare parts stock and found two sticks I could use :
- 4GB @ 2400 1Rx16
- 8GB @ 3200 1Rx8
Would the lower bandwidth of the 1Rx16 stick affect performance or should I go with the 8GB stick ? (I know this chip can only handle as high as 2400MHz in transfer speed anyways)

Yes, it likely wouldn't make a meaningful difference to go dual channel. Latency is likely more important than memory bandwidth in this case. Because of that, I would probably go with the 8 GB 1Rx8 stick.

Note that the device can apparently be a bit finicky when it comes to RAM compatibility, so you might have to try both of them and neither one is guaranteed to work.

 

[TheGeekn°72]

Yes, it likely wouldn't make a meaningful difference to go dual channel. Latency is likely more important than memory bandwidth in this case. Because of that, I would probably go with the 8 GB 1Rx8 stick.

Note that the device can apparently be a bit finicky when it comes to RAM compatibility, so you might have to try both of them and neither one is guaranteed to work.

Thanks for the reply ! Good news is, I retrieved my package today, repasted the CPU, installed the NVMe, put the 4GB stick inside and booted it up, had a quick look through the BIOS options (although I just corrected the time difference with China's timezone, didn't have to update BIOS as it came flashed with the latest revision straight from factory), I am currently through the install process for OPNsense bare metal and pleased to report that it works fine so far !

Once I am done with this, I'll swap the RAM stick then, that's a quick operation anyways...

Will post a small gallery of pictures I took during that install process (once I'll be done figuring out why I can't access the damned web GUI '-_-)

 

[SlowmoDK]

I found that 2666 MHz ram was better than 3200Mhz .. i had 2 kits of 3200 that didn't work/not boot unit, but all my 2666Mhz sticks seems to downclock to and work at 2400MHZ for CPU.

So if u bought barebone unit ... don't buy 3200 buy max 2666 or 2400.. ideal look for the best timing sticks at 2400Mhz

Happy hunting crew

 

[TheGeekn°72]

I found that 2666 MHz ram was better than 3200Mhz .. i had 2 kits of 3200 that didn't work/not boot unit, but all my 2666Mhz sticks seems to downclock to and work at 2400MHZ for CPU.

So if u bought barebone unit ... don't buy 3200 buy max 2666 or 2400.. ideal look for the best timing sticks at 2400Mhz

Happy hunting crew

Oh I didn't buy anything, both my 4GB@2400 1Rx16 and 8GB@3200 1Rx8 were spare parts I already had on hand and they both work just fine anyway

My current problem is a bit more serious though... I can't get any of the stacked RJ-45 ports to work... I tried to get one of them as a LAN interface but none of them seem to be connecting as intended and I got no idea why ?
OPNsense's install went through just fine, I could assign a WAN port too, my ISP's router would detect it too but when I try to ping it, it tells me that nothing comes up ??
I'll try reinstalling OPNsense, idk why that would do something like that...

 

[SlowmoDK]

My current problem is a bit more serious though... I can't get any of the stacked RJ-45 ports to work... I tried to get one of them as a LAN interface but none of them seem to be connecting as intended and I got no idea why ?
OPNsense's install went through just fine, I could assign a WAN port too, my ISP's router would detect it too but when I try to ping it, it tells me that nothing comes up ??
I'll try reinstalling OPNsense, idk why that would do something like that...

Figuring out the port layout might be the most frustrating bit about this unit..

My recommendation is to assign ip address to all ports (many different lan ports) and use ping/dhcp to see witch one u connect to

 

[TheGeekn°72]

My recommendation is to assign ip address to all ports (many lans) and use ping/dhcp to see witch one u connect to

That's a pretty good idea, I'll test it out right away !

 

[TheGeekn°72]

My recommendation is to assign ip address to all ports (many different lan ports) and use ping/dhcp to see witch one u connect to

UPDATE : I've assigned an network address to all RJ-45 ports, enabled DHCP, plugged-

wait a minute, I just got a realisation while typing this

...

Just got back from testing. The interfaces didn't let me connect because, and I shit you not, they didn't negotiate.
I was plugging my laptop's GbE port into it, it did NOT want to let me connect whatsoever. I then switched over to my USB-C/2.5G adapter, it linked up immediately and DHCP'd on one of the assigned networks...

How the hell is it that the ports aren't negotiating ???? I sure hope I can enable negotiation because I call BS

 

[SlowmoDK]

So it seems to me that the issue is with the laptop nic

If a usb nic gets ip and works

 

[TheGeekn°72]

So it seems to me that the issue is with the laptop nic

If a usb nic gets ip and works

but it shouldn't be the issue though, I made sure the laptop could negotiate speed, how come the router doesn't try to detect/match the port ?
I use this GbE port often and it always work, why wouldn't it operate normally here ? this is very curious

UPDATE : WHAT THE ??? ok so it detected the 2.5G USB NIC but it negotiated to 100mbps ???

UPDATE 2 : currently running an update, forgot about doing that since I was stuck on that issue, hopefully, that'll fix... whatever's going on here

UPDATE 3 : ran the update, the laptop's 2.5G NIC does appear to be interacting at 2.5G with OPNsense, finally, still no luck with the regular GbE port though... I'll have to look what did I put in as settings in my working OPNsense so that I could get this one to work, although I expect plenty more hurdles sadly... urgh...

 

[TheGeekn°72]

So it seems to me that the issue is with the laptop nic

After hours of very frustrating testing, I have come to the bitter conclusion that you were right
I tried another PC's GbE port, it recognized immediately, autonegotiated just fine...
I have no idea why that port doesn't work on that router tho, there's no reason for it to simply not work

 

[MacD87]

https://www.amazon.com/dp/B09QMC1458?ref=ppx_yo2ov_dt_b_product_details&th=1

added this 5v usb fan to the unit just sat it on top of the device ( Q20331G9 ) and it dropped temps from idle 43c to about 26c. just an FYI

 

[blunden]

For those of you who bought one of these, did the SATA power cable you received look like the one in this picture from the review?

I'm especially interested in the pinout on the motherboard side.

https://www.servethehome.com/wp-content/uploads/2024/01/Qotom-C3758-C3758R-Fanless-Internal-Configured.jpg

My unit came with a 90W CWT power supply by the way. Nice to get one from a PSU ODM/manufacturer that's actually used by well known brands.

 

[TheGeekn°72]

https://www.amazon.com/dp/B09QMC1458?ref=ppx_yo2ov_dt_b_product_details&th=1

added this 5v usb fan to the unit just sat it on top of the device ( Q20331G9 ) and it dropped temps from idle 43c to about 26c. just an FYI

DUDE LMAO, I bought the same thing, put it in the same way and obtained the same results hahahahaha
(Too damn bad we can't post pics, just one is "too large"... '-_-)

For those of you who bought one of these, did the SATA power cable you received look like the one in this picture from the review? I'm especially interested in the pinout on the motherboard side.

yup that's the one, no idea about the pinout though, I assume they're for power eh

 

[kryptonian]

Funny thing, similar thing is also available from TekLager.se. The one with C3558R: TLSense C3558R: 2x 10Gbps SFP+, 7x 2.5Gbps LAN, Atom C3558R CPU, and there used to be one with C3758, and I have the latter version.

Something to keep in mind, is that it seems the SFP+ ports might have some uh, driver issues in the Linux kernel, and as a result most of the times it will not link up even if the SFP+ module is regonized. If you use the Intel's out-of-tree driver for it however, it does link-up and work perfectly with 10G but your milage may vary. I'm currently building VyOS with the out-of-tree driver as a result, and releases and source is available: Releases · samipsolutions/vyos-build. (Related VyOS task: T5619 Update the Intel ixgbe driver due to issues with Intel X533)

 

[kryptonian]

Last 4 entries in syslog for ixgbe ... it even sees my DAC, then no more messages, and link drops from switch

Jan 07 02:07:07 kernel: ixgbe 0000:0d:00.1 eno4: entered allmulticast mode
Jan 07 02:07:07 kernel: ixgbe 0000:0d:00.1 eno4: entered promiscuous mode
Jan 07 02:07:07 kernel: ixgbe 0000:0d:00.1: registered PHC device on eno4
Jan 07 02:07:07 kernel: ixgbe 0000:0d:00.1 eno4: detected SFP+: 4

Same entries but for 2.5 gb

Jan 07 02:07:07 kernel: igc 0000:08:00.0 enp8s0: entered allmulticast mode
Jan 07 02:07:07 kernel: igc 0000:08:00.0 enp8s0: entered promiscuous mode
Jan 07 02:07:10 kernel: igc 0000:08:00.0 enp8s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX

Link never comes up in a functional state on the X553

Your follow-up acticle on this, indicates to the reader that this issue is fixed.

Maybe include edit to clarify that several others don't' see working x553 as you have, with the same setup ?

Cheers and fingers crossed we get this fixed soon

The problem is the ixgbe driver in the kernel unfortunately. The out-of-tree driver works, in-kernel one does not. I haven't bisected the differences, but BSD also works out of the box so something is weird with the negotiation on the in-kernel driver.
I would love to know how it worked out of the box on the review unit.

Please try this VyOS build and it should just work: Release v1.4-ixgbe-rolling-05012024T0310 · samipsolutions/vyos-build

For reference my VyOS's mod info on ixgbe running the custom VyOS 1.4 with the out-of-tree driver:

$ modinfo ixgbe
filename:       /lib/modules/6.1.61-amd64-vyos/updates/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version:        5.19.6
license:        GPL
description:    Intel(R) 10GbE PCI Express Linux Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
srcversion:     3EFD1D228DB9666D073C855
alias:          pci:v00008086d000015E5sv*sd*bc*sc*i*
alias:          pci:v00008086d000015E4sv*sd*bc*sc*i*
alias:          pci:v00008086d000015CEsv*sd*bc*sc*i*
alias:          pci:v00008086d000015CCsv*sd*bc*sc*i*
alias:          pci:v00008086d000015CAsv*sd*bc*sc*i*
alias:          pci:v00008086d000015C8sv*sd*bc*sc*i*
alias:          pci:v00008086d000015C7sv*sd*bc*sc*i*
alias:          pci:v00008086d000015C6sv*sd*bc*sc*i*
alias:          pci:v00008086d000015C4sv*sd*bc*sc*i*
alias:          pci:v00008086d000015C3sv*sd*bc*sc*i*
alias:          pci:v00008086d000015C2sv*sd*bc*sc*i*
alias:          pci:v00008086d000015AEsv*sd*bc*sc*i*
alias:          pci:v00008086d000015ADsv*sd*bc*sc*i*
alias:          pci:v00008086d000015ACsv*sd*bc*sc*i*
alias:          pci:v00008086d000015ABsv*sd*bc*sc*i*
alias:          pci:v00008086d000015B0sv*sd*bc*sc*i*
alias:          pci:v00008086d000015AAsv*sd*bc*sc*i*
alias:          pci:v00008086d000015D1sv*sd*bc*sc*i*
alias:          pci:v00008086d00001563sv*sd*bc*sc*i*
alias:          pci:v00008086d00001560sv*sd*bc*sc*i*
alias:          pci:v00008086d00001558sv*sd*bc*sc*i*
alias:          pci:v00008086d0000154Asv*sd*bc*sc*i*
alias:          pci:v00008086d00001557sv*sd*bc*sc*i*
alias:          pci:v00008086d0000154Dsv*sd*bc*sc*i*
alias:          pci:v00008086d00001528sv*sd*bc*sc*i*
alias:          pci:v00008086d000010F8sv*sd*bc*sc*i*
alias:          pci:v00008086d0000151Csv*sd*bc*sc*i*
alias:          pci:v00008086d00001529sv*sd*bc*sc*i*
alias:          pci:v00008086d0000152Asv*sd*bc*sc*i*
alias:          pci:v00008086d000010F9sv*sd*bc*sc*i*
alias:          pci:v00008086d00001514sv*sd*bc*sc*i*
alias:          pci:v00008086d00001507sv*sd*bc*sc*i*
alias:          pci:v00008086d000010FBsv*sd*bc*sc*i*
alias:          pci:v00008086d00001517sv*sd*bc*sc*i*
alias:          pci:v00008086d000010FCsv*sd*bc*sc*i*
alias:          pci:v00008086d000010F7sv*sd*bc*sc*i*
alias:          pci:v00008086d00001508sv*sd*bc*sc*i*
alias:          pci:v00008086d000010DBsv*sd*bc*sc*i*
alias:          pci:v00008086d000010F4sv*sd*bc*sc*i*
alias:          pci:v00008086d000010E1sv*sd*bc*sc*i*
alias:          pci:v00008086d000010F1sv*sd*bc*sc*i*
alias:          pci:v00008086d000010ECsv*sd*bc*sc*i*
alias:          pci:v00008086d000010DDsv*sd*bc*sc*i*
alias:          pci:v00008086d0000150Bsv*sd*bc*sc*i*
alias:          pci:v00008086d000010C8sv*sd*bc*sc*i*
alias:          pci:v00008086d000010C7sv*sd*bc*sc*i*
alias:          pci:v00008086d000010C6sv*sd*bc*sc*i*
alias:          pci:v00008086d000010B6sv*sd*bc*sc*i*
depends:
retpoline:      Y
name:           ixgbe
vermagic:       6.1.61-amd64-vyos SMP preempt mod_unload modversions
parm:           IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2 (array of int)
parm:           InterruptType:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default IntMode (deprecated) (array of int)
parm:           MQ:Disable or enable Multiple Queues, default 1 (array of int)
parm:           DCA:Disable or enable Direct Cache Access, 0=disabled, 1=descriptor only, 2=descriptor and data (array of int)
parm:           RSS:Number of Receive-Side Scaling Descriptor Queues, default 0=number of cpus (array of int)
parm:           VMDQ:Number of Virtual Machine Device Queues: 0/1 = disable (1 queue) 2-16 enable (default=8) (array of int)
parm:           max_vfs:Number of Virtual Functions: 0 = disable (default), 1-63 = enable this many VFs (array of int)
parm:           VEPA:VEPA Bridge Mode: 0 = VEB (default), 1 = VEPA (array of int)
parm:           InterruptThrottleRate:Maximum interrupts per second, per vector, (0,1,956-488281), default 1 (array of int)
parm:           LLIPort:Low Latency Interrupt TCP Port (0-65535) (array of int)
parm:           LLIPush:Low Latency Interrupt on TCP Push flag (0,1) (array of int)
parm:           LLISize:Low Latency Interrupt on Packet Size (0-1500) (array of int)
parm:           LLIEType:Low Latency Interrupt Ethernet Protocol Type (array of int)
parm:           LLIVLANP:Low Latency Interrupt on VLAN priority threshold (array of int)
parm:           FdirPballoc:Flow Director packet buffer allocation level:
            1 = 8k hash filters or 2k perfect filters
            2 = 16k hash filters or 4k perfect filters
            3 = 32k hash filters or 8k perfect filters (array of int)
parm:           AtrSampleRate:Software ATR Tx packet sample rate (array of int)
parm:           FCoE:Disable or enable FCoE Offload, default 1 (array of int)
parm:           MDD:Malicious Driver Detection: (0,1), default 1 = on (array of int)
parm:           LRO:Large Receive Offload (0,1), default 0 = off (array of int)
parm:           allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599 based adapters, default 0 = Disable (array of int)
parm:           dmac_watchdog:DMA coalescing watchdog in microseconds (0,41-10000), default 0 = off (array of int)
parm:           vxlan_rx:VXLAN receive checksum offload (0,1), default 1 = Enable (array of int)

 

[blunden]

yup that's the one, no idea about the pinout though, I assume they're for power eh

Thanks for confirming. It turned out that the forum user I bought it from mistakenly included the cable from a different mini PC that used the same connector on the motherboard side, but a different pinout. Easy mistake to make when trying to ship it to me as soon as possible. I would also have assumed that the motherboard side connector has a standardized pinout, but no. Once we figured that out, he was able to find the correct cable and send it to me. I just wanted to confirm that Qotom hadn't made any changes to that part since they clearly multi-source some components. Otherwise I wouldn't have gotten a higher quality power supply from a reputable PSU manufacturer (CWT).

Yes, it's 5V and ground.

Funny thing, similar thing is also available from TekLager.se. The one with C3558R: TLSense C3558R: 2x 10Gbps SFP+, 7x 2.5Gbps LAN, Atom C3558R CPU, and there used to be one with C3758, and I have the latter version.

Yeah, I saw that too when looking at their offerings since people on Swedish forums tend to suggest Teklager when people are asking about mini-PCs for routing.

I would say it's pretty clear that Qotom is the OEM they use for those two TLSense boxes. TekLager then add a sizeable margin on top for warehousing the units locally, support and other services like preinstalling your OS of choice.

The problem is the ixgbe driver in the kernel unfortunately. The out-of-tree driver works, in-kernel one does not. I haven't bisected the differences, but BSD also works out of the box so something is weird with the negotiation on the in-kernel driver.
I would love to know how it worked out of the box on the review unit.

Please try this VyOS build and it should just work: Release v1.4-ixgbe-rolling-05012024T0310 · samipsolutions/vyos-build

The negotiation behavior implemented by the potentially problematic commit mentioned in your links and also mentioned in here previously seems to be required on devices on both ends for that change to work reliably.

Since Patrick tested with two devices that both had that driver change, it makes sense that it worked in their tests. The problem is that most other devices and OS:es don't.

Thanks for the VyOS builds btw. I'll give those a go when experimenting with VyOS, which I'm kind of interested in doing.

 

[kryptonian]

The negotiation behavior implemented by the potentially problematic commit mentioned in your links and also mentioned in here previously seems to be required on devices on both ends for that change to work reliably.

Since Patrick tested with two devices that both had that driver change, it makes sense that it worked in their tests. The problem is that most other devices and OS:es don't.

My VyOS running the linked image is hooked up to a Juniper EX3300, running JunOS 15.1R7-S4 and works fine. Milage may definitely vary at least with the problematic commit.