Qotom Denverton fanless system with 4 SFP+

[sko]

Is there a BSD hypervisor I could try?

On FreeBSD you have bhyve if you really need a fully-fledged VM, otherwise just use jails...

 

[sukerman]

I am a complete Linux noob so am on a very steep learning curve.
I setup my Qotom with Proxmox and have Home Assistant running sweetly but wanted either pFSense or OPNsense to run too.

After days of wondering why whatever I tried the 10Gb SFP+ ports wouldn't work, reading this thread finally helped me understand that the fault lies with Proxmox having drivers for the Intel X553 SFP+. I'd tried every combination of transceivers and dac leads thinking it was a hardware issue/user error before this.

The Qotom 10Gb SFPs work fine if I run OPNSense directly on it. I don't have the Linux knowledge to run an OOC driver for Proxmox and I don't want to make the system less secure.

I would still like to run both HA and a firewall on the Qotom as this was my intended use for it;
-Is there a realistic chance Proxmox might add the full drivers to their distribution?
-Should I look at another hypervisor solution with drivers for the X553s?
-Would it be a crazy idea to use TrueNAS Scale as the host for HA and OPNSense as VMs? - I have at least got that to work on another system (though not sure if this would drive the X553s)?

Thanks for any advice!

Installing the OOT drivers for Proxmox is not hard, you need to do this: (but search for it I typed this from memory).

- apt install gcc make pve-kernel-headers
- reboot
- download and unzip the drivers you can get from the intel site,
- go into the src folder and type make install

That said you probably shouldn't if you don't know what you're doing. I've resorted to running proxmox 7.4, works out of the box. because I have 64Gb of ram I don't want to waste so I want to run some other vm's not just pfsense / opnsense bare metal. Updates stopped for Proxmox 7 31-July-2024 though..... But I'm not too worried as long as the VM's are up to date and there's no exposed ports on Proxmox.

I'm still getting bad performance with pfsense through bare metal or not, between two machines connected with DAC to the same 10G switch. iperf3 maxes out at 2.5Gb even if I run lots of threads. At 4 threads I get less that 1Gbit/s. Something is not right but I am not getting an rx_errors any more. Dissappointed to still be messing about trying everything to get this unit to run properly almost a month later. Impossible to know if I have a faulty unit or not, if it was Amazon I'd be returning it already, but I don't know how much hassle it is to return to AliExpress.

Whether I'm missing some miracle bios setting is also impossible to know.

Anybody got this working properly?

 

[blunden]

I found this too which sounded encouraging....

SOLVED - x553 networking support

Some more info from SuperMicro, at the moment the board don't support any form of Legacy boot and only option is to install with UEFI boot support. Wow that's a new one. Not a real problem, but certainly unusual.

www.truenas.com

That's talking about FreeNAS Core, which is based on FreeBSD, not Linux.

 

[Guinan]

That's talking about FreeNAS Core, which is based on FreeBSD, not Linux.

Thank you....hadn't spotted that.

I've had a bit of success binding the 10Gb SFPs to the OPNSense VM. That seems to work for now.
OPNSense sees them and I can get a connection. Will keep on experimenting and if I succeed will post a guide for other noobs....

 

[Vesalius]

Potential fix found. It is not a Proxmox or Truenas issue, but has always been a direct linux Kernel issue that affected all distributions using them.

[SOLVED] - intel x553 SFP+ ixgbe no go on PVE8

Did you do a make AND make install Then reboot? I'll assume this was directed at me. Yes, I did. I haven't been able to find a combination that achieves ~10/~10 Gbps in both directions. Usually incoming is fine but outgoing is ridiculously low which is just... odd. It implies the optics...

forum.proxmox.com

There is a problematic commit from March 2022 in kernel 6.1 (Non-functional ixgbe driver between Intel X553 chipset and Cisco switch via kernel >=6.1 under Debian)
It was later reverted in 6.9 and backported to LTS Kernels 6.6 and 6.1 in May 2024.
Unfortunately PVE does not use LTS kernels and as such does not profit from the kernel maintainers work.

I manually applied the revert of the patch (https://patchwork.kernel.org/projec...karound-v1-1-50f80f261c94@intel.com/#25860468) to a 6.8.8-4 pve kernel I built from source.

As soon as I booted from this patched kernel all the SFP+ came up and connected to my switch.
Without the patch only one of the SFP+ comes up (eno4, port on the bottom right.)

If the proxmox team can apply this patch to their pve kernel this will fix the SFP+ on all platforms using Intel C3xxx/X553.

This thread, intel x553 SFP+ ixgbe no go on PVE8, on the proxmox forums, has posts that will walk you through the commands for with manual install or DKMS install of the latest 5.20.10 OOT ixgbe intel driver that works.

 

[farmerj]

I've had one of these running for just under 3 months now (24/7, opnsense fw) with no issues, however today the device has just suddenly stopped working. Like something has shorted something on the board. The light on the included power supply flashes and thats about it. Just wanted to see if anyone else has experienced this and/or experienced Qotom's after sales support / warranty process? I brought mine via aliexpress.

Mine does this cold booting when using the 4th 10G port. Seems to work okay without anything in the 4th 10G port.

 

[oks2024]

Hello!

I just received my Qotom mini pc (C3758 version) and I'm having some issue with the ram. I bought 2x32 Gb of Timetec 2666 MHz.

When I use the two memory stick I have no signal, no beep. I waited a long time (more than 10 minutes) and nothing.

If I put only one memory stick on the bottom slot (marked as SODIMM1 if I remember correctly, I don't have the pc with me right now), same behavior, nothing happens. I tried both memory sticks.

But I use only one memory stick on the top slot, then after a while I have a beep, video output, and I was able to access bios and install proxmox. Again here I tried the two memory sticks, and everything is fine.

Do you know if it's a standard behavior when the memory is not fully compatible ? Or does it shows that one of my ram slot is not working properly ?

Is there a bios option that would improve ram compatibility ? I tried a few things, but only managed to make it worse .

Also, do you know where I can find a manual, or any kind of documentation for this PC. The Qotom website is not really helpful.
Thanks!

 

[VivienM]

Mine does this cold booting when using the 4th 10G port. Seems to work okay without anything in the 4th 10G port.

Mine seems to be doing exactly that now too...

 

[xephael]

Mine does this cold booting when using the 4th 10G port. Seems to work okay without anything in the 4th 10G port.

Mine seems to be doing exactly that now too...

On eno4 or the top left port it's failing to pull or recognize SFP+s for me in dmesg or ethtool -m eno4 (works on eno1-3)

Using intel 6.6.30-1-lts ixbge driver.

Anyone else see similar?

 

[Vesalius]

Proxmox released proxmox-kernel-signed-6.8.12-1 to the no-subscription repository, which sources Ubuntu-6.8.0-43.43 (where ubuntu finally reverts the ixgbe 553 breakage from back in Linux Kernel 6.1).

Installed and works after reboot without the OOT intel IXGBE installed for me. Have not done any iperf.

 

[blunden]

Mine does this cold booting when using the 4th 10G port. Seems to work okay without anything in the 4th 10G port.

By 4th 10G port you mean the one in the bottom when looking at the ports head on, right? In other words, eth3?

SFP+:

eth0 eth2
eth1 eth3

I was planning to use the 1st and 4th ports for WAN and LAN respectively because I wanted to minimize heat transfer between the two ports as much as possible since I'm forced to use a 10GBASE-T transceiver (one of the expensive lower power ones) in my WAN port. I could just switch to the other two ports though, as long as I know which one to avoid.

 

[VivienM]

By 4th 10G port you mean the one in the bottom when looking at the ports head on, right? In other words, eth3?

SFP+:

eth0 eth2
eth1 eth3

I was planning to use the 1st and 4th ports for WAN and LAN respectively because I wanted to minimize heat transfer between the two ports as much as possible since I'm forced to use a 10GBASE-T transceiver (one of the expensive lower power ones) in my WAN port. I could just switch to the other two ports though, as long as I know which one to avoid.

That's what was going on with me, yup. The thing wouldn't come to life with a 10GBaseT transceiver (not plugged into anything) on what you call eth3.

And I had the same idea - eth0 for DAC LAN, eth3 for future WAN.

 

[oks2024]

Hello!

I just received my Qotom mini pc (C3758 version) and I'm having some issue with the ram. I bought 2x32 Gb of Timetec 2666 MHz.

When I use the two memory stick I have no signal, no beep. I waited a long time (more than 10 minutes) and nothing.

If I put only one memory stick on the bottom slot (marked as SODIMM1 if I remember correctly, I don't have the pc with me right now), same behavior, nothing happens. I tried both memory sticks.

But I use only one memory stick on the top slot, then after a while I have a beep, video output, and I was able to access bios and install proxmox. Again here I tried the two memory sticks, and everything is fine.

Do you know if it's a standard behavior when the memory is not fully compatible ? Or does it shows that one of my ram slot is not working properly ?

Is there a bios option that would improve ram compatibility ? I tried a few things, but only managed to make it worse .

Also, do you know where I can find a manual, or any kind of documentation for this PC. The Qotom website is not really helpful.
Thanks!

Quick update just in case someone has the same questions.

The ram was the issue. I ordred a set of 2x32gb corsair vengeance, and it boots without issue, with one stick or two.

So now proxmox is up and running, and I can start doing some tests.

 

[Ctullu]

First, thanks for any assistance here, I am a newb at this type of setup. Has anyone else had this issue with proxmox?

To set the stage here, I am using the 1U rackmount device and the SFP+ interface names in proxmox are eno1, eno2, eno3, eno4 and the RJ45 interfaces are enp4s0, enp5s0, enp6s0, enp7s0, and enp8s0. Nothing unusual to me here.

My interfaces are assigned to Linux bridges and assigned to VMs (OPNSense and HomeAssistant). The SFP+ interface is set for a bridge passed to OPNSense for my LAN interface, which works great.

My issue is that the only SFP+ interface that will work for me is eno4 (the top left SFP+ port when looking at the device head-on). If I select any other SFP interface for the Linux bridge, the connection seems to go down. I have tried each of the other three SFP interfaces, but once they are activated (after selecting "apply configuration") the connection seems to drop (the lights for that particular SFP port will turn off and stop blinking).

I am using some cheap Amazon SFP+ adapter cables (https://a.co/d/abK9Dtr). I am unsure if those might be the issue or something else in my setup.

Additionally, in case it make a difference, I am running this through a Mokerlink 2G080610GSM.

 

[blunden]

I am using some cheap Amazon SFP+ adapter cables (https://a.co/d/abK9Dtr). I am unsure if those might be the issue or something else in my setup.

Additionally, in case it make a difference, I am running this through a Mokerlink 2G080610GSM.

At this point I can't really explain why only one port works for you. I assume you have checked that you are using Proxmox with a Linux kernel that have the revert needed to make the X533 work in Linux again?

I have two of those DAC cables as well and they work fine in the tests I've done, as long as the switch properly calibrates the signal values. My cheap HiSource switch does so just fine, while I had to get Hasivo to release a new firmware update with improved DAC support. Mokerlink is just rebranding Hasivo switches btw.

Can you link the switch you have? I didn't find anything by searching that model number.

Have you tried it with any other 10G gear?

 

[Ctullu]

At this point I can't really explain why only one port works for you. I assume you have checked that you are using Proxmox with a Linux kernel that have the revert needed to make the X533 work in Linux again?

I have two of those DAC cables as well and they work fine in the tests I've done, as long as the switch properly calibrates the signal values. My cheap HiSource switch does so just fine, while I had to get Hasivo to release a new firmware update with improved DAC support. Mokerlink is just rebranding Hasivo switches btw.

Can you link the switch you have? I didn't find anything by searching that model number.

Have you tried it with any other 10G gear?

First, thanks for the quick reply.

For the kernel question, I will have to go back and look. I honestly don’t remember how I have it configured off the top of my head.

Here is the switch I am using:

Amazon.com

i haven’t looked into firmware updates, but now I’m having problems accessing the switch’s web management page, but I am probably just forgetting something simple (I changed the access IP from the default at some point, maybe a factory reset will fix this)

 

[blunden]

First, thanks for the quick reply.

For the kernel question, I will have to go back and look. I honestly don’t remember how I have it configured off the top of my head.

Here is the switch I am using:

Amazon.com

i haven’t looked into firmware updates, but now I’m having problems accessing the switch’s web management page, but I am probably just forgetting something simple (I changed the access IP from the default at some point, maybe a factory reset will fix this)

The issue I had with my RTL9301 based Hasivo switch was that it didn't properly calibrate signal values when you plugged in a DAC. Since DACs essentially just extend the serial communication normally used to communicate with the transceiver modules (a distance of few centimeters) across a cable directly to another device (up to 7 meters), the signal needs to be adapted to support that. Otherwise it will become unstable or simply not link up at all.

Are you sure that you saved that change to the startup config? By default, all changes are made to the running config (in ram). You need to explicitly tell it to write it to the startup config stored in flash using the Save button that might not be visible if your browser window isn't wide enough.

If you have CLI access, you can usually view the running and startup config there.

There are usually hidden pages where you can change the factory default configuration if you can't get your IP address change to stick any other way.

 

[blunden]

I seem to recall people here claiming that the SFP+ vendor lock was disabled by default on this unit, but that doesn't appear to be the case for me except for DAC cables (which Intel never vendor locks). It refuses to link with transceivers that don't have Intel's vendor OUI, even if I unload the driver and then reload it again with allow_unsupported_sfp=1,1,1,1.

FlexOptix was nice enough to release a new vendor configuration for me within a few hours so it's all good, but it's still irritating that the lock is still there. My Intel coded Ipolex fiber transceivers also work, but I had hoped such things wouldn't be necessary. Unfortunately there doesn't seem to be any X553 unlocking tools available either.

EDIT: Actually, I got a non-Intel transceiver working now. Perhaps it's just extremely picky about other parts of the EEPROM contents.

 

[rukiri]

Technical issues aside, is anyone getting anything close to the 18gbps routing/firewall performance of a Netgate 8200?
It sounds like people aren't seeing more than 7gbs?

 

[Ctullu]

I seem to recall people here claiming that the SFP+ vendor lock was disabled by default on this unit, but that doesn't appear to be the case for me except for DAC cables (which Intel never vendor locks). It refuses to link with transceivers that don't have Intel's vendor OUI, even if I unload the driver and then reload it again with allow_unsupported_sfp=1,1,1,1.

FlexOptix was nice enough to release a new vendor configuration for me within a few hours so it's all good, but it's still irritating that the lock is still there. My Intel coded Ipolex fiber transceivers also work, but I had hoped such things wouldn't be necessary. Unfortunately there doesn't seem to be any X553 unlocking tools available either.

EDIT: Actually, I got a non-Intel transceiver working now. Perhaps it's just extremely picky about other parts of the EEPROM contents.

I’m
In either case, it’s something to look into. On the switch side of things, I can use any of the SFP+ ports on the switch with the one I have working with my OPNSense VM, no issues there. I might look at my config more closely, haven’t had much time between work and other stuff around the house (just moved to a new home, which is also why I am upgrading my network now).