Qotom Denverton fanless system with 4 SFP+

[sko]

Bought the rack mount as well. Noticed that if I have a RJ45 SFP+ Module in one of the sfp cages, it will not cold boot. Power supply lights come on, but power light on the panel does not and it appears dead.

I suspect the unit won't boot because of too much load on the SFP+ ports.
The SFP+ specification allows for either <1W (type 1) or <1.5W (type 2) power consumption - *all* copper modules are WAY over those limits, so they may or may not work and may or may not damage the port either due to high currents or temperatures...

 

[blunden]

I suspect the unit won't boot because of too much load on the SFP+ ports.
The SFP+ specification allows for either <1W (type 1) or <1.5W (type 2) power consumption - *all* copper modules are WAY over those limits, so they may or may not work and may or may not damage the port either due to high currents or temperatures...

While most 10GBASE-T modules are indeed in the 3W range, there are some more modern (and expensive) ones with lower power consumption (1.8W or less) and longer range (100 meter vs. 30 meter).

https://www.fs.com/de-en/products/154916.html?attribute=70728&id=3351321

Ubiquiti also sell one that's a bit cheaper.

It's certainly a possible reason though.

 

[Easynet]

Hi all,

I want to share my experience with QOTOM QDNV01 / Q20331G9 C3758R. This device can handle 10GbE traffic, but I have no clue what happen with my device that all I226-V NICs are triggering errors on PCIe root ports similar to these:
Hardware error from APEI Generic Hardware Error Source: 0
It has been corrected by h/w and requires no further action

Also the driver is crashing randomly. I tried to change the RAM memory, power supply, booting old Linux Kernels. Even in Windows 11 if you set as router and you push traffic between two 2.5GbE you will get WHEA error in System Logs exactly for the same PCIe root ports.

I've opened a thread to Intel community forum for this issue: Intel(R) Atom(TM) CPU C3758R with 5x I226V - Hardware error from APEI Generic Hardware Error Source
This issue is happening when you do bidirectional traffic, like iperf3 -c IP --bidir. In only one direction per port doesn't trigger these errors.
Also I have few videos capturing the kernel errors and driver crash regarding to this issue. I can share them from my Youtube channel.

Does anybody faced this issue with this device?

The seller told me to send it back. He doesn't want to change it for another one because is afraid to not face the same issue. He will refund the amount.

 

[farmerj]

I suspect the unit won't boot because of too much load on the SFP+ ports.
The SFP+ specification allows for either <1W (type 1) or <1.5W (type 2) power consumption - *all* copper modules are WAY over those limits, so they may or may not work and may or may not damage the port either due to high currents or temperatures...

Further testing shows it only has this problem when using the 4th port.

 

[Pheckphul]

Alright, yeah i didn't think the aspeed would support video over usb-c.
That's odd tho, no output from the console port? Whats it there for than i wonder...

Can't wait to hear how others have done it! (looking like ill use my old 17" with vga in the attic)

As I recall, all I had to do was enable console redirection in the BIOS and connect with a standard rollover/Cisco terminal cable. Since this is aimed at being a firewall, down to using a standard Cisco-type RJ45 serial port, not sure why Qotom didn't enable this by default. The decision was probably made by the same person who designed the 1U rack enclosure.

 

[sko]

not sure why Qotom didn't enable this by default. The decision was probably made by the same person who designed the 1U rack enclosure.

I also can't understand this on *all* those network appliances. But as you said: this decision is probably made by the same clown that puts (multiple!) HDMI connectors on such an appliance - but no VGA so you could hook it up to a (networked) KVM swtich... I suppose they think everyone will drag around a spare monitor and keyboard to the rack/comms cabinet for setup or troubleshooting...

 

[Jakorosin]

So... been reading, and watching and deciding and what not, and finally cut the knot. bought the 1u rack version C3758.
Planning to proxmox it with pfsense to use it primarily as router, using disk passthrough for the pfsense that incase proxmox dies, i can boot from the passthrough device directly.

The only thing i'm really wondering about that i cannot find...
How do you guys do the initial installation? Do you actually use the VGA port, is the USB-C display ready, maybe even install proxmox using the console port?

Would love to hear from you guys

Wanted to warn you really quick: you absolutely need to use Proxmox 7.x and not upgrade, or upgrade through to 8.x but pin the kernel at 5.x - Linux 6.x has broken compatibility for the x553 SFP+ drivers and there is no sign of anyone in development really caring about the issue, because it affects so few users. See here.

That aside, I'm really happy with mine. It's running Arista NGFW, a small HomeAssistant instance, Koha, and my Unifi controller, and doing great at it.

 

[blunden]

Further testing shows it only has this problem when using the 4th port.

That's good to know.

Later this year I'll possibly have to use mine with an RJ45 transceiver. That's unless the existing run is too long and I need to use my cheap HiSource switch as a signal repeater, but at that point I'll probably just run fiber from the HiSource to the Qotom.

 

[jsc0]

Has anyone tried adding a fan to the 1u version? What size fan fits and how did you power it?

 

[blunden]

Has anyone tried adding a fan to the 1u version? What size fan fits and how did you power it?

It already has a fan that pulls in air from the bottom as far as I know. Are you talking about adding a second one?

 

[Caennanu]

As I recall, all I had to do was enable console redirection in the BIOS and connect with a standard rollover/Cisco terminal cable. Since this is aimed at being a firewall, down to using a standard Cisco-type RJ45 serial port, not sure why Qotom didn't enable this by default. The decision was probably made by the same person who designed the 1U rack enclosure.

Thank you for this. This should most definately help! But ill have to hook it up to a screen first to do that... kinda defeats the purpose

Wanted to warn you really quick: you absolutely need to use Proxmox 7.x and not upgrade, or upgrade through to 8.x but pin the kernel at 5.x - Linux 6.x has broken compatibility for the x553 SFP+ drivers and there is no sign of anyone in development really caring about the issue, because it affects so few users. See here.

That aside, I'm really happy with mine. It's running Arista NGFW, a small HomeAssistant instance, Koha, and my Unifi controller, and doing great at it.

Alright, that is very good to know, don't want bugs now!

---
update: received the 1u rack version this afternoon (7 day delivery including weekend). Now to wait for the miss's to stop using her screen so i can 'borrow' the VGA to hdmi cable... :d

 

[jsc0]

It already has a fan that pulls in air from the bottom as far as I know. Are you talking about adding a second one?

Yes. Seems like these boards get really hot, and I'm planning on putting mine in a hot closest. I was hoping to order a few fans before my Qotom is delivered.

 

[Caennanu]

Wanted to warn you really quick: you absolutely need to use Proxmox 7.x and not upgrade, or upgrade through to 8.x but pin the kernel at 5.x - Linux 6.x has broken compatibility for the x553 SFP+ drivers and there is no sign of anyone in development really caring about the issue, because it affects so few users. See here.

That aside, I'm really happy with mine. It's running Arista NGFW, a small HomeAssistant instance, Koha, and my Unifi controller, and doing great at it.

i'm installing, or rather trying to, right now.
But for some reason its not seeing my Sata m.2 drives.
Been trying to figure out what im missing in the bios, so far no succes.

Any suggestions?

Or does it only support pci-e? .....

 

[sko]

That's not a SATA port, it's PCIe x2

 

[Caennanu]

That's not a SATA port, it's PCIe x2

yeah... i was afraid off that :S
Ah well, time to return the Sata ones!

Was just hoping maybe it would have a Sata mode, but i geuss it doesnt.

 

[farmerj]

Has anyone tried adding a fan to the 1u version? What size fan fits and how did you power it?

Put a 40x40x10 5V fan on top of a couple of _very_ low profile nvme ssd heat sinks, complete with my super custom coated paper clip retainer . 5v power can be had from the other "sata power" 4 pin header, or from one of the 10pin! usb headers. Still experimenting around with it. There is room for another 40x40x10 fan outboard fresh air side of the power supply, and of course, 12v available directly from the power supply.

Think the ultimate solution would be to mount sata drive on 1" or taller standoffs (if you are using one), then cut a huge 80x80 hole in the bottom of the middle of the case there. There has to be clearance on the bottom of the case for the other fan, so nothing lost (except any chance of returning it) putting another hole on that side. Mount 80mm fan 10mm or 20mm thick and run it slowly for minimal noise. Should get airflow out both ends of the case and in the same direction as the existing fan. Going to see how I get along with just the nvme cooling setup first before adding any more fans. Have a couple intel optane P1600X in there and those doogers used to run hot nvme-pci 48C case open, 59C case closed. The fan I mounted on the nvmes works much better than I expected, nvme-pci 39C with case open with fan control turned all the way down. CPUs with case open 48C-44. All of these temps are in cool 21C room, very lightly loaded.

The unoccupied 4 pin sata power connector is a "Mini Micro Jst 2.0 Ph 4 Pin Connector" pitch 2mm/0.078 and... I have wayyy toooo many I ordered off amazon. No idea what size really the usb connectors are, no standard 9 pin header will work of course, and a 10 pin I ordered was way too large. Motherboard front panel jumpers will work, but only across the rows, pitch is wrong. Planning to control fan speed with this: https://www.amazon.com/dp/B07P2BLG2L?ref=ppx_yo2ov_dt_b_product_details&th=1 Right now I'm using a usb version of the same controller.

 

[farmerj]

Wanted to warn you really quick: you absolutely need to use Proxmox 7.x and not upgrade, or upgrade through to 8.x but pin the kernel at 5.x - Linux 6.x has broken compatibility for the x553 SFP+ drivers and there is no sign of anyone in development really caring about the issue, because it affects so few users. See here.

That aside, I'm really happy with mine. It's running Arista NGFW, a small HomeAssistant instance, Koha, and my Unifi controller, and doing great at it.

this is the way: "use Proxmox 7.x and not upgrade, or upgrade through to 8.x but pin the kernel at 5.x"

you can build use the out of tree intel driver, however performance is horrible compared to 5.x in tree drivers. latest intel drivers: ~4G,10G rx/tx, Linux 5.x drivers: ~7G,10G.

I'm also pretty happy with it. ~40w from the wall, and would do better if I hadn't put 64G, 2x optanes, and one sata 2.5 in it. Lots of room to add whatever I please. Good for a homelab if you like to fiddle about with hardware/software.

Also, if you're using pfsense or opensense pcie passthrough will bypass this problem for those machines. On proxmox 7 when creating the devices, check the "PCI-Express" box only, no others. Device should end up looking like "0000:0d:00.0,pcie=1,rombar=0 Also, according to various sources, memory balloon=0, machine type q35. These settings seem working for me. Someone in this conversation has even posted performance tuning params. I've yet to get 10G into the house, so experimentation is just through a cheap switch. Have the latest bios. Have a selection of copper 10G cables, none over 1m, all seem to work fine. Have a couple of cisco sfp+s and fiber cable which also seem to work fine. Additonally, haven't found any way to make any of the 10G ports auto-neg or force them down to 1G. Also, if you're using 64G you may have to lower ram speed.

 

[Caennanu]

Also, if you're using pfsense or opensense pcie passthrough will bypass this problem for those machines. On proxmox 7 when creating the devices, check the "PCI-Express" box only, no others. Device should end up looking like "0000:0d:00.0,pcie=1,rombar=0 Also, according to various sources, memory balloon=0, machine type q35. These settings seem working for me. Someone in this conversation has even posted performance tuning params. I've yet to get 10G into the house, so experimentation is just through a cheap switch. Have the latest bios. Have a selection of copper 10G cables, none over 1m, all seem to work fine. Have a couple of cisco sfp+s and fiber cable which also seem to work fine. Additonally, haven't found any way to make any of the 10G ports auto-neg or force them down to 1G. Also, if you're using 64G you may have to lower ram speed.

Alright, yes i am using PCIE passthrough for the harddrive, all SFP's and 4 of 5 RJ45's.
(1 LAN port dedicated too Proxmox acces, till i can figure out how to bridge that one through pfsense)
I have however checked the 'All functions' checkbox and Rom-bar tho?

I have used UEFI with Q35., no ballooning.

Still tweaking and double checking ISP settings (PPPOE... and IGMP ITV) and portforwards / static ip's before i swap it over.
Just a bit too many docker containers and other server types relying on that.

 

[farmerj]

Alright, yes i am using PCIE passthrough for the harddrive, all SFP's and 4 of 5 RJ45's.
(1 LAN port dedicated too Proxmox acces, till i can figure out how to bridge that one through pfsense)
I have however checked the 'All functions' checkbox and Rom-bar tho?
View attachment 36508
I have used UEFI with Q35., no ballooning.

Still tweaking and double checking ISP settings (PPPOE... and IGMP ITV) and portforwards / static ip's before i swap it over.
Just a bit too many docker containers and other server types relying on that.

Found errors in dmesg if I checked anything other than just the pci-express in 7.x proxmox. Never found a really good explanation of what each option was for, other than you should have them checked if you are passing through a gpu. If anyone knows, pls point me to a doc or clue me in.

 

[Caennanu]

Found errors in dmesg if I checked anything other than just the pci-express in 7.x proxmox. Never found a really good explanation of what each option was for, other than you should have them checked if you are passing through a gpu. If anyone knows, pls point me to a doc or clue me in.

Alright, i have no errors whatsoever.... maybe its specific to your nic's?

Also, any suggestions on how to bridge the Proxmox nic? would feel kinda dumb to put a patch between port 5 and 1-4