Qotom Denverton fanless system with 4 SFP+

[blunden]

Fellas, I need your help

Since we have established that the SFP+ ports aren't vendor locked, I looked into the transceivers I need, however, I found several offers that *seem* to have what I want but I'm not sure which one to pick... I know 10Gtek, I got NICs from them before, works very well but the other brands linked down there, they seem to have good reviews and be cheaper but no idea if I should stick with a brand I know for an extra or go for a better price...
You guys see any discerning details between those that would be a dealbreaker of some sort maybe ?

https://amzn.eu/d/bfwjL0u

https://amzn.eu/d/5DrqADp

https://amzn.eu/d/dCUY9pp

https://amzn.eu/d/hgpfGsB

I bought DAC cables from H!Fiber and fiber transceivers from Ipolex late last year. Both of them list "Shenzen 10Gtek Transceivers Co., Ltd" as the manufacturer. As far as I understand, they are both either brands owned by 10Gtek or companies that use 10Gtek as an OEM.

QSFPTEK is a separate company with their own manufacturing though, as far as I know. I've heard about them but don't have any personal experience with their products. They themselves would certainly like to make you think they are the best in the (clearly biased) comparison below.

Compare 3 SFP manufacturers: QSFPTEK, FS.com and 10Gtek.com : transceivermakerのblog

Optical transceiver (also known as the fiber module) is an indispensable part of the fiber optic communication network. With the large number of modules used in the network, the network budget is constantly increasing. So more and more customers tend to

transceivermaker.livedoor.blog

In general, I would compare power consumption and overall specs. They are likely using very similar (or identical) hardware if the power consumption and range is the same. Some might show temperature and more detailed statistics than the others when queried though. You'd have to read the reviews to find that out.

EDIT: Some of those list 2.5G and 5G speeds. Keep in mind that those speeds will likely only work with devices that support those speeds, which many/most SFP+ devices you find don't. There are modules that can sync up at 10G on the SFP+ side and then rate limit down to those speeds on the RJ45 side, but only Aquantia based modules do that properly. As long as you intend to stick to 1G or 10G, it shouldn't matter though.

Also, keep in mind that they have been programmed with different vendor codes. It shouldn't matter in unlocked NICs and devices though.

 

[TheGeekn°72]

I bought DAC cables from H!Fiber and fiber transceivers from Ipolex late last year. Both of them list "Shenzen 10Gtek Transceivers Co., Ltd" as the manufacturer. As far as I understand, they are both either brands owned by 10Gtek or companies that use 10Gtek as an OEM.

Oh I see, that cuts down my potential options by half then since they're those brands you just mentioned

QSFPTEK is a separate company with their own manufacturing though, as far as I know. I've heard about them but don't have any personal experience with their products. They themselves would certainly like to make you think they are the best in the (clearly biased) comparison below.

Compare 3 SFP manufacturers: QSFPTEK, FS.com and 10Gtek.com : transceivermakerのblog

Optical transceiver (also known as the fiber module) is an indispensable part of the fiber optic communication network. With the large number of modules used in the network, the network budget is constantly increasing. So more and more customers tend to

transceivermaker.livedoor.blog

Yeah, I read the product page of their offer, I found quite the number of inconsistencies in the claims they were making like "this can't handle 1000mbps links" and then listing that data rate with a max cable range for it further down the slideshow

In general, I would compare power consumption and overall specs. They are likely using very similar (or identical) hardware if the power consumption and range is the same. Some might show temperature and more detailed statistics than the others when queried though. You'd have to read the reviews to find that out.

EDIT: Some of those list 2.5G and 5G speeds. Keep in mind that those speeds will likely only work with devices that support those speeds, which many/most SFP+ devices you find don't. There are modules that can sync up at 10G on the SFP+ side and then rate limit down to those speeds on the RJ45 side, but only Aquantia based modules do that properly. As long as you intend to stick to 1G or 10G, it shouldn't matter though.

I did try to look for reviews but annoyingly, they all say the same kind of things... thankfully some of those do mention working at 2.5G in a very common ISP router here so having the 10G/2.5G/1G array is already plenty enough for me, that's what I got, that's what I'll use, I don't have nor expect to acquire 5G rated hardware in pretty much any scenario
Aquantia based modules ? What are those ?

Also, keep in mind that they have been programmed with different vendor codes. It shouldn't matter in unlocked NICs and devices though.

Oh the only fiber device I own is the mini PC this thread is about and we've established it was unlocked so all the better for me, I'll just have to make sure a given PCIe NIC isn't vendor locked on the SFP port when I'll inevitably get one although that won't happen for a while still

 

[Patrick]

@Patrick the C3758 and C3758R have different PCIe lane counts (R gets +4). Any way to tell if this system design takes any advantage of that? Pretty much the only reason I could see in jumping to the "better" CPU since the compute difference doesn't seem to be worth it.

It does not from what I can see. Still x2 bifurcation.

Did you test if the SFP+ ports are vendor locked or not? That's pretty common on Intel NICs after all.

I just looked and we have some cheapo and random SFP+ optics in there, so I would say not.

@Patrick nice review

But regarding support, SOC is now so old that support is slipping ...

For instance the 10G X553's don't work in proxmox 8, Scale or any modern linux with kernel 6.x

This is not my experience. It was running in Ubuntu, but we installed fresh Poxmox VE 8.1.3 and iperf on both hosts, that is a very standard 10GbE result

 

[SlowmoDK]

This is not my experience. It was running in Ubuntu, but we installed fresh Poxmox VE 8.1.3 and iperf on both hosts, that is a very standard 10GbE result

Nice find

That's great news for this box, it must be the latest .3 update that fixed this issue

I tried latest Proxmox Backup Server on unit last night, that didn't work on the x553 NICs

 

[TheGeekn°72]

I just looked and we have some cheapo and random SFP+ optics in there, so I would say not.

This is not my experience. It was running in Ubuntu, but we installed fresh Poxmox VE 8.1.3 and iperf on both hosts, that is a very standard 10GbE result

Excellent ! I was concerned by those two aspects, although the people of this thread did essentially clear up the first one

Question : you guys mentioned in the video the SIM slot, saying that you'd come back to it later... and did not come back to it ?
Was it edited out or something ? is there something wrong with it ?
I was wondering, since it didn't appear to have antenna ports beyond the holes in the rear plate, what was it doing here, would a SIM chip be used as a safety measure maybe ? (also, do you know of any software/OS that can control/use SIMs ?)

 

[blunden]

I just looked and we have some cheapo and random SFP+ optics in there, so I would say not.

Lines up with what other reports here as well then. Thanks for providing additional confirmation.

Question : you guys mentioned in the video the SIM slot, saying that you'd come back to it later... and did not come back to it ?
Was it edited out or something ? is there something wrong with it ?
I was wondering, since it didn't appear to have antenna ports beyond the holes in the rear plate, what was it doing here, would a SIM chip be used as a safety measure maybe ? (also, do you know of any software/OS that can control/use SIMs ?)

My guess is that the SIM slot is there in case someone plugs in an 4G/5G module. It could also be there for when they produce a version of these boxes as an OEM/ODM for some other company.

I don't think the SIM card slot is meant to be used for TPM functionality, although I'm just speculating here.

 

[TheGeekn°72]

My guess is that the SIM slot is there in case someone plugs in an 4G/5G module. It could also be there for when they produce a version of these boxes as an OEM/ODM for some other company.

I don't think the SIM card slot is meant to be used for TPM functionality, although I'm just speculating here.

Hm, I suppose the M.2 slot that's not for storage is (probably ? I gotta double check) an E-key for those wifi/4G/5G modules then ?
I knew wifi modules went in there, didn't know 4G/5G modules for M.2 slots also were a thing though, I'll look it up too, that sounds interesting... I wonder if OPNsense supports 4G/5G modules too though, that would be interesting

 

[blunden]

Hm, I suppose the M.2 slot that's not for storage is (probably ? I gotta double check) an E-key for those wifi/4G/5G modules then ?
I knew wifi modules went in there, didn't know 4G/5G modules for M.2 slots also were a thing though, I'll look it up too, that sounds interesting... I wonder if OPNsense supports 4G/5G modules too though, that would be interesting

I don't remember what Key of M.2 they normally use (I think it be the same as what Wi-Fi modules use) but 4G/5G modules that connect over M.2 are pretty common in laptops, even though they might not necessarily use the PCI-E part of the M.2 connector.

Not sure about the support for 4G/5G modules in FreeBSD. Even Linux support isn't great sometimes. Besides driver support, there is also the issue of requiring proprietary FCC unlock sequences on newer modules due to US regulations (which unfortunately affect the rest of the world too). While the unlock sequences usually get reverse engineered, they sometimes get changed to make it harder because the vendors are required to make it more difficult to use these modules on unapproved platforms.

You can also have issues with firmware bugs that only meaningfully impact Linux and BSD, so hopefully your module is from a manufacturer who cares about solving such issues. I ran into those issues on my Lenovo work laptop, which Lenovo thankfully helped to resolve.

 

[Bobby Sharpe]

This is not my experience. It was running in Ubuntu, but we installed fresh Poxmox VE 8.1.3 and iperf on both hosts, that is a very standard 10GbE result

I'm wondering how you did get working that with Proxmox.

I have 2 boxes (Q20321G9 C3558R and Q20332G9 C3758). With OPNsense all interfaces are working incl. the 10Gb SFP+. With Proxmox the 10Gb are not working. It recognises the interfaces, but shows always LINK DOWN. As I understand it's because of Linux 6.x kernel. Hope it will be fixed soon.

 

[TheGeekn°72]

I don't remember what Key of M.2 they normally use (I think it be the same as what Wi-Fi modules use) but 4G/5G modules that connect over M.2 are pretty common in laptops, even though they might not necessarily use the PCI-E part of the M.2 connector.

As it turns out, yes, this is for 4G/5G but to my surprise, it is a B-key, would have thought that like wifi cards, it would have been an E-key... Oh well

Not sure about the support for 4G/5G modules in FreeBSD. Even Linux support isn't great sometimes. Besides driver support, there is also the issue of requiring proprietary FCC unlock sequences on newer modules due to US regulations (which unfortunately affect the rest of the world too). While the unlock sequences usually get reverse engineered, they sometimes get changed to make it harder because the vendors are required to make it more difficult to use these modules on unapproved platforms.

Looks like I'll be lurking around OPNsense/FreeBSD forums then...

You can also have issues with firmware bugs that only meaningfully impact Linux and BSD, so hopefully your module is from a manufacturer who cares about solving such issues. I ran into those issues on my Lenovo work laptop, which Lenovo thankfully helped to resolve.

Hmmmm, welp, that sounds like extensive research but that wouldn't be the first time I get over invested in something I probably won't proceed with

 

[blunden]

Hmmmm, welp, that sounds like extensive research but that wouldn't be the first time I get over invested in something I probably won't proceed with

Thankfully, there were already others with similar issues so I only had to participate in the thread and confirm that my logs were similar to the ones others were seeing. A couple of weeks or months later, I ended up having to install Windows in a KVM VM and passthrough the device in order to update the firmware as it took several additional months before they published the firmware update in LVFS to allow updating directly from Linux. Since then it has worked mostly fine.

 

[Patrick]

I'm wondering how you did get working that with Proxmox.

I have 2 boxes (Q20321G9 C3558R and Q20332G9 C3758). With OPNsense all interfaces are working incl. the 10Gb SFP+. With Proxmox the 10Gb are not working. It recognises the interfaces, but shows always LINK DOWN. As I understand it's because of Linux 6.x kernel. Hope it will be fixed soon.

Downloaded Proxmox VE 8.1.3. Installed it. Added SFP+ modules and connected OM4 fiber. Assigned IP addresses and set to autostart. Everything worked immediately.

 

[Bobby Sharpe]

Downloaded Proxmox VE 8.1.3. Installed it. Added SFP+ modules and connected OM4 fiber. Assigned IP addresses and set to autostart. Everything worked immediately.

Very strange. I have the same version of Proxmox (fresh installed). Neither a 10Gb DAC nor a 10Gb fiber is working. After power on the green LEDs are on as long as the UEFI boot is in process. As soon as the Proxmox operation system (Debian) is starting the LEDs goes off and later 'ip a' shows for these interfaces LINK DOWN.

However if I start a original Debian with Linux 5.x kernel all interfaces are working.

 

[SlowmoDK]

Maybe it's fixed in the enterprise repo, because i sure still have issues, not working for me here

@Patrick Can you please retest using switches and DAC not just directly between boxes ?

 

[SlowmoDK]

Very strange. I have the same version of Proxmox (fresh installed). Neither a 10Gb DAC nor a 10Gb fiber is working. After power on the green LEDs are on as long as the UEFI boot is in process. As soon as the Proxmox operation system (Debian) is starting the LEDs goes off and later 'ip a' shows for these interfaces LINK DOWN.

However if I start a original Debian with Linux 5.x kernel all interfaces are working.

Ya fresh install not working for me either

 

[SlowmoDK]

The only other thing that could be different i can think off, is the firmware version of the X553's

Not home atm, but will test later

How to Check the Firmware Version of Intel® Ethernet Adapters...

Command to identify the firmware version

www.intel.com

 

[Patrick]

Maybe it's fixed in the enterprise repo, because i sure still have issues, not working for me here

@Patrick Can you please retest using switches and DAC not just directly between boxes ?

Before we left Friday, we had them hooked up to a MikroTik switch via cheapo 10Gtek SFP+ optics.

 

[SlowmoDK]

Before we left Friday, we had them hooked up to a MikroTik switch via cheapo 10Gtek SFP+ optics.

I'm using the exact same setup ... during reboot link is active (visible in winbox) but as soon as proxmox loads link drops from switch..

Is this a subscription vs non-subscription issue perhaps ?


!!! CALLING ALL BUYERS OF THIS BOX !!!
(maybe a new tread is needed specific to this issue)

Can anyone else confirm Patrick findings of working X553 under kernel 6.xx ?

I could love an explanation, preferable with a working fix

This is my x553 info :

driver: ixgbe
version: 6.5.11-7-pve
firmware-version: 0x80000bd7
expansion-rom-version:
bus-info: 0000:0b:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

 

[SlowmoDK]

Last 4 entries in syslog for ixgbe ... it even sees my DAC, then no more messages, and link drops from switch

Jan 07 02:07:07 kernel: ixgbe 0000:0d:00.1 eno4: entered allmulticast mode
Jan 07 02:07:07 kernel: ixgbe 0000:0d:00.1 eno4: entered promiscuous mode
Jan 07 02:07:07 kernel: ixgbe 0000:0d:00.1: registered PHC device on eno4
Jan 07 02:07:07 kernel: ixgbe 0000:0d:00.1 eno4: detected SFP+: 4

Same entries but for 2.5 gb

Jan 07 02:07:07 kernel: igc 0000:08:00.0 enp8s0: entered allmulticast mode
Jan 07 02:07:07 kernel: igc 0000:08:00.0 enp8s0: entered promiscuous mode
Jan 07 02:07:10 kernel: igc 0000:08:00.0 enp8s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX

Link never comes up in a functional state on the X553

 

[SlowmoDK]

Before we left Friday, we had them hooked up to a MikroTik switch via cheapo 10Gtek SFP+ optics.

Your follow-up acticle on this, indicates to the reader that this issue is fixed.

Maybe include edit to clarify that several others don't' see working x553 as you have, with the same setup ?

Cheers and fingers crossed we get this fixed soon