No details, no ETAOh greatMy luck
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Happy belated New Year! I finally got my R86s G2 Neat little device - but still in the honeymoon phase - trying to get it to work they way I expected . . . One issue is the Mellanox ports - I got a DAC cable and it works fine with a QNAP 2108C switch - but wanted to use the other port with a 1Gbe SFP module - and even though I can see the module using ethtool (see below) - I can't get it to work. Tried both LinuxMint and Proxmox so far . . . Maybe this module is simply not compatible but not sure if I'm doing something wrong - device seems up but can't ping in or out - just itself:
lspci |grep -i mellanox
05:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
lshw -class network -businfo
Bus info Device Class Description
======================================================
pci@0000:01:00.0 enp1s0 network Ethernet Controller I225-V
pci@0000:02:00.0 enp2s0 network Ethernet Controller I225-V
pci@0000:03:00.0 enp3s0 network Ethernet Controller I225-V
pci@0000:05:00.0 enp5s0 network MT27500 Family [ConnectX-3]
pci@0000:05:00.0 enp5s0d1 network Ethernet interface
vmbr0 network Ethernet interface
root@R86SFox:~# ./mlxup
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX3
Part Number: MCX342A-XCC_Ax
Description: ConnectX-3 EN NIC for OCP;10GbE;dual-port SFP+;PCIe3.0 x8;IPMI disabled;R6
PSID: MT_1680110023
PCI Device Name: 0000:05:00.0
Port1 MAC: 0002c9ca13ec
Port2 MAC: 0002c9ca13ed
Versions: Current Available
FW 2.42.5000 2.42.5000
PXE 3.4.0752 3.4.0752
UEFI 14.11.0045 14.11.0045
Status: Up to date
dmesg | grep mlx
[ 1.531528] mlx4_core: Mellanox ConnectX core driver v4.0-0
[ 1.531551] mlx4_core: Initializing 0000:05:00.0
[ 8.945175] mlx4_core 0000:05:00.0: DMFS high rate steer mode is: disabled performance optimized steering
[ 8.945541] mlx4_core 0000:05:00.0: 31.504 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x4 link at 0000:00:1c.4 (capable of 63.008 Gb/s with 8.0 GT/s PCIe x8 link)
[ 8.987351] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
[ 8.987602] mlx4_en 0000:05:00.0: Activating port:1
[ 8.990136] mlx4_en: 0000:05:00.0: Port 1: Using 4 TX rings
[ 8.990143] mlx4_en: 0000:05:00.0: Port 1: Using 4 RX rings
[ 8.990538] mlx4_en: 0000:05:00.0: Port 1: Initializing port
[ 8.991237] mlx4_en 0000:05:00.0: registered PHC clock
[ 8.991432] mlx4_en 0000:05:00.0: Activating port:2
[ 8.991887] mlx4_en: 0000:05:00.0: Port 2: Using 4 TX rings
[ 8.991889] mlx4_en: 0000:05:00.0: Port 2: Using 4 RX rings
[ 8.992121] mlx4_en: 0000:05:00.0: Port 2: Initializing port
[ 8.992673] mlx4_core 0000:05:00.0 enp5s0: renamed from eth0
[ 9.000504] mlx4_en: enp5s0: Link Up
[ 9.017359] mlx4_core 0000:05:00.0 enp5s0d1: renamed from eth1
[ 9.020558] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0
[ 9.021495] <mlx4_ib> mlx4_ib_add: counter index 2 for port 1 allocated 1
[ 9.021498] <mlx4_ib> mlx4_ib_add: counter index 3 for port 2 allocated 1
[ 11.543580] mlx4_en: enp5s0d1: Steering Mode 1
[ 11.554598] mlx4_en: enp5s0d1: Link Down
[ 11.604491] mlx4_en: enp5s0: Steering Mode 1
[ 11.614840] mlx4_en: enp5s0: Link Up
lsmod | grep mlx
mlx4_ib 196608 0
ib_uverbs 147456 1 mlx4_ib
mlx4_en 122880 0
ib_core 385024 6 rdma_cm,mlx4_ib,iw_cm,ib_iser,ib_uverbs,ib_cm
mlx4_core 327680 2 mlx4_ib,mlx4_en
root@R86SFox:~# ethtool -i enp5s0d1
driver: mlx4_en
version: 4.0-0
firmware-version: 2.42.5000
expansion-rom-version:
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
root@R86SFox:~# ethtool -m enp5s0d1
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x00 (unknown or unspecified)
Transceiver codes : 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x01
Transceiver type : Ethernet: 1000BASE-T
Transceiver type : Extended: 100G AOC or 25GAUI C2M AOC with worst BER of 5x10^(-5)
Encoding : 0x01 (8B/10B)
BR, Nominal : 1300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 0m
Length (62.5um) : 0m
Length (Copper) : 100m
Length (OM3) : 0m
Laser wavelength : 0nm
Vendor name : QSFPTEK
Vendor OUI : 00:00:00
Vendor PN : QT-SFP-T
Vendor rev : A
Option values : 0x00 0x10
Option : TX_DISABLE implemented
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : BQT220317200
Date code : 220321
Optical diagnostics support : No
root@R86SFox:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:f0:cb:ee:c0:ab brd ff:ff:ff:ff:ff:ff
3: enp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:f0:cb:ee:c0:ac brd ff:ff:ff:ff:ff:ff
4: enp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:f0:cb:ee:c0:ad brd ff:ff:ff:ff:ff:ff
5: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether 00:02:c9:ca:13:ec brd ff:ff:ff:ff:ff:ff
6: enp5s0d1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 00:02:c9:ca:13:ed brd ff:ff:ff:ff:ff:ff
inet 192.168.30.19/24 scope global enp5s0d1
valid_lft forever preferred_lft forever
7: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:02:c9:ca:13:ec brd ff:ff:ff:ff:ff:ff
inet 192.168.10.19/24 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::202:c9ff:feca:13ec/64 scope link
valid_lft forever preferred_lft forever
Any hints welcome!
Neat little device - but still in the honeymoon phase - trying to get it to work they way I expected . . . One issue is the Mellanox ports - I got a DAC cable and it works fine with a QNAP 2108C switch - but wanted to use the other port with a 1Gbe SFP module - and even though I can see the module using ethtool (see below) - I can't get it to work. Tried both LinuxMint and Proxmox so far . . . Maybe this module is simply not compatible but not sure if I'm doing something wrong - device seems up but can't ping in or out - just itself:lspci |grep -i mellanox
05:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
lshw -class network -businfo
Bus info Device Class Description
======================================================
pci@0000:01:00.0 enp1s0 network Ethernet Controller I225-V
pci@0000:02:00.0 enp2s0 network Ethernet Controller I225-V
pci@0000:03:00.0 enp3s0 network Ethernet Controller I225-V
pci@0000:05:00.0 enp5s0 network MT27500 Family [ConnectX-3]
pci@0000:05:00.0 enp5s0d1 network Ethernet interface
vmbr0 network Ethernet interface
root@R86SFox:~# ./mlxup
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX3
Part Number: MCX342A-XCC_Ax
Description: ConnectX-3 EN NIC for OCP;10GbE;dual-port SFP+;PCIe3.0 x8;IPMI disabled;R6
PSID: MT_1680110023
PCI Device Name: 0000:05:00.0
Port1 MAC: 0002c9ca13ec
Port2 MAC: 0002c9ca13ed
Versions: Current Available
FW 2.42.5000 2.42.5000
PXE 3.4.0752 3.4.0752
UEFI 14.11.0045 14.11.0045
Status: Up to date
dmesg | grep mlx
[ 1.531528] mlx4_core: Mellanox ConnectX core driver v4.0-0
[ 1.531551] mlx4_core: Initializing 0000:05:00.0
[ 8.945175] mlx4_core 0000:05:00.0: DMFS high rate steer mode is: disabled performance optimized steering
[ 8.945541] mlx4_core 0000:05:00.0: 31.504 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x4 link at 0000:00:1c.4 (capable of 63.008 Gb/s with 8.0 GT/s PCIe x8 link)
[ 8.987351] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
[ 8.987602] mlx4_en 0000:05:00.0: Activating port:1
[ 8.990136] mlx4_en: 0000:05:00.0: Port 1: Using 4 TX rings
[ 8.990143] mlx4_en: 0000:05:00.0: Port 1: Using 4 RX rings
[ 8.990538] mlx4_en: 0000:05:00.0: Port 1: Initializing port
[ 8.991237] mlx4_en 0000:05:00.0: registered PHC clock
[ 8.991432] mlx4_en 0000:05:00.0: Activating port:2
[ 8.991887] mlx4_en: 0000:05:00.0: Port 2: Using 4 TX rings
[ 8.991889] mlx4_en: 0000:05:00.0: Port 2: Using 4 RX rings
[ 8.992121] mlx4_en: 0000:05:00.0: Port 2: Initializing port
[ 8.992673] mlx4_core 0000:05:00.0 enp5s0: renamed from eth0
[ 9.000504] mlx4_en: enp5s0: Link Up
[ 9.017359] mlx4_core 0000:05:00.0 enp5s0d1: renamed from eth1
[ 9.020558] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0
[ 9.021495] <mlx4_ib> mlx4_ib_add: counter index 2 for port 1 allocated 1
[ 9.021498] <mlx4_ib> mlx4_ib_add: counter index 3 for port 2 allocated 1
[ 11.543580] mlx4_en: enp5s0d1: Steering Mode 1
[ 11.554598] mlx4_en: enp5s0d1: Link Down
[ 11.604491] mlx4_en: enp5s0: Steering Mode 1
[ 11.614840] mlx4_en: enp5s0: Link Up
lsmod | grep mlx
mlx4_ib 196608 0
ib_uverbs 147456 1 mlx4_ib
mlx4_en 122880 0
ib_core 385024 6 rdma_cm,mlx4_ib,iw_cm,ib_iser,ib_uverbs,ib_cm
mlx4_core 327680 2 mlx4_ib,mlx4_en
root@R86SFox:~# ethtool -i enp5s0d1
driver: mlx4_en
version: 4.0-0
firmware-version: 2.42.5000
expansion-rom-version:
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
root@R86SFox:~# ethtool -m enp5s0d1
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x00 (unknown or unspecified)
Transceiver codes : 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x01
Transceiver type : Ethernet: 1000BASE-T
Transceiver type : Extended: 100G AOC or 25GAUI C2M AOC with worst BER of 5x10^(-5)
Encoding : 0x01 (8B/10B)
BR, Nominal : 1300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 0m
Length (62.5um) : 0m
Length (Copper) : 100m
Length (OM3) : 0m
Laser wavelength : 0nm
Vendor name : QSFPTEK
Vendor OUI : 00:00:00
Vendor PN : QT-SFP-T
Vendor rev : A
Option values : 0x00 0x10
Option : TX_DISABLE implemented
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : BQT220317200
Date code : 220321
Optical diagnostics support : No
root@R86SFox:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:f0:cb:ee:c0:ab brd ff:ff:ff:ff:ff:ff
3: enp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:f0:cb:ee:c0:ac brd ff:ff:ff:ff:ff:ff
4: enp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:f0:cb:ee:c0:ad brd ff:ff:ff:ff:ff:ff
5: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether 00:02:c9:ca:13:ec brd ff:ff:ff:ff:ff:ff
6: enp5s0d1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 00:02:c9:ca:13:ed brd ff:ff:ff:ff:ff:ff
inet 192.168.30.19/24 scope global enp5s0d1
valid_lft forever preferred_lft forever
7: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:02:c9:ca:13:ec brd ff:ff:ff:ff:ff:ff
inet 192.168.10.19/24 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::202:c9ff:feca:13ec/64 scope link
valid_lft forever preferred_lft forever
Any hints welcome!
Have you checked that you have set the cards to ethernet mode?
It seems a bit strange that the infiniband driver is loaded.
Thanks Bjorn - I did see that article and installed the MST tool - but can't seem to find out which mode the device is in or whether it's a VPI based model which support both IB & EN - but when I look at the command results - I see no indication of what mode it's in - the DAC cable seems to work fine connected to a QNAP 2108C switch -
I was unsure from the beginning whether the 1Gbe SFP module would work at all on an SFP+ cage - but seeing it under "ethtool -m" gave me hope!
I'm not so experienced with SFP in general and did much homework before I purchased this device . . .
root@R86SFox:~# mst status
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt4099_pciconf0 - PCI configuration cycles access.
domain:bus:dev.fn=0000:05:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
Chip revision is: 01
/dev/mst/mt4099_pci_cr0 - PCI direct access.
domain:bus:dev.fn=0000:05:00.0 bar=0x7fd00000 size=0x100000
Chip revision is: 01
but this says "ethernet":
lshw -class network -businfo
Bus info Device Class Description
======================================================
pci@0000:01:00.0 enp1s0 network Ethernet Controller I225-V
pci@0000:02:00.0 enp2s0 network Ethernet Controller I225-V
pci@0000:03:00.0 enp3s0 network Ethernet Controller I225-V
pci@0000:05:00.0 enp5s0 network MT27500 Family [ConnectX-3]
pci@0000:05:00.0 enp5s0d1 network Ethernet interface
vmbr0 network Ethernet interface
Well this says you have no carrier - so my guess is that the SFP module is not compatible and you need a SFP+ module - I have seen cards before that only supports 10gbps modules and not 1gbps.
That's what I suspected - but wasn't sure . . . Difficult to find any material on what is compatible and what not! I actually contacted the manufacturer and the only suggested one Huawei module . . . My question is whether SFP+ ports are backwards compatible with SFP in general . . . and secondlym why does ethtool see the device and gives no errors? i also see a link status light on my switch (Netgear GS308T) - so there seems to be a link connection of some sort . . . . so close yet so far! next choice is to purchase a SFP+ 10Gbe which does multispeed and use it as a 1Gbe device (the other end is the FTTH modem which only support 1GB) - oh well, back to the drawing board . . .
Strange - Patrick claimed that he could get 1Gb & 10Gb but not 2,5 or 5Gb from the ports . . .
It's really hit or miss with this - so I may have to change my intended NW design (wanted to use one of those SFP+ ports for the FTTH modem (normal 1GB Ethernet BaseT) so I could free up one on the 2,5GB NICs . . . guess that's not happening Attached is a PDF I found which lists "compatible" cables and receivers - there's only ONE RJ45 device from HP on that list . . .
Last edited:
Can confirm that the r86s handles rj45 sfp+ adapters just fine. I’m running two in mine - one Ubiquiti and one of the newer low power adapters from Fiberstore, both running at 10gbe
I’d guess the problem @rfox is seeing is a module incompatibility with the 1gbe sfp module.
Thanks - question is: If I purchase a 10Gbe SFP+ device which is also multispeed (1000, 2500, 5000 & 10000) can I run it at 1Gb speed? That might be an option - like i said, I wanted to use that port for my FTTH modem which only has a 1GB Ethernet port . . .
that frees up one of the 2.5GB portsNo. I have this ocp card and it support 10Gbase-t modules.
Did you look with this command ? it show config of the ports.
Code:
mlxconfig qMany modules and ports using sfp/sfp+ don't do autoconfig with speeds. Did you try to config the port speed to 1Gb/s ?
Code:
ethtool -s enpxxx speed 1000 autoneg offThis list of compatible parts doesn't exclude other hardware. Just try ...Strange - Patrick claimed that he could get 1Gb & 10Gb but not 2,5 or 5Gb from the ports . . .
Next question for those "power" professionals - I have a J4125 based system which I wanted to replace with the N5105 R86s device - on the J4125 model i can get all CPU governors - but the N5105 only two? This is my first N5105 based device . . .
J4125 - I set to ondemand:
root@FoxProxMini:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_available_governors
conservative ondemand userspace powersave performance schedutil
N5105:
root@R86SFox:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
performance powersave
I checked BIOS and "C States" are on - what else am I missing? Any hints welcome . . .
J4125 - I set to ondemand:
root@FoxProxMini:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_available_governors
conservative ondemand userspace powersave performance schedutil
N5105:
root@R86SFox:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
performance powersave
I checked BIOS and "C States" are on - what else am I missing? Any hints welcome . . .
Tried that command with no success - sending SFP module back and trying something else . . . Thanks for your help!
Interesting.
I don't have any insights but thought I'd offer a couple more datapoints.
- Same result on R86s N6005 box running Proxmox 7.3 - only "performance" and "powersave" governors available.
- Same result on Topton N6005 box running Proxmox 7.3 - only "performance" and "powersave" governors available.
- On J4125 box running Proxmox 7.3 you get "conservative ondemand userspace powersave performance schedutil"
So the change appears to be related to Jasper Lake chips.
After more research - it looks like it has to do with whether intel_pstate is enabled or disabled - If enabled, and it's a modern Intel processor - then you get only powersave and performance (because it's being managed by the CPU itself and NOT the kernel scheduler) - and supposedly powersave is more efficient as ondemand . . . I'm trying this out on a few other Proxmox boxes with different Intel CPUs (E-2278, J4125, i5-8529 and N5105)
Last edited:
I have used the 10gbe Onti (bought off AliExpress) sfp in this unit. It will connect internally at 10gbe,but will negotiate with 1, 2.5, 5 & 10 gbe network equipment.
I am just waiting for an optical GPON to connect directly with my ISP.
So far it copes really well with my WG VPN on full line capacity
Here are the SFP modules I have.
https://www.aliexpress.com/item/400...00001402045384!sea&curPageLogUid=hyvNXBSvvzxO
This definitely works. I don’t have 5Gb equipement, but I can connect to my 2.5Gb NIC at the proper speed. The SFP module connect to the r86s at 10Gbe but negotiates directly with your other equipment at the intermediate speeds.
I
I am just waiting for an optical GPON to connect directly with my ISP.
So far it copes really well with my WG VPN on full line capacity
Here are the SFP modules I have.
https://www.aliexpress.com/item/400...00001402045384!sea&curPageLogUid=hyvNXBSvvzxO
This definitely works. I don’t have 5Gb equipement, but I can connect to my 2.5Gb NIC at the proper speed. The SFP module connect to the r86s at 10Gbe but negotiates directly with your other equipment at the intermediate speeds.
I
Which one did you buy exactly? There appears to be one for HP and one "normal" - I guess for Cisco/Mikrotik ?!? Thx.