Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
D

dragonian

Member
Jan 3, 2020
47
30
18
I am running a 6450 and some ruckus r610s's. They work great. The APs are on the MGMT vlan (90), and have trunked vlans on other ssids.
I didn't find this problematic to setup.

FYI, I think I saw that the new fw train for ruckus ap's now will allow you to set the mgmt vlan to a tagged vlan. I haven't checked it out, as it's working fine... but you might want to check it out.

A couple of snippets from my config...
Code: 
vlan 10 name LAN by port
tagged ethe 1/1/15 ethe 1/1/21 to 1/1/24 ethe 1/2/1 ethe 1/2/4
untagged ethe 1/1/1 to 1/1/14 ethe 1/1/16 ethe 1/1/19 ethe 1/2/2 to 1/2/3
router-interface ve 10
spanning-tree
!
vlan 20 name GUEST by port
tagged ethe 1/1/15 ethe 1/1/21 to 1/1/24 ethe 1/2/1
spanning-tree
!
vlan 60 name CAMERA by port
tagged ethe 1/1/15 ethe 1/1/21 to 1/1/24 ethe 1/2/1
untagged ethe 1/1/20
router-interface ve 60
spanning-tree
!
vlan 90 name MGMT by port
tagged ethe 1/1/15 ethe 1/1/21 to 1/1/24 ethe 1/2/1 ethe 1/2/4
untagged ethe 1/1/17 to 1/1/18
router-interface ve 90
spanning-tree
!
interface ethernet 1/1/21
port-name ruckus-office
dual-mode  90
inline power
!
interface ethernet 1/1/22
port-name ruckus-basement
dual-mode  90
inline power
 
  • Like
Reactions: klui
K

klui

༺༻
Feb 3, 2019
919
526
93
dragonian said:
FYI, I think I saw that the new fw train for ruckus ap's now will allow you to set the mgmt vlan to a tagged vlan. I haven't checked it out, as it's working fine... but you might want to check it out.
Click to expand...
I didn't even notice that! Thanks.

System > IP Settings, Management Interface tab, Enable VLAN for Management Interface.
 
J

James Verbunk

New Member
Apr 13, 2018
10
0
1
19
dragonian said:
FYI, I think I saw that the new fw train for ruckus ap's now will allow you to set the mgmt vlan to a tagged vlan. I haven't checked it out, as it's working fine... but you might want to check it out.
Click to expand...
The mgmnt vlan config is only to set a 'stable' web address for the gui.

OK, so in the last 5 mins I got this working. I'm not sure of the mechanism that Ruckus uses to find AP cluster members but the issue turned out that I was blocking something critical on the upstream firewall (pfsense). I noticed that traceroute was always going through FW when I tried to reach other vlans so I got to thinking this was a router-on-a-stick config (which I do not want). So I blocked all traffic from the master AP and logged it. Going through line by line I made specific rules to allow or reject as needed and set these not to log in order to whittle down the log entries. At some point I hit something that made all the members join the cluster.

I'll go through and turn things off to try and cause a cluster heartbeat failure but I'm pretty sure one the master has the member IPs it will connect to them directly (based of off the AP cli config).

Now the big question - can anyone explain why router-on-a-stick config was the key here? I would assume that plugging in two APs should have been enough but I'm not sure why they didn't find each other? :/

Big thanks to @klui and @dragonian for taking the time to reply; hugely appreciated!
 
B

bween

New Member
Dec 24, 2021
10
1
3
Hi all, hoping someone will know how to solve a weird issue.

Switch: ICX 6610
Firewall: OPNsense, 10.10.99.2
DHCP (ics) + DNS (pihole) -> on the same rpi4: 10.10.90.2

Switch config (sh run):
Code: 
Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  no legacy-inline-power
stack disable
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 10 name servers by port
untagged ethe 1/1/13 to 1/1/16
router-interface ve 10
!
vlan 20 name trusted by port
untagged ethe 1/1/21 to 1/1/24
router-interface ve 20
!
vlan 30 name IOT by port
untagged ethe 1/1/25 to 1/1/28
router-interface ve 30
!
vlan 40 name untrusted by port
untagged ethe 1/1/29 to 1/1/30
router-interface ve 40
!
vlan 50 name IPcams by port
untagged ethe 1/1/37 to 1/1/42
router-interface ve 50
!
vlan 90 name DHCP-DNS by port
untagged ethe 1/1/2
router-interface ve 90
!
vlan 99 name TRANSIT by port
untagged ethe 1/1/1
router-interface ve 99
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
enable telnet authentication
hostname brocade
ip dhcp-client disable
ip dns server-address 10.10.90.2
ip route 0.0.0.0/0 10.10.99.2
!
username root password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT-06
!
!
ntp
disable serve
server 216.239.35.0
server 216.239.35.4
!
!
web-management https
!
!
!
!
!
!
!
interface ethernet 1/3/1
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
speed-duplex 10G-full
!
interface ethernet 1/3/5
speed-duplex 10G-full
!
interface ethernet 1/3/6
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
speed-duplex 10G-full
!
interface ve 1
ip address 192.168.1.2 255.255.255.0
!
interface ve 10
port-name VLAN-servers
ip address 10.10.10.1 255.255.255.0
ip helper-address 1 10.10.90.2
!
interface ve 20
port-name VLAN-trusted
ip address 10.10.20.1 255.255.255.0
ip helper-address 1 10.10.90.2
!
interface ve 30
port-name VLAN-IOT
ip address 10.10.30.1 255.255.255.0
ip helper-address 1 10.10.90.2
!
interface ve 40
port-name VLAN-untrusted
ip address 10.10.40.1 255.255.255.0
ip helper-address 1 10.10.90.2
!
interface ve 50
port-name VLAN-IPcams
ip address 10.10.50.1 255.255.255.0
ip helper-address 1 10.10.90.2
!
interface ve 90
port-name DHCP-DNS
ip address 10.10.90.1 255.255.255.0
ip helper-address 1 10.10.90.2
!
interface ve 99
port-name TRANSIT
ip address 10.10.99.1 255.255.255.252
!
!
!
!
!
!
!
!
!
end
/etc/dhcp/dhcpd.conf
Code: 
ddns-update-style none;
authoritative;
allow unknown-clients;
use-host-decl-names on;
log-facility local7;

#VLAN 10 servers
subnet 10.10.10.0 netmask 255.255.255.0 {
range 10.10.10.100 10.10.10.250;
option domain-name-servers 10.10.90.2;
option domain-name "servers";
option routers 10.10.10.1;
option broadcast-address 10.10.10.255;
default-lease-time 3600;
max-lease-time 7200;
}

#VLAN 20 trusted
subnet 10.10.20.0 netmask 255.255.255.0 {
range 10.10.20.100 10.10.20.250;
option domain-name-servers 10.10.90.2;
option domain-name "trusted";
option routers 10.10.20.1;
option broadcast-address 10.10.20.255;
default-lease-time 3600;
max-lease-time 7200;

#VLAN 30 IOT
subnet 10.10.30.0 netmask 255.255.255.0 {
range 10.10.30.100 10.10.30.250;
option domain-name-servers 10.10.90.2;
option domain-name "trusted";
option routers 10.10.30.1;
option broadcast-address 10.10.30.255;
default-lease-time 3600;
max-lease-time 7200;
}

#VLAN 40 untrusted
subnet 10.10.40.0 netmask 255.255.255.0 {
range 10.10.40.100 10.10.40.250;
option domain-name-servers 10.10.90.2;
option domain-name "trusted";
option routers 10.10.40.1;
option broadcast-address 10.10.40.255;
default-lease-time 3600;
max-lease-time 7200;
}

#VLAN 50 IPcams
subnet 10.10.50.0 netmask 255.255.255.0 {
range 10.10.50.100 10.10.50.250;
option domain-name-servers 10.10.90.2;
option domain-name "trusted";
option routers 10.10.50.1;
option broadcast-address 10.10.50.255;
default-lease-time 3600;
max-lease-time 7200;
}

#VLAN 90 DHCP=DNS
subnet 10.10.90.0 netmask 255.255.255.0 {
range 10.10.90.100 10.10.90.250;
option domain-name-servers 10.10.90.2;
option domain-name "DHCP-DNS";
option routers 10.10.90.1;
option broadcast-address 10.10.90.255;
default-lease-time 3600;
max-lease-time 7200;
}

The issue:
I've noticed after connecting the Rpi with ISC + pihole that my internet speed gets severely crippled (expected 400/10, getting 60/10). Without the DHCP server/pihole on the switch, I'm able to get the expected speeds from opnsense. This leads me to think there might be some issue with routing as the issue occurs only AFTER I add DHCP + pihole (maybe packets are getting lost/sent in a roundabout way?). I'm not totally sure where to start looking for the solution. Hoping someone might be able to point me in the right direction. Thanks!
 
Last edited:
K

koyetsu

New Member
Feb 28, 2021
21
12
3
SO trying to connect my ESXi host toi my ICX6610 via 1/2/1 or 1/2/6. VMWare says the link is up but brocade says it is down.
The 6610 sees the transceiver, mtu's are the same. any tips on this?
1640667550575.png
1640667577302.png
1640667612728.png
1640667680566.png

that is all the relevant stuff i can think of to this
NIC: HP InfiniBand FDR/Ethernet 10Gb/40Gb 2P 544FLR-QSFP Adapter
Transceivers: KAIAM QSFP+ 40G-LR4 Lite
 
Last edited:
J

jasonwc

Member
Dec 31, 2018
49
18
8
koyetsu said:
SO trying to connect my ESXi host toi my ICX6610 via 1/2/1 or 1/2/6. VMWare says the link is up but brocade says it is down.
The 6610 sees the transceiver, mtu's are the same. any tips on this?
View attachment 20931
View attachment 20932
View attachment 20933
View attachment 20934

that is all the relevant stuff i can think of to this
NIC: HP InfiniBand FDR/Ethernet 10Gb/40Gb 2P 544FLR-QSFP Adapter
Transceivers: KAIAM QSFP+ 40G-LR4 Lite
Click to expand...
It seems that your NIC natively supports 40Gb ethernet but you may have to set the port to ethernet mode.

On my cross-flashed MCX354A-FCBT, the following command sets both ports to ethernet mode:

Code: 
#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
For other useful commands, see:
I am using the same 40G-LR4 Lite transceivers to connect a Debian server to my ICX6610 and it worked immediately. I have the HP 649281-B21 NIC, cross-flashed to a MCX354A-FCBT with the OEM firmware, with ethernet mode enabled on both ports. I also removed the Flexboot ROM to speed bootup. The switch and server immediately showed an active 40Gb duplex connection and I get DOM on the server using ethtool -m.

Code: 
SSH@ICX6610-48p>show media e 1/2/1
Port   1/2/1:Type  : 40G QSFP Module
Vendor Name: KAIAM CORP       Serial Num: KD60629451      Revision: 1A

SSH@ICX6610-48p>show int e 1/2/1
40GigabitEthernet1/2/1 is up, line protocol is up
  Port up for 4 day(s) 12 hour(s) 5 minute(s) 33 second(s)
  Hardware is 40GigabitEthernet, address is 609c.9f46.394c (bia 609c.9f46.397d)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual 40Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual none

root@storage-server:~# ethtool -m enp2s0
        Identifier                                : 0x0d (QSFP+)
        Extended identifier                       : 0x80
        Extended identifier description           : 2.5W max. Power consumption
        Extended identifier description           : No CDR in TX, No CDR in RX
        Extended identifier description           : High Power Class (> 3.5 W) not enabled
        Connector                                 : 0x07 (LC)
        Transceiver codes                         : 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00
        Transceiver type                          : 40G Ethernet: 40G Base-LR4
        Encoding                                  : 0x00 (unspecified)
        BR, Nominal                               : 10300Mbps
        Rate identifier                           : 0x00
        Length (SMF,km)                           : 2km
        Length (OM3 50um)                         : 0m
        Length (OM2 50um)                         : 0m
        Length (OM1 62.5um)                       : 0m
        Length (Copper or Active cable)           : 0m
        Transmitter technology                    : 0x40 (1310 nm DFB)
        Laser wavelength                          : 1310.000nm
        Laser wavelength tolerance                : 6.500nm
        Vendor name                               : KAIAM CORP
        Vendor OUI                                : 14:ed:e4
        Vendor PN                                 : XQX2502
        Vendor rev                                : 1A
        Vendor SN                                 : KD60629247
        Date code                                 : 16062900
        Revision Compliance                       : SFF-8636 Rev 1.5
        Module temperature                        : 48.07 degrees C / 118.53 degrees F
        Module voltage                            : 3.2435 V
        Alarm/warning flags implemented           : No
        Laser tx bias current (Channel 1)         : 45.744 mA
        Laser tx bias current (Channel 2)         : 39.236 mA
        Laser tx bias current (Channel 3)         : 41.058 mA
        Laser tx bias current (Channel 4)         : 40.114 mA
        Transmit avg optical power (Channel 1)    : 1.5046 mW / 1.77 dBm
        Transmit avg optical power (Channel 2)    : 1.5329 mW / 1.86 dBm
        Transmit avg optical power (Channel 3)    : 1.5074 mW / 1.78 dBm
        Transmit avg optical power (Channel 4)    : 1.5194 mW / 1.82 dBm
        Rcvr signal avg optical power(Channel 1)  : 0.8579 mW / -0.67 dBm
        Rcvr signal avg optical power(Channel 2)  : 0.7901 mW / -1.02 dBm
        Rcvr signal avg optical power(Channel 3)  : 0.5906 mW / -2.29 dBm
        Rcvr signal avg optical power(Channel 4)  : 0.5509 mW / -2.59 dBm
 
Last edited:
D

divide_by_zero

New Member
Dec 25, 2021
4
0
1
Re: Setting up Brocade ICX6430 - I'm getting an error while attempting Initial Configure. When I enter "ip dhcp-client disable" I get "invalid input -> disable".
Code: 
ICX6430-24P Switch(config)#ip dhcp-client disable
Invalid input -> disable
Type ? for a list
ICX6430-24P Switch(config)#
I tried ip dhcp-client ? and "disable" is not one of the 5 listed commands. "Enable" is one of the commands and I entered it, then entered "disable" again and it returned Invalid input as before.

Suggestions, please?
 
K

kpfleming

Active Member
Dec 28, 2021
416
214
43
Pelham NY USA
Hello everyone! Just joined the forum after a friendly geek neighbor clued me in about the ICX boxes and I bought some to rebuild my home LAN (no lab, I yeet to production).

I've got four 7150-C12Ps in a ring stack config; three in the basement of the house, one in the garage, connected via 10Gb SM BiDi links with direct burial cable. I've been mostly using the stack as a fancy layer 2 device since I put it in, but am now starting to move core routing over to it (replacing an EdgeRouter 4). I went with this configuration to provide hardware redundancy since both my wife and I are full-time WFH people, and this way I can have most of our network gear (APs, NAS, etc.) connected across two or more boxes using LAGs and won't lose any connectivity if one of the 7150s fails.

I'm running 09.0.10 firmware, i jumped to 09.0.00 right after it was released since I was having lots of really strange problems with IPv6 traffic on 08.0.95, and those went away with the 09 firmware. I haven't run into any bugs or other weird behavior with the 09 series yet (knock on wood).

Thanks to everyone who has shared their knowledge here and made it possible for us to use this awesome hardware in our home networks. I'm to start watching videos to learn how to setup access-lists on my VLANs, they are very different from VyOS :)
 
  • Like
Reactions: jasonwc and itronin
2

2JZ-GTE

New Member
Mar 30, 2021
7
4
3
pinkypie said:
Hey guys need a little help please. I am trying to figure out how ACLs work. In one of Terry Henry's videos he says in version 8095 the ACLs are applied to the physical interface instead of the virtual interface. I've got a 6450 running 8030 so I figure that is why I couldn't recreate the results he was getting from applying the ACLs rules from the video.

I ran through some testing with a couple of vlans but I can not get the results that I would expect. I have tried various rules permitting and denying ICMP traffic between VLANs with no success. It either allows all or blocks all. I seem to have a misunderstanding of how ACLs work despite watching hours of video and reading.

I read that extended ACLs should be placed as close to the source as possible. So if I want to block VLAN3 from communicating with VLANx, then wouldn't the code be placed on the "out" interface of VLAN3?

Code: 
ip access-list extended block
remark block VLAN3 from communicating with other VLANs
   deny ip 192.168.3.0 0.0.0.255 192.168.0.0 0.0.255.255
remark allow traffic from other VLANs to flow out
  permit ip any any
Problem is when I do this, it allows all traffic. I am sure my problem is with a misunderstanding of how to correctly use "in" vs "out".

Any help in understanding and identifying the problem would be most appreciated.
Click to expand...
You apply the ACL to the IN direction as that is the closest to the source. The snippet below should block VLAN 3 from everything except established, dns, dhcp, and the internet. It will allow hosts in the same vlan to communicate with each other.

Code: 
interface ve 3
 ip access-group "ISONET" in
 ip address 192.168.3.254 255.255.255.0
 ip helper-address 1 10.2.1.1

ip access-list extended "ISONET"
 remark allow established connections
 permit tcp 192.168.3.0 0.0.0.255 any established
 remark allow DNS
 permit tcp 192.168.3.0 0.0.0.255 any eq dns
 permit udp 192.168.3.0 0.0.0.255 any eq dns
 remark allow DHCP
 permit udp 192.168.3.0 0.0.0.255 any eq bootps
 permit udp 192.168.3.0 0.0.0.255 any eq bootpc
 remark allow host to host in same vlan
 permit ip 192.168.3.0 0.0.0.255 192.168.3.0 0.0.0.255
 remark deny access to rest of LAN
 deny ip 192.168.3.0 0.0.0.255 192.168.0.0 0.0.255.255
 deny ip 192.168.3.0 0.0.0.255 172.16.0.0 0.15.255.255
 deny ip 192.168.3.0 0.0.0.255 10.0.0.0 0.255.255.255
 remark allow internet access
 permit ip any any
 
K

koyetsu

New Member
Feb 28, 2021
21
12
3
I was speaking to an Engineer where I work and he thinks my issue is actually my fiber. the transceivers are SMF and all i have is multimode om2/3. I have some Single Mode fiber on order and will be here tomorrow. will do further investigation on the card if that doesn't pan out.
 
K

koyetsu

New Member
Feb 28, 2021
21
12
3
divide_by_zero said:
Re: Setting up Brocade ICX6430 - I'm getting an error while attempting Initial Configure. When I enter "ip dhcp-client disable" I get "invalid input -> disable".
Code: 
ICX6430-24P Switch(config)#ip dhcp-client disable
Invalid input -> disable
Type ? for a list
ICX6430-24P Switch(config)#
I tried ip dhcp-client ? and "disable" is not one of the 5 listed commands. "Enable" is one of the commands and I entered it, then entered "disable" again and it returned Invalid input as before.

Suggestions, please?
Click to expand...
On the 6610 I had the same issue. It doesn't do anything with IP's and DHCP other than the management until you setup a VE for the vlan.

Code: 
conf t
vlan 50
tagged e 1/1/2
router-interface ve 50
int ve 50
The above commands enter config, goto vlan 50, setup a virtual ethernet device of 50, and then go into it. at that point the "ip" commands are available
 
fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,828
3,269
113
33
fohdeesha.com
koyetsu said:
SO trying to connect my ESXi host toi my ICX6610 via 1/2/1 or 1/2/6. VMWare says the link is up but brocade says it is down.
The 6610 sees the transceiver, mtu's are the same. any tips on this?
View attachment 20931
View attachment 20932
View attachment 20933
View attachment 20934

that is all the relevant stuff i can think of to this
NIC: HP InfiniBand FDR/Ethernet 10Gb/40Gb 2P 544FLR-QSFP Adapter
Transceivers: KAIAM QSFP+ 40G-LR4 Lite
Click to expand...
if you still havent gotten it to link up, try rebooting the switch now that the optic is inserted. sometimes they don't like new optics showing up in the stacking ports but a reboot typically makes the issue go away. also make sure youve done the relevant stuff here: FCX / ICX6610 - Fohdeesha Docs
 
K

koyetsu

New Member
Feb 28, 2021
21
12
3
fohdeesha said:
if you still havent gotten it to link up, try rebooting the switch now that the optic is inserted. sometimes they don't like new optics showing up in the stacking ports but a reboot typically makes the issue go away. also make sure youve done the relevant stuff here: FCX / ICX6610 - Fohdeesha Docs
Click to expand...
Rebooted switch, Swapped Transceivers, verified that it's in NIC mode not IB mode.
Using SMF, Verified that all the stacking info is cleared. I have to be missing something...

Edit: Plugged 2 transceivers into the 2 40gb ports and looped them and the ports came up. Problem is at the NIC...
 
Last edited:
  • Like
Reactions: fohdeesha
J

jasonwc

Member
Dec 31, 2018
49
18
8
Some of the Mellanox Infiniband/Ethernet QSFP cards will only do 10Gb Ethernet unless you crossflash different firmware. Are you certain your card supports 40Gb Ethernet? The model number you posted appears to, but obviously something isn’t working.
 
E

evanh

New Member
Oct 25, 2021
10
3
1
Any reason why this QSFP DAC wouldn't work, connecting 1/2/1 to a Mellanox cx354a card in an OpnSense box? Neither the switch nor the NIC think there's anything connected.

EDIT:
Code: 
telnet@ICX6610-24P Router#show interfaces brief

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/1/1      Up      Forward Full 1G    None  No  1    0   748e.f8e8.5d7a
1/1/2      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/3      Up      Forward Full 1G    None  No  1    0   748e.f8e8.5d7a
1/1/4      Up      Forward Full 1G    None  No  1    0   748e.f8e8.5d7a
1/1/5      Up      Forward Full 1G    None  No  1    0   748e.f8e8.5d7a
1/1/6      Up      Forward Full 1G    None  No  1    0   748e.f8e8.5d7a
1/1/7      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/8      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/9      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/10     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/11     Up      Forward Full 1G    None  No  1    0   748e.f8e8.5d7a
1/1/12     Up      Forward Full 100M  None  No  1    0   748e.f8e8.5d7a
1/1/13     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/14     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/15     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/16     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/17     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/18     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/19     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/20     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/21     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/22     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/23     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/24     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/1      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/2      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/3      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/4      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/5      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/6      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/7      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/8      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/9      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/10     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/3/1      Up      Forward Full 10G   None  No  1    0   748e.f8e8.5d7a
1/3/2      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/3/3      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/3/4      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/3/5      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/3/6      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/3/7      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/3/8      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
mgmt1      Down    None    None None  None  No  None 0   748e.f8e8.5d7a

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
ve1        Up      N/A     N/A  N/A   None  N/A N/A  N/A 748e.f8e8.5d7a
 
J

jasonwc

Member
Dec 31, 2018
49
18
8
evanh said:
Any reason why this QSFP DAC wouldn't work, connecting 1/2/1 to a Mellanox cx354a card in an OpnSense box? Neither the switch nor the NIC think there's anything connected.

EDIT:
Code: 
telnet@ICX6610-24P Router#show interfaces brief

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/1/1      Up      Forward Full 1G    None  No  1    0   748e.f8e8.5d7a
1/1/2      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/3      Up      Forward Full 1G    None  No  1    0   748e.f8e8.5d7a
1/1/4      Up      Forward Full 1G    None  No  1    0   748e.f8e8.5d7a
1/1/5      Up      Forward Full 1G    None  No  1    0   748e.f8e8.5d7a
1/1/6      Up      Forward Full 1G    None  No  1    0   748e.f8e8.5d7a
1/1/7      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/8      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/9      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/10     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/11     Up      Forward Full 1G    None  No  1    0   748e.f8e8.5d7a
1/1/12     Up      Forward Full 100M  None  No  1    0   748e.f8e8.5d7a
1/1/13     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/14     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/15     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/16     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/17     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/18     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/19     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/20     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/21     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/22     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/23     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/1/24     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/1      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/2      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/3      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/4      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/5      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/6      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/7      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/8      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/9      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/2/10     Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/3/1      Up      Forward Full 10G   None  No  1    0   748e.f8e8.5d7a
1/3/2      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/3/3      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/3/4      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/3/5      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/3/6      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/3/7      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
1/3/8      Down    None    None None  None  No  1    0   748e.f8e8.5d7a
mgmt1      Down    None    None None  None  No  None 0   748e.f8e8.5d7a

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
ve1        Up      N/A     N/A  N/A   None  N/A N/A  N/A 748e.f8e8.5d7a
Click to expand...
It should work assuming Opnsense has the necessary Mellanox driver loaded. Intel or Chelsio NiCs are generally recommended for pfsense due to driver support, and I would assume opnsense is similar.

The 10GTek 40G QSFP to 4 x SFP+ breakout cable works on my ICX6610 connecting to an Intel X520-DA2 NIC on pfsense.

A Google search yielded a result from 2018 indicating that you need to modify a boot setting to load the mlx4en driver. This may no longer apply to the latest version of opnsense. Mellanox ConnectX-3 support
EDIT: Here are updated instructions which work from the GUI and will persist a reboot. Also, it appears this is still required in 2021. Mellanox ConnectX-2 and up

Pfsense only added the mlx4 and mlx5 drivers in release 2.4.5. Feature #7537: Include mellanox mlx4 and mlx5 ethernet driver - pfSense - pfSense bugtracker
 
Last edited:
K

koyetsu

New Member
Feb 28, 2021
21
12
3
To the best of my knowledge yes. I also checked in the boot firmware and it appears to be in NIC mode...
 
E

evanh

New Member
Oct 25, 2021
10
3
1
jasonwc said:
A Google search yielded a result from 2018 indicating that you need to modify a boot setting to load the mlx4en driver. This may no longer apply to the latest version of opnsense. Mellanox ConnectX-3 support
EDIT: Here are updated instructions which work from the GUI and will persist a reboot. Also, it appears this is still required in 2021. Mellanox ConnectX-2 and up

Pfsense only added the mlx4 and mlx5 drivers in release 2.4.5. Feature #7537: Include mellanox mlx4 and mlx5 ethernet driver - pfSense - pfSense bugtracker
Click to expand...
Thanks for the reply. I had found that and added the commands, which worked just fine. The interfaces show up in OpnSense, just as disconnected.

I just tried connecting the cable in a loop from interface 1 to interface 2 on the mellanox nic, and it lit up just fine, so it seems like there's something on the switch side that's not recognizing or utilizing the cable.

EDIT2: Here's what the most relevant parts of my network setup look like, in case it's relevant (open to feedback here as well). Note that there wasn't a good 6610 equivalent in draw.io, so ignore the positional matchups. I added the interface ids for clarity.


EDIT:
Code: 
telnet@ICX6610-24P Router#show interfaces ethernet 1/2/1
40GigabitEthernet1/2/1 is down, line protocol is down
  Port down for 17 minute(s) 55 second(s)
  Hardware is 40GigabitEthernet, address is 748e.f8e8.5d7a (bia 748e.f8e8.5d93)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
  Configured mdi mode AUTO, actual unknown
  Member of L2 VLAN ID 1, port is untagged, port state is BLOCKING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is enabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  MTU 1500 bytes, encapsulation ethernet
  300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  607855000 packets input, 67135060808 bytes, 0 no buffer
  Received 210154120 broadcasts, 394335559 multicasts, 3365321 unicasts
  15 input errors, 1 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  601682076 packets output, 65644112862 bytes, 0 underruns
  Transmitted 214559764 broadcasts, 386674611 multicasts, 447701 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0            64811133                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                  30                   0
    6                   0                   0
    7                   1                   0
 
Last edited:
J

jasonwc

Member
Dec 31, 2018
49
18
8
evanh said:
Thanks for the reply. I had found that and added the commands, which worked just fine. The interfaces show up in OpnSense, just as disconnected.

I just tried connecting the cable in a loop from interface 1 to interface 2 on the mellanox nic, and it lit up just fine, so it seems like there's something on the switch side that's not recognizing or utilizing the cable.

EDIT2: Here's what the most relevant parts of my network setup look like, in case it's relevant (open to feedback here as well). Note that there wasn't a good 6610 equivalent in draw.io, so ignore the positional matchups. I added the interface ids for clarity.


EDIT:
Code: 
telnet@ICX6610-24P Router#show interfaces ethernet 1/2/1
40GigabitEthernet1/2/1 is down, line protocol is down
  Port down for 17 minute(s) 55 second(s)
  Hardware is 40GigabitEthernet, address is 748e.f8e8.5d7a (bia 748e.f8e8.5d93)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
  Configured mdi mode AUTO, actual unknown
  Member of L2 VLAN ID 1, port is untagged, port state is BLOCKING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is enabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  MTU 1500 bytes, encapsulation ethernet
  300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  607855000 packets input, 67135060808 bytes, 0 no buffer
  Received 210154120 broadcasts, 394335559 multicasts, 3365321 unicasts
  15 input errors, 1 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  601682076 packets output, 65644112862 bytes, 0 underruns
  Transmitted 214559764 broadcasts, 386674611 multicasts, 447701 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0            64811133                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                  30                   0
    6                   0                   0
    7                   1                   0
Click to expand...
Did you follow this step from the ICX6610 instructions to remove any existing stacking config?

Code: 
enable
conf t
stack unit 1
no stack-trunk 1/2/1 to 1/2/2
no stack-trunk 1/2/6 to 1/2/7
stack disable
exit
write mem
 
Last edited:
You must log in or register to reply here.
Top Bottom