Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Craig Curtin]

I believe 10gb ports 1 and 3 are stacking. Looking at your config... stacking enabled?
That could be the problem in this particular test.

edit - total guess on my part - don't have a 6450 - just read about when I was deciding what to buy... could be totally wrong guess.

FWIW on my 6610 I have to explicitly set speed and duplex on my FP 10Gbe ports. Don't know if that is the same on the 6450 but should
look something like:

!
interface ethernet 1/2/1
speed-duplex 10G-full
!
interface ethernet 1/2/2
speed-duplex 10G-full
!
interface ethernet 1/2/3
speed-duplex 10G-full
!
interface ethernet 1/2/4
speed-duplex 10G-full

edit - and using interfaces where /3/ becomes /2/ I simply changed it in the code section.

edit - further reading - looks like stacking is NOT enabled by default on 6450 and the default speed for the SFP+ ports is 10Gbe so the commands I put in are unnecessary.

Yep - i tried the speed-duplex commands on the 6450 last night and they were accepted but do not show up in the config anywhere.

Yes i believe that stacking on the 6450 is something that has to be enabled rather than defaulted to (again just from my reading and looking at the default config as generated)

Craig

 

[itronin]

In 6610 ports 1/2/x do not require any speed and duplex setting, but ports 1/3/x do require "speed-duplex 10G-full".
I don't know 6450.

I edited what I posted to match the module number on the 6450. correct on 6610 x/2/x does not require speed setting.

 

[AndroidCat]

@Craig Curtin
I found the original explanation about spanning tree, the one I had mentioned 2 pages before (which did not help you).
Quote:


STP is off by default with the l3 OS image (off globally and off per vlan), but remains "ON" per port, meaning ports are eligible to participate in an STP topology, but one hasn't been defined globally so they don't perform STP. They may still be blocking BPDU's though, so try turning off STP on each port connected to something related to sonos crap (every port connected to an access point and every port connected to a wired sonos device)

enable
conf t
interface ethernet 1/1/5
no spanning-tree

 

[Mushishi]

@Craig Curtin

I do have a R720 and the 6450 at home for my homelab but I am using the broadcom 10GB nic module and not a intel nic in it but if you want i can look into testing it with a ESXi image this weekend to see if I have the same problem.

I have listed the config I use on my 6450 just if something can help and i have STP enabled, and I do also have 10k MTU enabled, and I don't have any issuses with my 10g ports.

SSH@beefchunk(config)#show run
Current configuration:
!
ver 08.0.30uT313
!
stack unit 1
  module 1 icx6450-48-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
!
global-stp
!
!
lag TrueNAS dynamic id 2047
 ports ethernet 1/1/33 to 1/1/36
 primary-port 1/1/33
 lacp-timeout long
 deploy
 port-name "TrueNAS Docker" ethernet 1/1/33
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
 spanning-tree
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
jumbo
enable password-display
hostname beefchunk
ip dhcp-client disable
ip dns domain-list moonisp.dk
ip dns server-address 10.10.10.2 10.10.10.5
ip route 0.0.0.0/0 10.10.10.1
!
logging host 10.10.10.4
logging enable rfc5424
no telnet server
username root password ..........
snmp-server community ............
!
!
clock summer-time
clock timezone gmt GMT+03
!
!
ntp
 disable serve
 server 10.10.10.2
!
!
web-management https
web-management refresh port-statistic 10
web-management refresh front-panel 10
!
!
!
interface ethernet 1/1/25
 port-name Proxmox
!
interface ethernet 1/1/33
 port-name TrueNAS Docker
!
interface ethernet 1/1/47
 port-name Unraid
!
interface ethernet 1/1/48
 port-name Unraid
!
interface ethernet 1/2/1
 port-name TrueNAS
!
interface ethernet 1/2/2
 port-name Enterprise
!
interface ethernet 1/2/3
 port-name Dell
!
interface ve 1
 ip address 10.10.10.250 255.255.255.0
!
!
!
!
!
lldp run
!
!
!
!
end

And my show media:

Port 1/2/1:  Type : 10GE Twinax   1m (SFP +)
Port 1/2/2:  Type : 10GE SR 300m (SFP +)
Port 1/2/3:  Type : 10GE Twinax   5m (SFP +)
Port 1/2/4:  Type : EMPTY

 

[Craig Curtin]

@Craig Curtin

I do have a R720 and the 6450 at home for my homelab but I am using the broadcom 10GB nic module and not a intel nic in it but if you want i can look into testing it with a ESXi image this weekend to see if I have the same problem.

I have listed the config I use on my 6450 just if something can help and i have STP enabled, and I do also have 10k MTU enabled, and I don't have any issuses with my 10g ports.

SSH@beefchunk(config)#show run
Current configuration:
!
ver 08.0.30uT313
!
stack unit 1
  module 1 icx6450-48-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
!
global-stp
!
!
lag TrueNAS dynamic id 2047
ports ethernet 1/1/33 to 1/1/36
primary-port 1/1/33
lacp-timeout long
deploy
port-name "TrueNAS Docker" ethernet 1/1/33
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
spanning-tree
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
jumbo
enable password-display
hostname beefchunk
ip dhcp-client disable
ip dns domain-list moonisp.dk
ip dns server-address 10.10.10.2 10.10.10.5
ip route 0.0.0.0/0 10.10.10.1
!
logging host 10.10.10.4
logging enable rfc5424
no telnet server
username root password ..........
snmp-server community ............
!
!
clock summer-time
clock timezone gmt GMT+03
!
!
ntp
disable serve
server 10.10.10.2
!
!
web-management https
web-management refresh port-statistic 10
web-management refresh front-panel 10
!
!
!
interface ethernet 1/1/25
port-name Proxmox
!
interface ethernet 1/1/33
port-name TrueNAS Docker
!
interface ethernet 1/1/47
port-name Unraid
!
interface ethernet 1/1/48
port-name Unraid
!
interface ethernet 1/2/1
port-name TrueNAS
!
interface ethernet 1/2/2
port-name Enterprise
!
interface ethernet 1/2/3
port-name Dell
!
interface ve 1
ip address 10.10.10.250 255.255.255.0
!
!
!
!
!
lldp run
!
!
!
!
end

And my show media:

Port 1/2/1:  Type : 10GE Twinax   1m (SFP +)
Port 1/2/2:  Type : 10GE SR 300m (SFP +)
Port 1/2/3:  Type : 10GE Twinax   5m (SFP +)
Port 1/2/4:  Type : EMPTY

Thanks for that - would really appreciate it if you could do some testing with ESXi just to put my mind at ease - i setup ESxi 7.01 yesterday on one of my Compaq/HP units and attached it to the 6610 (spare) and will start testing and screen shotting today to see if i can get to the bottom of it. (I also have 2 x 6450s here that i will do the same testing with once i am sure i have a working config.

I have a couple of leftover R710s here that i can also fire up - but they do not have onboard 10GB so will have to put in an Intel or Mellanox card for that testing

looks like i can going to have a big weekend of testing !

on your 6450 the 1/2/2 that reports as 10GE SR is that a twisted pair module as well or just some short range fibre ?

Craig

 

[Craig Curtin]

@Craig Curtin
I found the original explanation about spanning tree, the one I had mentioned 2 pages before (which did not help you).
Quote:


STP is off by default with the l3 OS image (off globally and off per vlan), but remains "ON" per port, meaning ports are eligible to participate in an STP topology, but one hasn't been defined globally so they don't perform STP. They may still be blocking BPDU's though, so try turning off STP on each port connected to something related to sonos crap (every port connected to an access point and every port connected to a wired sonos device)

enable
conf t
interface ethernet 1/1/5
no spanning-tree

Hmm - that is an interesting find as it does state the ports are blocked (can not understand why still though)

I will give the explicit no spanning-tree and see how i go

Craig

 

[Craig Curtin]

Hmm - that is an interesting find as it does state the ports are blocked (can not understand why still though)

I will give the explicit no spanning-tree and see how i go

Craig

Just confirming i am running with the no spanning-tree (per port) for the 10GB ports already - here is my currently running config on the live system - currently with 2 x Dell Hosts to 10GB fibre and the 3rd one waiting to connect for me to solve this problem (or for some more DAC cables to arrive)

SSH@6610-Basement#show run
Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-48-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack disable
!
!
!
lag ToCisco dynamic id 1
ports ethernet 1/1/37 to 1/1/38
primary-port 1/1/37
deploy
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 100 name data-100 by port
tagged ethe 1/1/37 to 1/1/38 ethe 1/2/4 to 1/2/5 ethe 1/3/1 to 1/3/4 ethe 1/3/8
router-interface ve 100
!
vlan 101 name iot-101 by port
tagged ethe 1/1/37 to 1/1/38 ethe 1/2/4 to 1/2/5 ethe 1/3/1 ethe 1/3/8
!
vlan 102 name crypto-102 by port
tagged ethe 1/1/1 to 1/1/2 ethe 1/2/4 to 1/2/5 ethe 1/3/8
!
vlan 200 name storage-200 by port
tagged ethe 1/1/37 to 1/1/38 ethe 1/2/3 ethe 1/2/5 ethe 1/2/7 ethe 1/2/9 ethe 1/3/7 to 1/3/8
router-interface ve 200
!
vlan 202 name prosis-202 by port
tagged ethe 1/1/1 to 1/1/3 ethe 1/1/37 to 1/1/38 ethe 1/2/4 to 1/2/5 ethe 1/3/1 to 1/3/2 ethe 1/3/8
router-interface ve 202
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
jumbo
enable aaa console
hostname 6610-Basement
ip dhcp-client disable
ip dns server-address 172.16.100.1
ip route 0.0.0.0/0 172.16.100.1
!
logging host 172.16.100.26
no telnet server
username root password .....
!
!
!
!
ntp
disable serve
server 172.16.100.1
!
!
no web-management http
!
!
!
!
!
!
!
interface ethernet 1/1/1
port-name ESXI-Host1 - Onboard Management port
!
interface ethernet 1/1/2
port-name ESXI-Host2 - Onboard Management port
!
interface ethernet 1/1/3
port-name ESXI-Host-3 - Onboard Mangement port
!
interface ethernet 1/2/1
no spanning-tree
!
interface ethernet 1/2/2
no spanning-tree
!
interface ethernet 1/2/3
port-name Host-2-I520d-Port-2-VMNIC2
no spanning-tree
!
interface ethernet 1/2/4
port-name Host-3-I520s-VMNIC1
no spanning-tree
!
interface ethernet 1/2/5
port-name Host-1-I520s-VMNIC1-Bottom-Slot
no spanning-tree
!
interface ethernet 1/2/6
no spanning-tree
!
interface ethernet 1/2/7
port-name Host-1-I520s-VMNCI3-Top-Slot
no spanning-tree
!
interface ethernet 1/2/8
no spanning-tree
!
interface ethernet 1/2/9
port-name Host-3-I520-Port-2-VMNIC2
no spanning-tree
!
interface ethernet 1/2/10
port-name Host-2-I520d-Port-1-VMNIC1
no spanning-tree
!
interface ethernet 1/3/7
port-name Mediastore-Fibre
speed-duplex 10G-full
no spanning-tree
!
interface ethernet 1/3/8
speed-duplex 10G-full
no spanning-tree
!
interface ve 1
ip address 192.168.1.254 255.255.255.0
!
interface ve 100
ip address 172.16.100.254 255.255.255.0
!
interface ve 200
ip address 172.16.200.254 255.255.255.0
!
interface ve 202
ip address 192.168.202.254 255.255.255.0
!
!
!
!
!
!
!
!
!
end

SSH@6610-Basement#


Craig

 

[Mushishi]

Thanks for that - would really appreciate it if you could do some testing with ESXi just to put my mind at ease - i setup ESxi 7.01 yesterday on one of my Compaq/HP units and attached it to the 6610 (spare) and will start testing and screen shotting today to see if i can get to the bottom of it. (I also have 2 x 6450s here that i will do the same testing with once i am sure i have a working config.

I have a couple of leftover R710s here that i can also fire up - but they do not have onboard 10GB so will have to put in an Intel or Mellanox card for that testing

looks like i can going to have a big weekend of testing !

on your 6450 the 1/2/2 that reports as 10GE SR is that a twisted pair module as well or just some short range fibre ?

Craig

Not a problem at all. I had been thinking about giving ESXi a test again as I had not used it after they removed support for socket 1366 cpu's so I was not able to use a newer version on my old HP DL380 server.

After looking at the spare 10G nics i have at home i did see that i had a extra ConnectX-3 card that i can test with together with the Broadcom 57800S daughter card I have in the system just to test a nic that you have also tested with.

If you want also i can change the password on my config and give you a copy if i can get it to save to a tftp server. Just note i have the 48 port non-poe not sure if that will work on a poe/24-port switch.

Regarding 1/2/2 yes that is a fiber module as my desktop is nowhere near my servers/switch. I did not even think about the SR might mean short range.

 

[Zackey_TNT]

Hey everyone.
My Brocade ICX 6610 keeps randomly rebooting itself every so often (like once a month)
Can anyone please advise how to review the logs to find the problem? Or share if you have had the same experience.
It has 1 fan and 1 PSU.

 

[Rttg]

Had a similar random reboot a few days ago, and @AndroidCat pointed me on the right direction (I spun up a syslog-NRG container to consume the logs)

You need to stream your logs to external syslog server.
For instance:

logging host <IP>  udp-port 1514
logging enable rfc5424

 

[itronin]

Hey everyone.
My Brocade ICX 6610 keeps randomly rebooting itself every so often (like once a month)
Can anyone please advise how to review the logs to find the problem? Or share if you have had the same experience.
It has 1 fan and 1 PSU.

Is your ICX 6610 plugged into a UPS? If so, mfg and model please.

 

[Mushishi]

@Craig Curtin Just did a quick test here in the afternoon with a fresh install of the Dell ESXi. Forgot to test the the ConnectX though but the broadcom port from the intergraded module did work fine.

I did not setup any VM's and my management network was on the 2x1G ports on the broadcom. Just to see if it would disable the port as soon as i removed it from a vSwitch. I do also not run vsphere on it just the buildin webinterface.

 

[rootpeer]

I would like to ask for some help, as I have no experience with L3 switching (total noob).

I have two different sites that are connected via a pair of Ubiquiti Nanobeam APs. Both sites have a pfSense router in place and are using different LAN subnets and are in general separate from each other.

The point of the Ubiquiti bridge was to use the DSL of site-1 as a backup internet connection for site-2. Thus, I have connected the Ubiquiti AP at site-1 directly to pfSense-1 on a separate interface and the other AP at site-2 also directly to pfSense-2 as a WAN interface. That lets me access site-1's internet and LAN from site-2 but not the opposite.

Now the situation has arisen that I would like to be able to access both LANs from both locations, regardless of if pfSense-1 or pfSense-2 are online or not, basically access static servers on either LAN (VLAN 1) from either LAN (VLAN 1) in case one or both pfSense routers are online.

I have Ruckus ICX6450s at both locations.

My question is, is there a configuration recipe that would allow me to connect the Nanobeams to the switches, then use the L3 functionality to route between the native (V)LANs of each switch?

Edit: Is the following thought process correct?:

1. Connect the nanobeams to the switches on a new VLAN (e.g. VLAN 800)
2. Set up VEs (VE 80) on VLAN 800 and tag or untag the interfaces where the Nanobeams are connected
3. Set static route on site-1: LAN-IP-ADDRESS-SPACE of site-2 via VE 80
4. Set static route on site-2: LAN-IP-ADDRESS-SPACE of site-1 via VE 80
5. Profit

 

[Craig Curtin]

@Craig Curtin Just did a quick test here in the afternoon with a fresh install of the Dell ESXi. Forgot to test the the ConnectX though but the broadcom port from the intergraded module did work fine.

I did not setup any VM's and my management network was on the 2x1G ports on the broadcom. Just to see if it would disable the port as soon as i removed it from a vSwitch. I do also not run vsphere on it just the buildin webinterface.

View attachment 25563

Good one thanks - really appreciate the effort.

OK i have advanced a bit further i think - still have not stood up the Mellanox cards yet - that is the job for this afternoon.

However what i have discovered (i think) is that i might have a DAC compatibiity issue - as far as i knew i thought that a QSFP to SFP+ breakout cable would be a passive device - but i have tried the BIOS update process for the 82599 based Intel cards and i seem to be makig a bit of progress with this.

The thing that first put me onto it - was in the testing i stood up a Linux host with a dual port 520-da2 in it and connected that with a SFP+ AOC 10 metre cable back to my switch - came up the first time and did not think anything of it - then did some VLANning and it stayed up - but then tried a reboot and it would not come back up - went into DMESG and found the card had disabled the port because it did not like the SFP+ module.

So found the method using Ethtool to patch the BIOS on the card - https://forums.servethehome.com/ind...m-to-unlock-all-sfp-transceivers.24634/page-1 and performed that - and then it was working fine.

So thought this card was worth a try in the ESXi servers now it was patched and known to work - so moved it across and it appears to have brought up both links OK

Not definitive yet and have to perform more testing with the VLANs etc to see what happens - will also try a Mellanox card in the same box and let you know

Craig

 

[Mushishi]

@Craig Curtin well that seems like to be good news. In regards to your mellanox cards could they be OEM branded and then locked to that OEM's modules.? I faintly remember that mine was branded to someone but it is a few years ago i bought that card, and i know that i flashed it with stock firmware as some of the first.

The Intel X520 i have in my desktop was a sun branded card that allowed unsupported modules OOB. I did check with ethtool.

 

[Junction Runner]

edit: Ok so got a lot of it straightened out, I think part due to order of operations and flipping between the web ui, which sucks for this, and cli.

vlan and dhcp is working for vlan200 now on assigned ports, I just need to get my ubiquity ap to properly worth with the vlan for the guest network only.

But yeah, if anyone else reading this has issues with setting up vlans, ignore the web ui entirely except for a visual check after.

 

[Cobra0101]

Does anyone know how compatible the SFP+ ports are on the Brocade ICX6450 i.e. are they coded/password to stop 3rd party devices being used?

 

[Craig Curtin]

edit: Ok so got a lot of it straightened out, I think part due to order of operations and flipping between the web ui, which sucks for this, and cli.

vlan and dhcp is working for vlan200 now on assigned ports, I just need to get my ubiquity ap to properly worth with the vlan for the guest network only.

But yeah, if anyone else reading this has issues with setting up vlans, ignore the web ui entirely except for a visual check after.

OK so a couple of things

1) Post up your config
2) I assume based on what you have stated that you have a spare interface in OpnSense you can plug into the switch for the 200 VLAN ?
3) So the steps on the switch would be

a) Create VLAN 200 - login as root etc, then en, then conf t, then VLAN 200
b) add the ports to the VLAN - the least disruptive way is to have a spare port on OpnSense and plug it into one port and the AP into the other
c) lets say opnsense in port 1/1/34 and the ubiquiti is 1/1/35
d) so as per step a above we should still be in VLAN 200 config
e) type tag e 1/1/34 - this will remove it from VLAN 1 and put it into VLAN 200 as a tagged port only
f) then type tag e 1/1/35 - same as above

4) At this stage you have two devices that will have to support VLAN tagging and will only have access to VLAN 200
5) on Opensense you need to make sure the interface that is attached to port 1/1/34 has a tagged VLAN 200 defined on it
6) You need to assign it a valid IP address in the 200 subnet and then make sure the DHCP server is applied to that interface and is giving out IP addresses relevant to that subnet
7) You need to make sure the AP has a valid IP in the subnet and that you have turned off the DHCP server on the AP and it is bridging between the wireless interface and the LAN and it is putting devices into the 200 VLAN when they are successfully authenticated

Craig

 

[Craig Curtin]

Does anyone know how compatible the SFP+ ports are on the Brocade ICX6450 i.e. are they coded/password to stop 3rd party devices being used?

The Brocade are meant to be one of the more flexible in terms of which coding they support - but it appears to be difficult to get down to a deep level to actually confirm if it is accepted - beyond doing a show media - i am going through some problems (see my posts above) that i "think" may be related to transceiver issues and coding - but the show media commands are not complaining about any of the devices so it is just a feeling at this stage

Craig

 

[Craig Curtin]

@Craig Curtin well that seems like to be good news. In regards to your mellanox cards could they be OEM branded and then locked to that OEM's modules.? I faintly remember that mine was branded to someone but it is a few years ago i bought that card, and i know that i flashed it with stock firmware as some of the first.

The Intel X520 i have in my desktop was a sun branded card that allowed unsupported modules OOB. I did check with ethtool.

Nope still have not had the Mellanoxs in there yet - all testing done to this point with INtel 520DA-2 and Intel 540T

Hopefully - time permitting - will be onto the mellanox to start with this afternoon

The Mellanox all the took the flash to IB/ETH dual mode with no problem (not that i plan on using IB mode)

Craig