Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[jkieffer72]

So have you reset this switch and followed the OP to set it up - or is this one you have inherited with unknown settings ?

Try the base firmware reload from scratch - assuming a restart gives you something to work with and allows you into the initial boot screen.

Have you looked at flow control settings and is this a known console cable you are using - or just a spare cable lying around ?

Craig

New to me switch from eBay. Looks hardly used, but does not appear to be at default configuration.

I am trying to follow the guide to reset as configuration (password) is unknown. I have not been able to access the serial console yet.

The management (network) port pulls an IP Address from DHCP. I can reach the web interface, but again password is unknown and not any of the defaults I have found referenced online.

Flow control is set to none. The console cable was new in a package of accessories that came with the switch. (Power cords, rack hardware, stacking cables)

Going to go try again a couple of different ways...

 

[Craig Curtin]

New to me switch from eBay. Looks hardly used, but does not appear to be at default configuration.

I am trying to follow the guide to reset as configuration (password) is unknown. I have not been able to access the serial console yet.

The management (network) port pulls an IP Address from DHCP. I can reach the web interface, but again password is unknown and not any of the defaults I have found referenced online.

Flow control is set to none. The console cable was new in a package of accessories that came with the switch. (Power cords, rack hardware, stacking cables)

Going to go try again a couple of different ways...

OK well we would expect the serial cable to work - just confirming is it a DB9 serial or a USB Serial ?

Forgive me for asking - but you are plugging it into the correct (serial/console) management port - not the LAN management port (as they are both RJ45)

DO you have any other apps on your machine that would be accessing the serial port ?

If it is a USB cable then when you plug it into your machine you should hear the usual windows tones for a newly found device and the toast icon should popup saying it has been found

You can then go into device manager and look for the newly created serial port.

Craig

 

[jkieffer72]

OK well we would expect the serial cable to work - just confirming is it a DB9 serial or a USB Serial ?

Forgive me for asking - but you are plugging it into the correct (serial/console) management port - not the LAN management port (as they are both RJ45)

DO you have any other apps on your machine that would be accessing the serial port ?

If it is a USB cable then when you plug it into your machine you should hear the usual windows tones for a newly found device and the toast icon should popup saying it has been found

You can then go into device manager and look for the newly created serial port.

Craig

Thanks for the input.

I think it is something with my USB to Serial adapter.

I just tried from another computer that has a serial port and was able to connect without issue.

Edit/Update- basic configuration is done now. I have access via ssh and http. Now to see if I can get the rest of the network connected.. and figure out how to quiet it down further...

 

[Craig Curtin]

Thanks for the input.

I think it is something with my USB to Serial adapter.

I just tried from another computer that has a serial port and was able to connect without issue.

Edit/Update- basic configuration is done now. I have access via ssh and http. Now to see if I can get the rest of the network connected.. and figure out how to quiet it down further...

Good one - thanks for letting us know

Craig

 

[ramicio]

Okay, here's what I've been through hardware-wise

1. New DAC cables (two), one end coded for Brocade, the other end for Intel.
2. Another XL710 NIC

The behavior has not changed. I cannot use DHCP, and I can only establish a link via static settings. When I do form a static connection, these are the errors I'm getting on the interface in Ubuntu:

eth2: flags=4163(UP,BROADCAST,RUNNING,MULTICAST)  mtu 1500
    inet 192.168.1.43  netmask 255.255.255.0   broadcast 192.168.1.255
    inet6 fe80::3efd:feff:feb2:cd30  prefixlen 64  scopeid 0x20(link)
    ether 3c:fd:fe:b2:cd:30  txqueuelen 1000  (Ethernet)
    RX packets 741  bytes 46100 (46.1 KB)
    RX errors 382  dropped 197  overruns 0  frame 382
    TX packets 39  bytes 6887 (6.8 KB)
    TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Pinging is fine to and from, and even with larger byte sizes like 1,400 bytes.

On the switch there are zero RX or TX errors:

Spoiler: Switch output for 1/2/1

40GigabitEthernet1/2/1 is up, line protocol is up
  Port up for 16 minute(s) 37 second(s)
  Hardware is 40GigabitEthernet, address is 748e.f8fe.93f6 (bia 748e.f8fe.940f)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual 40Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual none
  Member of L2 VLAN ID 1, port is untagged, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is enabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  MTU 1500 bytes, encapsulation ethernet
  300 second input rate: 48 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 2168 bits/sec, 3 packets/sec, 0.00% utilization
  56352 packets input, 6296114 bytes, 0 no buffer
  Received 503 broadcasts, 7331 multicasts, 48518 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  350121 packets output, 73835998 bytes, 0 underruns
  Transmitted 143184 broadcasts, 121844 multicasts, 85093 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0              347387                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                  19                   0
    6                   0                   0
    7                2719                   0

Tried disabling LLDP for that port (a suggestion), made things worse, re-enabled. Port 1/2/6 acts the same.

Should I be looking for a new switch (bad ports), or different type of NIC?

EDIT:
If I ping with 1,450 bytes I get 0% packet loss. If I ping with 1,451 bytes or more, I get 100% packet loss.

 

[Craig Curtin]

Okay, here's what I've been through hardware-wise

1. New DAC cables (two), one end coded for Brocade, the other end for Intel.
2. Another XL710 NIC

The behavior has not changed. I cannot use DHCP, and I can only establish a link via static settings. When I do form a static connection, these are the errors I'm getting on the interface in Ubuntu:

eth2: flags=4163(UP,BROADCAST,RUNNING,MULTICAST)  mtu 1500
    inet 192.168.1.43  netmask 255.255.255.0   broadcast 192.168.1.255
    inet6 fe80::3efd:feff:feb2:cd30  prefixlen 64  scopeid 0x20(link)
    ether 3c:fd:fe:b2:cd:30  txqueuelen 1000  (Ethernet)
    RX packets 741  bytes 46100 (46.1 KB)
    RX errors 382  dropped 197  overruns 0  frame 382
    TX packets 39  bytes 6887 (6.8 KB)
    TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Pinging is fine to and from, and even with larger byte sizes like 1,400 bytes.

On the switch there are zero RX or TX errors:

Spoiler: Switch output for 1/2/1

40GigabitEthernet1/2/1 is up, line protocol is up
  Port up for 16 minute(s) 37 second(s)
  Hardware is 40GigabitEthernet, address is 748e.f8fe.93f6 (bia 748e.f8fe.940f)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual 40Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual none
  Member of L2 VLAN ID 1, port is untagged, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is enabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  MTU 1500 bytes, encapsulation ethernet
  300 second input rate: 48 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 2168 bits/sec, 3 packets/sec, 0.00% utilization
  56352 packets input, 6296114 bytes, 0 no buffer
  Received 503 broadcasts, 7331 multicasts, 48518 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  350121 packets output, 73835998 bytes, 0 underruns
  Transmitted 143184 broadcasts, 121844 multicasts, 85093 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0              347387                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                  19                   0
    6                   0                   0
    7                2719                   0

Tried disabling LLDP for that port (a suggestion), made things worse, re-enabled. Port 1/2/6 acts the same.

Should I be looking for a new switch (bad ports), or different type of NIC?

EDIT:
If I ping with 1,450 bytes I get 0% packet loss. If I ping with 1,451 bytes or more, I get 100% packet loss.

Post your switch config again - i have not used the 40G ports (but am about to when my cables arrive) - is there anything special that needs to be done to put them into 40Gb mode instead of stacking mode ?

Craig

 

[ramicio]

Post your switch config again - i have not used the 40G ports (but am about to when my cables arrive) - is there anything special that needs to be done to put them into 40Gb mode instead of stacking mode ?

Craig

Spoiler: show run

Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
  module 1 icx6610-24p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
stack disable
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
hostname switcheroo
ip dhcp-client disable
!
no telnet server
username root password .....
!
!
!
!
!
!
!
!
!
interface ethernet 1/1/3
 inline power
!
interface ethernet 1/1/6
 inline power
!
interface ethernet 1/1/7
 inline power
!
interface ethernet 1/3/1
 speed-duplex 10G-full
!
interface ethernet 1/3/2
 speed-duplex 10G-full
!
interface ethernet 1/3/3
 speed-duplex 10G-full
!
interface ethernet 1/3/4
 speed-duplex 10G-full
!
interface ethernet 1/3/5
 speed-duplex 10G-full
!
interface ethernet 1/3/6
 speed-duplex 10G-full
!
interface ethernet 1/3/7
 speed-duplex 10G-full
!
interface ethernet 1/3/8
 speed-duplex 10G-full
!
interface ve 1
 ip address 192.168.1.5 255.255.255.0
!
!
!
!
!
lldp run
!
!
!
!
end

 

[itronin]

Post your switch config again - i have not used the 40G ports (but am about to when my cables arrive) - is there anything special that needs to be done to put them into 40Gb mode instead of stacking mode ?

Craig

Its in the guide. basically disable stacking

 

[ramicio]

So, I used the Ethernet Port Config utility to set the port from 1x40g to 4x10g. It gives me 4 interfaces on the server. On the switch I moved the DAC over to the 1/2/2 - 1/2/5 breakout port. Gave /etc/network/interfaces 4 entries, used DHCP. Each interface pulled an address right away and each interface has ZERO errors. Are these cards not compatible with this switch when it comes to full 40g? Or can 1/2/1 and 1/2/6 both be dead whilst the 2 breakout ports are fine?

 

[ramicio]

I've also noticed that none of the "XL" LEDs on the front ever come on. Not during boot, and not even when a link is established (albeit with RX errors on the server). XL2-5 should at least be lit, because that is a successful link without any errors on the counters.

 

[ramicio]

I got a partial refund for the switch. Is it worth it buying another one? Is there a known issue with these particular NICs?

 

[Craig Curtin]

I got a partial refund for the switch. Is it worth it buying another one? Is there a known issue with these particular NICs?

I do not believe there are any known issues with those NICs.

I have just setup my Mellanox CX354 (single port) with a 40GB QSFP cable straight into 1/2/1 and 1/2/6 and had no problems.

Craig

 

[ramicio]

There is some physical weirdness going on inside of this switch. I opened it up and took out the 40g board. There was a piece of something down in the pins of one of the headers on the motherboard that are used to connect to the 40g board. So someone was definitely in there before tinkering, maybe trying to solve the same problem I am having. I blew it out with canned air, as well as the other header and the opposing connectors on the 40g board, but it made no difference. It just stinks I got the last one from the seller, or I would have just opted for a replacement. Are the "XL" LEDs on the front supposed to light up?

 

[itronin]

I apologize in advance If I am in the wrong spot, a bit NEW to these forums. I was looking into replacing two of my Ubiquiti Edge devices (EdgeRouter 10x and EdgeSwitch 10x) with a single Mikrotik CSS326-24G-2S+RM and pfSense. I came across a link in a reddit post to this thread. Based on the first post I was looking at a Brocade ICX6450-48P 48-Port PoE and not a 6610 due to noise. The 6450 48-P looks like I could also replace my EdgeSwitch 16 150W which powers some POE cameras and allow some room for me to play around with.

I have 3 EXSI servers for my home lab, multiple POEs, 1 POE AP, a couple of vlans, and general day to day home use (Plex, Media, Phones, Work, etc). Thinking about adding a freenas device or maybe some shared storage on day.

This thread which has some current posts and the 6450 48-P are both multiple years old, is the 6450 still a decent switch to purchase?

Is the licensing Guide V2 in the post still valid licensing instructions?

I am open to recommendations also.



Thank You in advance for answering an amateur’s questions!

TLDR - Yes.

Longer read - I'd try and get through the first 40 pages of posts in this thread. This is still a very active thread - start at the end and go backwards - you'll see. Fewer posts because people find the answers in this thread.

 

[smitty870]

TLDR - Yes.

Longer read - I'd try and get through the first 40 pages of posts in this thread. This is still a very active thread - start at the end and go backwards - you'll see. Fewer posts because people find the answers in this thread.

Thanks, I had seen it was still active and was not sure where to start or what to search for. I'll follow your advice!

 

[Craig Curtin]

Just wanted to ask - can the 4 ports on the back of the 6610 all be used as 40GB for NIC attachment - or do the breakout ports have to be used as breakout ports

I still have not managed to solve my problem with the disabling of ports in ESXi when i make VLAN changes - but by using 40GB NICs i have two servers up and running directly attached to the QSFP stacking ports in the rear - in my home lab - would be nice if i could get my last host connected at 40GB by removing the fanout out cable and direct attaching with a QSFP nic

Craig

 

[LodeRunner]

Just wanted to ask - can the 4 ports on the back of the 6610 all be used as 40GB for NIC attachment - or do the breakout ports have to be used as breakout ports

I still have not managed to solve my problem with the disabling of ports in ESXi when i make VLAN changes - but by using 40GB NICs i have two servers up and running directly attached to the QSFP stacking ports in the rear - in my home lab - would be nice if i could get my last host connected at 40GB by removing the fanout out cable and direct attaching with a QSFP nic

Craig

Pretty sure the two breakout ports are hardcoded to be so on the 6610.

 

[NablaSquaredG]

or do the breakout ports have to be used as breakout ports

TL,DR: Yes.

 

[vangoose]

ICX Firmware update video

 

[Craig Curtin]

6450 - which do i believe ?

Guys - 2 x 6450 - licensed and configured as per OP document - no issues

I have a couple of different SFP+ cables

1) 10GTek - AOC SFP+ 20m fibre - as per here - https://www.amazon.com.au/dp/B08TSZ5TH1?ref=ppx_yo2ov_dt_b_product_details&th=1
2) No name 1m DAC SFP+
3) Dell 5m DAC SFP+ - known to work between 2 x 6610 and a 6610 and an Intel 10GB card

When i link the two 6450 through their SFP+ ports i get this

SSH@afsydney-6450-Bottom>show int e 1/2/3

10GigabitEthernet1/2/3 is up, line protocol is up
  Port up for 4 minute(s) 5 second(s)
  Hardware is 10GigabitEthernet, address is cc4e.24a8.4540 (bia cc4e.24a8.4573)
  Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx
  Member of L2 VLAN ID 1, port is untagged, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Flow Control is enabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  MTU 1500 bytes, encapsulation ethernet
  300 second input rate: 344 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 14528 bits/sec, 11 packets/sec, 0.00% utilization
  904 packets input, 97366 bytes, 0 no buffer
  Received 10 broadcasts, 0 multicasts, 894 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  7803 packets output, 1149522 bytes, 0 underruns
  Transmitted 2831 broadcasts, 4135 multicasts, 837 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0                   0                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                   0                   0
    6                   0                   0
    7                   0                   0
SSH@afsydney-6450-Bottom>

So it thinks the port is running at 10G and i get the same report on the other end - however a show media does this




SSH@afsydney-6450-Bottom>show media e 1/2/3
Port   1/2/3: Type  : 1GE M-SX(SFP)
             Vendor: Intel Corp         Version: A
             Part# : FTLX8571D3BCV-IT   Serial#: INA10M20058


and when i switch to the Copper DAC i get this (yet it is passing traffic between switches so seems to work ?)



SSH@afsydney-6450-Bottom>show media e 1/2/3
Port   1/2/3: Type  : 10GE  Passive Twinax  1m (SFP +) (Not supported)
             Vendor: FCI Electronics    Version: G
             Part# : 10110818-2010LF    Serial#: CN1913SA102L021
SSH@afsydney-6450-Bottom>


And this is with a Dell 10GB DAC 5m cable that i know works between two of my 6610 switches and also from an Intel client to a 6610 at 10GB



SSH@afsydney-6450-Bottom>show media e 1/2/3
Port   1/2/3: Type  : 10GE  Passive Twinax  5m (SFP +) (Not supported)
             Vendor: Molex Inc.         Version:
             Part# : 74752-9507         Serial#: 012730026

So does it look like the Intel coded cable is working at 10GB ? (i will test shortly with a client card to the switch)

And should i be concerned about the not supported statement after both of the DACs ?


Craig