Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

Anyone else got authentication issues via ssh on ICX6610 switches when key-authentication is enabled?

Debug session says:

....
debug1: Offering public key: /home/robin/.ssh/id_rsa.pub RSA SHA256:GKW7yzA1J1qkr1Cr9MhUwAbHbF2NrIPEgZXeOUOz3Us explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: no key from blob. pkalg ssh-rsa: invalid format
ssh_dispatch_run_fatal: Connection to 10.10.2.50 port 22: invalid format

When I add ip ssh key-authentication no, it works fine...

SW version is 8030t, my SSH key is 4096 bits long

I have not setup the pub-key-file on the switch itself, however I wouldn't expect to see weird issues even if it should be set up

the 6 series only supports 2048 length keys, you might be making it barf by offering too thicc of bits even when it doesn't have a pubkey

 

[richtj99]

PoE units don't really use anymore power than their non-PoE counterparts unless you're actively powering PoE devices. You can also just remove the entire PoE daughterboard, it's the one on top

Is this the part (yellow?) Unplug, remove board? Can I take a daughter board from a 6450 to use in its place or is the issue something beyond the POE daughterboard?

 

[fohdeesha]

Is this the part (yellow?) Unplug, remove board? Can I take a daughter board from a 6450 to use in its place or is the issue something beyond the POE daughterboard?

That's the one, remove the cables with it too. Problem is almost definitely on the board somewhere, you could swap another one in if you had an exact replacement from the same model switch and same port count of that model. Something from a different model entirely like a 6450 will not work

 

[NablaSquaredG]

the 6 series only supports 2048 length keys, you might be making it barf by offering too thicc of bits even when it doesn't have a pubkey

But it works via password based authentication... If the unit can't handle 4096 bits, password auth shouldn't work (except if Brocade messed up, which would, given my experiences with the switch so far, really surprise me)


On another note:
I just tried to configure the 10GBe Ports for stacking and... failed.

I've googled the error ("1/2/2 must be one of the stacking ports: 1/2/1 1/2/6") and literally the only result is this thread.

That's a bit disappointing... But I can't imagine that these ports do not support stacking.
The manual has an entire section dedicated to them with sentences like

"Periodic background diagnosis provides warning messages when any 4 x10-Gbps subport is down."

The most common connection error in forming an ICX 6610 stack is connecting a 40 Gbps port to a 4 x10-Gbps port, possibly because the two port types use the same type of cable. When this happens, the system may show one end is up, and one end is down. The stack cannot be formed, and the periodic background diagnosis does not run.

For example, the active controller console displays the following messages for an ICX 6610 stack if a 4 x10-Gbps sub-port is down. *** Warning! miss 4*10G link 5/2/8(down)to 1/2/8(down). Stack can still work. *** Warning! U1, dir=1, 4*10G ports: 1/2/8 are down. *** Warning! U5, dir=1, 4*10G ports: 5/2/8 are down. Please use "show stack conn" to view detailed connections You can suppress the error messages by configuring "stack suppress-warning"

Edit:
Uhh, I THINK I've got it!

Will add more infos later

 

[NablaSquaredG]

So apparently it's not possible to ONLY use the 4x10G Port for stacking.

You can either have:

• 4x10G and 40G as data port (no stacking)
• 40G as stacking port and 4x10G as data ports
• 40G and 4x10G as stacking port

4x10G as stacking port and 40G as data ports is NOT possible. Why?

SSH@tbm-switch-leaf-public-1(config-unit-1)#stack-port 1/2/2
Error! 1/2/2 must be one of the stacking ports: 1/2/1 1/2/6.

So that's where my issues started.
stack secure-setup does NOT use the 4x10G ports by default.

So what do you need to do? Well, trunk the 40G ports together with the 4x10G ports.

It's as easy as

stack-trunk 1/2/1 to 1/2/5
stack-trunk 1/2/6 to 1/2/10

First command trunks the upper two ports, second command trunks the lower two ports.

If you run show running-config you should see:

stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-trunk 1/2/1 to 1/2/2
  stack-trunk 1/2/6 to 1/2/7
  stack-port 1/2/1 1/2/6

(don't wonder that it's only to 1/2/2 and to 1/2/7 - It is not unusual that the first port serves as an identifier for the other ones)

You can then use stack secure-setup to set up everything. NOTE: I have NOT tested this in a three device setup (only two devices), so YMMV.

You will then see something like

   active                                                                     
     +---+        +---+                                                       
 =2/6| 1 |2/1=-2/6| 2 |2/1-                                                   
 |   +---+        +---+   |                                                   
 |                        |                                                   
 |------------------------|                                                   

*** Warning! detect 2 trunk-to-port links. The stack may not work.
  U1 stack-trunk (1/2/1-1/2/2) to U2 stack-port (2/2/6)
  U1 stack-trunk (1/2/6-1/2/7) to U2 stack-port (2/2/1)

U#  Stack-port1                                  Stack-port2           
1   up (1/2/1-1/2/2)                             up (1/2/6-1/2/7)       
    up ports: 1/2/1, 1/2/2, 1/2/3, 1/2/4, 1/2/5
    up ports: 1/2/6, 1/2/7, 1/2/8, 1/2/9, 1/2/10

2   up (2/2/1)                                   up (2/2/6)

Go ahead and trunk the other ports

(config-unit-2)#stack-trunk 2/2/1 to 2/2/5
(config-unit-2)#stack-trunk 2/2/6 to 2/2/10

and there you go:

SSH@tbm-switch-leaf-public-1#show stack stack-ports
    active       standby                                                       
     +---+        +---+                                                       
 =2/6| 1 |2/1==2/6| 2 |2/1=                                                   
 |   +---+        +---+   |                                                   
 |                        |                                                   
 |------------------------|                                                   

U#  Stack-port1                                  Stack-port2           
1   up (1/2/1-1/2/2)                             up (1/2/6-1/2/7)       
    up ports: 1/2/1, 1/2/2, 1/2/3, 1/2/4, 1/2/5
    up ports: 1/2/6, 1/2/7, 1/2/8, 1/2/9, 1/2/10

2   up (2/2/1-2/2/2)                             up (2/2/6-2/2/7)       
    up ports: 2/2/1, 2/2/2, 2/2/3, 2/2/4, 2/2/5
    up ports: 2/2/6, 2/2/7, 2/2/8, 2/2/9, 2/2/10

TL;DR: Ignore the Brötcom docs, they're not complete...

 

[Propaganda]

The guide for the icx7250 provides some old firmware, is there any reason not to use the latest version on this site?
RUCKUS ICX 7250 Switch Hardware Installation Guide | Technical Documents | Ruckus Wireless Support

 

[ClintE]

ICX6610-48P arrived today from eBay. Followed the docs and it worked perfectly. It's all up and updated and connected. Thank you @fohdeesha for the guide. No way I would have ever figured out how to get this thing going on my own.

So far the only problem is the fan in one of the two supplies is much louder than the other. They're both Rev. A but I think it's actually just the fan going out or a bad bearing or something. I guess I'll just have to learn to live with it as the replacement supplies are incredibly expensive compared to the whole switch with two supplies and fan modules.

I'm still waiting for a few cards to arrive to try out 10/40gbe but for now it's working nicely at 1g.

Not sure, but I don't think you need two PSU's in any of the 6610 models; they're for redundancy and don't split the load. If I'm wrong about this, someone please correct me.

 

[tozmo]

What do the CONN jumpers do? My 7450-32zp has busted 2.5 ports, but there is a CONN16 3-pin header next to it. There are a lot of headers on these switches, but what purpose do they serve?

The attached pic shows the two processors over the 2.5gbe ports (right) and the regular ones (left). The 8 Poe ports are off screen to the left and do not have the jumper

 

[fohdeesha]

What do the CONN jumpers do? My 7450-32zp has busted 2.5 ports, but there is a CONN16 3-pin header next to it. There are a lot of headers on these switches, but what purpose do they serve?

can you attach a picture? They're debug headers for factory use, stuff like CPLD JTAG, CPU JTAG, etc. 3-pin might be a debug jumper though, not sure. never seen a 3 pin in these

 

[fohdeesha]

The guide for the icx7250 provides some old firmware, is there any reason not to use the latest version on this site?
RUCKUS ICX 7250 Switch Hardware Installation Guide | Technical Documents | Ruckus Wireless Support

Haven't had a chance to re-write the guide with new instructions for getting to the recommended 8092 train - it's the new UFI images that require a bit more steps for switches with older firmware. Been working on it this week though. Note the very latest is the 8095 train, but it's highly unadvised to run it on anything resembling production, it's basically the beta branch. The recommended stable train is 8092, 8092e I believe being the latest

 

[NablaSquaredG]

What do you think about my stacking "guide" @fohdeesha ?

 

[Jason Antes]

Here is a link to the BNA versions that I've uploaded. Haven't had a chance to test on 2019 yet for the latest version from Ruckus. Again, I'd use that version if you are wanting to manage IP as it's the most up to date with security fixes. If you need FC management then use the 14.3 version.

BNA

 

[Scarlet]

The link requires a google login, is that intended?

 

[JMHershey125]

Good afternoon,

I just picked up a ICX6610 switch, does this mean I have a fully licensed switch?

ICX6610-48P Router>show lic

Index    Lic Mode        Lic Name               Lid/Serial No  Lic Type    Status     Lic Period    Lic Capacity     

Stack unit 1:

1        Node Lock       ICX6610-PREM-LIC-SW    dzmHKHFlFIh    Normal      Invalid    Unlimited     0

2        Node Lock       ICX6610-10G-LIC-POD    dzrHKFHlHdN    Normal      Active     Unlimited     8

3        Node Lock       ICX6610-ADV-LIC-SW     dzrHKFHlHdN    Normal      Active     Unlimited     1

4        Node Lock       ICX-MACSEC-LIC         dzrHKFHlHdN    Normal      Active     Unlimited     1

ICX6610-48P Router>

 

[Jason Antes]

The link requires a google login, is that intended?

Try it again, I missed a setting and corrected it. Anyone should be able to get at them now.

 

[fohdeesha]

Good afternoon,

I just picked up a ICX6610 switch, does this mean I have a fully licensed switch?

ICX6610-48P Router>show lic

Index    Lic Mode        Lic Name               Lid/Serial No  Lic Type    Status     Lic Period    Lic Capacity    

Stack unit 1:

1        Node Lock       ICX6610-PREM-LIC-SW    dzmHKHFlFIh    Normal      Invalid    Unlimited     0

2        Node Lock       ICX6610-10G-LIC-POD    dzrHKFHlHdN    Normal      Active     Unlimited     8

3        Node Lock       ICX6610-ADV-LIC-SW     dzrHKFHlHdN    Normal      Active     Unlimited     1

4        Node Lock       ICX-MACSEC-LIC         dzrHKFHlHdN    Normal      Active     Unlimited     1

ICX6610-48P Router>

yes, with my licenses. Where did it come from?

 

[JMHershey125]

I bought it on eBay for $175 from a seller who pulled it from his homelab. the listing didn’t say anything about licenses, so I figured I should check on here.

 

[fohdeesha]

I bought it on eBay for $175 from a seller who pulled it from his homelab. the listing didn’t say anything about licenses, so I figured I should check on here.

Well I'll give him a pass for not advertising the licenses, but he should have deleted those before selling the switch

 

[epicurean]

For the icx7150 c12 , is there a way to completely reset the switch but keep the licenses intact?

 

[LodeRunner]

For the icx7150 c12 , is there a way to completely reset the switch but keep the licenses intact?

The license is honor based. Nuke the switch, follow fohdeesha's 7150 guide.