Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[joshdave]

I recently purchased a ICX6450-24p. I have NTP set up, and it properly sets the clock not long after the unit is booted. However I noticed that during boot and shortly after boot, the clock is set to 1969. Not a biggie, but just curious if this is normal for this switch? I tested the battery (BR2032), and it looks good (2.83v).

 

[epicurean]

Im wondering if this has to do with the pfsense firewall & the speed of Vlan 1 talking to vlan 80? I notice with my firewall, even with port rules enabled for inter-vlan traffic, there is a speed difference in transferring files from vlan 20 (nas no internet access) to my main vlan (25 with internet). Some VM's on Vlan 20 get better transfer speeds within vlan 20.

I switched a bunch of things to 10G & it runs better but I noticed that with a ton of IP cameras, two NVRS (dont ask), that there was so much traffic on my 1gb switches that my 1gb Verizon FIOS connection was only getting 400 speeds. When I turned off my two NVR's the speeds went to the mid 900 range which is closer to what I would expect. Once the NVR's went back on speeds slowed the whole network down.

When I moved from Cisco 1gb to these 6450/7250 units & made sure the only inter switch connectivity was through the 10gb connections it has been great.

I do have a lag setup so I am at 30gb between switches except:

Verizon Fios ONT to Firewall (1gb) to Unifi switch (1gb fiber) to icx6450.

just some thoughts - not sure if this is helpful.

I do have another 6450 (non POE) and a unifi switch 16 XB Is it possible to use them to offset this loss in bandwidth from the POE cameras
? or is there some setting in pfsense that can mitigate this?

 

[fohdeesha]

Hi - I purchased a 7250-48p - first one came with no working serial port, seller sent a 'tested' replacement - serial does work, as soon as it boots up I get this error:

U1-MSG: PoE Severe Error: Power being injected on port 1/1/33. No new PDs can get powered on this unit.Configure "no inline power" on all Switch to Switch connected ports of this unit and peer unit(s) to resolve the issue.

I tried disabling POE on all ports, just port 33, etc. I am not sure what to do or if this is fixable.

I took a photo of the inside (attached).

ICX7250-48P Router# show inline power

Power Capacity:         Total is 740000 mWatts. Current Free is 740000 mWatts.

Power Allocations:      Requests Honored 48 times


Port   Admin   Oper    ---Power(mWatts)---  PD Type  PD Class  Pri  Fault/
        State   State   Consumed  Allocated                          Error
--------------------------------------------------------------------------
  1/1/1 On      Off            0          0  n/a      n/a         3  n/a
  1/1/2 On      Off            0          0  n/a      n/a         3  n/a
  1/1/3 On      Off            0          0  n/a      n/a         3  n/a
  1/1/4 On      Off            0          0  n/a      n/a         3  n/a
  1/1/5 On      Off            0          0  n/a      n/a         3  n/a
  1/1/6 On      Off            0          0  n/a      n/a         3  n/a
  1/1/7 On      Off            0          0  n/a      n/a         3  n/a
  1/1/8 On      Off            0          0  n/a      n/a         3  n/a
  1/1/9 On      Off            0          0  n/a      n/a         3  n/a
1/1/10 On      Off            0          0  n/a      n/a         3  n/a
1/1/11 On      Off            0          0  n/a      n/a         3  n/a
1/1/12 On      Off            0          0  n/a      n/a         3  n/a
1/1/13 On      Off            0          0  n/a      n/a         3  n/a
1/1/14 On      Off            0          0  n/a      n/a         3  n/a
1/1/15 On      Off            0          0  n/a      n/a         3  n/a
1/1/16 On      Off            0          0  n/a      n/a         3  n/a
1/1/17 On      Off            0          0  n/a      n/a         3  n/a
1/1/18 On      Off            0          0  n/a      n/a         3  n/a
1/1/19 On      Off            0          0  n/a      n/a         3  n/a
1/1/20 On      Off            0          0  n/a      n/a         3  n/a
1/1/21 On      Off            0          0  n/a      n/a         3  n/a
1/1/22 On      Off            0          0  n/a      n/a         3  n/a
1/1/23 On      Off            0          0  n/a      n/a         3  n/a
1/1/24 On      Off            0          0  n/a      n/a         3  n/a
1/1/25 On      Off            0          0  n/a      n/a         3  n/a
1/1/26 On      Off            0          0  n/a      n/a         3  n/a
1/1/27 On      Off            0          0  n/a      n/a         3  n/a
1/1/28 On      Off            0          0  n/a      n/a         3  n/a
1/1/29 On      Off            0          0  n/a      n/a         3  n/a
1/1/30 On      Off            0          0  n/a      n/a         3  n/a
1/1/31 On      Off            0          0  n/a      n/a         3  n/a
1/1/32 On      Off            0          0  n/a      n/a         3  n/a
1/1/33 On      Off            0          0  n/a      n/a         3  voltage applied from ext src
1/1/34 On      Off            0          0  n/a      n/a         3  n/a
1/1/35 On      Off            0          0  n/a      n/a         3  n/a
1/1/36 On      Off            0          0  n/a      n/a         3  n/a
1/1/37 On      Off            0          0  n/a      n/a         3  n/a
1/1/38 On      Off            0          0  n/a      n/a         3  n/a
1/1/39 On      Off            0          0  n/a      n/a         3  n/a
1/1/40 On      Off            0          0  n/a      n/a         3  n/a
1/1/41 On      Off            0          0  n/a      n/a         3  n/a
1/1/42 On      Off            0          0  n/a      n/a         3  n/a
1/1/43 On      Off            0          0  n/a      n/a         3  n/a
1/1/44 On      Off            0          0  n/a      n/a         3  n/a
1/1/45 On      Off            0          0  n/a      n/a         3  n/a
1/1/46 On      Off            0          0  n/a      n/a         3  n/a
1/1/47 On      Off            0          0  n/a      n/a         3  n/a
1/1/48 On      Off            0          0  n/a      n/a         3  n/a
--------------------------------------------------------------------------
Total                         0          0

I disable port 33 poe then plug two poe devices into other random ports with POE enabled.

ICX7250-48P Router# U1-MSG: PoE Severe Error: PD on port 1/1/10 cannot be powered due to power being injected on another port of this unit.Configure "no inline power" on all Switch to Switch connected ports of this unit and peer unit(s) to resolve the issue.

U1-MSG: PoE Severe Error: PD on port 1/1/3 cannot be powered due to power being injected on another port of this unit.Configure "no inline power" on all Switch to Switch connected ports of this unit and peer unit(s) to resolve the issue.

Sort of at a loss on what to try?

Any suggestions would be great.

yeah that poe board is fried, tell the seller he's a dingus

 

[fohdeesha]

I recently purchased a ICX6450-24p. I have NTP set up, and it properly sets the clock not long after the unit is booted. However I noticed that during boot and shortly after boot, the clock is set to 1969. Not a biggie, but just curious if this is normal for this switch? I tested the battery (BR2032), and it looks good (2.83v).

normal

 

[fohdeesha]

big improvements & features coming to the guide soon

 

[richtj99]

I do have another 6450 (non POE) and a unifi switch 16 XB Is it possible to use them to offset this loss in bandwidth from the POE cameras
? or is there some setting in pfsense that can mitigate this?

What are the specs of your PFsense box? How is the CPU / Mem usage?

Unifi XG (10gb?) - What VMS/NVR for the cameas? If it is Blue Iris - what are the stream megabits?

Is it 1 vlan for you 1 vlan for cameras? I am happy to help where I can but switching to 10gb was pretty immediate for me. Again my vlans are all set with all cameras on the same camera vlan.

yeah that poe board is fried, tell the seller he's a dingus

Yeah I figured - bummer - can I safely use it as a non-poe unit? I can make an offer on it vs making him pay for shipping again. Then again I am not sure if a 7250-48p with POE disabled will use more electric than its worth. I want to make sure the rest of the switch isnt going to blow up. can I take the POE part of the board out or swap it?

 

[epicurean]

What are the specs of your PFsense box? How is the CPU / Mem usage?

Unifi XG (10gb?) - What VMS/NVR for the cameas? If it is Blue Iris - what are the stream megabits?

Is it 1 vlan for you 1 vlan for cameras? I am happy to help where I can but switching to 10gb was pretty immediate for me. Again my vlans are all set with all cameras on the same camera vlan.


My pfsense is an i3 4xxxT CPU with 16GB, on an supermicro X10SLV motherboard
Yes, the Unifi 16XG has SFP+ ports and 4 10 GB RJ45 ports. I do have blue iris, but I yet to connec yet and trying to figure out the network bog down. The blue iris PC is a dedicated 1151 motherboard on a 1GB connection.

My cameras are on vlan 80. I have a different vlan for IOT devices. All other devices are on my main LAN

 

[SuperMiguel]

For a 3 ft run I'd use a dac, more robust. Stay at 5m or less, the 6450 doesn't like 10m dacs

Pardon Our Interruption...

www.ebay.com

Any recommendation on network cards?

Was thinking to get these: 0N20KJ DELL BROADCOM 57810 10GB DUAL PORT PCI-E SFP+ NETWORK CARD N20KJ Y40PH | eBay

 

[Propaganda]

What for a icx7250, what is the logic level of miniusb/uart? rs232/TTL(1.8V)/TTL(3.3V)/TTL(5V)/other? Thanks!

 

[fohdeesha]

What for a icx7250, what is the logic level of miniusb/uart? rs232/TTL(1.8V)/TTL(3.3V)/TTL(5V)/other? Thanks!

it's standard RS232

 

[fohdeesha]

What are the specs of your PFsense box? How is the CPU / Mem usage?

Unifi XG (10gb?) - What VMS/NVR for the cameas? If it is Blue Iris - what are the stream megabits?

Is it 1 vlan for you 1 vlan for cameras? I am happy to help where I can but switching to 10gb was pretty immediate for me. Again my vlans are all set with all cameras on the same camera vlan.



Yeah I figured - bummer - can I safely use it as a non-poe unit? I can make an offer on it vs making him pay for shipping again. Then again I am not sure if a 7250-48p with POE disabled will use more electric than its worth. I want to make sure the rest of the switch isnt going to blow up. can I take the POE part of the board out or swap it?

PoE units don't really use anymore power than their non-PoE counterparts unless you're actively powering PoE devices. You can also just remove the entire PoE daughterboard, it's the one on top

 

[richtj99]

That all sounds good to me. Is the SFP+ port plugged into the 6450? I dont think you need it but can you pop a 10gb sfp+ card into the PFSsense box going to the Unifi?

I have close to 50 cameras on my network all on 1 vlan & am not seeing a slowdown.

I dont use PFsense anymore but it did have good reporting broken out on lan/vlan (wan too) - what activity level is it showing?

Is the 6450 set for auto speed?

show int brief?

to confirm the slowdown is on the Lan vlan?

My pfsense is an i3 4xxxT CPU with 16GB, on an supermicro X10SLV motherboard
Yes, the Unifi 16XG has SFP+ ports and 4 10 GB RJ45 ports. I do have blue iris, but I yet to connec yet and trying to figure out the network bog down. The blue iris PC is a dedicated 1151 motherboard on a 1GB connection.

My cameras are on vlan 80. I have a different vlan for IOT devices. All other devices are on my main LAN

 

[NablaSquaredG]

Anyone else got authentication issues via ssh on ICX6610 switches when key-authentication is enabled?

Debug session says:

....
debug1: Offering public key: /home/robin/.ssh/id_rsa.pub RSA SHA256:GKW7yzA1J1qkr1Cr9MhUwAbHbF2NrIPEgZXeOUOz3Us explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: no key from blob. pkalg ssh-rsa: invalid format
ssh_dispatch_run_fatal: Connection to 10.10.2.50 port 22: invalid format

When I add ip ssh key-authentication no, it works fine...

SW version is 8030t, my SSH key is 4096 bits long

I have not setup the pub-key-file on the switch itself, however I wouldn't expect to see weird issues even if it should be set up

 

[fohdeesha]

Anyone else got authentication issues via ssh on ICX6610 switches when key-authentication is enabled?

Debug session says:

....
debug1: Offering public key: /home/robin/.ssh/id_rsa.pub RSA SHA256:GKW7yzA1J1qkr1Cr9MhUwAbHbF2NrIPEgZXeOUOz3Us explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: no key from blob. pkalg ssh-rsa: invalid format
ssh_dispatch_run_fatal: Connection to 10.10.2.50 port 22: invalid format

When I add ip ssh key-authentication no, it works fine...

SW version is 8030t, my SSH key is 4096 bits long

I have not setup the pub-key-file on the switch itself, however I wouldn't expect to see weird issues even if it should be set up

the 6 series only supports 2048 length keys, you might be making it barf by offering too thicc of bits even when it doesn't have a pubkey

 

[richtj99]

PoE units don't really use anymore power than their non-PoE counterparts unless you're actively powering PoE devices. You can also just remove the entire PoE daughterboard, it's the one on top

Is this the part (yellow?) Unplug, remove board? Can I take a daughter board from a 6450 to use in its place or is the issue something beyond the POE daughterboard?

 

[fohdeesha]

Is this the part (yellow?) Unplug, remove board? Can I take a daughter board from a 6450 to use in its place or is the issue something beyond the POE daughterboard?

That's the one, remove the cables with it too. Problem is almost definitely on the board somewhere, you could swap another one in if you had an exact replacement from the same model switch and same port count of that model. Something from a different model entirely like a 6450 will not work

 

[NablaSquaredG]

the 6 series only supports 2048 length keys, you might be making it barf by offering too thicc of bits even when it doesn't have a pubkey

But it works via password based authentication... If the unit can't handle 4096 bits, password auth shouldn't work (except if Brocade messed up, which would, given my experiences with the switch so far, really surprise me)


On another note:
I just tried to configure the 10GBe Ports for stacking and... failed.

I've googled the error ("1/2/2 must be one of the stacking ports: 1/2/1 1/2/6") and literally the only result is this thread.

That's a bit disappointing... But I can't imagine that these ports do not support stacking.
The manual has an entire section dedicated to them with sentences like

"Periodic background diagnosis provides warning messages when any 4 x10-Gbps subport is down."

The most common connection error in forming an ICX 6610 stack is connecting a 40 Gbps port to a 4 x10-Gbps port, possibly because the two port types use the same type of cable. When this happens, the system may show one end is up, and one end is down. The stack cannot be formed, and the periodic background diagnosis does not run.

For example, the active controller console displays the following messages for an ICX 6610 stack if a 4 x10-Gbps sub-port is down. *** Warning! miss 4*10G link 5/2/8(down)to 1/2/8(down). Stack can still work. *** Warning! U1, dir=1, 4*10G ports: 1/2/8 are down. *** Warning! U5, dir=1, 4*10G ports: 5/2/8 are down. Please use "show stack conn" to view detailed connections You can suppress the error messages by configuring "stack suppress-warning"

Edit:
Uhh, I THINK I've got it!

Will add more infos later

 

[NablaSquaredG]

So apparently it's not possible to ONLY use the 4x10G Port for stacking.

You can either have:

• 4x10G and 40G as data port (no stacking)
• 40G as stacking port and 4x10G as data ports
• 40G and 4x10G as stacking port

4x10G as stacking port and 40G as data ports is NOT possible. Why?

SSH@tbm-switch-leaf-public-1(config-unit-1)#stack-port 1/2/2
Error! 1/2/2 must be one of the stacking ports: 1/2/1 1/2/6.

So that's where my issues started.
stack secure-setup does NOT use the 4x10G ports by default.

So what do you need to do? Well, trunk the 40G ports together with the 4x10G ports.

It's as easy as

stack-trunk 1/2/1 to 1/2/5
stack-trunk 1/2/6 to 1/2/10

First command trunks the upper two ports, second command trunks the lower two ports.

If you run show running-config you should see:

stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-trunk 1/2/1 to 1/2/2
  stack-trunk 1/2/6 to 1/2/7
  stack-port 1/2/1 1/2/6

(don't wonder that it's only to 1/2/2 and to 1/2/7 - It is not unusual that the first port serves as an identifier for the other ones)

You can then use stack secure-setup to set up everything. NOTE: I have NOT tested this in a three device setup (only two devices), so YMMV.

You will then see something like

   active                                                                     
     +---+        +---+                                                       
 =2/6| 1 |2/1=-2/6| 2 |2/1-                                                   
 |   +---+        +---+   |                                                   
 |                        |                                                   
 |------------------------|                                                   

*** Warning! detect 2 trunk-to-port links. The stack may not work.
  U1 stack-trunk (1/2/1-1/2/2) to U2 stack-port (2/2/6)
  U1 stack-trunk (1/2/6-1/2/7) to U2 stack-port (2/2/1)

U#  Stack-port1                                  Stack-port2           
1   up (1/2/1-1/2/2)                             up (1/2/6-1/2/7)       
    up ports: 1/2/1, 1/2/2, 1/2/3, 1/2/4, 1/2/5
    up ports: 1/2/6, 1/2/7, 1/2/8, 1/2/9, 1/2/10

2   up (2/2/1)                                   up (2/2/6)

Go ahead and trunk the other ports

(config-unit-2)#stack-trunk 2/2/1 to 2/2/5
(config-unit-2)#stack-trunk 2/2/6 to 2/2/10

and there you go:

SSH@tbm-switch-leaf-public-1#show stack stack-ports
    active       standby                                                       
     +---+        +---+                                                       
 =2/6| 1 |2/1==2/6| 2 |2/1=                                                   
 |   +---+        +---+   |                                                   
 |                        |                                                   
 |------------------------|                                                   

U#  Stack-port1                                  Stack-port2           
1   up (1/2/1-1/2/2)                             up (1/2/6-1/2/7)       
    up ports: 1/2/1, 1/2/2, 1/2/3, 1/2/4, 1/2/5
    up ports: 1/2/6, 1/2/7, 1/2/8, 1/2/9, 1/2/10

2   up (2/2/1-2/2/2)                             up (2/2/6-2/2/7)       
    up ports: 2/2/1, 2/2/2, 2/2/3, 2/2/4, 2/2/5
    up ports: 2/2/6, 2/2/7, 2/2/8, 2/2/9, 2/2/10

TL;DR: Ignore the Brötcom docs, they're not complete...

 

[Propaganda]

The guide for the icx7250 provides some old firmware, is there any reason not to use the latest version on this site?
RUCKUS ICX 7250 Switch Hardware Installation Guide | Technical Documents | Ruckus Wireless Support

 

[ClintE]

ICX6610-48P arrived today from eBay. Followed the docs and it worked perfectly. It's all up and updated and connected. Thank you @fohdeesha for the guide. No way I would have ever figured out how to get this thing going on my own.

So far the only problem is the fan in one of the two supplies is much louder than the other. They're both Rev. A but I think it's actually just the fan going out or a bad bearing or something. I guess I'll just have to learn to live with it as the replacement supplies are incredibly expensive compared to the whole switch with two supplies and fan modules.

I'm still waiting for a few cards to arrive to try out 10/40gbe but for now it's working nicely at 1g.

Not sure, but I don't think you need two PSU's in any of the 6610 models; they're for redundancy and don't split the load. If I'm wrong about this, someone please correct me.