Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[mrizzo]

And a few additional questions regarding my specific configuration. What we need is very simple - we are using this switch to A) terminate a GRE tunnel and B) have uplink failover in place from the two fiber drops that datacenter gives us.

Looking at the Ruckus guide, it looks like a dynamic LAG is what I need? Is there anything specific I need to do for it to failover automatically or is that inherent to a dynamic LAG?

Secondly, regarding the GRE tunnel, I came across this in the Ruckus guide:

Layer 3 requirements:
The LAG is rejected if any of the secondary LAG port has any Layer 3 configurations, such as IPv4 or IPv6 address, OSPF, RIP, RIPng, IS-IS, and so on.

I will be terminating a GRE tunnel on the switch - does that mean I should only terminate a tunnel on the primary LAG port? What if the primary LAG port goes down? Will the secondary LAG port still work with the tunnel?

 

[Jaspah]

I'm sure its the Mellanox - I ordered QSFP+ cables made for Mellanox / Cisco and they would only work at 10Gbps - once reprogrammed to Brocade / Brocade they showed up as 40Gbps even though I used a Cisco n3k switch

Weird. I have a MCX354A-FCBT and have used a 1M Mellanox QSFP+ DAC and a 5M HPE "BladeSystem" QSFP+ DAC without any issues on my 6610. It doesn't seem to be picky at all.

 

[tubs-ffm]

I am locked out from my ICX 7250.

I uploaded my config file to the ICX via TFTP. This before I downloaded and updated a little bit.
But after reload I cannot login any more to the console and ssh. The password is not accepted.

Did I missed something to do before uploading the file?
Any easy why to reset the password without starting from scratch?

 

[tubs-ffm]

I am locked out from my ICX 7250.

I uploaded my config file to the ICX via TFTP. This before I downloaded and updated a little bit.
But after reload I cannot login any more to the console and ssh. The password is not accepted.

Did I missed something to do before uploading the file?

Yes, I was stupid, but I found the answer by myself.

When I copy the the config from console with show run, the password is not included for security reasons. Only five dots are shown. And when I upload this config again, guess what? My new password is ......

!
telnet timeout 10
no telnet server
username admin password .....
!
!

 

[texteditor]

ICX7150-C12P here, what's the correct way forward if flashing to bootrom isn't an option (FIPS isn't enabled, seems to be the layer 2 package right now,

ICX7150-C12 Switch#copy tftp flash 192.168.1.30 mnz10114.bin
  client-certificate       client RSA certificate
  client-private-key       client RSA private key
  fips-primary-sig         Primary signature file
  fips-secondary-sig       Secondary signature file
  fips-ufi-primary-sig     Primary ufi signature file
  fips-ufi-secondary-sig   Secondary ufi signature file
  local-pri                Primary code image on the local unit
  local-sec                Secondary code image on the local unit
  pe-id-pri                Copy PE primary image to unit specified by unit-id
  pe-id-sec                Copy PE secondary image to unit specified by unit-id
  primary                  Primary code image
  secondary                Secondary code image
  trust-certificate        SSL Trust certificate

Looks like the seller was kind enough to bump me up to the newer version :/

ICX7150-C12 Switch#show version
  Copyright (c) Ruckus Networks, Inc. All rights reserved.
    UNIT 1: compiled on Sep 22 2019 at 23:54:26 labeled as SPS08090d
      (28660224 bytes) from Primary SPS08090d.bin (UFI)
        SW: Version 08.0.90dT211
      Compressed Primary Boot Code size = 786944, Version:10.1.15T225 (mnz10115)
       Compiled on Thu Jan 31 07:08:55 2019

  HW: Stackable ICX7150-C12-POE
==========================================================================
UNIT 1: SL 1: ICX7150-C12-2X1G POE 12-port Management Module
      Serial  #:FEK3833R0RC
      Software Package: BASE_SOFT_PACKAGE   
      Current License: 2X1G 
      P-ASIC  0: type B160, rev 11  Chip BCM56160_B0
==========================================================================
UNIT 1: SL 2: ICX7150-2X1GC 2-port 2G Module
==========================================================================
UNIT 1: SL 3: ICX7150-2X10GF 2-port 20G Module
==========================================================================
 1000 MHz ARM processor ARMv7 88 MHz bus
 8192 KB boot flash memory
 2048 MB code flash memory
 1024 MB DRAM

 

[texteditor]

Looking further into it, the secondary flash seems to at least be an SPR package

Attempting downgrade via USB image auto-loading, we'll see

 

[Hakujou]

I played a little bit with PSU and fans of ICX6610 yesterday, did some interesting findings.

Went to PSU rev A and one fan in intake to PSU rev C exhaust with one fan, then two.

Exhaust PSU seems to be MUCH more silent than intake but will require two back fans modules if you want decent temperature. Those tests are made in my garage in winter, where ambiant temp is about 10C (so pretty low).
(The bump in the graph is moving from PSU intake to PSU exhaust w/ one fan, then it goes back to normal once I add a second fan module).

Single PSU Rev.C in exhaust mode is SO MUCH MORE quiet than single PSU Rev.A intake, you would not believe it. It used to litterally scream (high pitch noise, very loud) to being inaudible over servers and Fortigate around it.



To sum it up:

1. Single PSU Rev.A intake + one fan = ~45C, VERY LOUD
2. Single PSU Rev.C exhaust + one fan = ~71C, quiet
3. Single PSU Rev.C exhaust + two fans = ~46C, quiet

I would have loved to test Rev.A vs Rev.C in same configuration (intake/exhaust) as I heard intake vs exhaust makes a lot of difference in noise levels.

 

[mwarps]

For another data point for those trying to make things a little quieter, I threw a Delta EFB0412VHD-F00 into my ICX7250-24 and it worked well. Switch boots fine, and it's significantly quieter than the stock Foxconn monstrosity.

EDIT: After bringing the switch back up, the fan was not sufficient to keep the ASIC temps in the low-range fan speed. More work is required.

 

[Ken Jacques]

For another data point for those trying to make things a little quieter, I threw a Delta EFB0412VHD-F00 into my ICX7250-24 and it worked well. Switch boots fine, and it's significantly quieter than the stock Foxconn monstrosity.

Curious on the decibel level with your replacement?

 

[mwarps]

Curious on the decibel level with your replacement?

Sadly I have no scientific method to measure this. My "butt dyno" says it's much, much quieter.

 

[Ken Jacques]

LOL, you could try using a decibel meeter app on your phone.

 

[Rand__]

I just received my ICX6610. I updated the firmware per the guide linked in the OP. However, I am confused about the part where I switch from the management port to a normal port to be able to SSH to the switch. Isn't that the point of the management port?

I just want to make sure I am doing things correctly before I rack the switch at the datacenter. My plan was to use our OOB link to connect to the management port to be able to SSH. Do I need to connect it to a normal port instead? And if so, what's the point of the management port?

As I learned the hard way - the management port 'reuses' a MAC of one of the regular switches ports (see https://forums.servethehome.com/ind...erful-10gbe-40gbe-switching.21107/post-289899).
So when those two end up on the same network (as is likely in a home setup) then there are collisions.

If you have a completely separate circuit for OOB management then it should be fine.

 

[Zalouma]

Big fan of the ICX, currently I have some FastIron FCX 648S POE, I know these does BGP with advanced license package but super expensive in my opinion to obtain aside that they require site id and alot of complication these days, all the ones i own come with the basic package, any suggestion on getting the license cheap?!

I ended up going to Arbua for this option however would love to use these as well for BGP if there is a cheaper way for the license

 

[tubs-ffm]

For another data point for those trying to make things a little quieter, I threw a Delta EFB0412VHD-F00 into my ICX7250-24 and it worked well. Switch boots fine, and it's significantly quieter than the stock Foxconn monstrosity.

I am curious. What about AISIC temperatures in your ICX 7250-24?

I did the same on my ICX 7250-24P. This device even has two fans. But the two Delta EFB0412VHD-F00 were not sufficient to keep the ASIC temperature down and frequently step 2 of the fan control jumped in. I had to add an additional fan on top of the cooling fins for the ASIC.

To follow-up on this topic.

Additionally, to the two Delta EFB0412VHD-F00 in the chassis today I installed the Sunon MF60101V3-1000U-A99 on top of the ASIC. I connected the Sunon in parallel to one of the chassis fans so it also will slow down in fan mode 1. After running a couple of hours in idle mode at room temperature I get these stable temperatures. Look OK to me. Unfortunately, I did not note the temperatures in the original setup with Foxconn fans.

Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 62.4 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 42.5 deg-C

Just in case someone is asking. I am not planning to use heavy PoE load. Two devices only.

 

[mrizzo]

As I learned the hard way - the management port 'reuses' a MAC of one of the regular switches ports (see https://forums.servethehome.com/ind...erful-10gbe-40gbe-switching.21107/post-289899).
So when those two end up on the same network (as is likely in a home setup) then there are collisions.

If you have a completely separate circuit for OOB management then it should be fine.

Thank you!

 

[tubs-ffm]

As I learned the hard way - the management port 'reuses' a MAC of one of the regular switches ports (see https://forums.servethehome.com/ind...erful-10gbe-40gbe-switching.21107/post-289899).
So when those two end up on the same network (as is likely in a home setup) then there are collisions.

If you have a completely separate circuit for OOB management then it should be fine.

Or using a virtual interface. This is what I found in the manual:

NOTE
All physical IP interfaces on Ruckus FastIron Layer 3 devices share the same MAC address. For this reason, if more than
one connection is made between two devices, one of which is a Ruckus FastIron Layer 3 device, Ruckus recommends
the use of virtual interfaces. It is not recommended to connect two or more physical IP interfaces between two routers.

 

[mwarps]

I am curious. What about AISIC temperatures in your ICX 7250-24?

I did the same on my ICX 7250-24P. This device even has two fans. But the two Delta EFB0412VHD-F00 were not sufficient to keep the ASIC temperature down and frequently step 2 of the fan control jumped in. I had to add an additional fan on top of the cooling fins for the ASIC.

So after getting the switch back up and running (very much in testing mode here).. The fan was not sufficient to keep the ASIC temps in low-range territory. I'm going to either 3d print a duct to work with a few extra 12V low noise fans from the side vent, or I'm going to have to put something directly on the ASIC.

 

[tubs-ffm]

So after getting the switch back up and running (very much in testing mode here).. The fan was not sufficient to keep the ASIC temps in low-range territory. I'm going to either 3d print a duct to work with a few extra 12V low noise fans from the side vent, or I'm going to have to put something directly on the ASIC.

With a low-profile fan on top of the ASIC my temperature is fine. The Sunon MF60101V3-1000U-A99 is 10 mm in high. I fixed it with a simple elastic band to the cooler. I connected it in parallel to one of the chassis fans. So, it also will at reduced voltage. My backup plan was to go to permanent 12 V. Sunon MF60101V1-1000U-A99 would be another option I red here in this thread.

My concern is the PSU temperature. My unit is with PoE. Im am not planning to use a lot of PoE power. But even in idle mode the PSU of this device needs more cooling. I am not far away from fan level 2 jump in temperature. Now it is winter. In summer I guess this will not work.

Cutting out the metal in front of the outlet fans is a recommendation for better air flow and for lower noise. But as long I am not yet sure if I will keep my device, I do not want to do anything I cannot revert.

 

[mrizzo]

So I have a question about redundancy. I picked up a second ICX6610 and have two fiber links run to my cabinet. My plan was to set them up as follows:

Switch A with uplink A connected to each server's eth0 interface
Switch B with uplink B connected to each server's eth1 interface

Is there any way to set this up so that if uplink A fails, switch A can use uplink B? Would a stacking configuration accomplish that?

 

[ArmedAviator]

@mrizzo ,

This is possible with a stacking configuration. Stack the switches, ideally with 2x 40Gbit DACs in the rear ports and set up a LAG as usual with the appropriate ports.