Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[richtj99]

Stacking and a simple LAG trunk are two very different things. Also, be aware that in a LAG while you'd have 30Gb of aggregate bandwidth, you'll see single stream speeds no faster than 10Gb. I am unsure how bandwidth works over stacking, having never deployed it; I suspect it's a LAG underneath along with extra settings for the various coordination required for stacking and failover.

Thats interesting - it might be worth playing around & testing - is there a command / way to speed test a 'theoretical' transfer between two lagged switches (which I have setup already)?

I have two 7250's I can setup as a standalone stack to test - again assuming I can use a command to do a speed test.

For ICX 7k series, the default is hitless stacking; and whichever unit has highest stack priority is the master; upon failure, next highest takes over; see Traditional stack unit priority (ruckuswireless.com)

Thats interesting - does this mean if I have 3 switches,

stack ID #1 (priority 100)
stack ID #2 (priority 50)
stack ID #3 (priority 10)

If #1 dies, does that mean #2 is the new master and reprograms the other switches?

If Stack ID #4 (priority 5) is added, will it be auto reprogrammed by #2?

If so, if I change Stack ID #4 to Priority 100 - will it be the new master & take all its existing info & send it down stream?

I am equating it to a raid rebuild (sort of).

 

[LodeRunner]

Thats interesting - it might be worth playing around & testing - is there a command / way to speed test a 'theoretical' transfer between two lagged switches (which I have setup already)?

I have two 7250's I can setup as a standalone stack to test - again assuming I can use a command to do a speed test.




Thats interesting - does this mean if I have 3 switches,

stack ID #1 (priority 100)
stack ID #2 (priority 50)
stack ID #3 (priority 10)

If #1 dies, does that mean #2 is the new master and reprograms the other switches?

If Stack ID #4 (priority 5) is added, will it be auto reprogrammed by #2?

If so, if I change Stack ID #4 to Priority 100 - will it be the new master & take all its existing info & send it down stream?

I am equating it to a raid rebuild (sort of).

For speed testing, iPerf is the usual go to. But there's nothing theoretical about what I said regarding LAGs. If you have 3x 10 Gb links between switches, you'd need 3x 10 Gb streams on both ends to saturate it. If you had a 40 Gbe adapter on each end and a 3x 10Gb LAG on the switches between, absent any multithreading or intelligence, your single flow speed will be limited to 10 Gb between switches.

Stack configuration: once stacked, the switches have a unified configuration. If you change the stack priority, the only config change that happens is the stack priority. In a stack each switch does NOT have its own unique config. The management IP will move to whichever switch is the stack master. When you add a switch to a stack, its local configuration is removed and replaced with the stack config. So adding S4 then changing priority to get it promoted to master doesn't do anything configuration-wise as S4 will have loaded the stack config upon joining.

 

[seany]

you can try a different 40gb port on the 6610, but that points to a media error, as in bad optic on either side, or a bad cable/connection/dirty fiber ends. how long is the fiber?

30m. I'd be surprised if there were major issues with all new parts from FS. I'll track down my cleaning supplies and go from there I guess.

 

[Jason Antes]

30m. I'd be surprised if there were major issues with all new parts from FS. I'll track down my cleaning supplies and go from there I guess.

If this were FC I'd tell you to look at the Tx/Rx power, don't know if that's possible with these switches. Look at errors on the port and see if you have CRC, discards, etc and maybe you'll find what's going on. The type of error should point you in a direction to look for issues. Low power on an SFP usually means the SFP is bad. CRC is usually physical connection. Discards being config issues. etc. Seeing all the CRC I'd look at the cable possibly being an issue. Can you see light at both ends? (hint: use a mirror or a piece of paper in a darkened room, don't hold it up to your eye)

 

[up-n-atom]

Who wants to live dangerously?

I gave the FI reversing roughly a 2 year hiatus, but today my ADD got the better of me and FI nostalgia took precedence over work. 2.5Gbps was still on my agenda despite the interfering hardware. A big FU to high quality engineering and the DS100DF410 tucked inside!

I couldn't bother to dust off my Ghidra script FISeek - Pastebin.com to seek hidden commands in newer firmware, nor determine what disassembled firmware version was before me... It's dated and the hardcoded addresses within the script will not work on other versions but who cares (FYI It was 8080b and @fohdeesha has the list buried in their guide if you're interested).

5-10 minutes into it, I had what I needed, Broadcom Shell access! Don't know why it evaded me last time, since it is deadly obvious. I knew the command was tor but I was oblivious to enabling it. Stare at the disassembly for too long and you lose sight of things.

Now that I have the holy grail of switch privileged access, it's a no brainer to try out 2.5G. Guess what? It doesn't work outside of the switch ASIC Obvious, I know... Maybe the 7150 lineup isn't up to the engineering prowess of the 7250? I can only hope it's equal to that of Ubiquiti hardware Someone, please run dm i2c or take pictures of your 7150 PCB.

FYI You can replace dm with dd for the commands below.

Anyway, to enable access to Broadcom Shell, run dm device-command tor or the cheeky dm device-command 2762233 (It spells out BROCADE on a touch tone phone).

DANGER!!! YOU CAN ABSOLUTELY BRICK OR DAMAGE YOUR SWITCH! TREAD CAREFULLY!

Run tor help and get intimate with your switch.

For extra help you can view code at src/appl/diag · bcm-6.5.7_yocto-2.5.1 · Kosta Zertsekel / bcm-sdk

Eg. My 2.5G adventure (Yes, the WarpCore phys support 2500) [Internet] Bypassing the HH3K up to 2.5Gbps using a BCM57810S NIC - Bell Canada | DSLReports Forums

phy 1/2/1
Port xe0 (PHY addr 0xc1): WC-B1/13/0 (Warpcore PHY Driver)
        0x00: 0x0000    0x01: 0x0000    0x02: 0x0000    0x03: 0x0000
        0x04: 0x0000    0x05: 0x0000    0x06: 0x0000    0x07: 0x0000
...
tor phy info
  xe0( 50)   143  bff0    c1    c1              WC-B1/13/0     250000
  xe1( 51)   143  bff0    c1    c1              WC-B1/13/1     250000
  xe2( 52)   143  bff0    c1    c1              WC-B1/13/2     250000
...

enable
dm device-command tor
tor port xe2 speed=2500
tor port xe2
ping 192.168.1.10

:(

If someone discovers something of interest please do share because I've only played for a few hours.

 

[supacupa]

Well crap. I got some AT24C164's to use to unlock my ICX6610, but upon opening it up it appears there's no place to install the EEPROM. How do I unlock the ICX6610 so I can use the 10 gbit ports?

It looks like I may have incorrectly assumed this applied to the ICX6610 as well: EEPROM Unlocks - Fohdeesha Docs

Sidenote: It looks like I have 10x AT24C164's I don't need. I'm in the US is anyone needs one quickly. Just DM me for more info.

It looks like a software license might be my only option for this device.

 

[cesmith9999]

It looks like a software license might be my only option for this device.

@supacupa re-read the first page about the license.

Chris

 

[fohdeesha]

Well crap. I got some AT24C164's to use to unlock my ICX6610, but upon opening it up it appears there's no place to install the EEPROM. How do I unlock the ICX6610 so I can use the 10 gbit ports?

It looks like I may have incorrectly assumed this applied to the ICX6610 as well: EEPROM Unlocks - Fohdeesha Docs

Sidenote: It looks like I have 10x AT24C164's I don't need. I'm in the US is anyone needs one quickly. Just DM me for more info.

It looks like a software license might be my only option for this device.

Second sentence in the entire thread

 

[supacupa]

@supacupa re-read the first page about the license.

Second sentence in the entire thread

Yup, PM sent. I thought this needed a firmware unlock, originally. Thanks!

 

[supacupa]

Works perfectly! Supa easy to do. Thanks!

 

[ozkan]

Hi all, I need a small help. I have Brocade ICX 7250-48 without POE (non-POE) . Basically single PSU without FAN. Running 8030u firmware. But when I upgrade 8080 or 8070 I got error at boot;

FATAL ERROR: At least one good PSU with working Fan should be present in the system
System is shutting down!!!

I can only boot 8030x; here is the sh chasis output;

ICX7250-48 Switch#sh chassis
The stack unit 1 chassis info:

Power supply 1 present, status failed
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 56.7 deg-C

Fan speed switching temperature thresholds:
Speed 1: NM<----->98 deg-C
Speed 2: 67<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction: Front to Back
Fan 2 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 57.4 deg-C (Sensor 1)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-C
Boot Prom MAC : 609c.9f4b.a8a0
Management MAC: 609c.9f4b.a8a0

 

[tozmo]

Well, it requires a fan (not the one in the PSU) in order to fully boot. You can do some command line stuff, but to actually run the switch it requires a fan module. I'll sell you mine once I finalize my fan mod that DodgyRoute is helping me concoct.
Edit: Had 7450 stuck in my head, Kylie Minogue style

 

[fohdeesha]

the 7250 doesn't have fan or PSU modules, they're all built in. Sounds like @ozkan your fan is failed or not being detected properly, on the integrated units without hotswap stuff it will report that as a generic PSU failure sometimes. Check that the fan is spinning

 

[Freebsd1976]

new 7250-24 run hot in summer (ambient temperature is 32C) , cut some chopstick to attach mb6010v3-000c-a99 and fin , temperature drop from 82C to 72C. not so good due to mb6010v3-000c-a99 run slow at 4.5v when fan on speed 1.



edit:
make fan contact fin closely , now temp drop to 69C

 

[ozkan]

the 7250 doesn't have fan or PSU modules, they're all built in. Sounds like @ozkan your fan is failed or not being detected properly, on the integrated units without hotswap stuff it will report that as a generic PSU failure sometimes. Check that the fan is spinning

one of the fans was not spinning before upgrade, I cleaned it with sprey. After cleaning it started spinning, then I booted the switch and upgraded. Also checked the fan, speed is normal. It was FAN1. I’ll try replacing the fan.

 

[richtj99]

Hi - still playing with the 2x 7250 units.

I have stacking working with two 7250's & I like being able to control both from 'one' interface. I think that sounds good but I hate losing a 10gb port just for the 'management' end of things.

Can I stack using a 1gb port?

I would prefer a 80gb lag vs a 10gb stack port & a 70gb lag (OCD portion for me).

Is the 'whole' 10gb port used up in the stack? Does it send any data or just management linking?

 

[Drewy]

@richtj99
you don’t need a lag between switches that are stacked. Stacking makes them 1 logical switch. All vlans etc are available on both switches. The stack link\connection handles all management and inter switch traffic. You don’t have to run additional lags between the switches, the stack does it all.

 

[SuperMiguel]

Does stacking only work with certain port? Certain cable?

I have two 6610 want to use the 10g for stacking

 

[ozkan]

the 7250 doesn't have fan or PSU modules, they're all built in. Sounds like @ozkan your fan is failed or not being detected properly, on the integrated units without hotswap stuff it will report that as a generic PSU failure sometimes. Check that the fan is spinning

I think I found the problem, but have no idea how to solve. There is small warning when I boot the switch with 8030u;

ICX7250-48 Switch>Successfuly set IngressClassPort on CPU port to 2

PS 1, Internal Power supply detected with no power.

ICX7250-48 Switch>en

Is there any idea what causes this ?

Edit :

I've got my hands dirty and checked every point that measurable. PSU output seems good;

EMPTY
Yellow : 12.01V
Yellow : 12.01V
EMPTY
Black : GND
Black : GND

There seems 2x 12V feeding DC converter board.

12V DC connector on switch board;

1. Orange : 3.4V 6. Orange : 3.4V
2. Brown : 3.4V 7. Brown : 3.4V
3. Green : 3.4V 8. Yellow : 3.4V
4. Black : GND 9. Yellow : 11.7V
5. Black : GND 10. Yellow: 11.7V

Fan pins also read;
11.67V

PSU seems OK, but I powered the switch on, PWR led is orange and not green. I think I need to find 2x 12V DC PSU to test if inline PSU is the problem.

 

[CorvetteGS]

Haven't had a chance to re-write the guide with new instructions for getting to the recommended 8092 train - it's the new UFI images that require a bit more steps for switches with older firmware. Been working on it this week though.

Hey @fohdeesha is this going to happen in the near future? Just picked up a new switch and I'm debating on waiting for the revised guide before messing with it or if I should just use the guide as it stands now and do it all over when you update it with the latest firmware. Thanks for your hard work!