Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

HomelabN0rth

New Member
Jan 30, 2019
3
2
3
damn, I was hoping to extract those development images before wiping them :p
Oh sorry, I was in TERMINALDAD> mode - I wasn't going to let it "win", ya know? I have to admit, the documentation from Rukus is what saved the day. It wasn't all where I was looking for it but some Google-fu with the error messages and commands was enough to bring up pages where they posted how to do each individual part, even if each part was on a separate document LOL
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
1,945
1,777
113
29
fohdeesha.com
:p I will say you got lucky, half of the "development" switches I have seen with these bootloaders have an "update_boot" routine that lays the bootloader file down unmodified at the beginning of flash (which worked for whatever development bootloaders they were using). However the "production" bootloader files that we all have have around 30 lines or so of metadata at the beginning that needs to be stripped off first, if you flash those using the update_boot of a dev bootloader and it lays it down raw, it'll brick it. It sounds like yours already had a production bootloader though, that just happened to be in diag mode with a bunch of strange boot environment variables set for the broadcom image (similar to @britinpdx ). I almost want to ask g-electronic where he is getting these because they keep coming with the strangest stuff I've seen
 
Last edited:
  • Like
Reactions: HomelabN0rth

fohdeesha

Kaini Industries
Nov 20, 2016
1,945
1,777
113
29
fohdeesha.com
I checked their source and it appears in the patch file for the 6650 and 7250 but not the 6450 or 7650.
Nice find. after skimming through the patch files some more it turns out the secret mode is still there, they just renamed the routine and environment variable.

for the ICX6450:
(it also requires setting of pp_init for this mode to occur)

setenv pp_init_enable 1
setenv diag_cmd_on 1
saveenv
reset

##go back to normal
setenv pp_init_enable
setenv diag_cmd_on
saveenv
reset


For the ICX7650:

setenv diag_mode_on 1
saveenv
reset

##go back to normal
setenv diag_mode_on
saveenv
reset
 
  • Like
Reactions: sean

Skippy96

New Member
May 14, 2019
19
6
3
Is there any way to leverage all the access we have with these dev builds to modify the firmware to allow for fan modifications?
My 6610 is getting too loud for my office :)
 

sean

Member
Sep 26, 2013
64
30
18
CT
Nice find. after skimming through the patch files some more it turns out the secret mode is still there, they just renamed the routine and environment variable.
Nice. I checked the patch files but they're big and I missed anything obvious. I now see the pp_init_enable and hilariously a comment about hiding the diag commands.

Is there any way to leverage all the access we have with these dev builds to modify the firmware to allow for fan modifications?
I defer to fohdeesha but for me, the other models using u-boot and Linux means Arris has to release their modifications. It's then "easy" (the u-boot patch is 400k lines) to go through their changes and find stuff they've hidden. In the case of Linux, it's easier to take their firmware images and open them to see the surrounding environment and how it boots. Actual hardware control however is still limited to FastIron and their kernel module. Those are proprietary and much harder to inspect, much less modify.

The 6610 has none of this however. Someone else was working on a way to lie about the fan RPMs and I think that will be the quickest/only way to modify the fans.
 

spali

Member
Nov 4, 2018
32
3
8
I’d take 2 of the 40 gig ports on each 6610 and make them a stack
Then I’d take the other 2 ports off the 6610s and make an mlag to your arista. That way you get failover stack connections.
I'm new to mlag, but aren't the links then limited to 8x 10g per switch?
Thought a second time about it.
10g downlinks wouldn't be a bottleneck, due i will not serve higher bandwidth from the 6610.
Can i use a standard QSFP DAC in the breakout ports to uplink to an arista QSFP port just set to breakout mode? Kind of breakout to breakout?

Second question... My two 6110's I ordered each a single Rev. A PSU ... I have read the whole thread now and because they seem not to spin down when in "single mode" and because the intake version RPS16-I is really rare to find. I would like to ask if anyone which has a lot of equipment could do a test for me.
I'm thinking if it could work to put as second a non POE together with a POE PSU in each switch to let them spin down. If this would work I would just not have the redundancy in POE?
Hope explained it halfway understandable :rolleyes:
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,945
1,777
113
29
fohdeesha.com
Is there any way to leverage all the access we have with these dev builds to modify the firmware to allow for fan modifications?
My 6610 is getting too loud for my office :)

like Sean said, on either the linux based models, or the non-linux based models, both of them use a proprietary module/binary to control the fans once booted, so nothing we can do there. you can control them all you want from the bootloader, but as soon as the fastiron app boots it takes over chassis control
 

Emanuele

New Member
Mar 1, 2019
16
1
3
Hi all! Thank to @kiteboarder I landed to this thread, @fohdeesha great work here :) ,and after a couple of pages I decided to go all in for 2x ICX 6610-24 (non poe version).
@fohdeesha great work
One of them arrived without rack ears, I read here where to find a set (they cost a lot BTW).
The problem has presented the moment I turned it on.
It has INTAKE FAN and EXHAUST PSU, so it is returning a critical error and reboot.
IMG_20190608_130515.jpg
Again and again obviously.
IMG_20190608_125719.jpg
So, I can't play with my new toy :(
I wrote down to the seller but I don't think he has lots of spare parts of this switch.

Does anyone here is willing to sell 4x INTAKE PSU to EU (Italy)? Maybe not too much €$£?
 

spali

Member
Nov 4, 2018
32
3
8
Does anyone here is willing to sell 4x INTAKE PSU to EU (Italy)? Maybe not too much €$£?
Searched a lot in the last days, they are almost not available in Europe for a descent price.
But if switching airflow to exhaust is an option, I would get new FAN's. There are currently available for 100$ inc. shipping to Europe on ebay.
 

Emanuele

New Member
Mar 1, 2019
16
1
3
:eek: I did pay 480€ inc. shipping per the 2x switch, I can't afford 100$ per a single fan module... it's way overpriced...
 

spali

Member
Nov 4, 2018
32
3
8
:eek: I did pay 480€ inc. shipping per the 2x switch, I can't afford 100$ per a single fan module... it's way overpriced...
I know, the prices in Europe are horrible and most times shipping from US is cheaper even the shipping costs are height. Just ordered two intake FANs from them, because my two 6610-48P-I that are on the way, have only one FAN and one PSU. PSU is also Rev. A :(
But was the best deal I found in the last time for 400$ inc shipping each. And I'm hopping to find some intake PSUs in the future. But mine work at least, but loud. But worst case would be for me to stack and just shut one of them down and put the PSU into the other to let the PSU fans spin down.
In your case it would be probably best to search for a good deal of any 6610 where you just take the fans out. Would be cheaper than buying them extra. Except someone in europe has some spare parts.
 

Emanuele

New Member
Mar 1, 2019
16
1
3
You are probably right, prices can't be helped. I'll start looking around for spare parts/broken 6610, if someone want to sell something, here I am :)
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
damn, I was hoping to extract those development images before wiping them :p
I mean, you could order a switch from g-electronic and dump them then return it as "faulty"

#JustSayin'

RE Rack ears: I think we've hit the point where modelling rack ears for the various switches would be a smart move so we can print our own and skip the $50-60 tax every time a switch ships without them. I'm eyeballing a couple of 3d printers at the moment, if no one else bites I'll do the work in the next couple of months if I continue to have free time like I do right now.

If anyone has experience doing simple static load modelling and can point me at some basic resources for designing durable parts I'd appreciate it.
 
Last edited:

Hakujou

New Member
Apr 28, 2019
19
6
3
I mean, you could order a switch from g-electronic and dump them then return it as "faulty"

#JustSayin'
While g-electronic has mostly bad switches and they probably just test them by checking they power on (to be fair, they don't claim otherwise), their customer service has always been honest and refunded everyone without trouble (they even let me kept my faulty switch while refunding me entirely).

That kind of behavior is just dishonest and could lead to a more troublesome customer service.
 
  • Like
Reactions: yukaia and Emanuele

ewer0012

Member
Feb 10, 2019
79
24
8
I managed to pick up an ICX6450 for $100 and an ICX7250 for $150 on Friday. Hoping these will do the trick so I can get away from MikroTik.

Thanks for sharing, OP.
 
  • Like
Reactions: FRPII

BeTeP

Well-Known Member
Mar 23, 2019
520
324
63
RE Rack ears: I think we've hit the point where modelling rack ears for the various switches would be a smart move so we can print our own and skip the $50-60 tax every time a switch ships without them.
Are we still talking about 2 small pieces of angled steel with a few holes? I just grab ears from any old piece of equipment and drill the missing mounting holes if any. If you don't have any laying around - Cisco 1U rack ears sell for like $5/pair shipped from China.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Are we still talking about 2 small pieces of angled steel with a few holes? I just grab ears from any old piece of equipment and drill the missing mounting holes if any. If you don't have any laying around - Cisco 1U rack ears sell for like $5/pair shipped from China.
Yeah, we are. Rack ears are one of those things that are ludicrously priced when you want the actual OEM part. Being able to 3d-print them out of like $10 worth of nylon would be kind of nice.
 

cuco

Member
Feb 13, 2018
51
2
8
35
Yeah, we are. Rack ears are one of those things that are ludicrously priced when you want the actual OEM part. Being able to 3d-print them out of like $10 worth of nylon would be kind of nice.
I would also appreciate it and go all in
 

nthu9280

Well-Known Member
Feb 3, 2016
1,588
441
83
San Antonio, TX
I don't have any experience with 3D printing but not sure if the material used for making the brackets has enough strength to withstand this application especially for 6610. The rear of the switch sags due to the weight. Imagine the twist excerted even on the OEM metal brackets. It may be ok for 6450.
 
Last edited: