Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[kapone]

@kapone Fairly certain that is not the case. One of these devices is my OSX laptop that I've used for a ton of debugging and pinging from, and the other is a OPNsense machine that is definitely responsive to pings because I've gotten it to respond to pings in the past. Also, the switch itself can ping all of these devices using the built-in ping tool.

That's...weird.

Routes ARE default ON for VLANs in the 6610, I've never seen it not be on by default. A few suggestions/things to try, possibly?

- Instead of VLAN1, move that device to a different VLAN, eliminating any "default VLAN" issues. Test
- Factory reset the 6610 and try again?
- Instead of a /30 on that VLAN, try /24 just for shits and giggles.

 

[Phlesh]

@kapone I'll try anything at this point!

I've got a VLAN 16 and a VLAN 32 now. Same setup -- x.x.[32/16].10 IP's for two different devices, one on each subnet. Neither can ping the other at this moment.

Anything else I can check? Is there a way I could have inadvertently disabled inter-VLAN routing?

 

[Phlesh]

Another question, might help me debug: is there any way from the serial port to watch some of the packets going to/from a given IP, and provide some info on what switching decisions are being made for each packet?

 

[kapone]

Do a "sh run" and paste the output.

 

[fohdeesha]

have just picked up a couple of icx6610s and I am trying to get the console serial port set to 115200, and what I did for the 6650 does not work.

Setting this in the u-boot on the 6650 I used

On the 6610s there is nothing. I read that others just used buadrate, as they are not u-boot

Where does one change the baud rate for the serial console on these?

Thanks in advance.

setenv is for the u-boot based switches, the 6610 runs brocades custom bootloader not based on u-boot. I can't remember if I ever found it on those. I thought I remember it being something under the "set" command in the bootloader but not seeing it there right off the bat, could be hidden

 

[fohdeesha]

updating some stuff for a colo customer I set up ages ago and think I found a record for their setup, ICX6610 with zero issues:

STACKID 1  system uptime is 1102 day(s) 18 hour(s) 55 minute(s) 33 second(s)
The system started at 02:03:04 GMT-05 Sat Jan 26 2019

 

[tinfoil3d]

There's no ECC onboard right? I mean, a lot of things can happen in 3 years but since it only has something like 64 or 128mb chances of bitflip in a critical area isnt that high either i guess

 

[fohdeesha]

There's no ECC onboard right? I mean, a lot of things can happen in 3 years but since it only has something like 64 or 128mb chances of bitflip in a critical area isnt that high either i guess

the memory in these switches is ECC, 512MB PC2-5300 Reg ECC specifically in the case of the 6610 - with 64 or 128mb they wouldn't even be able to uncompress the OS image into RAM during boot

 

[tinfoil3d]

that explains a lot. but to tell you the truth i have raspi and even desktop computers running for way longer than that and still doing fine. no ecc whatsoever

 

[linuxsrc]

setenv is for the u-boot based switches, the 6610 runs brocades custom bootloader not based on u-boot. I can't remember if I ever found it on those. I thought I remember it being something under the "set" command in the bootloader but not seeing it there right off the bat, could be hidden

Monitor>set ?
debug-task Debug task
sample-rate Sample task
sample-task Sample task
ether-address Configure Ethernet address

Nothing here that seems to make you think you can change the rate
This is why I am confused, I tried this with expecting it to be similar to what the 7xxx versions did.

Monitor>set baudrate 115200
Syntax error, 'baudrate'

Thinking like you, I tried it just in case it was hidden.

Strange how, nothing points to baud rate being able to be changed, even in the official docs. Brocade seems to never mention it, and I have spent a bit of time combing through them to find any mention of baud rate outside, use 9600 blah blah blah, to connect to the console.

 

[fohdeesha]

Nothing here that seems to make you think you can change the rate
This is why I am confused, I tried this with expecting it to be similar to what the 7xxx versions did.



Thinking like you, I tried it just in case it was hidden.

Strange how, nothing points to baud rate being able to be changed, even in the official docs. Brocade seems to never mention it, and I have spent a bit of time combing through them to find any mention of baud rate outside, use 9600 blah blah blah, to connect to the console.

it's entirely possible it's hardcoded to 9600 baud on the old powerPC custom bootloaders, they're pretty basic. Is there a specific reason you need to increase it?

 

[linuxsrc]

it's entirely possible it's hardcoded to 9600 baud on the old powerPC custom bootloaders, they're pretty basic. Is there a specific reason you need to increase it?

Just for constancy, all of my hardware runs at 115200, and I hate the slow interface as well. It is not the end of the world.
Also, it is one of those, I want it to work this way why can't it kind of thinking.

 

[klui]

Maybe when you perform file transfers, but can you actually tell the difference between 9600 and 115200 interactively? I can't. Besides if you have connection managers or console servers, the configurations are managed anyway.

 

[RobstarUSA]

Does anyone know if this will work @ 2.5Gbit/s in the 6610?

 

[victorhooi]

Thanks for the informative (and very long...haha) thread with good info.

If we wanted a 48-port POE switch today, which would you pick between the 6610-48P and the 7250-48P?

The 6610 seems to have the advantage of the 4 x 40Gbe QSFP ports (although not really sure how popular 40Gbe is these days) and slightly cheaper second-hand.

However, the 7250 is non-EOL, and still getting software updates.

Are there any reasons (apart from cost) you'd still get the 6610 today?

 

[klui]

I'd only get the 6610 for its 40G ports if I don't have any other 40G-capable switch. After acquiring the 6610 I've obtained quite a few other switches that are 40G capable.

Be aware its 40G ports are hard configured: 2 are 40G, 2 are split, and they can't be changed. Their configuration isn't a problem for stacking but can be a hassle if you need a different topology for using them as access/trunks to non-6610 devices.

 

[linuxsrc]

Maybe when you perform file transfers, but can you actually tell the difference between 9600 and 115200 interactively? I can't. Besides if you have connection managers or console servers, the configurations are managed anyway.

Only if I had a console server or connection manager. That would make my life easier 15 console cables connected to a raspberry pi are a mess. (any recommendations?)

 

[klui]

You use Linux, based on your user name? You can associate a host with a comment in your command line history and search for the host name. If you're running Windows PuTTY is a perfectly fine way to use connection strings to a host.

I don't use a Pi to perform KVM functions so don't know how you launch your terminals. If you can parse the host name/device where you want to connect you can have some .rc file that will provide the correct settings. Lots of possibilities.

 

[vfxer]

I just want to know whether I can use 8x10G ports on Brocade ICX7250 for downstream other than uplinking and stacking. Can I use those 8 x 10G ports to connect my servers to them and have internal routing between servers?


Or can I use 2 x 10G for uplink to a router (e.g. Opnsese/PFsense) and other 6 x 10Gb connects to my 3 servers with LACP (for example: 2 x ports will be LACP’d and connect to a 10Gb optical adapter in one of my servers and so on)

 

[LodeRunner]

I just want to know whether I can use 8x10G ports on Brocade ICX7250 for downstream other than uplinking and stacking. Can I use those 8 x 10G ports to connect my servers to them and have internal routing between servers?


Or can I use 2 x 10G for uplink to a router (e.g. Opnsese/PFsense) and other 6 x 10Gb connects to my 3 servers with LACP (for example: 2 x ports will be LACP’d and connect to a 10Gb optical adapter in one of my servers and so on)

You can. They may be called uplinks, but they're just regular switch ports. No magic to them.