Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[jode]

Ideas ?

If I understand you correctly, you have two 6450 switches in your LAN and they are connected via a 10gb link.

To simplify configuration and keep configurations in sync between these two switches the easiest way is to "stack" them. Once stacked, they work in unison and when connected to one switch you have control over the other as well (e.g. configure ports of the other switch).

Assuming you downloaded the package from the first post look for "fastiron-08030b-switchstackingguide.pdf" for the documentation of this feature.

 

[logixworx]

The manual for Brocade 6430/6450 has instructions for wall mounting the 6430, but says that the 6450p cannot be wall mounted. Is there a specific reason why this is so? Weight? convection? I need to vertically mount my 6450p using a common 4u wall mounted rack that can hold a 150 lbs load. Will i have any issues?

 

[NablaSquaredG]

The manual for Brocade 6430/6450 has instructions for wall mounting the 6430, but says that the 6450p cannot be wall mounted. Is there a specific reason why this is so? Weight? convection? I need to vertically mount my 6450p using a common 4u wall mounted rack that can hold a 150 lbs load. Will i have any issues?

I don't think you'll have any issues. The rack mount angles are pretty sturdy and you don't really have any leverage when mounting vertically

 

[macboy80]

I have just received my first ICX 6450-48P. I'm fairly savvy with tinkering, however I cannot get the serial connection to come up. My worry is that something more sinister is afoot. When powering on, the fans come up, very loud like full speed, and never go down, however the only lights that come on are the 4 X1-X4 lights. They stay solid amber for more than 5 minutes, and the PWR / DIAG lights never come on. I do get blinking green link lights on the management port as well.

Does anyone have an idea? Apologies in advance if this has been covered. I can't figure out how to search through a almost 500 page thread.

 

[NablaSquaredG]

When powering on, the fans come up, very loud like full speed, and never go down,

At some point, they should spin down. Is it really "never" or how long have you waited?

however I cannot get the serial connection to come up.

Just a quick sanity check:
You are connecting your Cisco-Style Console cable to the top port on the left, not the bottom one?

 

[macboy80]

At some point, they should spin down. Is it really "never" or how long have you waited?

I left it running as I researched, first that the usb-serial drivers were updated and then as I tried to find information on the power light not coming up. It was certainly more than 5 mins and probably more than 10 mins.

Just a quick sanity check:
You are connecting your Cisco-Style Console cable to the top port on the left, not the bottom one?

Yes. The cable is the one that came with the unit from the seller. A grey, Foxconn cable with RJ-45 on both ends and plugged in to a DB9 connector on one side. It is the one that is seen in many for-sale listings on the web. I am using this SABRENT USB 2.0 to Serial (9 Pin) DB 9 RS 232 Converter Cable, Prolific Chipset with Putty.

I was hoping some of you may have seen this behavior and could offer some assurance before went down the path of learning how to test this in Linux.

What I am most unsure of is:
1. The lack of PWR / DIAG lights, with it seemingly hung on the X1-4 lights. (Brocade docs seem to say call support if you don't have a PWR light.)
2. The amount of time it should take to show serial output / control the fan speed. (On a PC, full fans are usually NO POST.)
3. What kind of weird Win11 / driver options may be interfering with serial cable. (In the advanced properties, I'm seeing fifo and modem options.)

 

[JD7]

I left it running as I researched, first that the usb-serial drivers were updated and then as I tried to find information on the power light not coming up. It was certainly more than 5 mins and probably more than 10 mins.


Yes. The cable is the one that came with the unit from the seller. A grey, Foxconn cable with RJ-45 on both ends and plugged in to a DB9 connector on one side. It is the one that is seen in many for-sale listings on the web. I am using this SABRENT USB 2.0 to Serial (9 Pin) DB 9 RS 232 Converter Cable, Prolific Chipset with Putty.

I was hoping some of you may have seen this behavior and could offer some assurance before went down the path of learning how to test this in Linux.

What I am most unsure of is:
1. The lack of PWR / DIAG lights, with it seemingly hung on the X1-4 lights. (Brocade docs seem to say call support if you don't have a PWR light.)
2. The amount of time it should take to show serial output / control the fan speed. (On a PC, full fans are usually NO POST.)
3. What kind of weird Win11 / driver options may be interfering with serial cable. (In the advanced properties, I'm seeing fifo and modem options.)

I can test it tomorrow when I get home but 5mins at the most for it to show on the screen however if you have never used that adaptor or cable to get into any other switches before it might be your drivers and cable I use a win10 machine to get into my stuff. if its not configured your not gonna see a lot of lights until you enable some stuff like POE and such or give it a live connection to your network, it can still be running full fans and post but should slow down after its finished.

 

[fohdeesha]

I left it running as I researched, first that the usb-serial drivers were updated and then as I tried to find information on the power light not coming up. It was certainly more than 5 mins and probably more than 10 mins.


Yes. The cable is the one that came with the unit from the seller. A grey, Foxconn cable with RJ-45 on both ends and plugged in to a DB9 connector on one side. It is the one that is seen in many for-sale listings on the web. I am using this SABRENT USB 2.0 to Serial (9 Pin) DB 9 RS 232 Converter Cable, Prolific Chipset with Putty.

I was hoping some of you may have seen this behavior and could offer some assurance before went down the path of learning how to test this in Linux.

What I am most unsure of is:
1. The lack of PWR / DIAG lights, with it seemingly hung on the X1-4 lights. (Brocade docs seem to say call support if you don't have a PWR light.)
2. The amount of time it should take to show serial output / control the fan speed. (On a PC, full fans are usually NO POST.)
3. What kind of weird Win11 / driver options may be interfering with serial cable. (In the advanced properties, I'm seeing fifo and modem options.)

yeah that sounds dead

 

[macboy80]

Thank you for the replies. I have a different usb to serial cable coming and am going to try with Linux as well. I will advise when I know more. In the meantime, if anyone has any experience with the boot up indicator light pattern, I'd love to know what it should look like it.

Update / Edit: I've now used a FTDI and Prolific serial cable on both Windows and Ubuntu. No serial output, the continued odd boot indications, and your advice have led me to return the switch as faulty.

Update 2: I bought another one, and everything worked just as expected per @fohdeesha excellent guide. FYI. While in the bootloader, the fans do spin at 100%, and the lights are just like my broken switch with no PWR / DIAG and X1-4 lit amber. All that changes as expected as soon as the boot process starts.

 

[Matt0925]

Full disclaimer, I'm very new to home networking, managed switches and especially level 3 switches. So I have no idea if this is normal behavior, I have something configured wrong or if my switch has an issue.

I purchased a 7250 to replace an Aruba 2500. Followed the guide to get it up and running, place it into service. Runs for a while, then start having disconnects across the network. Oddly enough PC shows still connected but no Internet. Then it'll completely disconnect. Running OPNSense for a router, at the same time it started having errors. I disconnected the 7250, completely restarted the router and errors went away. They seemed to be an IP config issue?

Tonight I tried to reintroduced the 7250. I set it to run as level 2 since I don't know anything about level 3. Ran fine about 45 mins. Then disconnects started again. I bypassed the 7250, started looking into what it could be, checking the log I did get this-

May 13 23:08:15:I:System: Interface ethernet 1/2/8, state down
May 13 23:06:49:I:System: Interface ethernet 1/1/22, state down
May 13 23:06:40:I:System: Interface ethernet 1/1/29, state down
May 13 23:06:31:I:System: Interface ethernet 1/2/5, state down
May 13 23:06:27:I:System: Interface ethernet 1/2/7, state down
May 13 22:19:08:I:System: Interface ethernet 1/2/5, state up
May 13 22:18:59:I:System: Interface ethernet 1/2/5, state down
May 13 22:16:55:I:NTP: The system clock is not synchronized to any time source.
May 13 22:10:01:I:System: Interface ethernet 1/2/7, state up
May 13 22:09:57:I:System: Interface ethernet 1/2/7, state down
May 13 22:09:32:I:System: Interface ethernet 1/2/7, state up
May 13 22:09:27:I:System: Interface ethernet 1/2/7, state down
May 13 22:09:18:I:System: Interface ethernet 1/2/7, state up
May 13 22:09:01:I:System: Interface ethernet 1/2/7, state down
May 13 22:09:01:I:System: Interface ethernet 1/2/7, state up
May 13 22:08:49:I:System: Interface ethernet 1/2/7, state down

So ports appear to be up and down for some reason. I'm going to assume this is not typical behavior. Not sure if I've misconfigured something or if it's a hardware issue? Otherwise I have no idea why I'm getting constant disconnects

 

[TonyArrr]

Full disclaimer, I'm very new to home networking, managed switches and especially level 3 switches. So I have no idea if this is normal behavior, I have something configured wrong or if my switch has an issue.

I purchased a 7250 to replace an Aruba 2500. Followed the guide to get it up and running, place it into service. Runs for a while, then start having disconnects across the network. Oddly enough PC shows still connected but no Internet. Then it'll completely disconnect. Running OPNSense for a router, at the same time it started having errors. I disconnected the 7250, completely restarted the router and errors went away. They seemed to be an IP config issue?

Tonight I tried to reintroduced the 7250. I set it to run as level 2 since I don't know anything about level 3. Ran fine about 45 mins. Then disconnects started again. I bypassed the 7250, started looking into what it could be, checking the log I did get this-

May 13 23:08:15:I:System: Interface ethernet 1/2/8, state down
May 13 23:06:49:I:System: Interface ethernet 1/1/22, state down
May 13 23:06:40:I:System: Interface ethernet 1/1/29, state down
May 13 23:06:31:I:System: Interface ethernet 1/2/5, state down
May 13 23:06:27:I:System: Interface ethernet 1/2/7, state down
May 13 22:19:08:I:System: Interface ethernet 1/2/5, state up
May 13 22:18:59:I:System: Interface ethernet 1/2/5, state down
May 13 22:16:55:I:NTP: The system clock is not synchronized to any time source.
May 13 22:10:01:I:System: Interface ethernet 1/2/7, state up
May 13 22:09:57:I:System: Interface ethernet 1/2/7, state down
May 13 22:09:32:I:System: Interface ethernet 1/2/7, state up
May 13 22:09:27:I:System: Interface ethernet 1/2/7, state down
May 13 22:09:18:I:System: Interface ethernet 1/2/7, state up
May 13 22:09:01:I:System: Interface ethernet 1/2/7, state down
May 13 22:09:01:I:System: Interface ethernet 1/2/7, state up
May 13 22:08:49:I:System: Interface ethernet 1/2/7, state down

So ports appear to be up and down for some reason. I'm going to assume this is not typical behavior. Not sure if I've misconfigured something or if it's a hardware issue? Otherwise I have no idea why I'm getting constant disconnects

Could be helpful to share your config from the switch so folks can look over it and point out anything that might cause problems.
What is connected to what port?

 

[Matt0925]

Could be helpful to share your config from the switch so folks can look over it and point out anything that might cause problems.
What is connected to what port?

Sorry, probably should have thought of that. Config-

SSH@brocade(config)#show config!Startup-config data location is flash memory - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

As for ports-

1/2/5- unRaid connected through Amazon DAC cable
1/2/7- OPNSense connected through ethernet using SPF+ to RJ45 transceiver
1/2/8- PC connected through ethernet using SPF+ to RJ45 transceiver

The one gig ports had a the backup unRaid connection, a UPS and a Nvidia Shield.

 

[fohdeesha]

Sorry, probably should have thought of that. Config-

SSH@brocade(config)#show config!Startup-config data location is flash memory - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

As for ports-

1/2/5- unRaid connected through Amazon DAC cable
1/2/7- OPNSense connected through ethernet using SPF+ to RJ45 transceiver
1/2/8- PC connected through ethernet using SPF+ to RJ45 transceiver

The one gig ports had a the backup unRaid connection, a UPS and a Nvidia Shield.

looks fine to me, altho the "ip route 192.168.1.0/24 192.168.1.1" statement is redundant as you already have a default pointing to the same IP. that much flapping on every single port really looks like a hardware fault, can you unplug one connection at a time and see if it goes away? this includes the whole SFP+ copper module, maybe one of them is bad and friggin things up

 

[Matt0925]

looks fine to me, altho the "ip route 192.168.1.0/24 192.168.1.1" statement is redundant as you already have a default pointing to the same IP. that much flapping on every single port really looks like a hardware fault, can you unplug one connection at a time and see if it goes away? this includes the whole SFP+ copper module, maybe one of them is bad and friggin things up

Thanks for the thoughts. I probably added the duplicate when I was trying to figure out why my connection was dropping, at first I figured I messed up setting things up so I was reentering some of it. So the line "ip route 0.0.0.0/0 192.168.1.1" accomplishes the same thing? I can remove the redundant one, though it doesn't sound like it's the cause of my issues?

Right now I just have it connected to the router through a dumb switch into one of the 1gig ports and the UPS plugged into it. Not showing a dropped connection on either of those since yesterday. Both transceivers are still plugged in, though nothing into them. I'll remove one, add my PC to the other and give it some time, see if that causes a problem, add unRaid and work my way up.

My other issue is I'm on a timeline, return window closes in just a couple days so I'm inclined to ship this one back and grab another off eBay instead. But I'll report back with any findings, thank you

Next morning edit-

Started with the PC last night, no drops for about an hour. Added the unRaid server through the DAC cable, no drops. I then moved PC to another port and the other transceiver, then not thinking put the PC to sleep so of course it disconnected. But started back up this morning and it's had no disconnects since. So at this point it does not seem to be the PC, unRaid, DAC cable or either transceivers causing issues.

Leaves two options, OPNSense or the backup unRaid connection. My guess is OPNSense, though not a clue what the cause could be. Regardless when I get a chance I'll connect it as well and see.

 

[fohdeesha]

Thanks for the thoughts. I probably added the duplicate when I was trying to figure out why my connection was dropping, at first I figured I messed up setting things up so I was reentering some of it. So the line "ip route 0.0.0.0/0 192.168.1.1" accomplishes the same thing? I can remove the redundant one, though it doesn't sound like it's the cause of my issues?

Right now I just have it connected to the router through a dumb switch into one of the 1gig ports and the UPS plugged into it. Not showing a dropped connection on either of those since yesterday. Both transceivers are still plugged in, though nothing into them. I'll remove one, add my PC to the other and give it some time, see if that causes a problem, add unRaid and work my way up.

My other issue is I'm on a timeline, return window closes in just a couple days so I'm inclined to ship this one back and grab another off eBay instead. But I'll report back with any findings, thank you

Next morning edit-

Started with the PC last night, no drops for about an hour. Added the unRaid server through the DAC cable, no drops. I then moved PC to another port and the other transceiver, then not thinking put the PC to sleep so of course it disconnected. But started back up this morning and it's had no disconnects since. So at this point it does not seem to be the PC, unRaid, DAC cable or either transceivers causing issues.

Leaves two options, OPNSense or the backup unRaid connection. My guess is OPNSense, though not a clue what the cause could be. Regardless when I get a chance I'll connect it as well and see.

wait, backup unraid connection? so unraid is connected to the switch twice?

 

[Matt0925]

wait, backup unraid connection? so unraid is connected to the switch twice?

I have unRaid set up to use two NICs, I believe bonded is the term? It'll run with one or the other and is supposed to work with both as well.

Currently 10 gig NIC is connected to the 7250 through a DAC cable. 1 gig is connected to a regular switch that's also connected between the router and 7250. In the past I did have both connected to the 7250 however

 

[fohdeesha]

I have unRaid set up to use two NICs, I believe bonded is the term? It'll run with one or the other and is supposed to work with both as well.

Currently 10 gig NIC is connected to the 7250 through a DAC cable. 1 gig is connected to a regular switch that's also connected between the router and 7250. In the past I did have both connected to the 7250 however

if it's a plain bond then you've created a loop by plugging both in and that would be why all the ports are flapping, although I would have expected to see loop detected/stp messages in your logs - I guess stp defaults to off in the l3 image though, so just the broadcast storm probably started taking ports down. In unraid you'd need to configure something aware like failover or lacp or active/passive bonds etc

 

[Matt0925]

if it's a plain bond then you've created a loop by plugging both in and that would be why all the ports are flapping, although I would have expected to see loop detected/stp messages in your logs - I guess stp defaults to off in the l3 image though, so just the broadcast storm probably started taking ports down. In unraid you'd need to configure something aware like failover or lacp or active/passive bonds etc

I thought I'd followed directions to get it set up to support both being connected but certainly possible I have an error somewhere, I'll check on those settings and adjust though realistically I probably don't need both connections.

Is having the two connections on different switches enough to avoid the issue? I guess I'd have thought I'd still run into the same problem

Edit- So unraid has several options for bonds. Reading through them I'd think my current one would be the right choice? I have Mode 1 (active-backup) selected.



Rest of the settings, just in case it helps-

 

[fohdeesha]

I thought I'd followed directions to get it set up to support both being connected but certainly possible I have an error somewhere, I'll check on those settings and adjust though realistically I probably don't need both connections.

Is having the two connections on different switches enough to avoid the issue? I guess I'd have thought I'd still run into the same problem

Edit- So unraid has several options for bonds. Reading through them I'd think my current one would be the right choice? I have Mode 1 (active-backup) selected.

View attachment 36723

Rest of the settings, just in case it helps-
View attachment 36724

the two connections being on different switches wouldn't cause a loop if the other switch has stp enabled (which most do by default). I have never used unraid and know nothing about it, but active/backup like you show wouldn't cause a loop unless something was misconfigured. However "use bridging = yes" might, not sure what unraid means there. I also see three ports in the bond, not two? I would just plug everything back into the icx except for this unraid second/third connection, and see if your link flaps go away

 

[Matt0925]

the two connections being on different switches wouldn't cause a loop if the other switch has stp enabled (which most do by default). I have never used unraid and know nothing about it, but active/backup like you show wouldn't cause a loop unless something was misconfigured. However "use bridging = yes" might, not sure what unraid means there. I also see three ports in the bond, not two? I would just plug everything back into the icx except for this unraid second/third connection, and see if your link flaps go away

The other switch being an unmanaged switch I have no idea how it handles STP. But as of right now I have (nearly) everything plugged back into the 7250- router and PC though the transceivers and unraid through the DAC. Coming up on an hour now and so far it is stable, about now is typically when I start seeing issues.

Bridging doesn't look like it would be the culprit- "Bridging is a feature which creates a virtual bridge and allows VMs and Docker containers to communicate directly with the physical Ethernet port. Both bonding and bridging can be combined to let VMs or containers communicate over a bonded interface."

The NIC I added for 10 gig has two SFP+ ports on it, then the 1 gig port on the motherboard. I have port 1 on the NIC and the MB ports connected, though the second port on the NIC is also in the bond. I found this post on the unraid forum, it looks like mixing different network ports doesn't always play nice so thinking you are right on the issue. Assuming disconnects don't start up again I'll just disconnect the 1gig line and call it a day