Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Scarlet]

You should be able to change the stacking configuration. This is how it looks on my 6450:

Spoiler: 6450 Stack configuration

stack unit 1

  module 1 icx6450-24p-poe-port-management-module

  module 2 icx6450-sfp-plus-4port-40g-module

  priority 255

  stack-port 1/2/1 1/2/3

stack unit 2

  module 1 icx6450-24p-poe-port-management-module

  module 2 icx6450-sfp-plus-4port-40g-module

  priority 240

  stack-port 2/2/1

stack enable

On unit 2 I set the stack ports using

SSH@ICX6450-24P Router switch-01(config)#stack unit 1
SSH@ICX6450-24P Router switch-01(config-unit-1)#stack-port 2/2/1

I would expect that something similar is available on the 7250 no matter how the initial configuration was done.

 

[gertsch]

yours came with the layer2 only firmware - scroll down to the note where it starts with "Note: If during the router-interface ve 1 command earlier " - read that blurb on how to get over to the l3 image

So I'm very new to networking and build up my homelab as I go at the moment, so happy to learn
I just got the 6430-C12 for very cheap and tried to follow the guide. I guess this model does not run the Layer3 fw right?
Is this also the reason, that the " ip route 0.0.0.0/0 192.168.0.1" command is throwing an invalid input error?

 

[Propaganda]

Do you have two cables connecting the switches (using four ports)? If so, that's why it built a ring stack, because that's the preferred configuration (since it provides redundancy).

You should be able to change the stacking configuration. This is how it looks on my 6450:

Spoiler: 6450 Stack configuration

stack unit 1

  module 1 icx6450-24p-poe-port-management-module

  module 2 icx6450-sfp-plus-4port-40g-module

  priority 255

  stack-port 1/2/1 1/2/3

stack unit 2

  module 1 icx6450-24p-poe-port-management-module

  module 2 icx6450-sfp-plus-4port-40g-module

  priority 240

  stack-port 2/2/1

stack enable

On unit 2 I set the stack ports using

SSH@ICX6450-24P Router switch-01(config)#stack unit 1
SSH@ICX6450-24P Router switch-01(config-unit-1)#stack-port 2/2/1

I would expect that something similar is available on the 7250 no matter how the initial configuration was done.

I am pretty sure I originally setup the stack using only a single cable (1/2/1 <->2/2/1) but it seemed to automatically consume 4 ports. I don't really know if the redundancy is worth the loss of 2 ports for my setup at the moment. Nearly all the devices connected to the 10G ports are already doubled up failsafe between the switch's and I am not sure how often a 3M DAC cable breaks but I'll have spares around if it does. I think I figured out a way to remove the ports with the following command, not really setup to test it at the moment since I have things torn apart at the moment.

stack unit 1
no multi-stack-port 1/2/3 and 2/2/3
exit

 

[jasonwc]

So I'm very new to networking and build up my homelab as I go at the moment, so happy to learn
I just got the 6430-C12 for very cheap and tried to follow the guide. I guess this model does not run the Layer3 fw right?
Is this also the reason, that the " ip route 0.0.0.0/0 192.168.0.1" command is throwing an invalid input error?

https://webresources.ruckuswireless.com/pdf/datasheets/ds-icx-6430-6450.pdf

See page 6. It appears that all of the 6430 series are Layer 2-only.

 

[Wolfcastle]

I apologize if this has been answered already but this is a monster of a thread and I haven't read through it all.
1) Is there any reason the ICX 7450 isn't being considered in the top post? Are licenses also available for this model? Seems like a good option compared to the ICX 6610 for lower power/quieter operation unless I am missing something?
2) The top post lists the ICX 6610-24P port as ~80 watts, I am guessing that is no POE, no traffic, is that correct? The 7450-24 seems like a really good option at ~65W for 10% traffic and likely a newer feature set.

I am putting in a 6610 in my electrical room where noise doesn't matter but I am debating whether it's worthwhile to have a 7450 or 7250 switch in my office closet to service the second floor of my house vs having a Mikrotik 10G SFP switch. Cost is pretty similar between the two small/simple switches vs the Brocade prices on eBay but the power consumption and noise in a closet with little ventilation makes me hesitate. The 7450 would be pretty nice in that would have a 40G uplink to connect to my 6610 and obviously the Brocades are L3 switches but I don't think that makes any difference for the handful of devices on the "daughter" switch (all wireless will be run off the 6610).

Would love to have advice if anyone has done something similar.

 

[jasonwc]

The ICX6610 offers 16 10G (8 SFP+ and 2 x QSFP+ breakout ports) + 2 x 40G QSFP+ ports built-in. The most common layout I've seen for the ICX7450 is 4 x 10G SFP + 2 x 40G QSFP+. You can get a max of 3 40G QSFP+ ports with no 10G ports or a max of 12 10G ports with no 40G ports. While the ICX6610 only offers PoE+ (30W), the 7450 also offers high-power (90W) PoH ports. The ICX6610 supports line-rate MACSEC encryption on the 8 SFP+ ports, which would be useful if you were running fiber outdoors to connect two buildings. So, I think it comes down to what features are important to you, how much noise you can tolerate, and how important power consumption is to you.

In terms of noise, I really don't think the ICX6610 is bad with two Rev B/C (front to back exhaust) power supplies and two fans. It's actually similar to what I measured for an ICX6450-48p with all 3 fans connected.

Regarding power consumption, the first post indicates that the power consumption isn't particularly linked to the amount of traffic. I think power consumption only increased a few watts going from 10% to 100%.

 

[Wolfcastle]

The ICX6610 offers 16 10G (8 SFP+ and 2 x QSFP+ breakout ports) + 2 x 40G QSFP+ ports built-in. The most common layout I've seen for the ICX7450 is 4 x 10G SFP + 2 x 40G QSFP+. You can get a max of 3 40G QSFP+ ports with no 10G ports or a max of 12 10G ports with no 40G ports. While the ICX6610 only offers PoE+ (30W), the 7450 also offers high-power (90W) PoH ports. So, I think it comes down to how what features are important to you, how much noise you can tolerate, and how important power consumption is to you.

In terms of noise, I really don't think the ICX6610 is bad with two Rev B/C (front to back exhaust) power supplies and two fans. It's actually similar to what I measured for an ICX6450-48p with all 3 fans connected.

Yes that layout for the 7450 is what I am seeing on eBay as well. I just wanted to make sure there are no licensing limitations since there are workarounds for the models in the first post of the thread.

For my specific case, I am not using any POE on my office switch (looking at the 24 non P version), if I ever do need to run anything POE from the office switch I would add in a POE injector. The only real constraint is that my office has a couple of PCs that I want to connect at 10G to the servers on the 6610. 40G uplink is a nice to have but realistically 10G is more than enough for now.

In terms of noise, you're probably looking at them in a rackmount config with good ventilation not thrown into a closet with almost no ventilation. I don't think I am comfortable throwing 100+W 24/7 into that closet!

 

[kpfleming]

I just wanted to make sure there are no licensing limitations since there are workarounds for the models in the first post of the thread.

The 7xxx series use honor-based licensing, so that shouldn't be an issue.

 

[Mithril]

Has anyone used the external power option for the switches that support it like the 6450? I'd be keen on a pinout and if it's possible to wire up your own DC-DC external supply.

 

[evanh]

If instead you want working 40G, and your current card can't be flashed, many folks here can confirm that the HP 649281-B21 can be flashed easily using the above instructions. I've already done two cards myself, and they only cost $30. Unlike the CX311A NICs, these have working digital optical monitoring, and it seems to work with any manufacturer, since it works with the KAIAM transceivers with detailed info.

Just want to say thanks again for your guidance and precedence! My m.2 > pcie riser arrived today so I was able to run both cards on the linux box long enough to flash them including the PSID change. Put the card back in my OpnSense box and booted and... no connection... but then remembered I still had it in the 4x10 slot on the switch. One slot over into 1/2/1 and we're showing full 40G connection! Huzzah!

 

[MrSliff]

Call me lucky, shot a ICX6450-48 today for just 30 bucks.

Looking forward to eat some beef

 

[covfefe]

• Comair Rotron "Gryphon" GDA4028-12BB - discontinued, alas
  If you can find these? BUY THESE. 40x40x28mm, but 11CFM, 0.34in H2O, 8800RPM, but just 31.4dBA! I am still mad they discontinued them. No, I am not selling any of my spares.

I got 3 of them.. but they came with bare wires.. so I ordered ends to put on them..

I'm looking for the wire diagram for the original fans to know which wire is which wire.. the original fans three wires green yellow and black ( from left to right) but I dont know what green= or yellow= or black= , I assume positive, negative and tachometer.

Did you end up figuring out the wiring? Just about to order a Gryphon GDA4028 for my ICX7250-24.
My best guess is :

Fan Red-positive     <-> Brocade Yellow=12V
Fan Black-negative   <-> Brocade Black=Ground
Fan White-Tachometer <-> Brocade Red=Sensor

 

[casualpie]

Hi folks. Long time lurker, first time poster. I have a recently acquired Brocade 6610 that seems to get in a pissing match with my server every morning before it just stops talking to it like a jilted lover. Wondering if anyone else has had similar issues.

Server is running latest edition of Proxmox.
NIC is a Mellanox CX354-FCCT, latest firmware. Port being used is set to ethernet mode. Used from ebay, but shows no problems elsewhere.
Cable is a FS.com QSFP+ 40gbps passive DAC. Brand new from vendor.
Switch is a 6610-48p, updated and fully licensed thanks to a rockstar member here we all know and appreciate. Used from a local sale.

At some point overnight, the 40gb connection will start to flap, up and down within a second, every second until link error disable threshold is reached and the port goes into err-disable for 5 seconds. It'll repeat the cycle until the port is eventually released from err-disable, but the link doesn't come back up. Then I wake up and go replug the DAC and it works fine again all day. Looking up the statistics on the switch shows several hundred InErrors accumulate, but that's all. The server shows no signs anything is wrong (other than losing connection). If it's worth noting, during the day the interface will accumulate a few more InErrors (less than 10) but with no effect on performance or link. Heavy traffic doesn't seem to be the issue, as it successfully performed a VM backup over the connection about an hour before the incident.

I've tried switching NICs and DACs for identical models and configuration. Tried changing to the other QSFP port (yes the 40gb port not the breakouts). Flow control is enabled on the switch and NIC (ethtool says the interface supports pause frame use Symmetric Receive-only, but ethtool -a lists TX and RX = on). Link error disable is set on the interface with a 5 second recovery time. Errdisable recovery is set for all causes and 10 seconds recovery. Network is flat, no VLANs or special configurations apart from the error handling mentioned.

Unfortunately I don't have another switch to test with, but directly linking two servers together on the same port/DAC results in a stable connection with no hiccups.

Has anyone had a similar issue before, or am I chasing ghosts when they're in the house next door? Any troubleshooting tips would be appreciated as well.

 

[MrSliff]

I'm searching for Service Manuals and Schematics for the ICX7250 POE Version Switch.

Does anybody know where i can find these? Ive seen a defective switch where one MOSFET in the intermediate circuit of the power supply exploded, also one Capacitor is damaged by the explosion. I may get it if i can get Service Documents. These Documents would make the repair process much easier, since i could look for the right replacement part. I dont know if the text on the second MOSFET is still readable (would replace both then).

 

[kpfleming]

I'm searching for Service Manuals and Schematics for the ICX7250 POE Version Switch.

Does anybody know where i can find these? Ive seen a defective switch where one MOSFET in the intermediate circuit of the power supply exploded, also one Capacitor is damaged by the explosion. I may get it if i can get Service Documents. These Documents would make the repair process much easier, since i could look for the right replacement part. I dont know if the text on the second MOSFET is still readable (would replace both then).

The place to ask would be the Ruckus Wireless forums.

 

[MrSliff]

The place to ask would be the Ruckus Wireless forums.

Thanks, i will check if i can get them there.

 

[withoutink]

Brand new here, have yet to read through the 329 pages, but I did ping pong around some search results in this thread. I am considering purchasing a Ruckus ICX7150 - C12P - 2X1G from eBay, but wondering if that is advisable. I saw a couple mentions of PSU's dying, and since these are not warrantied past their original purchaser. I pinged the seller, and they said its in like new condition, and is working perfectly.

I don't mind purchasing, if this won't be a headache down the road.

If the PSU is the only issue what does it cost to replace the PSU?

Thanks in advance

 

[kpfleming]

Brand new here, have yet to read through the 329 pages, but I did ping pong around some search results in this thread. I am considering purchasing a Ruckus ICX7150 - C12P - 2X1G from eBay, but wondering if that is advisable. I saw a couple mentions of PSU's dying, and since these are not warrantied past their original purchaser. I pinged the seller, and they said its in like new condition, and is working perfectly.

I don't mind purchasing, if this won't be a headache down the road.

If the PSU is the only issue what does it cost to replace the PSU?

Thanks in advance

I bought four, one died about 48 hours after I received it and plugged it in (PSU failure), seller replaced it and all four have been fine for months since. Given their capabilities and the relatively low cost on eBay, buy at least one more than you actually need and test them all well

 

[RobstarUSA]

Hi folks. Long time lurker, first time poster. I have a recently acquired Brocade 6610 that seems to get in a pissing match with my server every morning before it just stops talking to it like a jilted lover. Wondering if anyone else has had similar issues.

Server is running latest edition of Proxmox.
NIC is a Mellanox CX354-FCCT, latest firmware. Port being used is set to ethernet mode. Used from ebay, but shows no problems elsewhere.
Cable is a FS.com QSFP+ 40gbps passive DAC. Brand new from vendor.
Switch is a 6610-48p, updated and fully licensed thanks to a rockstar member here we all know and appreciate. Used from a local sale.

At some point overnight, the 40gb connection will start to flap, up and down within a second, every second until link error disable threshold is reached and the port goes into err-disable for 5 seconds. It'll repeat the cycle until the port is eventually released from err-disable, but the link doesn't come back up. Then I wake up and go replug the DAC and it works fine again all day. Looking up the statistics on the switch shows several hundred InErrors accumulate, but that's all. The server shows no signs anything is wrong (other than losing connection). If it's worth noting, during the day the interface will accumulate a few more InErrors (less than 10) but with no effect on performance or link. Heavy traffic doesn't seem to be the issue, as it successfully performed a VM backup over the connection about an hour before the incident.

I've tried switching NICs and DACs for identical models and configuration. Tried changing to the other QSFP port (yes the 40gb port not the breakouts). Flow control is enabled on the switch and NIC (ethtool says the interface supports pause frame use Symmetric Receive-only, but ethtool -a lists TX and RX = on). Link error disable is set on the interface with a 5 second recovery time. Errdisable recovery is set for all causes and 10 seconds recovery. Network is flat, no VLANs or special configurations apart from the error handling mentioned.

Unfortunately I don't have another switch to test with, but directly linking two servers together on the same port/DAC results in a stable connection with no hiccups.

Has anyone had a similar issue before, or am I chasing ghosts when they're in the house next door? Any troubleshooting tips would be appreciated as well.

Have you tried disabling flow control? I've never seen it work well in a modern network.
Edit: I don't think this will fix your L1 issues however. In any case I have propxmox ve on a 10Gbit/s link & it's rock solid, so I don't think it's anythign with proxmox but perhaps an transceiver/nic/switch compatibility issue. What device does the nic show up as in proxmox? Which kernel are you on ?

 

[luks]

Does anyone have experience with the ICX7450? How does it compare to the ICX6610? How loud is it? I need a switch that supports both 10Gb and 40Gb. I'm thinking between ICX6610, ICX7450 and VDX6740 (fiber models).