Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[klui]

I am also seeing this message repeating every few seconds & I am not sure how to shut it off?

port 1/2/8 is not capable of digital optical monitoring.

Try using the clear command.

 

[Fallen Kell]

Oh, that's lovely, I might have to get one of those; I currently carry around a regular USB to DB9 dongle and my Cisco cable. I do have some switches that use different pin outs though, so have DB( as the common element is handy.

If you have a serial cable setup that works with CISCO it will also work with the 6610. The reason for the confusion is because most people on this thread do not also have CISCO gear and/or proper serial support. So the general recommendation are the USB serial RJ45 connectors that are linked by most people. You don't need them if you have computers with serial ports, or an adapter that supports a DB9 or DB25 and appropriate adapters to RJ45 using the CISCO pin outs. Again, most people now don't know what that is in the first place...

Also as stated by the others, a properly wiped switch will be essentially booting to a stage that you can load the new firmware. Read the guide posted on the first page and it will walk you through loading the firmware which includes the OS. As mentioned, you will need a serial connection initially since your switch does not have an active working OS. After you have an OS, you can setup the network management port (I connected mine to a port that I set to a management VLAN, but ideally, you connect this to a different switch since if the switch is not responding properly, you might not be able to connect to the management port that way).

Be prepared for the 6610 to be EXTREMELY LOUD until after you load a proper OS. It will run all fans at 100% in this state because it doesn't have the smarts/controls at this point to sense temperatures and adjust speeds lower. So don't think it is that you have a broken switch and/or something that is way to loud for your use case until after you get the OS loaded (usually 1-2 minutes after booting).

 

[jht3]

Hey @jht3 I am interesting in testing out VyOS on the edge as a firewall with a transit vlan and my 6450 L3 switch hosting multiple Internal vlans. I've not found any howtoo's specifically for setting this up in the VyOS cli. Do you know of any or have some basic tips beyond the nat masquerade rule here?

its a very basic setup, if you follow the vyos documentation. configure your interfaces, add a static route, add NAT rules, configure firewall. and any other services you want or need. i've been running vyos as my main fw/router for 4+ years now after growing weary of pfsense and its complete lack of a CLI, missing support for VLANs on Xenserver, etc.
or i can send you my basic config if that would help get you started.

 

[Vesalius]

its a very basic setup, if you follow the vyos documentation. configure your interfaces, add a static route, add NAT rules, configure firewall. and any other services you want or need. i've been running vyos as my main fw/router for 4+ years now after growing weary of pfsense and its complete lack of a CLI, missing support for VLANs on Xenserver, etc.
or i can send you my basic config if that would help get you started.

That would be great thanks.

 

[coxhaus]

So I'm trying to figure out what went wrong trying to replicate kapone's post but can't seem to find the culprit of it.

First of all, you'll have to forgive me if I don't make too much sense, since English is not my first language but always try to do my best to communicate with it.

Right now my home gear network consists of the following:

PC Engines APU2 - PfSense
Cisco SG300-52 L3 enabled
ICX7250-48 L3 10Gb license (Thanks @fohdeesha)
ICX6610-48 Fully licensed too (1 PSU rev3, 1 FAN) - Not in use, too loud after a few minutes (maybe will sell it since PSU and fan will cost me as much as another unit)

What I am trying to achive?

Have a native L3 network after a while since FW was handling the VLANs as a Router on a Stick approach, right now it can't handle inter-vlan 1Gb network traffic after doing so for a "some time", since I have the gear that can do all the L3 at its core.

My core networking/services/servers (more APU2s) are on a 12U startech rack is hangin on my apartment's entrance corner and my office is a few meters away cabled with 6 CAT6 ethernet drops, maybe more, but don't tell my wife.

My first approach was to use the ICX7250 as a Core switch for my place on the aforementioned startech rack, and the ICX6610 for my 24U rack with 6 SM servers all with 10Gb NICs and a 40Gbps NIC on my main ESX/NAS server, short long story, as I mentioned the switch is too loud to have it 24/7 on my apartment I didn't even setup up correctly on both ends. - Currently discarded until further notice or until christmas bonus. lol

Then I tried to use the SG300 as my Core L3 switch and the ICX7250 as my rack switch with inter-VLAN routing on my main LAN, this worked "well" can access the SVIs, setup the firewall rules, static routes both the switches and FW can see and communicate, everything was ok but then I realized L3 routing was performed at the Cisco so 10Gb traffic was limited to 1Gb as you might guessed. - Discarded for the time being, maybe will get back to this if could find the routing issue on the ICX.

So third attempt, since the APU has 3 ethernet ports (WAN, LAN, OPT1) tried the Kapone's post guideline, using the OPT1 which was unused to connect one of the cable drops to my office directly to the ICX, created a /30 transit VLAN, gateway, static route on FW, static route on SW, FW rules and what not, but can't communicate from my main home network to the VLANs associated on the ICX, I'm still using the SG300 on my LAN, nothing has done yet in there (No L3 switching, VLANs SVIs, nothing really, just a dumb SW ATM), was thinking to use it as an access L2 SW for the VLANs needed for the APU2s VMs and LXC containers using a second drop back from my rack to the startech rack.

So with all this, which approach will be the best to execute, and more important, am I missing something on my config, steps that might be overlooked?

Basically TDLR;

Need to setup a L3 network using Cisco SG300, ICX7250 and pfsense, but have failed doing so.

Here's the precious data if needed:



If you need more info I'd gladly provide it.

I was using a Cisco SG300-28 switch in L3 10 years ago. They are a nice switch but old. I have changed the last couple years and use a Cisco SG350-10P now. I have a Cisco SG350X-24 but the fans bother me. I used pfsense for about a year and switched to a Cisco RV340 router as all the DHCP and local routing was on my L3 switch. Pfsense updates did not work as they usually broke something and I got tired of testing my router over and over. So, I dropped pfsense. I ran Untangle as a UTM behind my Cisco router for 4 or 5 years. Maybe that is why I did not appreciate pfsense as I ran Untangle as a UTM device before pfsense.

Using an L3 switch is still better than L2 if you are using network VLANs. I always assign a network to a VLAN. I use several in my home.
Using a Cisco SG300 switch in L3 will require you to turn on L3 mode. This is only required on these older Cisco switches. The Cisco SG350 switches do not have a mode.

 

[richtj99]

Thats pretty interesting - I am a bit more confused -

MPT/MPO seems like a great option - I am not sure if I can use it in my conduit - it goes under my driveway & is fairly old - I think there is some water in it - can the MPT/MPO be used outdoors (in a conduit that could be wet?)

What's the difference between a preterminated fiber 12 strand OM3 & a 12 strand OM3 MTP/MPO? Seems like it is the same thing?

This near switch #1:

MTP Fiber Assembly - 12 Strand OM3 Plenum, MTP Male to Generic LC Duplex

Save time & money with MTP/MPO installations. Pull one cable rather than 12+. Discount Low Voltage has the MTP/MPO cable you need now at wholesale prices. With MTP cables, it's possible to run a single cable that automatically terminates up to 72 s

www.discount-low-voltage.com

This through the conduit:

MTP Fiber Assembly - 12 Strand OM3 Plenum, MTP Female to MTP Female

Save time & money with MTP/MPO installations. Pull one cable rather than 12+. Discount Low Voltage has the MTP/MPO cable you need now at wholesale prices. With MTP cables, it's possible to run a single cable that automatically terminates up to 72 sepa

www.discount-low-voltage.com

This near switch #2:

MTP Fiber Assembly - 12 Strand OM3 Plenum, MTP Male to Generic LC Duplex

Save time & money with MTP/MPO installations. Pull one cable rather than 12+. Discount Low Voltage has the MTP/MPO cable you need now at wholesale prices. With MTP cables, it's possible to run a single cable that automatically terminates up to 72 s

www.discount-low-voltage.com

I guess the only downside might be damaging the cable when plugging it in?

If your main concern is fitting it down an existing conduit, I'd go with MPT/MPO terminated cables, and get fanouts that plug into them, rather than trying to stuff the full 12xLC fanout down the conduit.

Try using the clear command.

Thank you! Worked perfect!

 

[ViciousXUSMC]

Picked up two more ICX 6450's
Has anybody found the "perfect fan" yet for these?

Back in the day I used Noctua NF-A4x20 FLX fans and that worked for me. That switch is silent and still running strong.
However I dont use much PoE on it (and I found out I have a bad PoE module in that switch), thinking maybe something with a bit more CFM for the next mod because it may be running a PoE Camera array.

 

[tozmo]

I have a 7450 with broken 2.5gb ports, 2psu, no fans. Before I start hacking cisco usb cables to set this up, does connecting the 2 psu offset the OS requirement for a fan? Or do I need to hack in a fan like dodgy route ? Or just buy a 6610 fan?

 

[NateS]

Thats pretty interesting - I am a bit more confused -

MPT/MPO seems like a great option - I am not sure if I can use it in my conduit - it goes under my driveway & is fairly old - I think there is some water in it - can the MPT/MPO be used outdoors (in a conduit that could be wet?)

Well, no fiber connector is going to be very happy being dragged through muddy water, but once it's installed the connectors will be hanging out the ends of the conduit, and you could clean them and keep them dry at that point. The fiber itself should be fine (but double check that the jacket is rated for water contact; probably most are).

If I were you, I'd leave the dust caps on the connectors and then tape the whole thing up well with something water-resistant like electrical tape before you actually pull it through the conduit. That should keep most of the water out while you're installing it, but I'd also recommend picking up a cleaning kit for whatever connecter type you ultimately decide on to clean out any water that does find its way in.

What's the difference between a preterminated fiber 12 strand OM3 & a 12 strand OM3 MTP/MPO? Seems like it is the same thing?

The former is just preterminated with some unspecified connector(s), which could be 12xLC, 1xMPO, 12xSC, etc. The latter specifies that it's terminated with MPT/MPO connectors specifically.

This near switch #1:

MTP Fiber Assembly - 12 Strand OM3 Plenum, MTP Male to Generic LC Duplex

Save time & money with MTP/MPO installations. Pull one cable rather than 12+. Discount Low Voltage has the MTP/MPO cable you need now at wholesale prices. With MTP cables, it's possible to run a single cable that automatically terminates up to 72 s

www.discount-low-voltage.com

This through the conduit:

MTP Fiber Assembly - 12 Strand OM3 Plenum, MTP Female to MTP Female

Save time & money with MTP/MPO installations. Pull one cable rather than 12+. Discount Low Voltage has the MTP/MPO cable you need now at wholesale prices. With MTP cables, it's possible to run a single cable that automatically terminates up to 72 sepa

www.discount-low-voltage.com

This near switch #2:

MTP Fiber Assembly - 12 Strand OM3 Plenum, MTP Male to Generic LC Duplex

Save time & money with MTP/MPO installations. Pull one cable rather than 12+. Discount Low Voltage has the MTP/MPO cable you need now at wholesale prices. With MTP cables, it's possible to run a single cable that automatically terminates up to 72 s

www.discount-low-voltage.com

I guess the only downside might be damaging the cable when plugging it in?

I suppose you do have two more connectors in the chain, but I don't think the added potential for damage from that is worth worrying about. The biggest downside is just that it costs a bit more, and planning it out is a little more complex.

One thing you do have to keep in mind is that there are multiple ways to split out the 12 fibers into send and receive pairs, and reverse each pair such that a send port always hits a receive port. In this case, I'd recommend keeping your main underground fiber a straight-through, type A cable, and then use the fanouts on each end to split off the pairs, but there are other valid methods too. See here for more info: Understanding Polarity in MTP-12 Fiber Cable System

Also, if you want either end to have a patch panel, rather than a fanout, check out the MTP/MPO cassettes from fs: FHD MTP to LC Fiber Cassettes

 

[infoMatt]

Picked up two more ICX 6450's
Has anybody found the "perfect fan" yet for these?

I've used two Sunon MagLev KDE1204PKVX with two small capacitors between +12V and GND to smooth out the PWM noise... it's not silent by any means but it's bearable when you sit beside it and basically silent when closed inside a rack.

 

[3nodeproblem]

I currently have a fanmodded ICX6450-48P ("ICX1") running as my home prod switch. Apart from VLANs I'm not making use of much fancy functionality. Haven't looked into STP etc. I have not stacked before.

Now I am moving to a new physical location and want to do the migration with minimal downtime. I have just received and prepared a secondary ICX6450-48P ("ICX2") - not fanmodded but otherwise identical. I want ICX1 to be in the new location and not make use of ICX2 except as having on hand as a cold spare. Can someone tell me if the following plan makes sense or if I'm missing something:

1. Stack ICX1 and ICX2 both in the old physical location, with ICX1 as master. Linear traditional stack. ICX1 as active controller, ICX2 as standby.
2. Configure VLAN ports on ICX2 to be identical to ICX1.
3. Migrate over all ports/links one by one from ICX1 to ICX2 until ICX1 is unpopulated.
4. Power off ICX1 (thereby breaking the stack)
5. Move ICX1 to new physical location.
6. Connect ICX1 to internet, add a wireguard VPN host on the "server" subnet and VLAN so the servers can span across the two physical locations
7. Move hosts to new physical location until ICX2 is unpopulated.
8. Power off ICX2 and put it in storage

Also, is anyone aware of if it's possible to stack using RJ45 Cat6 transceivers in the stacking ports or if I need to get a "proper" cable like this one? Brocade XBR-TWX-01.5 10G SFP+ DAC Twinax Cable

 

[LodeRunner]

I currently have a fanmodded ICX6450-48P ("ICX1") running as my home prod switch. Apart from VLANs I'm not making use of much fancy functionality. Haven't looked into STP etc. I have not stacked before.

Now I am moving to a new physical location and want to do the migration with minimal downtime. I have just received and prepared a secondary ICX6450-48P ("ICX2") - not fanmodded but otherwise identical. I want ICX1 to be in the new location and not make use of ICX2 except as having on hand as a cold spare. Can someone tell me if the following plan makes sense or if I'm missing something:

1. Stack ICX1 and ICX2 both in the old physical location, with ICX1 as master. Linear traditional stack. ICX1 as active controller, ICX2 as standby.
2. Configure VLAN ports on ICX2 to be identical to ICX1.
3. Migrate over all ports/links one by one from ICX1 to ICX2 until ICX1 is unpopulated.
4. Power off ICX1 (thereby breaking the stack)
5. Move ICX1 to new physical location.
6. Connect ICX1 to internet, add a wireguard VPN host on the "server" subnet and VLAN so the servers can span across the two physical locations
7. Move hosts to new physical location until ICX2 is unpopulated.
8. Power off ICX2

Also, is anyone aware of if it's possible to stack using RJ45 Cat6 transceivers in the stacking ports or if I need to get a "proper" cable like this one? Brocade XBR-TWX-01.5 10G SFP+ DAC Twinax Cable

If desired use is only as a cold spare, use TFTP to dump the running config, and load it on the second switch as the startup config, then power it off.

Stacking will probably not do what you want for this use case, it normally adds the second switch as a ‘module’ to the first switch.

 

[3nodeproblem]

If desired use is only as a cold spare, use TFTP to dump the running config, and load it on the second switch as the startup config, then power it off.

Stacking will probably not do what you want for this use case, it normally adds the second switch as a ‘module’ to the first switch.

Thanks for replying. The reason for stacking is so I can reuse ICX1 in the new location without having to take the whole network offline during the migration - so it would be stacked only so I can replace ICX1 with ICX2 in the old location.

 

[dennisp]

I currently have a fanmodded ICX6450-48P ("ICX1") running as my home prod switch. Apart from VLANs I'm not making use of much fancy functionality. Haven't looked into STP etc. I have not stacked before.

Now I am moving to a new physical location and want to do the migration with minimal downtime. I have just received and prepared a secondary ICX6450-48P ("ICX2") - not fanmodded but otherwise identical. I want ICX1 to be in the new location and not make use of ICX2 except as having on hand as a cold spare. Can someone tell me if the following plan makes sense or if I'm missing something:

1. Stack ICX1 and ICX2 both in the old physical location, with ICX1 as master. Linear traditional stack. ICX1 as active controller, ICX2 as standby.
2. Configure VLAN ports on ICX2 to be identical to ICX1.
3. Migrate over all ports/links one by one from ICX1 to ICX2 until ICX1 is unpopulated.
4. Power off ICX1 (thereby breaking the stack)
5. Move ICX1 to new physical location.
6. Connect ICX1 to internet, add a wireguard VPN host on the "server" subnet and VLAN so the servers can span across the two physical locations
7. Move hosts to new physical location until ICX2 is unpopulated.
8. Power off ICX2 and put it in storage

Also, is anyone aware of if it's possible to stack using RJ45 Cat6 transceivers in the stacking ports or if I need to get a "proper" cable like this one? Brocade XBR-TWX-01.5 10G SFP+ DAC Twinax Cable

Your process looks good, but I'd use a regular uplink between the switches instead of stacking. Stacking adds unnecessary complexity

 

[3nodeproblem]

Your process looks good, but I'd use a regular uplink between the switches instead of stacking. Stacking adds unnecessary complexity

That's a good point actually. So just connect the two over trunk ports right?

 

[dennisp]

Exactly. You could even use a regular 1G port

 

[BecauseScience]

I apologize for what sounds like the dumbest noob question ever.

What's the command to cleanly power down a 6450? I'm connected via serial admin port.

I searched the manuals then googled for an additional 20 minutes with no luck.

----

This USB to serial cable works great on my 6450 and it uses a FTDI chip to boot:
Asunflower 12 Ft FTDI USB to RJ45 Router Console Cable - RS232

 

[fohdeesha]

I apologize for what sounds like the dumbest noob question ever.

What's the command to cleanly power down a 6450? I'm connected via serial admin port.

I searched the manuals then googled for an additional 20 minutes with no luck.

----

This USB to serial cable works great on my 6450 and it uses a FTDI chip to boot:
Asunflower 12 Ft FTDI USB to RJ45 Router Console Cable - RS232

There isn't one, it's a read only filesystem designed to have the power yanked unless it's in the middle of a firmware upgrade or something

 

[ccie4526]

So right now these guys are at $90, and I was able to get one at best offer of $80. They countered my $70 offer.

Brocade ICX6450-24P 24-Port PoE+ Gigabit Enterprise Class Stackable Switch | eBay

Find many great new & used options and get the best deals for Brocade ICX6450-24P 24-Port PoE+ Gigabit Enterprise Class Stackable Switch at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[dennisp]

So right now these guys are at $90, and I was able to get one at best offer of $80. They countered my $70 offer.

Brocade ICX6450-24P 24-Port PoE+ Gigabit Enterprise Class Stackable Switch | eBay

Find many great new & used options and get the best deals for Brocade ICX6450-24P 24-Port PoE+ Gigabit Enterprise Class Stackable Switch at the best online prices at eBay! Free shipping for many products!

www.ebay.com

I have a stack of three of them from this seller in front of me right now, no issues with them. They came wiped with an old version of the OS, 7-something, no "bonus" licenses other than the 2 default stacking ports. This seller has been moving down from about 150, I got mine 2 weeks ago for 100/ea.