Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

coxhaus

Active Member
Jul 7, 2020
109
36
28
So I'm trying to figure out what went wrong trying to replicate kapone's post but can't seem to find the culprit of it.

First of all, you'll have to forgive me if I don't make too much sense, since English is not my first language but always try to do my best to communicate with it.

Right now my home gear network consists of the following:

PC Engines APU2 - PfSense
Cisco SG300-52 L3 enabled
ICX7250-48 L3 10Gb license (Thanks @fohdeesha)
ICX6610-48 Fully licensed too (1 PSU rev3, 1 FAN) - Not in use, too loud after a few minutes (maybe will sell it since PSU and fan will cost me as much as another unit)

What I am trying to achive?

Have a native L3 network after a while since FW was handling the VLANs as a Router on a Stick approach, right now it can't handle inter-vlan 1Gb network traffic after doing so for a "some time", since I have the gear that can do all the L3 at its core.

My core networking/services/servers (more APU2s) are on a 12U startech rack is hangin on my apartment's entrance corner and my office is a few meters away cabled with 6 CAT6 ethernet drops, maybe more, but don't tell my wife.

My first approach was to use the ICX7250 as a Core switch for my place on the aforementioned startech rack, and the ICX6610 for my 24U rack with 6 SM servers all with 10Gb NICs and a 40Gbps NIC on my main ESX/NAS server, short long story, as I mentioned the switch is too loud to have it 24/7 on my apartment I didn't even setup up correctly on both ends. - Currently discarded until further notice or until christmas bonus. lol

Then I tried to use the SG300 as my Core L3 switch and the ICX7250 as my rack switch with inter-VLAN routing on my main LAN, this worked "well" can access the SVIs, setup the firewall rules, static routes both the switches and FW can see and communicate, everything was ok but then I realized L3 routing was performed at the Cisco so 10Gb traffic was limited to 1Gb as you might guessed. - Discarded for the time being, maybe will get back to this if could find the routing issue on the ICX.

So third attempt, since the APU has 3 ethernet ports (WAN, LAN, OPT1) tried the Kapone's post guideline, using the OPT1 which was unused to connect one of the cable drops to my office directly to the ICX, created a /30 transit VLAN, gateway, static route on FW, static route on SW, FW rules and what not, but can't communicate from my main home network to the VLANs associated on the ICX, I'm still using the SG300 on my LAN, nothing has done yet in there (No L3 switching, VLANs SVIs, nothing really, just a dumb SW ATM), was thinking to use it as an access L2 SW for the VLANs needed for the APU2s VMs and LXC containers using a second drop back from my rack to the startech rack.

So with all this, which approach will be the best to execute, and more important, am I missing something on my config, steps that might be overlooked?

Basically TDLR;

Need to setup a L3 network using Cisco SG300, ICX7250 and pfsense, but have failed doing so.


Here's the precious data if needed:



If you need more info I'd gladly provide it.
I was using a Cisco SG300-28 switch in L3 10 years ago. They are a nice switch but old. I have changed the last couple years and use a Cisco SG350-10P now. I have a Cisco SG350X-24 but the fans bother me. I used pfsense for about a year and switched to a Cisco RV340 router as all the DHCP and local routing was on my L3 switch. Pfsense updates did not work as they usually broke something and I got tired of testing my router over and over. So, I dropped pfsense. I ran Untangle as a UTM behind my Cisco router for 4 or 5 years. Maybe that is why I did not appreciate pfsense as I ran Untangle as a UTM device before pfsense.

Using an L3 switch is still better than L2 if you are using network VLANs. I always assign a network to a VLAN. I use several in my home.
Using a Cisco SG300 switch in L3 will require you to turn on L3 mode. This is only required on these older Cisco switches. The Cisco SG350 switches do not have a mode.
 
Last edited:
  • Like
Reactions: vpadro and Wesumat

richtj99

Member
Jul 8, 2017
67
1
8
50
Thats pretty interesting - I am a bit more confused -

MPT/MPO seems like a great option - I am not sure if I can use it in my conduit - it goes under my driveway & is fairly old - I think there is some water in it - can the MPT/MPO be used outdoors (in a conduit that could be wet?)

What's the difference between a preterminated fiber 12 strand OM3 & a 12 strand OM3 MTP/MPO? Seems like it is the same thing?

This near switch #1:

This through the conduit:

This near switch #2:

I guess the only downside might be damaging the cable when plugging it in?



If your main concern is fitting it down an existing conduit, I'd go with MPT/MPO terminated cables, and get fanouts that plug into them, rather than trying to stuff the full 12xLC fanout down the conduit.

Try using the clear command.
Thank you! Worked perfect!
 

ViciousXUSMC

Active Member
Nov 27, 2016
264
140
43
41
Picked up two more ICX 6450's
Has anybody found the "perfect fan" yet for these?

Back in the day I used Noctua NF-A4x20 FLX fans and that worked for me. That switch is silent and still running strong.
However I dont use much PoE on it (and I found out I have a bad PoE module in that switch), thinking maybe something with a bit more CFM for the next mod because it may be running a PoE Camera array.
 
  • Like
Reactions: BecauseScience

tozmo

Active Member
Feb 1, 2017
142
102
43
74
I have a 7450 with broken 2.5gb ports, 2psu, no fans. Before I start hacking cisco usb cables to set this up, does connecting the 2 psu offset the OS requirement for a fan? Or do I need to hack in a fan like dodgy route ? Or just buy a 6610 fan?
 

NateS

Active Member
Apr 19, 2021
159
91
28
Sacramento, CA, US
Thats pretty interesting - I am a bit more confused -

MPT/MPO seems like a great option - I am not sure if I can use it in my conduit - it goes under my driveway & is fairly old - I think there is some water in it - can the MPT/MPO be used outdoors (in a conduit that could be wet?)
Well, no fiber connector is going to be very happy being dragged through muddy water, but once it's installed the connectors will be hanging out the ends of the conduit, and you could clean them and keep them dry at that point. The fiber itself should be fine (but double check that the jacket is rated for water contact; probably most are).

If I were you, I'd leave the dust caps on the connectors and then tape the whole thing up well with something water-resistant like electrical tape before you actually pull it through the conduit. That should keep most of the water out while you're installing it, but I'd also recommend picking up a cleaning kit for whatever connecter type you ultimately decide on to clean out any water that does find its way in.

What's the difference between a preterminated fiber 12 strand OM3 & a 12 strand OM3 MTP/MPO? Seems like it is the same thing?
The former is just preterminated with some unspecified connector(s), which could be 12xLC, 1xMPO, 12xSC, etc. The latter specifies that it's terminated with MPT/MPO connectors specifically.

This near switch #1:

This through the conduit:

This near switch #2:

I guess the only downside might be damaging the cable when plugging it in?
I suppose you do have two more connectors in the chain, but I don't think the added potential for damage from that is worth worrying about. The biggest downside is just that it costs a bit more, and planning it out is a little more complex.

One thing you do have to keep in mind is that there are multiple ways to split out the 12 fibers into send and receive pairs, and reverse each pair such that a send port always hits a receive port. In this case, I'd recommend keeping your main underground fiber a straight-through, type A cable, and then use the fanouts on each end to split off the pairs, but there are other valid methods too. See here for more info: Understanding Polarity in MTP-12 Fiber Cable System

Also, if you want either end to have a patch panel, rather than a fanout, check out the MTP/MPO cassettes from fs: FHD MTP to LC Fiber Cassettes
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
Picked up two more ICX 6450's
Has anybody found the "perfect fan" yet for these?
I've used two Sunon MagLev KDE1204PKVX with two small capacitors between +12V and GND to smooth out the PWM noise... it's not silent by any means but it's bearable when you sit beside it and basically silent when closed inside a rack.
 
  • Like
Reactions: fohdeesha

3nodeproblem

Member
Jun 28, 2020
48
11
8
I currently have a fanmodded ICX6450-48P ("ICX1") running as my home prod switch. Apart from VLANs I'm not making use of much fancy functionality. Haven't looked into STP etc. I have not stacked before.

Now I am moving to a new physical location and want to do the migration with minimal downtime. I have just received and prepared a secondary ICX6450-48P ("ICX2") - not fanmodded but otherwise identical. I want ICX1 to be in the new location and not make use of ICX2 except as having on hand as a cold spare. Can someone tell me if the following plan makes sense or if I'm missing something:

1. Stack ICX1 and ICX2 both in the old physical location, with ICX1 as master. Linear traditional stack. ICX1 as active controller, ICX2 as standby.
2. Configure VLAN ports on ICX2 to be identical to ICX1.
3. Migrate over all ports/links one by one from ICX1 to ICX2 until ICX1 is unpopulated.
4. Power off ICX1 (thereby breaking the stack)
5. Move ICX1 to new physical location.
6. Connect ICX1 to internet, add a wireguard VPN host on the "server" subnet and VLAN so the servers can span across the two physical locations
7. Move hosts to new physical location until ICX2 is unpopulated.
8. Power off ICX2 and put it in storage

Also, is anyone aware of if it's possible to stack using RJ45 Cat6 transceivers in the stacking ports or if I need to get a "proper" cable like this one? Brocade XBR-TWX-01.5 10G SFP+ DAC Twinax Cable
 
Last edited:

LodeRunner

Active Member
Apr 27, 2019
540
227
43
I currently have a fanmodded ICX6450-48P ("ICX1") running as my home prod switch. Apart from VLANs I'm not making use of much fancy functionality. Haven't looked into STP etc. I have not stacked before.

Now I am moving to a new physical location and want to do the migration with minimal downtime. I have just received and prepared a secondary ICX6450-48P ("ICX2") - not fanmodded but otherwise identical. I want ICX1 to be in the new location and not make use of ICX2 except as having on hand as a cold spare. Can someone tell me if the following plan makes sense or if I'm missing something:

1. Stack ICX1 and ICX2 both in the old physical location, with ICX1 as master. Linear traditional stack. ICX1 as active controller, ICX2 as standby.
2. Configure VLAN ports on ICX2 to be identical to ICX1.
3. Migrate over all ports/links one by one from ICX1 to ICX2 until ICX1 is unpopulated.
4. Power off ICX1 (thereby breaking the stack)
5. Move ICX1 to new physical location.
6. Connect ICX1 to internet, add a wireguard VPN host on the "server" subnet and VLAN so the servers can span across the two physical locations
7. Move hosts to new physical location until ICX2 is unpopulated.
8. Power off ICX2

Also, is anyone aware of if it's possible to stack using RJ45 Cat6 transceivers in the stacking ports or if I need to get a "proper" cable like this one? Brocade XBR-TWX-01.5 10G SFP+ DAC Twinax Cable
If desired use is only as a cold spare, use TFTP to dump the running config, and load it on the second switch as the startup config, then power it off.

Stacking will probably not do what you want for this use case, it normally adds the second switch as a ‘module’ to the first switch.
 

3nodeproblem

Member
Jun 28, 2020
48
11
8
If desired use is only as a cold spare, use TFTP to dump the running config, and load it on the second switch as the startup config, then power it off.

Stacking will probably not do what you want for this use case, it normally adds the second switch as a ‘module’ to the first switch.
Thanks for replying. The reason for stacking is so I can reuse ICX1 in the new location without having to take the whole network offline during the migration - so it would be stacked only so I can replace ICX1 with ICX2 in the old location.
 

dennisp

New Member
Apr 1, 2021
18
13
3
I currently have a fanmodded ICX6450-48P ("ICX1") running as my home prod switch. Apart from VLANs I'm not making use of much fancy functionality. Haven't looked into STP etc. I have not stacked before.

Now I am moving to a new physical location and want to do the migration with minimal downtime. I have just received and prepared a secondary ICX6450-48P ("ICX2") - not fanmodded but otherwise identical. I want ICX1 to be in the new location and not make use of ICX2 except as having on hand as a cold spare. Can someone tell me if the following plan makes sense or if I'm missing something:

1. Stack ICX1 and ICX2 both in the old physical location, with ICX1 as master. Linear traditional stack. ICX1 as active controller, ICX2 as standby.
2. Configure VLAN ports on ICX2 to be identical to ICX1.
3. Migrate over all ports/links one by one from ICX1 to ICX2 until ICX1 is unpopulated.
4. Power off ICX1 (thereby breaking the stack)
5. Move ICX1 to new physical location.
6. Connect ICX1 to internet, add a wireguard VPN host on the "server" subnet and VLAN so the servers can span across the two physical locations
7. Move hosts to new physical location until ICX2 is unpopulated.
8. Power off ICX2 and put it in storage

Also, is anyone aware of if it's possible to stack using RJ45 Cat6 transceivers in the stacking ports or if I need to get a "proper" cable like this one? Brocade XBR-TWX-01.5 10G SFP+ DAC Twinax Cable
Your process looks good, but I'd use a regular uplink between the switches instead of stacking. Stacking adds unnecessary complexity
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,726
3,066
113
33
fohdeesha.com
I apologize for what sounds like the dumbest noob question ever.

What's the command to cleanly power down a 6450? I'm connected via serial admin port.

I searched the manuals then googled for an additional 20 minutes with no luck.

----

This USB to serial cable works great on my 6450 and it uses a FTDI chip to boot:
Asunflower 12 Ft FTDI USB to RJ45 Router Console Cable - RS232
There isn't one, it's a read only filesystem designed to have the power yanked unless it's in the middle of a firmware upgrade or something
 
  • Like
Reactions: 3nodeproblem

ccie4526

Member
Jan 25, 2021
82
53
18
So right now these guys are at $90, and I was able to get one at best offer of $80. They countered my $70 offer.
 

dennisp

New Member
Apr 1, 2021
18
13
3
So right now these guys are at $90, and I was able to get one at best offer of $80. They countered my $70 offer.
I have a stack of three of them from this seller in front of me right now, no issues with them. They came wiped with an old version of the OS, 7-something, no "bonus" licenses other than the 2 default stacking ports. This seller has been moving down from about 150, I got mine 2 weeks ago for 100/ea.
 

NV43

New Member
Jun 9, 2021
2
0
1
So right now these guys are at $90, and I was able to get one at best offer of $80. They countered my $70 offer.
I also have one of these on the way that I got for $80. Should be here Monday. It'll be my first real switch so I'm pretty excited to tinker with it.

I'm also interested in what fans others are using for a fan mod.
 

BecauseScience

New Member
Feb 3, 2016
19
2
3
Could you use the 6450-24P as a simple L2 switch as well?
You should buy an unmanaged L2 switch if that's you're use case. Some (most?) are fanless and there's no complex setup.

The 6450 has non-silent fans. I consider them fairly loud.

It takes time and effort to update the firmware and get them into a known state. Getting and keeping a TFTP server working is a pain in the butt on both Windows and Linux and that's only part of it.

You have to buy a special console cable for the setup also.