Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[dontwanna]

Also, while waiting for your 6610s, go grab a console cable, if you don't already have one. You'll need it.

Thanks, I don't have a console cable yet. I'm looking at the pictures of those 6610-48p switches, and can't see any weird ports on them (like usb, serial etc), does it mean that I'm gonna be plugging the console cable into one of those rj45 ports in the back? So the console cable is going to be rj45<=>rj45 then? Can't I just make one by myself then, with a crimp tool?

 

[darthray]

Thanks, I don't have a console cable yet. I'm looking at the pictures of those 6610-48p switches, and can't see any weird ports on them (like usb, serial etc), does it mean that I'm gonna be plugging the console cable into one of those rj45 ports in the back? So the console cable is going to be rj45<=>rj45 then? Can't I just make one by myself then, with a crimp tool?

That's right. You'll be plugging the cable onto the RJ45 port on the right (closest to the 40Gbps ports, the one with the ones-and-zeroes label under it). Those console cables are going USB <-> RJ45. The USB portion has a USB-to-serial chip embedded in them with the correct wiring going to the RJ45 end.

You can probably make one yourself but you'll need a serial port or USB-serial converter anyways. They are using the same RJ45 connector as network cables do, but the wiring and protocol running on that port is different from Ethernet.

 

[fohdeesha]

I did the clear:

SSH@ICX7250-48P Router(config)#clear cable-diagnostics tdr 1/2/1

Then I went out of conf t

SSH@ICX7250-48P Router#phy cable-diagnostics tdr 1/2/1
        This feature is only supported when the interface is configured for Auto-Negotiation

I tried it with 1/2/8 (not part of a lag)

SSH@ICX7250-48P Router#phy cable-diagnostics tdr 1/2/8
        This feature is only supported when the interface is configured for Auto-Negotiation

I might be close?

1/2/1 is part of a lag, 1/2/8 is plugged into a unifi 1gb optic so its slowed to 1gb

next I turned off a part of my lag:

lag uplink
disable e 1/2/2

long story short, disabled a port from each lag, tried again, still didnt work.

Any other ideas?

TDR / cable phy is only for native copper ports

 

[dontwanna]

You'll be plugging the cable onto the RJ45 port on the right (closest to the 40Gbps ports, the one with the ones-and-zeroes label under it). Those console cables are going USB <-> RJ45. The USB portion has a USB-to-serial chip embedded in them with the correct wiring going to the RJ45 end.

Wait, do I need a "usb<=>rj45" console cable, or an "rj45<=>rj45" console cable? It's just from reading the fohdeesha's update guides I've got the impression that one end of the cable goes into the switch (and that's rj45, at least in case of icx6610), and the other end is connected to my network (so also rj45):

Connect to the switches serial/console port using a program like Putty (9600 8N1), and connect the dedicated management port to your network (do not use a "normal" port).

So I thought I'll just need to assemble the wires in some specific order and crimp a regular cat5 cable. Google found me this, which suggested it's as simple as swapping some pins and then crimping as usual:

What is the name of the cable (RJ45 to RJ45) that can be used to connect from a KVM to Cisco serial port?

As everybody knows Cisco's console ports are RJ45. However, the cable they come bundled with is DB9 to RJ45. We have a KVM that allows managing network devices via a serial port but the port on the...

serverfault.com

That amazon link in your post lists all kinds of cables, there are rj45<=>rj45 there, rj45<=>usb, rj45<=>db9, so that got me confused. Do I need an rj45<=>rj45 serial cable, like this one? https://www.amazon.com/Cisco-Console-Rollover-Cable-RJ45/dp/B004Z9YG5W/
I don't see any chips embedded in the pictures.

Lol, I'll probably have a hard time learning these switches if I'm so dumb that I can't even figure out the serial cable thing.

 

[infoMatt]

@dontwanna The serial cable is... well, serial You'd have to connect it to a serial port of you computer, not on the network (that's the management interface, a NIC essentially bound to the switch CPU); if you have a modern computer that doesn't have a serial port you can use a USB-DB9 adapter or you can buy some "console cable" that has the adapter chip inside the USB connector and on the other end has a "rollover" RJ45.

 

[darthray]

Wait, do I need a "usb<=>rj45" console cable, or an "rj45<=>rj45" console cable? It's just from reading the fohdeesha's update guides I've got the impression that one end of the cable goes into the switch (and that's rj45, at least in case of icx6610), and the other end is connected to my network (so also rj45):

So I thought I'll just need to assemble the wires in some specific order and crimp a regular cat5 cable. Google found me this, which suggested it's as simple as swapping some pins and then crimping as usual:

What is the name of the cable (RJ45 to RJ45) that can be used to connect from a KVM to Cisco serial port?

As everybody knows Cisco's console ports are RJ45. However, the cable they come bundled with is DB9 to RJ45. We have a KVM that allows managing network devices via a serial port but the port on the...

serverfault.com

That amazon link in your post lists all kinds of cables, there are rj45<=>rj45 there, rj45<=>usb, rj45<=>db9, so that got me confused. Do I need an rj45<=>rj45 serial cable, like this one? https://www.amazon.com/Cisco-Console-Rollover-Cable-RJ45/dp/B004Z9YG5W/
I don't see any chips embedded in the pictures.

Lol, I'll probably have a hard time learning these switches if I'm so dumb that I can't even figure out the serial cable thing.

Sorry for the confusion. I hadn't noticed that RJ45 <-> RJ45 cable was part of that list.

Here's an example of the cable you'll need (that's the one I have):

AmazonSmile: OIKWAN Console Cable,USB Console Cable, USB to RJ45 Console Cable with FTDI chip Compatible with Cisco, Huawei,HP,Arista,Opengear,Aruba，Juniper Routers/Switches for Laptops in Windows, Mac, Linux: Computers & Accessories

It's an USB <-> RJ45 cable. It has an embedded USB-to-serial chip on the USB side of the cable and it has the correct pinout on the RJ45 side. There exist many ways to connect to equipment over serial, but this is the one that will work with the 6610s.

 

[LodeRunner]

Sorry for the confusion. I hadn't noticed that RJ45 <-> RJ45 cable was part of that list.

Here's an example of the cable you'll need (that's the one I have):

AmazonSmile: OIKWAN Console Cable,USB Console Cable, USB to RJ45 Console Cable with FTDI chip Compatible with Cisco, Huawei,HP,Arista,Opengear,Aruba，Juniper Routers/Switches for Laptops in Windows, Mac, Linux: Computers & Accessories

It's an USB <-> RJ45 cable. It has an embedded USB-to-serial chip on the USB side of the cable and it has the correct pinout on the RJ45 side. There exist many ways to connect to equipment over serial, but this is the one that will work with the 6610s.

Oh, that's lovely, I might have to get one of those; I currently carry around a regular USB to DB9 dongle and my Cisco cable. I do have some switches that use different pin outs though, so have DB( as the common element is handy.

 

[Vesalius]

@itronin
i guess i didn't think it was relevant? my fw is Vyos.

and you are spot on. no, i don't have a NAT rule for 172, only 10.0.0.0/16.

added the nat rule and problem solved!

vyos@vyos# show nat
source {
     rule 1 {
         outbound-interface eth0
         source {
             address 10.0.0.0/16
         }
         translation {
             address masquerade
         }
     }
     rule 2 {
         outbound-interface eth0
         source {
             address 172.16.0.0/30
         }
         translation {
             address masquerade
         }
     }
}

Hey @jht3 I am interesting in testing out VyOS on the edge as a firewall with a transit vlan and my 6450 L3 switch hosting multiple Internal vlans. I've not found any howtoo's specifically for setting this up in the VyOS cli. Do you know of any or have some basic tips beyond the nat masquerade rule here?

 

[richtj99]

So I am using a 6 strand OM3 fiber cable & am looking for another strand. I was looking at:

Pre-Terminated Fiber Cables – Discount Low Voltage

Save time during on-site optical fiber cable installations by ordering custom pre-terminated fiber cable assemblies from Discount-Low-Voltage.com. We can pre-terminate a variety of single mode and multimode fiber cables along with video snake cables for your networking applications.

www.discount-low-voltage.com

preterminated OM4 - they have 12 & 24 strand. I think 12 would be fine (though I thought 6 was fine in 2012). He said a 24 strand should fit OK in a 1" conduit but the conduit has been in the ground for 30 or so years & I have no idea if it will be OK.

I had my OM3 unused due to cost for termination until 3 years ago so this is appealing.

He did say that singlemode is more popular though.

Its a home network so I dont even really have a good reason on why I want to get my 30gb lag to 40gb beyond I dont have enough strands.

So the question is:

1. Any thoughts on a pretermintated 12 strand LC to LC?
2. As I am buying a new cable, should i consider other formats? SC, singlemode, etc?

I think my distance is less than 250 feet'ish.

 

[LodeRunner]

At that distance, I can't think of any reason to do SM rather than MM. You can do up to 100 Gb (with a bidi) over a single pair of MM OM4 up to 100 meters.

 

[NateS]

So the question is:

1. Any thoughts on a pretermintated 12 strand LC to LC?
2. As I am buying a new cable, should i consider other formats? SC, singlemode, etc?

If your main concern is fitting it down an existing conduit, I'd go with MPT/MPO terminated cables, and get fanouts that plug into them, rather than trying to stuff the full 12xLC fanout down the conduit.

 

[BecauseScience]

Why is 7250-48P so much more desirable than 6610-48P?

I'm trying to score a 7250-48P to run as the core switch from my home but I'm encountering ridiculous prices even using offers. Meanwhile, there are loads of (relatively) cheap 6610's available. It's kind of hard to believe a multi-hundred dollar difference is down to power usage and noise. Is there some other advantage I'm missing?

Wondering if I should buy a 6610 instead and save $$$...

 

[NateS]

Why is 7250-48P so much more desirable than 6610-48P?

I'm trying to score a 7250-48P to run as the core switch from my home but I'm encountering ridiculous prices even using offers. Meanwhile, there are loads of (relatively) cheap 6610's available. It's kind of hard to believe a multi-hundred dollar difference is down to power usage and noise. Is there some other advantage I'm missing?

Wondering if I should buy a 6610 instead and save $$$...

There are some other differences too: the 7250 runs newer FW, is linux-based instead of a custom OS, possibly more compatible with newer brocade/ruckus gear, and probably most importantly for Ebay, the licensing is honor-based rather than enforced, so people who are not on this forum can use all the 10g ports without having to track down and pay for an additional license.

The 6610 also has more ports (including some 40g), so if the added noise and power consumption can be tolerated it's definitely the better buy. For my network, I'm using 6610s as the core switches in a closet where I don't care about noise, and 6450s anywhere noise is a concern.

 

[richtj99]

I have a cat6 following my existing fiber - results are:

Port    Speed   Local pair      Pair Length     Remote pair     Pair status
----    -----   ----------      -----------     -----------     -----------
1/1/14    1G    Pair A          50-80M          Pair B          terminated
                Pair B          50-80M          Pair A          terminated
                Pair C          50-80M          Pair D          terminated
                Pair D          50-80M          Pair C          terminated

- still not sure what size cable to get as a replacement - 150 feet or 240 feet?

I am also seeing this message repeating every few seconds & I am not sure how to shut it off?

port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.
port 1/2/8 is not capable of digital optical monitoring.

 

[klui]

I am also seeing this message repeating every few seconds & I am not sure how to shut it off?

port 1/2/8 is not capable of digital optical monitoring.

Try using the clear command.

 

[Fallen Kell]

Oh, that's lovely, I might have to get one of those; I currently carry around a regular USB to DB9 dongle and my Cisco cable. I do have some switches that use different pin outs though, so have DB( as the common element is handy.

If you have a serial cable setup that works with CISCO it will also work with the 6610. The reason for the confusion is because most people on this thread do not also have CISCO gear and/or proper serial support. So the general recommendation are the USB serial RJ45 connectors that are linked by most people. You don't need them if you have computers with serial ports, or an adapter that supports a DB9 or DB25 and appropriate adapters to RJ45 using the CISCO pin outs. Again, most people now don't know what that is in the first place...

Also as stated by the others, a properly wiped switch will be essentially booting to a stage that you can load the new firmware. Read the guide posted on the first page and it will walk you through loading the firmware which includes the OS. As mentioned, you will need a serial connection initially since your switch does not have an active working OS. After you have an OS, you can setup the network management port (I connected mine to a port that I set to a management VLAN, but ideally, you connect this to a different switch since if the switch is not responding properly, you might not be able to connect to the management port that way).

Be prepared for the 6610 to be EXTREMELY LOUD until after you load a proper OS. It will run all fans at 100% in this state because it doesn't have the smarts/controls at this point to sense temperatures and adjust speeds lower. So don't think it is that you have a broken switch and/or something that is way to loud for your use case until after you get the OS loaded (usually 1-2 minutes after booting).

 

[jht3]

Hey @jht3 I am interesting in testing out VyOS on the edge as a firewall with a transit vlan and my 6450 L3 switch hosting multiple Internal vlans. I've not found any howtoo's specifically for setting this up in the VyOS cli. Do you know of any or have some basic tips beyond the nat masquerade rule here?

its a very basic setup, if you follow the vyos documentation. configure your interfaces, add a static route, add NAT rules, configure firewall. and any other services you want or need. i've been running vyos as my main fw/router for 4+ years now after growing weary of pfsense and its complete lack of a CLI, missing support for VLANs on Xenserver, etc.
or i can send you my basic config if that would help get you started.

 

[Vesalius]

its a very basic setup, if you follow the vyos documentation. configure your interfaces, add a static route, add NAT rules, configure firewall. and any other services you want or need. i've been running vyos as my main fw/router for 4+ years now after growing weary of pfsense and its complete lack of a CLI, missing support for VLANs on Xenserver, etc.
or i can send you my basic config if that would help get you started.

That would be great thanks.

 

[coxhaus]

So I'm trying to figure out what went wrong trying to replicate kapone's post but can't seem to find the culprit of it.

First of all, you'll have to forgive me if I don't make too much sense, since English is not my first language but always try to do my best to communicate with it.

Right now my home gear network consists of the following:

PC Engines APU2 - PfSense
Cisco SG300-52 L3 enabled
ICX7250-48 L3 10Gb license (Thanks @fohdeesha)
ICX6610-48 Fully licensed too (1 PSU rev3, 1 FAN) - Not in use, too loud after a few minutes (maybe will sell it since PSU and fan will cost me as much as another unit)

What I am trying to achive?

Have a native L3 network after a while since FW was handling the VLANs as a Router on a Stick approach, right now it can't handle inter-vlan 1Gb network traffic after doing so for a "some time", since I have the gear that can do all the L3 at its core.

My core networking/services/servers (more APU2s) are on a 12U startech rack is hangin on my apartment's entrance corner and my office is a few meters away cabled with 6 CAT6 ethernet drops, maybe more, but don't tell my wife.

My first approach was to use the ICX7250 as a Core switch for my place on the aforementioned startech rack, and the ICX6610 for my 24U rack with 6 SM servers all with 10Gb NICs and a 40Gbps NIC on my main ESX/NAS server, short long story, as I mentioned the switch is too loud to have it 24/7 on my apartment I didn't even setup up correctly on both ends. - Currently discarded until further notice or until christmas bonus. lol

Then I tried to use the SG300 as my Core L3 switch and the ICX7250 as my rack switch with inter-VLAN routing on my main LAN, this worked "well" can access the SVIs, setup the firewall rules, static routes both the switches and FW can see and communicate, everything was ok but then I realized L3 routing was performed at the Cisco so 10Gb traffic was limited to 1Gb as you might guessed. - Discarded for the time being, maybe will get back to this if could find the routing issue on the ICX.

So third attempt, since the APU has 3 ethernet ports (WAN, LAN, OPT1) tried the Kapone's post guideline, using the OPT1 which was unused to connect one of the cable drops to my office directly to the ICX, created a /30 transit VLAN, gateway, static route on FW, static route on SW, FW rules and what not, but can't communicate from my main home network to the VLANs associated on the ICX, I'm still using the SG300 on my LAN, nothing has done yet in there (No L3 switching, VLANs SVIs, nothing really, just a dumb SW ATM), was thinking to use it as an access L2 SW for the VLANs needed for the APU2s VMs and LXC containers using a second drop back from my rack to the startech rack.

So with all this, which approach will be the best to execute, and more important, am I missing something on my config, steps that might be overlooked?

Basically TDLR;

Need to setup a L3 network using Cisco SG300, ICX7250 and pfsense, but have failed doing so.

Here's the precious data if needed:



If you need more info I'd gladly provide it.

I was using a Cisco SG300-28 switch in L3 10 years ago. They are a nice switch but old. I have changed the last couple years and use a Cisco SG350-10P now. I have a Cisco SG350X-24 but the fans bother me. I used pfsense for about a year and switched to a Cisco RV340 router as all the DHCP and local routing was on my L3 switch. Pfsense updates did not work as they usually broke something and I got tired of testing my router over and over. So, I dropped pfsense. I ran Untangle as a UTM behind my Cisco router for 4 or 5 years. Maybe that is why I did not appreciate pfsense as I ran Untangle as a UTM device before pfsense.

Using an L3 switch is still better than L2 if you are using network VLANs. I always assign a network to a VLAN. I use several in my home.
Using a Cisco SG300 switch in L3 will require you to turn on L3 mode. This is only required on these older Cisco switches. The Cisco SG350 switches do not have a mode.

 

[richtj99]

Thats pretty interesting - I am a bit more confused -

MPT/MPO seems like a great option - I am not sure if I can use it in my conduit - it goes under my driveway & is fairly old - I think there is some water in it - can the MPT/MPO be used outdoors (in a conduit that could be wet?)

What's the difference between a preterminated fiber 12 strand OM3 & a 12 strand OM3 MTP/MPO? Seems like it is the same thing?

This near switch #1:

MTP Fiber Assembly - 12 Strand OM3 Plenum, MTP Male to Generic LC Duplex

Save time & money with MTP/MPO installations. Pull one cable rather than 12+. Discount Low Voltage has the MTP/MPO cable you need now at wholesale prices. With MTP cables, it's possible to run a single cable that automatically terminates up to 72 sepa

www.discount-low-voltage.com

This through the conduit:

Pre-Terminated Fiber Cables – Discount Low Voltage

Save time during on-site optical fiber cable installations by ordering custom pre-terminated fiber cable assemblies from Discount-Low-Voltage.com. We can pre-terminate a variety of single mode and multimode fiber cables along with video snake cables for your networking applications.

www.discount-low-voltage.com

This near switch #2:

MTP Fiber Assembly - 12 Strand OM3 Plenum, MTP Male to Generic LC Duplex

Save time & money with MTP/MPO installations. Pull one cable rather than 12+. Discount Low Voltage has the MTP/MPO cable you need now at wholesale prices. With MTP cables, it's possible to run a single cable that automatically terminates up to 72 sepa

www.discount-low-voltage.com

I guess the only downside might be damaging the cable when plugging it in?

If your main concern is fitting it down an existing conduit, I'd go with MPT/MPO terminated cables, and get fanouts that plug into them, rather than trying to stuff the full 12xLC fanout down the conduit.

Try using the clear command.

Thank you! Worked perfect!