Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

ok, bought a 50M 12Fiber MPO cable on bay. I'm attaching a diagram it came with.

I also bought because I found one super cheap, a Systimax Instapatch panel thinking I could run a 50M MPO from our server room to the video suite and fan out from there.

I used a paper clip and flipped one end of one of the MPO Cables I have and plugged it in. if you look at the photo of the inside of the instapatch the first MPO input is going to Alpha side 1-12 which makes sense to me. With the flipped MPO though the lasers are lighting up fibers 1,2,3,4 and if I use the non-flipped as it came MPO cable it lights up 9,10,11,12

Looking at the back of the 6610 I'm trying the two right side 40GB ports (fan out ports) and I disabled stacking etc during setup. in the Web UI I see everything I have connected and I see those ports as 4 10GB ports.

If I plug a LC cable into 1,2 or 3,4 (or 9,10,11,12 with other cable) I get connected but no internet (in a windows system) from a solarflare card.

LC transmitters work fine in all the 10GB front ports, and I would have thought that the instapatch would light up 1,3,5,7 or 2,4,6,8 and give me 4 channels.

Is this instapatch made to do something else? I see there are different versions of it,. Any help is appreciated.

Oh, last question, should I just buy an MPO Breakout cable and coupler to go at other end?

EDIT: 9,10,11,12 with non flipped cable

yeah, I bought a bunch of these, they weren't wired for breakout, they're just designed to be on each end of an MPO cable and use it trunk a bunch of LC connections - it's not cabled for breakout at all, nor breakout with a reverse connector. I took the time to rewire mine by carefully popping each connector out the back of the LC couplers, it took some time but it works. I can't remember how I figured out the fan-out, but I had this saved somewhere for it (the cables inside it should match the coloring on this, I think I just used this as a guide for which color pairs make up an LC connection on breakouts):

 

[kate]

Hello! Thanks a ton for this whole thread and to everyone contributing, it’s an awesome resource. I just got an ICX 7250-48P that I’m really excited about and trying to get set up.

So far I haven’t been able to get connected to the console, though. I’m using a janky cable I made with an rsr232-to-usb breakout adapter and using the correct settings I’m clearly getting data, I can see the boot process, and am able to send keystrokes, but everything displayed is completely garbled. It looks like what I’d expect from an incorrect baud rate, but I’ve tried just about every different baud rate for the connection and nothing works. I confirmed with the seller that the console port was working and that they used it at 9600 to factory reset just before shipping it, so I’m not sure what else to try with my set up to get it working. I have proper cables on the way, that’s my next best hope. I think I’ve tried everything I’ve been able to find searching through this thread and in other places.

Anyways I tried hooking the management port into my router and it happily got an address from DHCP and I was able to telnet into it, so the switch is working. Is it possible to flash a new bootrom through telnet? Poking around the CLI it looks like it’s possible to set boot options, including tftp boot, so I was wondering if that was an option and if anyone‘s tried it. I get nervous doing an update like that without access to the console, though.

 

[thebwack]

What are the Systimax and LC cable SKUs?

If your patch cables work for regular SFP+ ports then they are A-B cables, meaning TX on one end point to RX on the other. Because your MTP cable has method B polarity if things don't work probably your Systimax is method A. Then you will need to convert the LC cable to A-A. That's why it's important to know the specifications of the breakout cassette through its SKU. Good LC patch cords can have their polarity changed in the field but some can't.

Generally, to make things work with duplex fiber, there needs to be an odd number of polarity B cables in the chain.

Fiber Polarity Basics

Polarity defines direction of flow, such as the direction of a magnetic field or an electrical current. In fiber optics, polarity is directional; light signals travel through a fiber optic cable from one end to the other. A fiber optic link’s transmit signal (Tx) at end of the cable must match...

www.flukenetworks.com

Make sure you're not mixing OS2 cassette with OMx MTP trunk or vice versa. Your MTP appears to be OM4?

Thanks! That's all helpful to wrap my head around fiber. Seeing as these instapatch are not super valuable I think I'll try re-wiring it first like @fohdeesha did, see if I can get it working. I like the idea of having this instapatch at the other end for scalability later if I need to add more seats, I'll just run another trunk.

I'm attaching the model # and the LC cables (OM3) I've been using for reference

@fohdeesha - thanks for that pic and diagram!

 

[thebwack]

Also for reference, here's a few transceivers that work great in my 6610 so far. I know the RJ45 10GB ones can be pricy the 6COM I just got on Amazon 2 for $88 ($44 each). IDK if that's amazing for 10GB RJ45 but they're working good.

https://www.amazon.com/dp/B08G1PHYYV?ref_=cm_sw_r_cp_ud_dp_QA18YDT8NPKDV4H07ZQF

My Solarflare cards in my PCs have been way more picky about transceivers

 

[adman_c]

thanks for the donation! as for your routing issues, you can ensure the switch itself is routing correctly by running "ping 192.168.0.254 source 192.168.10.254" - I would be really surprised if that doesn't work. next step would be having a PC in 192.168.0.0/24 with its gateway set to 192.168.0.254 try and ping 192.168.10.254 - if that works (it should), then the switch is working fine, I'd imagine the rest of your issues are related to your firewall having gateways/IP interfaces in all these vlans that the switch is trying to route but hard to say from glancing at your config

Apparently all I had to do was donate to appease the networking gods! Well, not actually, but your bit of troubleshooting advice convinced me that it probably wasn't the switch, so I eliminated that from consideration. I thought I had already eliminated the firewall by previously yanking its connection from the switch and still seeing the problem. That meant it was probably on the client end. I'm now pretty sure my problem was
the default route on my proxmox servers--they were pointed at the firewall, so I think this was causing issues when I had a different gateway identified on my workstation. And wouldn't you know, they only machines I was testing across vlans were these proxmox boxes. I changed the default routes to point at the switch ip, and everything is working. Linespeed routing between my trusted VLANs. Hot diggity!!

 

[Serhan]

yeah, I bought a bunch of these, they weren't wired for breakout, they're just designed to be on each end of an MPO cable and use it trunk a bunch of LC connections - it's not cabled for breakout at all, nor breakout with a reverse connector. I took the time to rewire mine by carefully popping each connector out the back of the LC couplers, it took some time but it works. I can't remember how I figured out the fan-out, but I had this saved somewhere for it (the cables inside it should match the coloring on this, I think I just used this as a guide for which color pairs make up an LC connection on breakouts:

View attachment 22912

View attachment 22913

Why is the fiber stuff so complicated?

 

[fohdeesha]

Hello! Thanks a ton for this whole thread and to everyone contributing, it’s an awesome resource. I just got an ICX 7250-48P that I’m really excited about and trying to get set up.

So far I haven’t been able to get connected to the console, though. I’m using a janky cable I made with an rsr232-to-usb breakout adapter and using the correct settings I’m clearly getting data, I can see the boot process, and am able to send keystrokes, but everything displayed is completely garbled. It looks like what I’d expect from an incorrect baud rate, but I’ve tried just about every different baud rate for the connection and nothing works. I confirmed with the seller that the console port was working and that they used it at 9600 to factory reset just before shipping it, so I’m not sure what else to try with my set up to get it working. I have proper cables on the way, that’s my next best hope. I think I’ve tried everything I’ve been able to find searching through this thread and in other places.

Anyways I tried hooking the management port into my router and it happily got an address from DHCP and I was able to telnet into it, so the switch is working. Is it possible to flash a new bootrom through telnet? Poking around the CLI it looks like it’s possible to set boot options, including tftp boot, so I was wondering if that was an option and if anyone‘s tried it. I get nervous doing an update like that without access to the console, though.

so, the times I've seen this in the past (garbled output), it was the USB adapter itself. You can try cold-power cycling your PC completely, I've seen that fix this issue on cheap USB <> serial adapters, or if you have another usb <> rs232 cable you can try that too. If you have some other serial device, you could try using this usb adapter on that as well and see if you get garbled text output or not - obviously if the output is fine with the adapter on a different serial device, the issue isn't the usb adapter

 

[kate]

I already tried a different adapter that I’ve used in the past and that I know works, and it had the same output. They had totally different chips, too—one FTDI, one Prolific. It’s strange. I’m not sure why one of these rj45 to usb “Cisco console cables” (which probably has one of the same chips in it) would work any better short of general cable+signal quality, but I’ve read some posts where people with this problem resolved it by using one of those instead.

I saw a pinout for an rj45 serial cable that has RTS tied to ground and I tried that too without success (not surprising, the switch isn’t using flow control after all). I can’t figure out what else would be different.

 

[dasbooter]

I feel bad but is there anyway somebody could add to the sticky a serial console cable buying guide. All cables are not available in different places or are for whatever reason more expensive. I have looked through the thread and some of these cables would ship from overseas from where I am at. If somebody who knows what is important or what to look for could maybe get it stickied near the top it would really help.

I saw somewhere how fohdeesha got pretty frustrated with such questions and I sympathize as it doesn't really contribute to the knowledge base but as a noobie I dont know cables. This one is available to me is highly rated and is shipped from my continent. I have tried to match the pin outs but on the cable I am linking one of the grounds pin 5 is listed as data carrier detection. Will this matter?

Here is the cable in question Link: Amazon cable





Edit: This cable worked fine...its probably not rocket science. The factory command in the instructions didn't work for me because apparently my OS was a very old version but a quick search on this thread showed me how to reset to default and I was off and running. Understanding VLAN's and the LVL 3 capabilities is a whole other level to me compared to just running it as a dumb switch. It is intimidating and fun to try and understand this stuff as a just a guy who likes to fiddle with technology. I will keep at it. For now I have 48 port switch capable of powering my ubiquiti AP...Thanks so much fohdeesha

 

[adman_c]

Hello! Thanks a ton for this whole thread and to everyone contributing, it’s an awesome resource. I just got an ICX 7250-48P that I’m really excited about and trying to get set up.

So far I haven’t been able to get connected to the console, though. I’m using a janky cable I made with an rsr232-to-usb breakout adapter and using the correct settings I’m clearly getting data, I can see the boot process, and am able to send keystrokes, but everything displayed is completely garbled. It looks like what I’d expect from an incorrect baud rate, but I’ve tried just about every different baud rate for the connection and nothing works. I confirmed with the seller that the console port was working and that they used it at 9600 to factory reset just before shipping it, so I’m not sure what else to try with my set up to get it working. I have proper cables on the way, that’s my next best hope. I think I’ve tried everything I’ve been able to find searching through this thread and in other places.

Anyways I tried hooking the management port into my router and it happily got an address from DHCP and I was able to telnet into it, so the switch is working. Is it possible to flash a new bootrom through telnet? Poking around the CLI it looks like it’s possible to set boot options, including tftp boot, so I was wondering if that was an option and if anyone‘s tried it. I get nervous doing an update like that without access to the console, though.

This might be barking up the wrong tree, but have you tried a different usb to serial adapter? I made one out of a keystone and an old mini usb cable I had lying around per this post: https://forums.servethehome.com/ind...erful-10gbe-40gbe-switching.21107/post-316704
Took all of 5 minutes and worked first go round. I also have an unused Brocade oem mini usb serial cable if you’d like it.

 

[kate]

This might be barking up the wrong tree, but have you tried a different usb to serial adapter? I made one out of a keystone and an old mini usb cable I had lying around per this post: https://forums.servethehome.com/ind...erful-10gbe-40gbe-switching.21107/post-316704
Took all of 5 minutes and worked first go round. I also have an unused Brocade oem mini usb serial cable if you’d like it.

Yeah, I did. If you look at my last post up the page I wrote a little more about the other adapter I tried.

Thank you for that offer! I already ordered an oem cable that should be getting here Monday, though . I’m on my second cannibalized mini usb cable now so hopefully one of these other cables will work. If not I’m back to figuring out how to update the bootrom with only telnet access and no console.

 

[epicurean]

How do you change the ip address of a icx6450? I tried to disable the current fixed IP , to use the DHCP assigned with
"ip dhcp-client enable"
but it does not work.

 

[xtantaudio]

Hello, I am looking for some assistance with my Brocade ICX6650. I have followed the documentation on the first page from Fohdeesha in order to get the license and configuration done on the switch and I also purchased 3 of the 649281-B21 (originally HPE) cards and completed the firmware update on those units.

The 649281-B21 cards work when connected directly to each other, the link comes up between my test PC's. When I plug the QSFP Cable from one of the test PC's into the ICX6650, the link light never comes on, the NIC shows no cable connected. any suggestions ? I am REALLY hoping that the brocade switch is not bad on the 4 40Gb ports.

I have confirmed that the 2 40Gb Breakout ports DO work.

I did run the fpod-40g-enable group 1, 2 and 3 commands to "turn on" the 40Gb ports.

I have included a show ver of the switch in case it helps. It looks to me like I have everything right, so I am guessing I am missing something stupid here. If any more information is needed, I will provide it, I would REALLY like to get this working in my home lab.

 

[danb35]

If you'd followed the guide linked in the OP to set it up initially, it should be:

enable
configure terminal
interface ve 1
no ip address 192.168.1.50/24 # substitute whatever address you'd configured
exit
[code]
Then set a new address, or enable DHCP (though I don't think the latter would be recommended).

 

[Lone Wolf]

How do you change the ip address of a icx6450? I tried to disable the current fixed IP , to use the DHCP assigned with
"ip dhcp-client enable"
but it does not work.

Did you run 'factory set-default'? I believe when I first got my 6450 and ran that, it was able to grab a DHCP address by default. The guide then tells you to disable it. You could skip that part if you are wanting to do a static DHCP assignment.

 

[epicurean]

Did you run 'factory set-default'? I believe when I first got my 6450 and ran that, it was able to grab a DHCP address by default. The guide then tells you to disable it. You could skip that part if you are wanting to do a static DHCP assignment.

No, I did not run factory set-default . Would that set EVERYTHING to default, which is not my intention. I just want to be able to change the IP. Yes, I can also do a change in static IP assignment.

 

[Lone Wolf]

No, I did not run factory set-default . Would that set EVERYTHING to default, which is not my intention. I just want to be able to change the IP. Yes, I can also do a change in static IP assignment.

Yes, if you already have a lot of configuration in the switch it would make sense not to do a factory reset. I had assumed you had just received the switch and were setting it up from scratch, my bad.

I'm not sure how to set it back to dhcp to be honest. I would've thought your command of "ip dhcp-client enable" would do the trick.

 

[kpfleming]

The DHCP client won't attempt to obtain an address unless there are *no* addresses assigned to any interfaces in the switch. Please ensure that you have used `no ip ...` to remove all of them.

 

[kate]

Ok! I got the console working using an OEM Brocade Mini USB to DB9 cable from eBay, a DB9 to RJ45 adapter with configurable pinout, and an RJ45 to USB console cable. The RJ45 to USB console cable has the exact same FTDI chip as the adapter I was previously using so I think the garbled data issue I was having probably just comes down to signal quality. I‘d expect a DIY Mini USB to RJ45 keystone cable would work with this console cable like others have reported. Here are the pinouts I used, and links to products:

https://www.amazon.com/dp/B01AFNBC3K
RJ45 Pins :: Console Cable Pinout
1 :: RTS
2 :: DTR
3 :: TX
4 :: GND
5 :: GND
6 :: RX
7 :: DSR
8 :: CTS

https://www.amazon.com/dp/B00066HL3W
RJ45 Pins :: Adapter Colors
1 :: blue
2 :: orange
3 :: black
4 :: red
5 :: green
6 :: yellow
7 :: brown
8 :: white

DB9 Pins :: Adapter Colors
1 :: empty
2 :: black
3 :: yellow
4 :: orange
5 :: red+green
6 :: brown
7 :: white
8 :: blue
9 :: empty

The console cable has two grounds, so I had to tie together two of the DB9 adapter pins together like this:

 

[kate]

In case anybody else runs into this, the Brocade web management interface has a maximum password length of 31 characters ‍*facepalm*. It seems like it truncates anything longer than that, even if it's a valid password everywhere else on the system.