Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Gamble]

Hi Guys, trying to get my cabling in order before my new (to me) switches arrive. I want to use two of the 40gb ports on the back for stacking. The switches will be deployed about 150 feet apart. What's the correct way to cable this? My research leads me to believe I will need:

• 4x Brocade XBR-000232 QSFP transceivers
• 2x MTO/MTP cables

If that sounds right, what about the connectors for MTP, should they be type A straight through or type B cross-over? What about female or male, or does that not matter in this case? Just a little confused and want to avoid ordering the wrong cables, thanks!

 

[fohdeesha]

those transceivers will work, but you need reversed / type B / key up to key up fiber:

https://i.imgur.com/ZtbUv2Q.jpg

the MPO/MPT connectors need to be female, eg no pins sticking out (the pins are in the optic)

You're going to pay quite a bit for this, places like fiberstore do not sell MPO/MTP anywhere near this length, and 99% of the listings on ebay are key up to key down, which will not work. On top of that, remember that with mtp/mpo, for a single cable, you're actually buying 6 pairs of fiber. It's gunna be a few hundred dollars

this is the cheapest I could find that explicity states it's type B (flipped) 50M 12 STRAND MTP MPO OM3 50u 50/125 MICRON FIBER OPTIC PATCH CABLE 10GB AQUA | eBay

If you're feeling adventerous, you can also buy a straight through cable which is much cheaper, like: MTP Female 50M Interconnect Cable Assembly, Flat Ribbon, Plenum Rated, 12-Fibers | eBay

then use this method to pull the key housing off one end, and roll it over/flip it. DO NOT pull off the inner housing/spring clip area like he starts to do at 2 minutes, since you only need to flip the outer housing and are not changing the gender (you're only changing the polarity)

 

[Gamble]

those transceivers will work, but you need reversed / type B / key up to key up fiber:

https://i.imgur.com/ZtbUv2Q.jpg

the MPO/MPT connectors need to be female, eg no pins sticking out (the pins are in the optic)

You're going to pay quite a bit for this, places like fiberstore do not sell MPO/MTP anywhere near this length, and 99% of the listings on ebay are key up to key down, which will not work. On top of that, remember that with mtp/mpo, for a single cable, you're actually buying 6 pairs of fiber. It's gunna be a few hundred dollars

this is the cheapest I could find that explicity states it's type B (flipped) 50M 12 STRAND MTP MPO OM3 50u 50/125 MICRON FIBER OPTIC PATCH CABLE 10GB AQUA | eBay

If you're feeling adventerous, you can also buy a straight through cable which is much cheaper, like: MTP Female 50M Interconnect Cable Assembly, Flat Ribbon, Plenum Rated, 12-Fibers | eBay

then use this method to pull the key housing off one end, and roll it over/flip it. DO NOT pull off the inner housing/spring clip area like he starts to do at 2 minutes, since you only need to flip the outer housing and are not changing the gender (you're only changing the polarity)

Thanks for the response, very much appreciated! I hear you on the MPO cost, was estimating it to be around $350ish for cabling. If there's a cheaper way I'm definitely all ears and open to suggestions on how to best do this.

 

[BackupProphet]

You can ask fiberstore about this length, they do customization.

 

[fohdeesha]

Thanks for the response, very much appreciated! I hear you on the MPO cost, was estimating it to be around $350ish for cabling. If there's a cheaper way I'm definitely all ears and open to suggestions on how to best do this.

If it were me, I would pick these up for 80 bucks MTP Female 50M Interconnect Cable Assembly, Flat Ribbon, Plenum Rated, 12-Fibers | eBay

and using the paperclip method in the video, remove the key housing and rotate it 180 degrees on one end of the cable to turn it into a Type B. But I am cheap and adventurous (see: this entire thread)

If you don't feel comfortable doing that, you can spend some more $$ on a Type B cable from factory, like backupprophet mentioned FS will do custom orders. From past experience a 50 meter type B MPO cable will probably be around $150 before shipping for one cable

 

[Gamble]

If it were me, I would pick these up for 80 bucks MTP Female 50M Interconnect Cable Assembly, Flat Ribbon, Plenum Rated, 12-Fibers | eBay

and using the paperclip method in the video, remove the key housing and rotate it 180 degrees on one end of the cable to turn it into a Type B. But I am cheap and adventurous (see: this entire thread)

If you don't feel comfortable doing that, you can spend some more $$ on a Type B cable from factory, like backupprophet mentioned FS will do custom orders. From past experience a 50 meter type B MPO cable will probably be around $150 before shipping for one cable

I did look at FS and compared to the cable you linked. FS would be about double the cost, so now I have to ask myself if I'm feeling brave. Definitely seems worth the effort to give the paperclip method a try.

 

[K D]

VLAN is enabled in the Supermicro board (VLAN 55). All VLANs are defined in the Unifi configuration. I have a dhcp reservation in my USG for the Supermicro board. The board gets the pre-defined IP address. I have a bunch of devices set this way.

I connected a 1/2/2 on a brocade 6450 to a 10G SFP+ port on the Unifi with a DAC. Also connected a few other devices (on the default vlan).
vlan 55 tag eth 1/1/20 on the brocade and connected the SuperMicro to the port.

I was able to get to all the devices on the default vlan from any where. The supermicro is not reachable from outside the switch.

When I tried tvlan 55 ag eth 1/2/2, I lost connectivity to the switch. How do I set up 1/2/2 as a trunk port so that all the vlans are passed between the 2 switches correctly?

 

[fohdeesha]

it sounds like you're trying to run both untagged (vlan 1) and tagged (all other tagged vlans) traffic over the same port. for that you'll need to enable dual-mode on the port. By default, when you tag a port in a tagged vlan, it takes it out of the default untagged vlan (which is why you lost access). You need to tell the port to operate in dual-mode (eg stay in the untagged default vlan even if it's in tagged vlans). For dual-mode command to work it needs to be tagged in a non-default vlan first

enable
conf t
vlan 55
tagged e 1/2/2
interface e 1/2/2
dual-mode 1
#it'll go back to passing untagged vlan 1 traffic as well
#then tag it in any other tagged vlans you want it in
vlan 60
tagged e 1/2/2
vlan 70
tagged e 1/2/2
etc

 

[K D]

Argh. Rolled a chair over the console cable and now it doesnt work. . Ordered a couple on Amazon that should get here by Tuesday.

Note to self : Always keep spares.

Atleast the original config came up when i rebooted and everything else is working fine.

 

[nthu9280]

those transceivers will work, but you need reversed / type B / key up to key up fiber:

https://i.imgur.com/ZtbUv2Q.jpg

the MPO/MPT connectors need to be female, eg no pins sticking out (the pins are in the optic)

You're going to pay quite a bit for this, places like fiberstore do not sell MPO/MTP anywhere near this length, and 99% of the listings on ebay are key up to key down, which will not work. On top of that, remember that with mtp/mpo, for a single cable, you're actually buying 6 pairs of fiber. It's gunna be a few hundred dollars

this is the cheapest I could find that explicity states it's type B (flipped) 50M 12 STRAND MTP MPO OM3 50u 50/125 MICRON FIBER OPTIC PATCH CABLE 10GB AQUA | eBay

If you're feeling adventerous, you can also buy a straight through cable which is much cheaper, like: MTP Female 50M Interconnect Cable Assembly, Flat Ribbon, Plenum Rated, 12-Fibers | eBay

then use this method to pull the key housing off one end, and roll it over/flip it.

Would something like this work for this purpose by joining two mpo cables?

Lot of 8 Panduit FAMPOBLA Optical Fiber Adapter Connector OM3 Coupler | eBay

Edit: Fixed the link. trying to put links on mobile device doesn't always work for me.

 

[mb300sd]

Had a pretty good PM session with @fohdeesha about using the 6610 to flash optics. He pointed out that the manual explicitly says that 40G BiDi optics do not work, even though they actually work fine. I have 2 6610s linked with them right now. Might be a bit of useful info

 

[fohdeesha]

Would something like this work for this purpose by joining two mpo cables?

Lot of 8 Panduit FAMPOBLA Optical Fiber Adapter Connector OM3 Coupler | eBay

Edit: Fixed the link. trying to put links on mobile device doesn't always work for me.

It will only work to join a female (no pins) MPO connector with a male (has guide pins) connector. if you try to use it to join two common female MPO fibers, it will not work as there are no guide pins to align the near-microscopic fiber terminations. pretty common install error: Why the ICC MPO adapter panel doesn’t work for my MPO network? | ICC

Had a pretty good PM session with @fohdeesha about using the 6610 to flash optics. He pointed out that the manual explicitly says that 40G BiDi optics do not work, even though they actually work fine. I have 2 6610s linked with them right now. Might be a bit of useful info

Yup! some good info. Another thing Brocade was being quite conservative about (or maybe they just didn't want you buying BiDi optics from someone else, since they didn't sell any at the time). They say only SR4 optics are supported, but @mb300sd is using BiDi optics no problem.

Considering they are the highest power class possible, and draw as much power as allowed in the QSFP+ spec (class 4 / 3.5w) I think that greenlights the ICX6610 for being able to power pretty much any optic you'd need, including 40GBASE-LR4 long range optics, as they're the same power class (class 4) - if for some reason you want to stack across 10KM

 

[mb300sd]

Yup! some good info. Another thing Brocade was being quite conservative about (or maybe they just didn't want you buying BiDi optics from someone else, since they didn't sell any at the time). They say only SR4 optics are supported, but @mb300sd is using BiDi optics no problem.

Considering they are the highest power class possible, and draw as much power as allowed in the QSFP+ spec (class 4 / 3.5w) I think that greenlights the ICX6610 for being able to power pretty much any optic you'd need, including 40GBASE-LR4 long range optics, as they're the same power class (class 4) - if for some reason you want to stack across 10KM

I'd note that there might be a total power limit, I doubt that a switch manufacturer would have a separate 3.3v power rail for every slot, but it might be possible to damage something if you use 4x BiDi or LR4 modules in a switch. I've only tested with 2 with the other 2 slots empty. The goal of flashing the module was to get it to work in my Mellanox SX6018 which does restrict high power modules, but I'll never use all 18 ports on it. Mellanox appears to support LR4 modules in the lower port count switches, on certain ports, with an ini mod. I'm 90% sure this is to limit total power consumption/heat.

 

[epicurean]

I will getting a 6610-48E soon. Is there anyway to get all the 40GB ports to be like a switch, and not locked to some for stacking only? So I can do away with my oracle infiniband switch which I only use about 6 ports (out of 30!)?

 

[nthu9280]

I will getting a 6610-48E soon. Is there anyway to get all the 40GB ports to be like a switch, and not locked to some for stacking only? So I can do away with my oracle infiniband switch which I only use about 6 ports (out of 30!)?

Please check the first 2 posts on this thread about what can / can't be done.

You can use 2 as 40G ports. The other 2 work as 40G -> 4x10G.

 

[epicurean]

It's a marketing lie so people wouldn't realize it was half the price and had just as much 40gbE connectivity as their flagship SX800 chassis router at the time. all the rear QSFP ports (including the non-breakout 40gbE) work great for general use, just try not to publicize it too much

@fohdeesha, appreciate if you can help me with this. Will PM you when the 6610 reaches me.

 

[fohdeesha]

@fohdeesha, appreciate if you can help me with this. Will PM you when the 6610 reaches me.

https://forums.servethehome.com/index.php?threads/brocade-icx6450-icx6610-etc.21107/#post-196457

 

[roberth58]

getting an error upgrading 6450. when I enter "ip dhcp-client disable" I get "invalid input -> disable". Tried ip dhcp-client ? and disable is not one of the 5 listed inputs. Any ideas?

 

[fohdeesha]

yours came with the layer2 only firmware - scroll down to the note where it starts with "Note: If during the router-interface ve 1 command earlier " - read that blurb on how to get over to the l3 image

 

[roberth58]

yours came with the layer2 only firmware - scroll down to the note where it starts with "Note: If during the router-interface ve 1 command earlier " - read that blurb on how to get over to the l3 image

that was it, upgraded and all ports working. thanks for all the help.