Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[tubs-ffm]

If you just open up a command prompt and ping the switch, it should stay around 1-2ms until libreNMS's scheduled polling happens

Thank you.

My concern started when I observed the Gateway monitoring on OPNsense. (OK, my fault. What you do not know, does not make you concerned about.) Significant different values to the gateway on DSL modem than to directly connected switch. On OPNsense side both same NICs are same type and both directly connected via copper without LAGG and without VLAN in between. (see below)

Ping from Windows machine directly connected via fiber shows similar behavior. (see below)

When I will find time, I will check if there is any relation to other activities. In every case there is no load on my switch. This device is oversized for it purpose on mainly is doing nothing else than creating noise, generating heat and increasing the electricity bill.


OPNsense monitor


PING to switch

 

[NickM]

What specific type of fiber should I run in my house for 10g (maybe 40g) now, and hopefully maintaining future upgradability?

I thought I had this all figured out, and then realized the 10g SFP+ ($8) and 40g QSFP ($16) optical transceivers I was looking at had incompatible fiber runs, and now I'm secondguessing myself. I'd hoped "fiber is fiber" and you just upgraded the logic on the end.

For completeness, I have ConnectX-3 dual port QSFP cards (plus QSFP/SFP adapters), and an ICX7150 enroute. I'd considered running a direct connection between two computers via 40g QSFP in addition to the 10g connections to the switch


TLDR: Are there similarly cheap transceivers that would use the same fiber run for both 10g and 40g? What about 100g down the road?

If I should forget about the future and just run what works for now, let me know and tell me what fiber and transceivers to choose there, too

 

[LodeRunner]

There are 40 Gb transceivers that will use 2 fiber LC runs instead of the MPO/MPT type. I believe that would be a 40 Gb BiDi. I think there are also LC BiDi 100 Gb transceivers.

OM3 vs OM4 Multimode Fiber: What's the difference? is a good article about fiber types, speeds, and max run lengths.

Typically cheap does not apply to 40 and 100 Gb BiDi transceivers yet. A 40 Gb MPO/MPT from fs.com will be around $70; the BiDi LC connector version is around $300.

 

[ClintE]

We recently installed flexible conduit and pulled multimode fiber through the house. Currently using ICX 6610-24F in office (working on extensive fan mod) and very quiet Dell Powerconnect 5524's with Noctua fans in living room and bedroom. I have 2 or 3 Mellanox ConnectX-3 dual port QSFP cards and 3m DAC's, but not using them yet; really need 4m or 5m DAC's. Servers have Solarflare SFN6122 dual port SFP+ cards and workstations have Mellanox ConnectX-2 SFP+ single port cards. All SFP+ (and SFP 1Gb ethernet for the 24F's slots) tranceivers are Brocade. Everything was purchased used except for the fiber patch cables.

I ordered different lengths of some of the cheapest LC-LC duplex multimode patch cables from Amazon and they all work great. Should run 40Gb if needed, once I save up for transceivers.

 

[Jason Antes]

My pings were between different OS on the same or different vlans and saw the behavior. I never pinged the switch itself. Wasn't too worried about it but found it strange.

 

[NickM]

There are 40 Gb transceivers that will use 2 fiber LC runs instead of the MPO/MPT type. I believe that would be a 40 Gb BiDi. I think there are also LC BiDi 100 Gb transceivers.

OM3 vs OM4 Multimode Fiber: What's the difference? is a good article about fiber types, speeds, and max run lengths.

Typically cheap does not apply to 40 and 100 Gb BiDi transceivers yet

Thanks - knowing that it exists for 40 and 100Gb over LC/OM4 for 'upgrading later' assuages some of my concerns of just running LC-terminated OM4 for now (presuming that's the easiest/cheapest route for now?)

Are the breakout/feed-in casettes (or really -- I only need a single pair for now) for the MPO type fiber worth looking at, to run multiple strands and just use a single pair of it for 10g? Or is there some component in that chain that's going to be dramatically more expensive?

 

[tubs-ffm]

If you just open up a command prompt and ping the switch, it should stay around 1-2ms until libreNMS's scheduled polling happens

I did a couple of test port to port from one machine to and other machine. Vi copper and fiber. All OK. Only to the switch shows the peaks with slow ping response. So all, OK.

I switched of monitoring and now I can sleep well again.

 

[LodeRunner]

Thanks - knowing that it exists for 40 and 100Gb over LC/OM4 for 'upgrading later' assuages some of my concerns of just running LC-terminated OM4 for now (presuming that's the easiest/cheapest route for now?)

Are the breakout/feed-in casettes (or really -- I only need a single pair for now) for the MPO type fiber worth looking at, to run multiple strands and just use a single pair of it for 10g? Or is there some component in that chain that's going to be dramatically more expensive?

There are MPO/MPT to LC cassettes if you want to run multi fiber bundles. MTP®/MPO-LC - MTP®/MPO Fiber Cassettes - Fiber Optic Cassettes - FS
And mounting enclosures: 1U 19-in Blank Rack mount Fiber Patch Panel with Cable Management Panel and D-rings - FS (lots of other options on that page)

As to their value to you, that depends on what your goal is. Another option would be a multi fiber bundle terminated as LC on both ends and use LC-LC keystones.

 

[mattaw]

@fohdeesha Do you know how to configure flow control on the "unstacked" 40gbe ports? I am using two Connextx-3 (non-Pros) doing ROCEv1 (and 1.25!?! wth Mellanox) to do NVMe-oF between linux and Windows 10 Pro over RDMA.

I see from the Fastiron flow control guide you can enable symmetric flow control globally on the icx6610 to prevent packet loss globally (but not PFC) but all the QoS type settings that apply to the icx6610 are not visibly working on the stacking ports, i.e. you cannot query them with "show". My guess is the stacking ports have a type of PFC lossless running for stack config traffic due to the Queue 7->6 remap when stacking is on and they have not been freed using your procedure.

I appreciate your thoughts.

 

[tubs-ffm]

With a low-profile fan on top of the ASIC my temperature is fine. The Sunon MF60101V3-1000U-A99 is 10 mm in high. I fixed it with a simple elastic band to the cooler. I connected it in parallel to one of the chassis fans. So, it also will at reduced voltage. My backup plan was to go to permanent 12 V. Sunon MF60101V1-1000U-A99 would be another option I red here in this thread.

My concern is the PSU temperature. My unit is with PoE. Im am not planning to use a lot of PoE power. But even in idle mode the PSU of this device needs more cooling. I am not far away from fan level 2 jump in temperature. Now it is winter. In summer I guess this will not work.

Cutting out the metal in front of the outlet fans is a recommendation for better air flow and for lower noise. But as long I am not yet sure if I will keep my device, I do not want to do anything I cannot revert.

ICX 7250-24P exploded.

The joy with my ICX 7250-24P suddenly came to an end when the PSU exploded yesterday. Smoke, flames and electrical "bzzzz sound" included.

To go back in time. I modified my device by replacing the fans to for noise reasons. There is no heavy load on my device and I used only one PoE device. Temperature of both sensors are fine. See record of last 24 h till crash below. The device is running like this for 3 weeks or so. The up and down is because of changing ambient temperature during day and night. But suddenly an "explosion" and it was over.

I cannot say for sure if it is because of the modification I did or by chance. Theoretically the temperature at the sensor could be fine but not in other areas. In every case I would not recommend to anybody to do this type of modification.

Anybody who wants to exchange my broken ICX 7250-24P to a running ICX 7150-24 or ICX 7150-24P?

 

[GroundPepper]

FYI I was on the brocade site and noticed the latest firmware available for the ICX7250 is 08090d, while the tutorial ICX7250 / ICX7450 - Fohdeesha Docs contains a previous version (ICX7250 / ICX7450 - Fohdeesha Docs).

Does anyone know how what command to use to figure out the current firmware version installed, btw?

 

[tubs-ffm]

Does anyone know how what command to use to figure out the current firmware version installed, btw?

It is not a secret. It's written in the manual.
show version

 

[GroundPepper]

It is not a secret. It's written in the manual.
show version

I've been reading the command reference guide (Commscope Technical Content Portal). It's just the way it's organized, you can't search for "firmware version", you have know what command you want to run first, then you can look up what it does. And there are a lot of commands.

 

[klui]

I've been reading the command reference guide (Commscope Technical Content Portal). It's just the way it's organized, you can't search for "firmware version", you have know what command you want to run first, then you can look up what it does. And there are a lot of commands.

Use another term. Search for "cli flash version"

It's also better to download the PDF then search for "version" within the document.

 

[RoachedCoach]

I've been reading the command reference guide (Commscope Technical Content Portal). It's just the way it's organized, you can't search for "firmware version", you have know what command you want to run first, then you can look up what it does. And there are a lot of commands.

Alternatively, it's listed on the Device page in the WebUI.

 

[LodeRunner]

I'm rather partial to "sh ?" which will just give you a list of available commands in your current context. Some super advanced stuff is hidden by default which more or less falls under "If you don't know that it exists, you probably don't need it."

 

[tubs-ffm]

FYI I was on the brocade site and noticed the latest firmware available for the ICX7250 is 08090d,

One point I forgot in my previous answer. "latest firmware" depends on the definition. You may have looked on latest issued. There are several firmware paths that are getting maintained in parallel: 08080, 08090, 08092, 08095. The higher number is the newer path with newer functionality. The letters at the end is the revision number per path. Acc. feedback in this forum, 08092 can be considered as stable enough.

 

[eduncan911]

To the 7450 owners... Convince me to pickup an 7450 over an 6610 or another 7250.

I have an 7250-48P that I was planning on installing. But now that I've started curating parts for my rack, I need more than 8x 10Gbps. I actually need 12x 10Gbps, leaving not much room to expand in the future if I pickup a 2nd 7250.

I've read a lot about the 7450 in this thread. It's louder than the 6610(?), more power, not as much bandwidth as the 6610 (though all i need are 10Gbps, not 40Gbps - so I think). However, I really want the higher-powered ports as one RPi hat I was thinking of designing will require more power than standard PoE+ (for high-torque robotics) - not to mention newer access points 5/10 years from now.

So, I'm on the fence...

• Has anyone confirmed the rear 40Gbps break-out cables for the 7450 work as assignable ports (not as stackable)? I saw the data sheet list the breakout cables, but not much in this thread about assigning them ports/VLANs/etc.
• Convince me to not just buy another 7250-24/48. I mean, it will give me 14x SPF+ (have to use two to link them). That's $500+ in switches, when I could donate my 7250 to a school and just get a single 6610 or 7450 instead. And idle of 50W sounds nice, until you realize - I have to run two, for 100W idle! Which leads me to...
• The 6610-48P idles around 110W, which is around 2x 7250s. It also has 40Gbps ports to connect to my media/storage server. However, the 6610 doesn't have high-powered PoE ports (PoH).

So, the 7450-48P sounds like the winner due to being only a single switch, PoH, etc instead of running 2x 7250s. Plus the school I am working on would get the 7250 as a donation since I am rebuilding their network as well.

Sound is a little concern but I do plan on building a fully-enclosed rack for under the stairs.

 

[eduncan911]

I ordered a Dell Force10 DAC breakout to try. If it doesn't work on my 7450 I'll send it to someone with a 6610 free to verify compatibility for future reference.

Digging up this old post from 9 months ago...

@EngineerNate Did that cable work? Also, are we able to assign individual ports on the rear module to systems? Ashamed to say I actually dug through your user history and rear all of your posts since that one above. Didn't see any follow-up.

 

[LodeRunner]

7450 supports the 4x SFP+ modules in the back slots; I needed more ports and just bought another 4x10G module and it worked fine. As far as I have found, it does not have the commands to enable breakout and all the documentation I found for configuring 40g breakout applies to the 7750. I don't know why the breakout cable is listed unless when you use the breakout QSFP it auto-configures the breakout.

Edit:
Lack of breakout options on my 7450 might be because even with "stack disable" set it still says

stack unit 1
  module 1 icx7450-48p-poe-management-module
  module 2 icx7400-xgf-4port-40g-module
  module 3 icx7400-xgf-4port-40g-module
  module 4 icx7400-qsfp-1port-40g-module
  stack-port 1/4/1
stack disable

Guess I need to pull the config, nuke it to factory, and redo it, being sure to gut any stack setup first. Trying to remove the last stack port results in:

SSH@core(config-unit-1)#no stack-port 1/4/1
Error! cannot remove the only stack-port 1/4/1.