Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Jason Antes

Member
Feb 28, 2020
128
23
18
Twin Cities
Is there any rate limit by standard for PING / ICMP responses in Fastiron?

The gateway monitoring of OPNsense shows "poor" values for the connection to my ICX 7250 running 0.80.92e. Looking on LibreNMS I also see peeks in ping time over the day. ICMP rate limit? Something wrong? Or normal due to measurement principle?
View attachment 17736
I see the same issue with OPNSense and my 6610. Every 15-20th ping against any other system goes to 200ms from 1-2ms. It's kind of strange and sometimes causes an issue with Putty sessions. I figured it was an OPNSense issue with the 10Gbe card or something. Wish I knew more about troubleshooting these kinds of issues.
 
  • Like
Reactions: cesmith9999

ArmedAviator

Member
May 16, 2020
89
51
18
Ohio
Hmmm, I too just loked at my ICX latency in LibreNMS and see the same wild latency spikes up to about 130-160ms. I haven't had any connectivity issues as a result, though, so it's moot to me.
 

rocketpanda40

Member
Dec 12, 2019
32
19
8
Mine fluctuates alot as well, but not to the same heights y'alls seem to. It doesn't add too much to the data other than to show that your spikes up to 200ms probably have more to do with your traffic patterns and network topology than the device itself.icx7250-ping-latency.png
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,087
1,949
113
30
fohdeesha.com
ICMP pings to the switch itself are answered by the management CPU, which prioritizes ICMP replies very low compared to other activities, so some fluctuation is normal. That combined with the fact LibreNMS runs the ping during the same time as the scheduled SNMP polling, which puts a decent amount of load on the CPU to answer. If you just open up a command prompt and ping the switch, it should stay around 1-2ms until libreNMS's scheduled polling happens
 

tubs-ffm

Member
Sep 1, 2013
88
26
18
If you just open up a command prompt and ping the switch, it should stay around 1-2ms until libreNMS's scheduled polling happens
Thank you.

My concern started when I observed the Gateway monitoring on OPNsense. (OK, my fault. What you do not know, does not make you concerned about.) Significant different values to the gateway on DSL modem than to directly connected switch. On OPNsense side both same NICs are same type and both directly connected via copper without LAGG and without VLAN in between. (see below)

Ping from Windows machine directly connected via fiber shows similar behavior. (see below)

When I will find time, I will check if there is any relation to other activities. In every case there is no load on my switch. This device is oversized for it purpose on mainly is doing nothing else than creating noise, generating heat and increasing the electricity bill.


OPNsense monitor
gateway.PNG

PING to switch
ping.PNG
 

NickM

Member
Jul 19, 2015
36
7
8
36
What specific type of fiber should I run in my house for 10g (maybe 40g) now, and hopefully maintaining future upgradability?

I thought I had this all figured out, and then realized the 10g SFP+ ($8) and 40g QSFP ($16) optical transceivers I was looking at had incompatible fiber runs, and now I'm secondguessing myself. I'd hoped "fiber is fiber" and you just upgraded the logic on the end.

For completeness, I have ConnectX-3 dual port QSFP cards (plus QSFP/SFP adapters), and an ICX7150 enroute. I'd considered running a direct connection between two computers via 40g QSFP in addition to the 10g connections to the switch


TLDR: Are there similarly cheap transceivers that would use the same fiber run for both 10g and 40g? What about 100g down the road?

If I should forget about the future and just run what works for now, let me know and tell me what fiber and transceivers to choose there, too :)
 
  • Like
Reactions: tommybackeast

LodeRunner

Active Member
Apr 27, 2019
103
44
28
There are 40 Gb transceivers that will use 2 fiber LC runs instead of the MPO/MPT type. I believe that would be a 40 Gb BiDi. I think there are also LC BiDi 100 Gb transceivers.

OM3 vs OM4 Multimode Fiber: What's the difference? is a good article about fiber types, speeds, and max run lengths.

Typically cheap does not apply to 40 and 100 Gb BiDi transceivers yet. A 40 Gb MPO/MPT from fs.com will be around $70; the BiDi LC connector version is around $300.
 
  • Like
Reactions: NickM

ClintE

New Member
Feb 22, 2019
29
6
3
We recently installed flexible conduit and pulled multimode fiber through the house. Currently using ICX 6610-24F in office (working on extensive fan mod) and very quiet Dell Powerconnect 5524's with Noctua fans in living room and bedroom. I have 2 or 3 Mellanox ConnectX-3 dual port QSFP cards and 3m DAC's, but not using them yet; really need 4m or 5m DAC's. Servers have Solarflare SFN6122 dual port SFP+ cards and workstations have Mellanox ConnectX-2 SFP+ single port cards. All SFP+ (and SFP 1Gb ethernet for the 24F's slots) tranceivers are Brocade. Everything was purchased used except for the fiber patch cables.

I ordered different lengths of some of the cheapest LC-LC duplex multimode patch cables from Amazon and they all work great. Should run 40Gb if needed, once I save up for transceivers.
 

Jason Antes

Member
Feb 28, 2020
128
23
18
Twin Cities
My pings were between different OS on the same or different vlans and saw the behavior. I never pinged the switch itself. Wasn't too worried about it but found it strange.
 

NickM

Member
Jul 19, 2015
36
7
8
36
There are 40 Gb transceivers that will use 2 fiber LC runs instead of the MPO/MPT type. I believe that would be a 40 Gb BiDi. I think there are also LC BiDi 100 Gb transceivers.

OM3 vs OM4 Multimode Fiber: What's the difference? is a good article about fiber types, speeds, and max run lengths.

Typically cheap does not apply to 40 and 100 Gb BiDi transceivers yet
Thanks - knowing that it exists for 40 and 100Gb over LC/OM4 for 'upgrading later' assuages some of my concerns of just running LC-terminated OM4 for now (presuming that's the easiest/cheapest route for now?)

Are the breakout/feed-in casettes (or really -- I only need a single pair for now) for the MPO type fiber worth looking at, to run multiple strands and just use a single pair of it for 10g? Or is there some component in that chain that's going to be dramatically more expensive?
 

tubs-ffm

Member
Sep 1, 2013
88
26
18
If you just open up a command prompt and ping the switch, it should stay around 1-2ms until libreNMS's scheduled polling happens
I did a couple of test port to port from one machine to and other machine. Vi copper and fiber. All OK. Only to the switch shows the peaks with slow ping response. So all, OK.

I switched of monitoring and now I can sleep well again.
 

LodeRunner

Active Member
Apr 27, 2019
103
44
28
Thanks - knowing that it exists for 40 and 100Gb over LC/OM4 for 'upgrading later' assuages some of my concerns of just running LC-terminated OM4 for now (presuming that's the easiest/cheapest route for now?)

Are the breakout/feed-in casettes (or really -- I only need a single pair for now) for the MPO type fiber worth looking at, to run multiple strands and just use a single pair of it for 10g? Or is there some component in that chain that's going to be dramatically more expensive?
There are MPO/MPT to LC cassettes if you want to run multi fiber bundles. MTP®/MPO-LC - MTP®/MPO Fiber Cassettes - Fiber Optic Cassettes - FS
And mounting enclosures: 1U 19-in Blank Rack mount Fiber Patch Panel with Cable Management Panel and D-rings - FS (lots of other options on that page)

As to their value to you, that depends on what your goal is. Another option would be a multi fiber bundle terminated as LC on both ends and use LC-LC keystones.
 

mattaw

Member
Jul 30, 2018
56
15
8
@fohdeesha Do you know how to configure flow control on the "unstacked" 40gbe ports? I am using two Connextx-3 (non-Pros) doing ROCEv1 (and 1.25!?! wth Mellanox) to do NVMe-oF between linux and Windows 10 Pro over RDMA.

I see from the Fastiron flow control guide you can enable symmetric flow control globally on the icx6610 to prevent packet loss globally (but not PFC) but all the QoS type settings that apply to the icx6610 are not visibly working on the stacking ports, i.e. you cannot query them with "show". My guess is the stacking ports have a type of PFC lossless running for stack config traffic due to the Queue 7->6 remap when stacking is on and they have not been freed using your procedure.

I appreciate your thoughts.
 

tubs-ffm

Member
Sep 1, 2013
88
26
18
With a low-profile fan on top of the ASIC my temperature is fine. The Sunon MF60101V3-1000U-A99 is 10 mm in high. I fixed it with a simple elastic band to the cooler. I connected it in parallel to one of the chassis fans. So, it also will at reduced voltage. My backup plan was to go to permanent 12 V. Sunon MF60101V1-1000U-A99 would be another option I red here in this thread.

My concern is the PSU temperature. My unit is with PoE. Im am not planning to use a lot of PoE power. But even in idle mode the PSU of this device needs more cooling. I am not far away from fan level 2 jump in temperature. Now it is winter. In summer I guess this will not work.

Cutting out the metal in front of the outlet fans is a recommendation for better air flow and for lower noise. But as long I am not yet sure if I will keep my device, I do not want to do anything I cannot revert.
ICX 7250-24P exploded.

The joy with my ICX 7250-24P suddenly came to an end when the PSU exploded yesterday. Smoke, flames and electrical "bzzzz sound" included.

To go back in time. I modified my device by replacing the fans to for noise reasons. There is no heavy load on my device and I used only one PoE device. Temperature of both sensors are fine. See record of last 24 h till crash below. The device is running like this for 3 weeks or so. The up and down is because of changing ambient temperature during day and night. But suddenly an "explosion" and it was over.

I cannot say for sure if it is because of the modification I did or by chance. Theoretically the temperature at the sensor could be fine but not in other areas. In every case I would not recommend to anybody to do this type of modification.

Anybody who wants to exchange my broken ICX 7250-24P to a running ICX 7150-24 or ICX 7150-24P? :)

IMG_20210309_213938.jpg
7250-temp.png
 

klui

Active Member
Feb 3, 2019
280
119
43
I've been reading the command reference guide (Commscope Technical Content Portal). It's just the way it's organized, you can't search for "firmware version", you have know what command you want to run first, then you can look up what it does. And there are a lot of commands.
Use another term. Search for "cli flash version"

It's also better to download the PDF then search for "version" within the document.
 
  • Like
Reactions: GroundPepper

LodeRunner

Active Member
Apr 27, 2019
103
44
28
I'm rather partial to "sh ?" which will just give you a list of available commands in your current context. Some super advanced stuff is hidden by default which more or less falls under "If you don't know that it exists, you probably don't need it."