Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[jasonwc]

And ATT makes it near impossible to connect their ONT directly to a switch SFP+ transceiver, especially for greater than 1Gb service as the ont is built into the modem.

That's too bad. I read AT&T provides its own router with a 5G Base-T interface. Is that correct?

From the appearance and specifications, and Verizon's prior statements regarding working with Calix, it appears that FiOS will use this NG-PON2 ONT:

The GP1200NG GigaPoint ONU is a next-generation premise solution equipped with a 10G NG-PON2 WAN interface and a single 10G Ethernet service port to provide convenient, ultra high bandwidth service at the subscriber edge. The GP1200NG enables you to deliver up to 10G data and/or IPTV services to residential and small/medium business subscribers over our E9 system.

The spec sheet lists a single 1000/2500/10G Base-T port for connecting to the customer router. More details here: 10G PON ONTs/ONUs

 

[richtj99]

Hi - Thanks so much! The sh mac e 1/1/30 worked & showed the mac:

14a7.8b7d.7f1c which does match up to the mac I have but I am unable to ping it.

The IP should be 192.168.168.247 (it was set static), I cant see any IP. I am using ubiquiti for my wireless & the software does see the mac showing up as 192.168.1.108 which is the default dahua.

I manually changed the IP of the PC I was using to 192.168.1.213 & was unable to ping it on the vlan. Somehow the Unifi sees it which is on a separate vlan.

I am a bit at a loss on how to figure this out.

The camera is a PTZ & outside so the cold weather might be kicking on a heater pulling more power?

The router does not do DHCP but i have a windows 2019 server doing DHCP (not much else) on that vlan. Not sure how to check if it can see the device either as the camera isnt set for DHCP.

Thanks for helping me!

Rich

I have a bunch of Dahua cameras in a dedicated VLAN for the security camera network. I found them very easy to setup and haven't had any issues with them - but the very first thing I did was setup a static IP address. Since the port shows as up on the switch, I would check your DHCP server logs to see what IP address it assigned. The most likely scenario is that the router assigned a new IP. To find the MAC address of the attached camera, try:

show mac-address e 1/1/30

Another option would be to check the ARP table on the router. You could also do a scan for all addresses on the subnet depending on how many cameras you have on that VLAN. Once you find the IP, login and manually assign an IP.

I also find it odd that the inline power statistics shows the device is using 802.3at. Every Dahua camera I've seen lists 802.3af, including the PTZ cameras. My Dahua cameras all show 802.3af in inline power details.

1/1/3 On On 2600 15400 802.3af n/a 3 n/a
1/1/29 On On 3200 15400 802.3af n/a 3 n/a
1/1/30 On On 2200 15400 802.3af n/a 3 n/a
1/1/31 On On 5900 15400 802.3af n/a 3 n/a

 

[jasonwc]

And ATT makes it near impossible to connect their ONT directly to a switch SFP+ transceiver, especially for greater than 1Gb service as the ont is built into the modem.

Hi - Thanks so much! The sh mac e 1/1/30 worked & showed the mac:

14a7.8b7d.7f1c which does match up to the mac I have but I am unable to ping it.

The IP should be 192.168.168.247 (it was set static), I cant see any IP. I am using ubiquiti for my wireless & the software does see the mac showing up as 192.168.1.108 which is the default dahua.

I manually changed the IP of the PC I was using to 192.168.1.213 & was unable to ping it on the vlan. Somehow the Unifi sees it which is on a separate vlan.

I am a bit at a loss on how to figure this out.

The camera is a PTZ & outside so the cold weather might be kicking on a heater pulling more power?

The router does not do DHCP but i have a windows 2019 server doing DHCP (not much else) on that vlan. Not sure how to check if it can see the device either as the camera isnt set for DHCP.

Thanks for helping me!

Rich

If somehow the camera was reset to factory defaults on 192.168.1.108, I think you would be unable to access it because it's currently untagged on VLAN 68. If you have a 192.168.1.0/24 subnet, it's presumably VLAN1 (although it could be assigned to any VLAN ID). When you tried manually changing your PC IP to 192.168.1.213, did you also modify the PC switch port so that it was untagged on VLAN 68 - matching the camera?. If it was untagged on a different VLAN, the traffic would never have made it to the camera.

It's also possible the device is malfunctioning but the IP didn't change. In that case, I would disconnect the device from the switch for 60 seconds, thereby removing power to the device, and then re-connect it. Then see if you can ping the device at 192.168.168.247. To eliminate the possibility of firewall issues, I would untag your PC port in VLAN 68 so no routing is required to ping the camera. You can also check the router's ARP table to see if there has been any recent traffic from 192.168.168.247.

Your interface statistics noted both input and CRC errors, as well as 140+ dropped packets, which I haven't seen on my cameras. Are your other cameras working properly? Do they show packet errors? Has this camera been working for a while or did you only recently install it. If the latter, perhaps the cable or connections are subpar.

Have you had a lot of rain or snow recently? Is the camera IP67 rated? I would assume so if it's an outdoor PTZ. Still, it's possible the ethernet connector has been exposed to moisture. Dahua provides watertight connectors for the ethernet connection. Did you use it?

  Port up for 8 day(s) 17 hour(s) 15 minute(s) 19 second(s)
  Hardware is GigabitEthernet, address is 609c.9f46.396a (bia 609c.9f46.396a)
  Configured speed auto, actual 100Mbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual MDI
  Member of L2 VLAN ID 15, port is untagged, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  Port name is Backyard Camera
  Inter-Packet Gap (IPG) is 96 bit times
  MTU 10200 bytes, encapsulation ethernet
  300 second input rate: 13482440 bits/sec, 1148 packets/sec, 13.63% utilization
  300 second output rate: 303696 bits/sec, 587 packets/sec, 0.39% utilization
  631943572 packets input, 927033212318 bytes, 0 no buffer
  Received 14476 broadcasts, 2094 multicasts, 631927002 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  315611841 packets output, 20380943088 bytes, 0 underruns
  Transmitted 43405 broadcasts, 375203 multicasts, 315193233 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0           315242481                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                   0                   0
    6                   0                   0
    7              369361                   0

 

[bween]

That's too bad. I read AT&T provides its own router with a 5G Base-T interface. Is that correct?

From the appearance and specifications, and Verizon's prior statements regarding working with Calix, it appears that FiOS will use this NG-PON2 ONT:

The GP1200NG GigaPoint ONU is a next-generation premise solution equipped with a 10G NG-PON2 WAN interface and a single 10G Ethernet service port to provide convenient, ultra high bandwidth service at the subscriber edge. The GP1200NG enables you to deliver up to 10G data and/or IPTV services to residential and small/medium business subscribers over our E9 system.

The spec sheet lists a single 1000/2500/10G Base-T port for connecting to the customer router. More details here: 10G PON ONTs/ONUs

Yep, the included router has a 5g Base-T interface. (I think it's the yellow port) Using that port, I can see they actually overprovision my residence with something like 1.1-1.2 gbps. Maybe 5g will come soon!

Should also note that while I'd rather directly plug in my own equipment directly to the ONT, I haven't really had any problems with their ip passthrough either. I just go into the ATT router settings and disable everything except for an IP passthrough to my firewall. Haven't really noticed any performance degradation using their passthrough mode.

Pretty neat info about FiOS. Unfortunately not available in my area for the foreseeable future. Jealous!

 

[jasonwc]

Yep, the included router has a 5g Base-T interface. (I think it's the yellow port) Using that port, I can see they actually overprovision my residence with something like 1.1-1.2 gbps. Maybe 5g will come soon!

Should also note that while I'd rather directly plug in my own equipment directly to the ONT, I haven't really had any problems with their ip passthrough either. I just go into the ATT router settings and disable everything except for an IP passthrough to my firewall. Haven't really noticed any performance degradation using their passthrough mode.

Pretty neat info about FiOS. Unfortunately not available in my area for the foreseeable future. Jealous!

It's too bad none of these devices provide a combo 10G Base-T and SFP+ port. SFP+ switches and NICs are cheaper, use less power, and have lower latency. I even saw that some consumer routers are now including an SFP+ port. For example, What SFP+ modules are compatible with Nighthawk X10 R9000, R8900 and Nighthawk Pro Gaming XR700 routers? | Answer | NETGEAR Support.

 

[bween]

Hi, hoping someone can help with HomeKit and mDNS.
Setup:
- home assistant with homebridge to homekit, on servers vlan 10
- HomePod mini, on trusted vlan 20
- rpi running avahi-daemon, interface on vlan 10/20
- all routing done

Problem:
When my iPhone (connected to vlan 20) is on the home network, everything works fine. When my iPhone is not connected to the home network, all of the homebridge ONLY devices are not responding. Problem immediately goes away when the iPhone is connected to the home network.

Tried restarting devices, repairing things etc to no avail.

Is there some setting on the switch I need to enable for HomePod to recognize avahi as the mdns reflector? Maybe an ip helper address or something?

In case anyone is curious for the solution that worked on my network here it is.

- ditched avahi-daemon
- Created two interfaces on Home Assistant - one in vlan 10 for servers and one in vlan 20
- all routing still done on brocade (no special igmp or anything on here)
- changed broadcast filtering to unicast-ARP-only (aruba os 8.9)
- reboot all involved stuff like homepod etc
--> next step - reconfigure access lists to restrict home assistant device access from vlan 20

Found out about an iOS app called Discovery which can help figure out mDNS problems. This was super useful helping me figure out what mDNS was being dropped.

For me, the problem ended up being that aruba automatically filtered mDNS. While my phone could still see the devices (they communicated directly), as soon as my phone left the local network, the homepod could not discover the bridge. Changing the broadcast filtering to unicast-ARP-only allowed the homepod to discover home assistant's mdns and all was well.

 

[linuxsrc]

Hi all, I am looking at buying one of the ICX6650 could someone help me understand the differences in the module numbers I am finding on eBay

ICX6650-32-I-ADV 32-PORT
ICX6650-48-I-ADV
ICX6650-56E-ADV

Physically these look the same, but the prices range from 690.00 to 900.00 dollars
The 32-I-ADV is at the lowest and the 56E-ADV is at the highest.

I don't want to pay for something that is not needed to get the 48 10G ports I want.

Thanks

 

[jasonwc]

Hi all, I am looking at buying one of the ICX6650 could someone explain the differences in the module numbers I am finding on ebay



Physically these look the same, but the prices range from 690.00 to 900.00 dollars
The 32-I-ADV is at the lowest and the 56E-ADV is at the highest.

I don't want to pay for something that is not needed to get the 48 10G ports I want.

Thanks

The only thing you need to worry regarding the model # is the difference between the I and E models. The E models exhaust front to back which is quieter than the I models (back to front). Otherwise, the differences (32/48/56 port) and Advanced license only is relevant to licensing. You can use the guide to enable all features for free. So, you'll be able to utilize all 48 SFP+ ports, both breakout QSFP+ ports (8 more 10G ports), and all four 40G QSFP+ ports, and also use the advanced L3 features. So, any ICX6550 will give you 56 10G ports and four 40G ports.

The other difference you may see is that some switches may not come with two power supplies and two fans. On the ICX6610, there are 3 revisions of the power supply - A, B, and C. The A revision is significantly louder, even when mixed with a B or C. I'm not sure if this applies to the ICX6650. Anyone know?

 

[linuxsrc]

The only thing you need to worry about is the difference between the I and E models. The E models exhaust front to back which is quieter than the I models (back to front). Otherwise, the differences (32/48/56 port) and Advanced license only is relevant to licensing. You can use the guide to enable all features for free. So, you'll be able to utilize all 48 SFP+ ports, both breakout QSFP+ ports (8 more 10G ports), and all four 40G QSFP+ ports, and also use the advanced L3 features. So, any ICX6550 will give you 56 10G ports and four 40G ports.

The other difference you may see is that some switches may not come with two power supplies and two fans. On the ICX6610, there are 3 revisions of the power supply - A, B, and C. The A revision is significantly louder, even when mixed with a B or C. I'm not sure if this applies to the ICX6650. Anyone know?

To understand your statement correctly, the 32/48/56 is the number of aggregate ports or SFP+ ports only?
The following description is what is confusing me:

ICX6650-48-I-ADV 56 Port

The description says 48 in it be but s described as having 56 ports.
Thanks again.

Licensing, I get it now!

Thanks, I had to read better!

 

[Chewza]

@fohdeesha have a weird issue with a new to me ICX6610-24f.

As a starting point... It's been fully upgraded per your guide. (Thanks a ton for this btw!)

However I've got a Rev B PSU that refuses to behave.
The system boots, loud fans as one expects.
Loads config, PSU is quiet.
About a minute later I can hear the PSU fan spin up for seemingly no reason at all and just stays there. The switch doesn't have any activity, and no ethernet ports are connected.

SSH@brocade6610>show chassis
The stack unit 1 chassis info:

Power supply 1 not present
Power supply 2 (AC - Regular) present, status ok
     Model Number:    23-0000144-01
    Serial Number:    0XS     
    Firmware Ver:      B
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 not present
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 57.5 deg-C

Fan speed switching temperature thresholds:
        Speed 1: NM<----->80       deg-C
        Speed 2:       75<-----> 87 deg-C (shutdown)

Fan 2 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
    Current temperature : 55.5 deg-C
CPU Temperature Readings:
    Current temperature : 51.5 deg-C
sensor A Temperature Readings:                                   
    Current temperature : 35.0 deg-C
sensor B Temperature Readings:
    Current temperature : 48.5 deg-C
sensor C Temperature Readings:
    Current temperature : 35.5 deg-C
sensor D Temperature Readings:
    Current temperature : 31.0 deg-C
stacking card Temperature Readings:
    Current temperature : 57.5 deg-C
    Warning level.......: 84.0 deg-C
    Shutdown level......: 87.0 deg-C

Nothing from the chassis info seems out of whack, unless I'm missing something.
I have yet to perform testing with a Rev A PSU that was also included with the system, but have shied away from per the guidance that it's considerably louder.
I've also tried seating the PSU in each of the slots, along with moving the fan (fan is definitely NOT the culprit here)

Any suggestions would be greatly appreciated!

 

[jasonwc]

@fohdeesha have a weird issue with a new to me ICX6610-24f.

As a starting point... It's been fully upgraded per your guide. (Thanks a ton for this btw!)

However I've got a Rev B PSU that refuses to behave.
The system boots, loud fans as one expects.
Loads config, PSU is quiet.
About a minute later I can hear the PSU fan spin up for seemingly no reason at all and just stays there. The switch doesn't have any activity, and no ethernet ports are connected.

SSH@brocade6610>show chassis
The stack unit 1 chassis info:

Power supply 1 not present
Power supply 2 (AC - Regular) present, status ok
     Model Number:    23-0000144-01
    Serial Number:    0XS   
    Firmware Ver:      B
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 not present
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 57.5 deg-C

Fan speed switching temperature thresholds:
        Speed 1: NM<----->80       deg-C
        Speed 2:       75<-----> 87 deg-C (shutdown)

Fan 2 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
    Current temperature : 55.5 deg-C
CPU Temperature Readings:
    Current temperature : 51.5 deg-C
sensor A Temperature Readings:                                 
    Current temperature : 35.0 deg-C
sensor B Temperature Readings:
    Current temperature : 48.5 deg-C
sensor C Temperature Readings:
    Current temperature : 35.5 deg-C
sensor D Temperature Readings:
    Current temperature : 31.0 deg-C
stacking card Temperature Readings:
    Current temperature : 57.5 deg-C
    Warning level.......: 84.0 deg-C
    Shutdown level......: 87.0 deg-C

Nothing from the chassis info seems out of whack, unless I'm missing something.
I have yet to perform testing with a Rev A PSU that was also included with the system, but have shied away from per the guidance that it's considerably louder.
I've also tried seating the PSU in each of the slots, along with moving the fan (fan is definitely NOT the culprit here)

Any suggestions would be greatly appreciated!

The guide indicates that if you have only one PSU and one fan, to put both in the leftmost position, when the switch is viewed from the rear (qsfp ports). You mentioned trying different positions for the power supply but the fan position will also change the noise level.

The switch will run quietist with two power supplies and two fans, but I found I needed to reload the switch after connecting the second power supply to see the lower noise level. However, this is only true with B/C power supplies. With your A and B combo, you’re best off only using the B.

 

[adman_c]

I recently picked up a couple 7250s and transplanted my aftermarket fans (I haven't been able to find a record of which specific model, but I believe it's one of the Sunon maglev models discussed early in the thread) from my 6450s. My 6450s and 7250s are running the latest firmware on @fohdeesha 's site and don't throw any errors.

To my ear, the 6450s were still borderline too loud with the new fans. When I moved the fans into my 7250s, the temps are quite high and send the fans into the "speed 2" regime every 5 minutes or so. This is with ~20W of POE and 6 10G devices in each switch (half DAC, half RJ45 transceivers).

While I still have both my 6450s and 7250s I may make a separate thread about fans with noise samples since it seems like a frequent topic.

I found that the Sunon maglev fans made a horrible resonance when installed in my 6450, so I bought a bunch of other fans to try. I ended up settling on the first ones I tried after the Sunons. They were quieter than the stock fans and waaaaay less annoying than the Sunons. Check out rootwyrm's fan post of doom for some other suggestions, as the Deltas that I bought are OOS at the link I posted in my earlier post.

 

[Chewza]

The guide indicates that if you have only one PSU and one fan, to put both in the leftmost position, when the switch is viewed from the rear (qsfp ports). You mentioned trying different positions for the power supply but the fan position will also change the noise level.

The switch will run quietist with two power supplies and two fans, but I found I needed to reload the switch after connecting the second power supply to see the lower noise level. However, this is only true with B/C power supplies. With your A and B combo, you’re best off only using the B.

This didn't seem to be the case.
With either PSU and Fan in either of the left or right slots, the system would refuse to lower the fan speeds after a full boot.
For shits and giggles, I put the B PSU and fan in the left slots, and the A PSU in the right.
The system has yet to ramp it's fans up.

So I guess that’s my bad thinking I could get away with only 1 PSU.
I’ll keep an eye out for a Rev B PSU to match with my other.

Thanks!

 

[jasonwc]

This didn't seem to be the case.
With either PSU and Fan in either of the left or right slots, the system would refuse to lower the fan speeds after a full boot.
For shits and giggles, I put the B PSU and fan in the left slots, and the A PSU in the right.
The system has yet to ramp it's fans up.

So I guess that’s my bad thinking I could get away with only 1 PSU.
I’ll keep an eye out for a Rev B PSU to match with my other.

Thanks!

That’s odd. When you say the switch refused to lower the fan speeds, do you mean it audibly didn’t lower, or that “show chassis” continued to show the fan in high mode?

I have an ICX6610-48P with dual C power supplies. It definitely ramps down with either one or two power supplies and dual fans. However, with dual PSUs, the fans clearly run at a lower RPM and are noticeably quieter (49dB versus 55-57dB). Running with a single PSU and dual fans also runs cooler, presumably due to the higher fan RPM.

I also tested with a single PSU and single fan in the left slots. It ramped down as well but it was too hot for my preference. The second fan really drops the temperatures. I’m also using two 2.5W 40GBase-LR4 Lite optics and seven mostly 10G-LR optics, so the extra airflow helps.

 

[fohdeesha]

This didn't seem to be the case.
With either PSU and Fan in either of the left or right slots, the system would refuse to lower the fan speeds after a full boot.
For shits and giggles, I put the B PSU and fan in the left slots, and the A PSU in the right.
The system has yet to ramp it's fans up.

So I guess that’s my bad thinking I could get away with only 1 PSU.
I’ll keep an eye out for a Rev B PSU to match with my other.

Thanks!

pretty sure it just runs the fan annoyingly higher on the 24F version, as it presumably has 24x power hungry SFP transceivers to cool

 

[Chewza]

pretty sure it just runs the fan annoyingly higher on the 24F version, as it presumably has 24x power hungry SFP transceivers to cool

sorry, my bad, it's not the 24F, it's just the straight 24 (8x SFP+ in the front with 24x 1gbe RJ45, and the stacking module)

 

[fohdeesha]

sorry, my bad, it's not the 24F, it's just the straight 24 (8x SFP+ in the front with 24x 1gbe RJ45, and the stacking module)

correction: pretty sure I'm wrong and have no idea

 

[Chewza]

correction: pretty sure I'm wrong and have no idea

Regardless of whether you may or may not be wrong haha, I'm happy with the noise from the B+A PSU's for the time being.
Again, appreciate everything you've done for us with the documentation and testing!

 

[covfefe]

How did you guys install an ASIC fan in the ICX7250? I can't find good mounting points. Did you make holes in the top case, and fastened it there?

 

[Freebsd1976]

How did you guys install an ASIC fan in the ICX7250? I can't find good mounting points. Did you make holes in the top case, and fastened it there?

no need make holes , for mounting low speed 6010 or 6015 fan , tapping Screw is enough ,