Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[itronin]

Of course, after I post all of that, I see all the RX errors... gonna try a new cable.

see this post (and previous posts by OP). TLDR: While the 710 did not work, a low cost (used) mellanox CX3 did. Testing done using DAC, maybe AOC will work as it was not tested.

 

[selta]

Interesting. I see at least 2-3 people here have the XL710 and the same issue I am having. I'm not going to sink more money into 40GbE here though -- my R640 came with the XL710, and was the only reason I was going to try to use it. I'll just use 10GbE and call it good.

 

[tillburn]

So based on the configs you've posted, you should have a working configuration. All of the VLANs extend across the switches, and you've got an untagged port on VLAN 50 on both switches, so devices on those ports should be able to talk to each other.

What problems are you experiencing now?

I am reading up on spanning tree and rapid spanning tree, is there any primer you recommend? I feel like I am having some congestion on the network, specifically on the vlan carrying the cable modem as after about 6 hours the download speeds seems to be very slow, but if I reset both routers and the wireless AP the speeds for download are then just fine.

Physically it's laid out like this:

7250 port 1/1/1 vlan 50 ----->cable modem
7250 port 1/1/2 ------> unmanaged switch -----> 3x PC
7250 port 1/1/48 POE ----->wap (Vlans unimplemented yet).
7250 port 1/2/7 ----->10g Computer
7250 port 1/2/1 ----->10g LAG 1 all vlans -----> 6610

6610 port 1/1/1 vlan 50 ----->Pfsense VM
6610 port 1/1/2 Hypervisor 00
6610 port 1/1/3 Hypervisor 01
6610 port 1/1/48 POE ----> wap (Vlans unimplemented yet).
6610 port 1/3/1 -----> 10g LAG 1 all vlans------> 7250
6610 port 1/3/7 -----> 10g VM NAS
6610 port 1/3/8 -----> 10g VM Pfsense LAN

Any advice on spanning tree/rstp or if there potentially something else plaguing the network that I should be aware of please let me know.

 

[Coach43]

Not a network guy, but LOVE computers and what they can do!
Thank you Fohdeesha - I have TWO ICX6610's fully licensed and trunked via a 40Gb fiber optic link!
Instructions worked like a charm, but did not instill all layer 2 and layer 3 knowledge in me (I've been reading a lot of this 408 page forum etc, but still...)

The remote unit is a 6610P and I am now setting up security cameras. I'd like to isolate them onto their own vlan so they can't phone home to china or get accessed by the internet, but so they can still connect to my other internal systems. My firewall and DHCP are hosted upstream from the routers and that all works fine. I can easily add this new network there, I just need to make sure my 6610's know what's going on.

I think it's a few lines for those of you that know what you are doing.

My default network 192.168.1.0 and I'd like the new one to be 192.168.10.0.
I'm running router code.

Thanks in advance!


P.S. I'm interested in a 'Router Programming in a Nutshell' reference. Open to recommendations.

 

[Vilmalith]

Would the icx6450 poe be a noticable upgrade from some Netgear multigig switches, even with some stuff like aps running multigig?

 

[msg7086]

I wonder if there are other branded switches that are as affordable, power efficient, noise friendly, multi functional as those?

Looking for a new switch that's going to the furnace/tech room with 5 PoE and 8 non-PoE cat6 in a new construction home. My ICX6450-48P sounds a bit too hot, noisy, and overkill in that scenario, but I couldn't find much cheap alternative that can do some PoE, cool and quiet, and with a 10G SFP port. Considering mikrotik 8P-2S+ but it's $200+. Maybe I should just throw the 6450 in and call it a day.

Sorry for the derailment.

 

[kpfleming]

I am reading up on spanning tree and rapid spanning tree, is there any primer you recommend? I feel like I am having some congestion on the network, specifically on the vlan carrying the cable modem as after about 6 hours the download speeds seems to be very slow, but if I reset both routers and the wireless AP the speeds for download are then just fine.

Your connection configuration looks fine. You've got a 10G link between the ICX devices which should be plenty, and STP/RSTP should not be a concern if there aren't any switching loops. Since you only have two switches it's unlikely you have any loops.

You shouldn't need two physical links from the 6610 to the pfSense VM; one should be plenty, with two VLANs (one tagged and one untagged, until you setup the other VLANs) on it. Still, it's not a problem to have two links.

When you say 'reset both routers' are you referring to the ICX boxes?

 

[Mark]

Picked up an ICX-7650-48ZP on ebay and it has not arrived yet. Wondering if anyone can confirm actually how the breakouts work or don't? The documentation says the backports can be used for uplink but then that will not allow the front 4x10gb ports to work and there is no indication about the ability to actually breakout the back ports.

 

[tillburn]

Your connection configuration looks fine. You've got a 10G link between the ICX devices which should be plenty, and STP/RSTP should not be a concern if there aren't any switching loops. Since you only have two switches it's unlikely you have any loops.

You shouldn't need two physical links from the 6610 to the pfSense VM; one should be plenty, with two VLANs (one tagged and one untagged, until you setup the other VLANs) on it. Still, it's not a problem to have two links.

When you say 'reset both routers' are you referring to the ICX boxes?

Yes that was a typo, I did mean to say both switches (ICX boxes).

 

[TonyArrr]

Hi all, this thread is amazing, so much knowledge.
At 400 pages, I kind of wonder if it should be a subsection all of its own

I’ve been looking for a switch for home in advance of wiring up the apartment, and am very much sold on a 24 port 7250, or 7450 if I could find one functioning cheap enough.

I’ve read a bunch about fan mods, exploring moving enough air quietly enough, adding fans to the ASIC etc, and I’m wondering if anyone has explored installing larger heatsinks (or smaller ones with integrated fans), or more conductive thermal interface material, as a way to make it easier for heat to move out of the PCB and into the path of the airflow?

I haven’t seen anything going that way, aside from the fan mod attaching a fan on the existing ASIC heatsinks, or installing fans on the top casing blowing into the heatsinks.

Or are the heatsinks not removable? Obviously not gonna get better thermal conductivity than if they’re soldered to the ICs, but I haven’t found any info either way

 

[fleeball]

Hi All

Have had the ICX6610-48P running fine for ages. Decided to upgrade to the latest firmware(8030u) and bootloader as I was having problems with PoE. Firmware upgrade went all ok, but now I can not log into the web gui. I can get to the web gui via an ip address but it does not accept the username and password that was setup before the firmware upgrade. Previous username was root and password root also. I don't have a console cable lying around at the moment and need to enable PoE. Does anyone know what the default username and password are for the latest firmware? Or have any ideas as to how I can enable PoE without a console cable( I know close to impossible).

 

[audiobahn]

If you can ssh in, then follow the instructions here ICX6xxx Advanced - Fohdeesha Docs .

Sounds like you need...
aaa authentication web default local

If you can't ssh in, then start with FCX / ICX6610 - Fohdeesha Docs ...

 

[fleeball]

Thanks for the reply and I did look at those instructions, but a far as I can tell the commands can only be executed via the console port. If I try to SSH it is asking to login.

 

[audiobahn]

If nothing works you might just need to connect via the console port, worst case scenario you need to buy a LONG Ethernet cable they come in handy though I have a bad habit of cutting out smaller ones when I’m need until they’re no longer long enough!

 

[fleeball]

Yeap just ordered a console cable, seems as that is going to be the way I can get back into the switch. Thanks for your suggestions..

Update: Console cable resolved the issue.

 

[OKGolombRuler]

Having some trouble with DHCP relay (read: ip helper) on an ICX7150-24 L3 switch running 8.0.95h routing firmware. The DHCP server is running on a virtual Mikrotik, on an esxi server hung off one of the switchports. When I trunk the VLANs through to the mikrotik, and set up individual DHCP servers on each VLAN, no problem-- the MT L3 VLAN interface sees the broadcast, responds, everybody's happy. When I configure IP relay on the switch VEs, nothing happens-- the MT's loopback interface (192.168.101.1) with the dhcp server doesn't see packets, and I don't seem to see anything in the switch logs, even with a few debug flags (e.g. dhcp_relay) enabled. I confirmed that I can ping the MT loopback from the switch, and specifically from the VE source IP.

Here's a sample config on the VE port:

#sh run int ve7
interface ve 7
port-name guest VE
ip address 192.168.7.254 255.255.255.0
ip bootp-gateway 192.168.7.254
ip directed-broadcast
ip helper-address 1 192.168.101.1
ipv6 enable
ipv6 nd router-preference high
ipv6 nd ra-dns-server 2620:fe::11
ipv6 nd ra-dns-server 2620:fe::fe:11
!

and from the top-level config:

#sh run | incl dhcp
ip dhcp snooping vlan 7 9
ip dhcp-client disable
ip dhcp-server relay-agent-echo enable
dhcp snooping trust
dhcp snooping trust
dhcp snooping trust

#sh run | incl gate
ip bootp-gateway 192.168.7.254
ip bootp-gateway 192.168.9.254


#sh debug                                                                 
Debug message destination:  Console                                                                 
Debug message destination:  SSH session 1                                                           
Generic IP:                                                                                         
          DHCP SNOOPING:  debugging is on                                                           
          DHCP RELAY:  debugging is on                                                              
UDP:                                                                                                
          UDP:  debugging is on

I've reloaded the switch just to be sure.

Three questions for the Brain Trust:

1. What if any other commands might I want to run for debugging / diagnostic purposes?
2. Am I wrong in expecting the MT loopback (101.1) to see IPv4 UDP port 67 packets 'stamped' from 192.168.7.254?
3. Anybody who's gotten ip helper/dhcp relay working, got any ideas to try or config to share?


Actually I'll add a fourth, unrelated-but-tangential question to save thread clarity:

4. I'd like to be able to push traffic going 'up the stick' between the switch and the virtualized router 'out of' one ethernet port and 'back into' another, so I can jumper in (and then back around) a physical bump-in-the-wire device I'd like to play with, without changing any other configs. It seems (per the 8.0.95 L3 manual) that 7150s don't do VRFs. Any ideas on how to do this gracefully with my 7150?

 

[OKGolombRuler]

ETA the MT DHCP server config:

      name="guest1" interface=loopback relay=192.168.7.254 lease-time=2h address-pool=xx7x
      bootp-support=dynamic bootp-lease-time=lease-time server-address=192.168.101.1
      authoritative=yes use-radius=no conflict-detection=no lease-script=""
      allow-dual-stack-queue=no

The MT is running latest, 7.7 I believe.

 

[Matta]

Greetings !
Recently I've acquired ICX6610-48p and I just wanted to use it as a switch (I like those SFP+ ports) . I just gave it IP address within main subnet and that's it. Most of my equipment is Mikrotik (main router, switches, APs).
I have two virtual WLANs on all APs and everything is working like it should when Brocade is not in equation (clients are getting IP addresses, can access internet, speed limitation, etc.).
When I connect some APs (or switches with connected APs) to Brocade, wifi clients on virtual WLANs are not getting IP addresses.
Main subnet is working without issues, both wifi clients and computers connected to LAN ports.

My question is (since this is my first contact with industry-grade switch):
- Do I need to config VLANs also on Brocade, in order that virtual WLAN clients could get IP addresses and access internet without problems ?

It's maybe dumb for most of you but, as I said, this is my first with these kind of switches and I'm clueless.

Thanks !

 

[OKGolombRuler]

ETA the MT DHCP server config:

      name="guest1" interface=loopback relay=192.168.7.254 lease-time=2h address-pool=xx7x
      bootp-support=dynamic bootp-lease-time=lease-time server-address=192.168.101.1
      authoritative=yes use-radius=no conflict-detection=no lease-script=""
      allow-dual-stack-queue=no

The MT is running latest, 7.7 I believe.

Further edited to add, apparently I don't understand debug destinations. After getting a firehose to the face and a little no debug udp, I got some useful logs:

Debug: Jan 29 11:38:20 DHCP_RELAY : Relay agent rxed message on port v7
Debug: Jan 29 11:38:20 DHCP_RELAY : Received Bootp request msg on port v7
Debug: Jan 29 11:38:20 DHCP_RELAY : Bootp request packet send with GiAddr 192.168.7.254
Debug: Jan 29 11:38:20 DHCP_RELAY : Bootp request packet sent on port INVALID,With helper-address 192.168.101.1
Debug: Jan 29 11:38:20 DHCP_RELAY :restored gi_l2_action_required: 1
Debug: Jan 29 11:38:25 DHCP_RELAY : Relay agent rxed message on port v7
Debug: Jan 29 11:38:25 DHCP_RELAY : Received Bootp request msg on port v7
Debug: Jan 29 11:38:25 DHCP_RELAY : Bootp request packet send with GiAddr 192.168.7.254
Debug: Jan 29 11:38:25 DHCP_RELAY : Bootp request packet sent on port INVALID,With helper-address 192.168.101.1
Debug: Jan 29 11:38:25 DHCP_RELAY :restored gi_l2_action_required: 1
Debug: Jan 29 11:38:29 DHCP_RELAY : Relay agent rxed message on port v7
Debug: Jan 29 11:38:29 DHCP_RELAY : Received Bootp request msg on port v7
Debug: Jan 29 11:38:29 DHCP_RELAY : Bootp request packet send with GiAddr 192.168.7.254
Debug: Jan 29 11:38:29 DHCP_RELAY : Bootp request packet sent on port INVALID,With helper-address 192.168.101.1
Debug: Jan 29 11:38:29 DHCP_RELAY :restored gi_l2_action_required: 1

...which suggests that it's seeing the DHCP DISCOVER, stamping it correctly, setting the proper unicast destination... and then trying to send it out a nonexistent interface? Because the DHCP server is not on a directly-connected interface (have to route packet to get there)? Am I reading that right? Because it makes no sense to me that it wouldn't just... route that unicast traffic like all the rest of it...

 

[kpfleming]

Does the ICX have a route to 192.168.101.1? What is in the IPv4 routing table on the ICX?