Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[jbrukardt]

i've been eyeballing the ZP series, the 7650 if available. true 1/2.5/5/10 multigig on 24 of the ports, along with 90w POE is awesome.

 

[i386]

i've been eyeballing the ZP series, the 7650 if available. true 1/2.5/5/10 multigig on 24 of the ports, along with 90w POE is awesome.

I've a 7150-48zp and it's the loudest device in the rack. 90w per port needs a lot of cooling...

 

[DouglasteR]

I got another 6610, non POE this time

After the initial sanitization procedures, i can't access it's web config page, nor i can ping it.

I have set it's ip as 192.168.0.19, which is an unused ip on my subnet.

Strangely, it seems to be locked in "router" mode ?!




The switch is factory reseted and updated as instructed by Fohdeesha Docs.

 

[TonyArrr]

I got another 6610, non POE this time

After the initial sanitization procedures, i can't access it's web config page, nor i can ping it.

I have set it's ip as 192.168.0.29, which is an unused ip on my subnet.

Strangely, it seems to be locked in "router" mode ?!

View attachment 49447
View attachment 49448

The switch is factory reseted and updated as instructed by Fohdeesha Docs.

Just double checking you meant 192.168.0.19, not 29? Your screenshots show 19.
(This IS a mistake I would and have made, btw, so no harm no foul if it is!)

other than that, there is a command for enabling/disabling authentication for web, console, and other interfaces like that, maybe it needs to be set explicitly for the web interface? I just don’t recall what it is off the top of my head, but I do think it is mentioned in the Fohdeesha docs “further configuration” or something like that…
I’m out for the rest of the night but if no one else has found it by tomorrow morning I’ll take a gander and see if I can find it

 

[DouglasteR]

Just double checking you meant 192.168.0.19, not 29? Your screenshots show 19.
(This IS a mistake I would and have made, btw, so no harm no foul if it is!)

other than that, there is a command for enabling/disabling authentication for web, console, and other interfaces like that, maybe it needs to be set explicitly for the web interface? I just don’t recall what it is off the top of my head, but I do think it is mentioned in the Fohdeesha docs “further configuration” or something like that…
I’m out for the rest of the night but if no one else has found it by tomorrow morning I’ll take a gander and see if I can find it

Its totally a common mistake , but was a typo. (i have done it lots of times too lol)

But even before the web interface i should be able to ping the switch (like the one i did before).

What is catching my eye is the "router" in the console prompt, indicating that something may be wrong in it´s routing configs.

I will try to do a complete reset in it later today !

Thanks !

 

[DouglasteR]

Even more strange, i can ping the router from the 6610:


But i can't ping the 6610 from the router

 

[hmw]

i've been eyeballing the ZP series, the 7650 if available. true 1/2.5/5/10 multigig on 24 of the ports, along with 90w POE is awesome.

I have a post somewhere in this thread about the noise levels between a 6610 POE and a 7650 ZP. I had to add acoustic treatment to the rack AND it’s in the basement. And it’s still very clearly audible. Not to mention that it consumes 180W with just one PSU. It’s a great switch and gives me 10g everywhere + 100G to my virtualization servers, NAS and homelab but at this point I would rather use a non POE multi gig and the few POE ports perhaps power them with a smaller dedicated POE switch

edit: here’s the post

 

[DouglasteR]

Managed to fix all the issues by switch the serial cable, ugh !

Now the 6610 is operating correctly.

Thanks guys.

 

[DouglasteR]

Another question,

Is there a way to save the switch config to an external file ? For possible restore in the future ?

 

[TonyArrr]

Another question,

Is there a way to save the switch config to an external file ? For possible restore in the future ?

I genuinely am not aiming to be a smartarse, but here is a DuckDuckGo search. Seems there is a variety of ways, and even some GitHub repos of convenience utilities for it, so you’ve got some options

 

[i386]

Is there a way to save the switch config to an external file ? For possible restore in the future ?

This is what I do/use:
copy from switch to computer: copy startup-config tftp 192.168.1.3 7150-24P_01.config
copy from computer to switch: copy tftp startup-config 192.168.1.3 7150-24P_01.config

Replace "192.168.1.3" with your tftp hosts address and "7150-24P_01.config" with the name of your config file

 

[DouglasteR]

I genuinely am not aiming to be a smartarse, but here is a DuckDuckGo search. Seems there is a variety of ways, and even some GitHub repos of convenience utilities for it, so you’ve got some options

No problem at all, i even saw some options thru the web but i believe this thread is the GOAT for these things.

This is what I do/use:
copy from switch to computer: copy startup-config tftp 192.168.1.3 7150-24P_01.config
copy from computer to switch: copy tftp startup-config 192.168.1.3 7150-24P_01.config

Replace "192.168.1.3" with your tftp hosts address and "7150-24P_01.config" with the name of your config file

Great ! Thanks guys.

 

[seatrope]

Hi all!

I've been running for several years now very smoothly with a "core" ICX6610 performing L3 routing across several VLANs, with Opnsense firewall upstream (no routing, just firewalling) and a few "downstream" or "edge" switches (a mix of 6450 and 7150s), thanks to the extensive assistance from all the experts here including @kapone @kpfleming among others!

Recently I got a few more 7150s for expanding downstream/edge connectivity (i'm not sure if i'm using the terms core/edge appropriately).

The topology is:

ISP
|
Opnsense
|
ICX6610
/ / | \ \
7150 7150 7150 7150 7150

Even though things have been working, I realized I'm probably not configuring the downstream switches correctly. I have basically been mostly duplicating the config on the core ICX6610 in terms of VLANs, VEs, PIMs and ACLs, with the 6610 as the "gateway" address for the downstream 7150s.

It all works, but reading through some of the older posts it seems I should NOT be activating a VE/router interface on any of the "downstream" switches (7150s) but letting the 6610 do all the L3 routing between VLANs? Each 7150 does need to be VLAN aware as different ports are designated for different VLANs.

So, should the below be configured/active on any of the 7150 "downstream/edge" switches?
- VLANs (yes)
- VE/router interfaces (guessing NO)
- ACLs (unsure, perhaps only blocking access to the 7150 switch itself - all ACL blocks between VLANs to be done at the "core" 6610?)
- PIM (probably only at the 6610 as this is a routing function also)?

I guess the "advantage" to having L3 routing activated on the downstream 7150 is to preserve connectivity between devices on different VLANs on that 7150, IF the connection to the 6610 is down or the 6610 itself is down?

I have been searching this 553 page thread and have only found some peripheral information. My apologies if my search-fu was not sufficient!

Thanks everyone!

 

[Jelmer]

Both can work. L3 on the access switches saves bandwith if clients on the same switch in different vlans want to exchange bits.

 

[kapone]

experts here including @kapone @kpfleming among others!

Blast from the past! I've moved on to Mellanox SX6036 switches at my core, but...glad I could be helpful in the past.

ISP
|
Opnsense
|
ICX6610
/ / | \ \
7150 7150 7150 7150 7150

I too have multiple switches (have to/need to) and oddly I have a differing opinion from you in terms of connectivity. I did start off like you years and years (decades?) back, but it evolved. Now, mine's setup as:

ISP
|
SX6036 (designated VLAN 666 for WAN, no router interface on it)
/ / | \ \
switch1switch2 switch3 etc

Essentially, the WAN connection is terminated at the core switch, in a VLAN. Why? My firewall machine(s)/VMs can be anywhere on my network and as long as they have a connection that's been added to that specific VLAN, WAN access will work. It provides a lot of freedom in terms where to put the firewall, cabling etc etc. ANd if you wanna run an HA pair of firewalls...well, you have to do this (or a dedicated separate switch).

I have basically been mostly duplicating the config on the core ICX6610 in terms of VLANs, VEs, PIMs and ACLs, with the 6610 as the "gateway" address for the downstream 7150s

Pretty much. The downstream switches are essentially L2 extenders, not L3 routers.

I should NOT be activating a VE/router interface on any of the "downstream" switches

Correct.

but letting the 6610 do all the L3 routing between VLANs

Correct.

Each 7150 does need to be VLAN aware as different ports are designated for different VLANs

Correct.

So, should the below be configured/active on any of the 7150 "downstream/edge" switches?

- VLANs (yes) - Correct.
- VE/router interfaces (guessing NO) - Correct.
- ACLs (unsure, perhaps only blocking access to the 7150 switch itself - all ACL blocks between VLANs to be done at the "core" 6610?) - - Correct. You wanna do ACLs where the L3 routing is being done.
- PIM (probably only at the 6610 as this is a routing function also)? - Correct.

I guess the "advantage" to having L3 routing activated on the downstream 7150 is to preserve connectivity between devices on different VLANs on that 7150, IF the connection to the 6610 is down or the 6610 itself is down?

Correct. But this is a fairly edge case. I mean if your core is down...but pedantically if those two devices are on the same VLAN in those downstream switches, they can still talk, you just can't cross the L2 boundary, if the core switch is dead-ish.

You're pretty much spot on.

 

[seatrope]

@kapone thank you as always sad that you've moved on to the Mellanox but again appreciate all your help you've given me (and others)!

Yes, I thought about terminating the ISP to the 6610 in an isolated VLAN, but never got around to implementing it. Part of why is that, instead of having a HA firewall pair, I have a backup (shutdown) Atom 1U Opnsense on my rack. It's clearly labeled for my wife, if the main Opnsense Proxmox server dies, she can just unplug the two cables from the main FW and plug them into the labeled ports on the backup Opnsense and hit power on....

I guess there's probably a way to do the same with your setup without having to change routes on the switch (which my wife will never do)...

Edit - i guess you could just do something similar and it might be even easier as it would be 1 cable to change for my wife, instead of two.. as you are probably trunking VLAN 666 and whatever post-firewall VLAN back to the core switch on the same cable?

 

[seatrope]

Followup dumb question.

If I go the route of no VE/router interfaces on my downstream switches and assuming I'm still running L3 router software on the 7150 downstream switches, if I don't activate at least one VE on a management VLAN, is there any way to access the 7150 switch via SSH short of using the dedicated OOB Ethernet/serial port?

 

[kapone]

@kapone thank you as always sad that you've moved on to the Mellanox but again appreciate all your help you've given me (and others)!

Yes, I thought about terminating the ISP to the 6610 in an isolated VLAN, but never got around to implementing it. Part of why is that, instead of having a HA firewall pair, I have a backup (shutdown) Atom 1U Opnsense on my rack. It's clearly labeled for my wife, if the main Opnsense Proxmox server dies, she can just unplug the two cables from the main FW and plug them into the labeled ports on the backup Opnsense and hit power on....

I guess there's probably a way to do the same with your setup without having to change routes on the switch (which my wife will never do)...

Edit - i guess you could just do something similar and it might be even easier as it would be 1 cable to change for my wife, instead of two.. as you are probably trunking VLAN 666 and whatever post-firewall VLAN back to the core switch on the same cable?

There's scripts on the interwebs for it...but essentially CARP and VIPs. Basically:

- Your TRANSIT interface points to a VIP. Only the master *sense responds on it (the slave becomes a master, if the master is down). I use a /29 for my TRANSIT network so I can I have 8 IPs on it, although I really only need four.
- The "backup/slave" *sense essentially shuts down it's WAN interface if it's a slave. (Both the primary and slave have access to VLAN 666)
- The rules and states are synced between the two instances.

It's pretty seamless, less than a few pings for the switchover and completely automated.

I don't use *sense for DHCP/DNS anymore, I use Technitium, and that has it's own HA config. This is actually cleaner as Technitium handles DHCP/DNS/Ad blocking all in one place, and the firewall(s) just do...well, firewalling.

If I go the route of no VE/router interfaces on my downstream switches and assuming I'm still running L3 router software on the 7150 downstream switches, if I don't activate at least one VE on a management VLAN, is there any way to access the 7150 switch via SSH short of using the dedicated OOB Ethernet/serial port?

See...that sort of ties into my earlier response. My switches (all of them) get their OOB/management IPs from Technitium over DHCP. I have a single "jumpbox" that has access to this OOB network and it...just works? That jumpbox is a Windows 10 box as such, but severely hardened and only accessible by very very few IPs (My main wrkstation, my travel iPAD over wireguard, and my dev team lead in a foreign country...)

Edit: p.s. I have two circuits at home, a 10gb symmetric Comcast metro circuit and a 5gb symmetric FIOS circuit. It all performs...quite well. This is a Google fiber speed test (not many at these speeds) for my FIOS line.

 

[seatrope]

There's scripts on the interwebs for it...but essentially CARP and VIPs. Basically:

- Your TRANSIT interface points to a VIP. Only the master *sense responds on it (the slave becomes a master, if the master is down). I use a /29 for my TRANSIT network so I can I have 8 IPs on it, although I really only need four.
- The "backup/slave" *sense essentially shuts down it's WAN interface if it's a slave. (Both the primary and slave have access to VLAN 666)
- The rules and states are synced between the two instances.

It's pretty seamless, less than a few pings for the switchover and completely automated.

I don't use *sense for DHCP/DNS anymore, I use Technitium, and that has it's own HA config. This is actually cleaner as Technitium handles DHCP/DNS/Ad blocking all in one place, and the firewall(s) just do...well, firewalling.


See...that sort of ties into my earlier response. My switches (all of them) get their OOB/management IPs from Technitium over DHCP. I have a single "jumpbox" that has access to this OOB network and it...just works? That jumpbox is a Windows 10 box as such, but severely hardened and only accessible by very very few IPs (My main wrkstation, my travel iPAD over wireguard, and my dev team lead in a foreign country...)

Very cool. Never heard of Technitium. I never used *sense for DNS/DHCP. Always have a VRRP HA pair of piholes in my house, one in basement and one on 3rd floor both on UPSes. Seems there is significant feature overlap there though Technitium sounds like it does some things better.

 

[kapone]

Never heard of Technitium

Not surprised. It should have better visibility, but it doesn't, even though it's the closest replacement to a full featured DNS/DHCP (Windows AD vibes...) server. I think it's mostly because it's backed by a single developer and that poses some risks (even though he's super nice and responsive).