Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[audiobahn]

Thanks!.Wow, I gotta try that craft knife trick. Almost in despair that I cannot get a simple adapter module out of the switch

No problem. It really doesn't help that they are placed back to back so you have extremely limited space to work with. Open the latch and try pulling it out as you would normally do. It'll come up about 1mm which is just enough to see the little notch and put the blade in, lots of light helps. Once you get it to click it comes out VERY easy. Good luck.

 

[epicurean]

No problem. It really doesn't help that they are placed back to back so you have extremely limited space to work with. Open the latch and try pulling it out as you would normally do. It'll come up about 1mm which is just enough to see the little notch and put the blade in, lots of light helps. Once you get it to click it comes out VERY easy. Good luck.

You Sir are a life saver. The craft knife trick worked for me. Much thanks!

 

[audiobahn]

You Sir are a life saver. The craft knife trick worked for me. Much thanks!

Fantastic! Great to hear. Certainly a new life skill acquired for both of us. Let's hope we don't need it again.

 

[audiobahn]

Just a sanity question as I'm getting to know my new router...

All ports seem to work fine but the LED of port 1/1/11 was solid amber instead of solid green. I looked into it further and it seemed to sync as expected at 1Gbit and work fine however it showed 1 CRC error. After resetting / clearing its log the light went back to green and it doesn't seem to have any further problems.

So is that it? Problem fixed? Is it default behaviour to "latch" on Orange until the log is cleared, even if there's no issues?

 

[Jason Antes]

I recently decided to flip my ICX-6450-24 to the back of my rack to simplify some cable management. That part went well, but now I have a hot switch that's trying to pull air from the back of a crowded cabinet and it's hovering around 70C.

I know it's physically possible to flip the fan around where it sits, but is that going to provide adequate air flow? Or is there a way to mount it such that it blows out the side?

Update: I started reading the thread from the beginning and found the answer to my question:https://forums.servethehome.com/ind...gbe-40gbe-switching.21107/page-35#post-205200

In sum: yes just reversing the fan orientation is fine.

My switch has the port exhaust fans and I have a plenum for it to get fresh air from the front to the switch.

 

[tillburn]

I need some config advice.

I have a 7250 flashed, licensed, racked up with UPS + cable modem, POE wifi AP + home office all wired up working fine.
I have a 6610 flashed, licensed, racked up with UPS + NAS, Disk shelf, Hypervisor machines, misc servers, Starlink and POE wifi AP in the shop.

Physically like this: Home office, 6610 1/3/1 mm-LC OM4 fiber--------------------------------------> Shop, 7250 1/2/1 mm-LC OM4 fiber.

I only have the single mm-LC 100 meter fiber cable linking the shop to the office.

7250/48p is 10.10.100.2
6610/48p is 10.10.100.3

Can access both switches via ssh on the network.

Both switches have vlans configured this way:

vlan 1 name DEFAULT-VLAN by port
router-interface ve 1

vlan 10 name management by port
vlan 20 name servers by port
vlan 30 name IOT by port
vlan 40 name sec_cam by port
vlan 50 name cablenet by port
vlan 51 name starlink by port
vlan 777 name native by port

I would like to use vlans to tunnel the cable modem to the rack in the shop, host either a physical or VM instance of pfsense and divide up the network with vlans to keep things more secure. Setting up pfsense with dual wan isn't a problem, but tunnelling the cable modem to the shop is a hurdle I need to clear.

I have two EnGenius EWS357AP AP's that do have mesh, but I haven't been impressed with it's bandwidth or reliability (it's disabled). I intend to run both AP's with multiple SSIDs, tagging SSID's and using a management vlan to control them. I was able to pull that fiber through the electrical conduit from the house to the shop. I can't pull any more fiber through that conduit, it's full I was lucky to get this one cable through, took 3 attempts and lots of lube...

At first I thought I could stack them, run them as a single physical switch with one linear fiber connection, but then I found out that doesn't work because they aren't the same physical switch and don't have the same firmware... Can't buy another 7250 because the price is through the roof STH members must have gobbled up most of them

I very much want to do this myself where I am stuck is finding examples using brocades terms/commands and or translating cisco to brocade T Henry's videos are helpful, but vlans, VE's etc. are tripping me up. I read Kapone's post (page 75ish) in this thread and it seems similar to what I want to do but I have the extra element of having two switches, two locations and vlans.

When I add the physical ports to the vlans, wr mem on the 6610 in the shop and then do the same for the 7250 in the office I lose connection between the two. So clearly I am missing steps for a route, not adding the vlans to the port(s) on the switch(s) or something else that isn't jumping out at me.

Suggestions, pointers, anyone willing to translate above to a check list of things to do to get up and running I would appreciate it!

 

[AllenB]

Hi all, I've got a multigig question -- have searched this thread and found some partial answers, but nothing complete. Perhaps this will be of use to others.

I'm using an ICX7250-24p as my home switch. Love it. Got a server and two workstations on 10GbE, some APs on PoE, and then various other connections.

Now that faster Internet speeds are available, I'm looking at getting a DOCSIS modem with a 2.5GbE port. And that's where the trouble starts.

It appears that 2.5/5 GbE support was never added to the ICX7250, even in the latest 9.0.x releases. Is that correct? And if so, it strongly implies that using a "basic" multigig SFP+ adapter won't help, either.

I've seen references to one or more SFP+ to copper adapters which can handle the data rate conversion from 2.5/5.0 to 10 gbps. I've not been able to actually pin down model numbers, or find a search term that hits these.

So, is there a solution for getting 2.5 GbE into an ICX7250? I'd prefer to avoid a 2nd switch. The fallback is to put a 2.5GbE NIC into the server but that means losing all Internet connectivity if it goes down.

Thanks,
Allen

 

[LodeRunner]

I need some config advice.

I have a 7250 flashed, licensed, racked up with UPS + cable modem, POE wifi AP + home office all wired up working fine.
I have a 6610 flashed, licensed, racked up with UPS + NAS, Disk shelf, Hypervisor machines, misc servers, Starlink and POE wifi AP in the shop.

Physically like this: Home office, 6610 1/3/1 mm-LC OM4 fiber--------------------------------------> Shop, 7250 1/2/1 mm-LC OM4 fiber.

I only have the single mm-LC 100 meter fiber cable linking the shop to the office.

7250/48p is 10.10.100.2
6610/48p is 10.10.100.3

Can access both switches via ssh on the network.

Both switches have vlans configured this way:

vlan 1 name DEFAULT-VLAN by port
router-interface ve 1

vlan 10 name management by port
vlan 20 name servers by port
vlan 30 name IOT by port
vlan 40 name sec_cam by port
vlan 50 name cablenet by port
vlan 51 name starlink by port
vlan 777 name native by port

I would like to use vlans to tunnel the cable modem to the rack in the shop, host either a physical or VM instance of pfsense and divide up the network with vlans to keep things more secure. Setting up pfsense with dual wan isn't a problem, but tunnelling the cable modem to the shop is a hurdle I need to clear.

Addressing only the cable modem:
Shouldn't be a problem. Plug the cable modem into a port, set that port as untagged VLAN 50, make sure your trunk port to the other switch properly tagged, then if pfSense is physical, whatever port it's using for WAN, set to untagged VLAN 50. If you're trying to trunk everything into pfSense, can't help you there, I've never bothered with VLANs directly on pfSense. Mine's a VM, so one VNIC per interface instead of VLAN tagging and sub-interfaces.

I take an ethernet handoff from my ISPs ONT straight into a switch with a L2 untagged port, that gets handed to the VM cluster over a trunk port and pealed out as VNIC that pfSense sees as physical interface.

Using your VLANs and assuming you want to trunk all of them between both switches, here'res the basic commands for the 7250, where 1/2/1 is the 10G port you're using for uplink, and for the example, your cable modem is on 1/1/1:

vlan 10
tag e 1/2/1
vlan 20
tag e 1/2/1
vlan30
tag e 1/2/1
vlan 40
tag e 1/2/1
vlan 50
tag e 1/2/1
untag e 1/1/1

The 6610 side is probably the same unless you need tagged and untagged traffic on the same port, in which case there are extra steps that I am not conversant in.

You mention hypervisors, so if your pfSense is virtualized, then the ports to your VM hosts would also be tagged traffic, just like the inter-switch trunk ports. Then you'll setup VNICs using the relevant VLAN IDs.

For example, on my Hyper-V host, this is my opnSense VM:

VLAN 2 is the VLAN my fiber ONT is connected to. Each of those VNICs is tagged to a different VLAN. I personally prefer handling it at his level rather than inside opnSense.

 

[LodeRunner]

Hi all, I've got a multigig question -- have searched this thread and found some partial answers, but nothing complete. Perhaps this will be of use to others.

I'm using an ICX7250-24p as my home switch. Love it. Got a server and two workstations on 10GbE, some APs on PoE, and then various other connections.

Now that faster Internet speeds are available, I'm looking at getting a DOCSIS modem with a 2.5GbE port. And that's where the trouble starts.

It appears that 2.5/5 GbE support was never added to the ICX7250, even in the latest 9.0.x releases. Is that correct? And if so, it strongly implies that using a "basic" multigig SFP+ adapter won't help, either.

I've seen references to one or more SFP+ to copper adapters which can handle the data rate conversion from 2.5/5.0 to 10 gbps. I've not been able to actually pin down model numbers, or find a search term that hits these.

So, is there a solution for getting 2.5 GbE into an ICX7250? I'd prefer to avoid a 2nd switch. The fallback is to put a 2.5GbE NIC into the server but that means losing all Internet connectivity if it goes down.

Thanks,
Allen

Several threads here, this is the most recent one I recall: https://forums.servethehome.com/ind...y-sfp-adapters-to-connect-2-5-gbe-rj45.38585/
I have no direct experience with these modules. From what I've read AQS-107 based SFPs (real ones, not knock off 'compatible' ones) and the Mikrotik S+RJ10 seem like best bets. Buy from somewhere with an easy return policy.

I know you want to avoid a second switch, but the TEG-3102WS seems fairly priced for a 8x 2.5Gbe + 2x SFP+ switch, or the TEG-S762 for a 4x 2.5Gbe + 2x 10Gbe switch (you'd need a plain 10G RJ45 SFP on the 7250). QNAP also has some low port count 2.5Gbe switches with combo 10G SFP+/RJ45 ports.

 

[AllenB]

Several threads here, this is the most recent one I recall: https://forums.servethehome.com/ind...y-sfp-adapters-to-connect-2-5-gbe-rj45.38585/
I have no direct experience with these modules. From what I've read AQS-107 based SFPs (real ones, not knock off 'compatible' ones) and the Mikrotik S+RJ10 seem like best bets. Buy from somewhere with an easy return policy.

I know you want to avoid a second switch, but the TEG-3102WS seems fairly priced for a 8x 2.5Gbe + 2x SFP+ switch, or the TEG-S762 for a 4x 2.5Gbe + 2x 10Gbe switch (you'd need a plain 10G RJ45 SFP on the 7250). QNAP also has some low port count 2.5Gbe switches with combo 10G SFP+/RJ45 ports.

Great, thanks for clarifying. I confess, I searched this huge thread but not the other forum threads.

If I needed more than one or two multi-gig ports, I'd definitely go with a switch like one of those. For a single port though, fitting it into the existing switch makes the most sense. Some of the AQS-107 stuff is expensive enough to trade off against a 2nd switch, but the Microtik and perhaps others are affordable. I'll have a look at buying one of those. Thanks!

 

[clix00]

Maybe someone can help tell me what I am doing wrong here.

What my end result is to have this become my primary switch to consolidate my rack. Current network config has the primary + 4 VLANS. However, I am having a hard time getting my Brocade (FCX648S-HPOE) to pass DHCP from any VLAN ports.

Currently, VLAN1, is working for DHCP, but I cannot get any other VLAN to work. Do I need create a VE for every VLAN? While writing this, I did try to ping the routers for the other VLANs with no response from the Brocade, however, I can ping it from my local computer that is ported through the same unmanaged switch as the Brocade.

Any suggestions on what I am doing wrong to get the VLANs set correctly?

Spoiler: sh run

telnet@ToroTheBull#sh run
Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 fcx-48-poe-port-management-module
module 2 fcx-cx4-2-port-16g-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 29 by port
tagged ethe 1/1/3
untagged ethe 1/1/4
router-interface ve 29
!
!
!
!
!
hostname ToroTheBull
ip dhcp-client disable
!
!
!
!
!
!
!
!
!
!
interface ve 1
ip address 10.29.27.87 255.255.255.0
!
interface ve 29
ip address 192.168.100.2 255.255.255.0
ip helper-address 1 192.168.100.1
!
!
!
!
!
!
!
!
!
end

 

[kpfleming]

Where is your DHCP server running? Is it connected to a trunk port with membership in all of the VLANs with DHCP clients, and configured to serve each of those VLANs?

This is very likely not a problem with the switch configuration at all, but more likely not having the proper type of configuration in place across the switch and DHCP server.

 

[clix00]

Where is your DHCP server running? Is it connected to a trunk port with membership in all of the VLANs with DHCP clients, and configured to serve each of those VLANs?

This is very likely not a problem with the switch configuration at all, but more likely not having the proper type of configuration in place across the switch and DHCP server.

It is running on my EdgeRouterX--all ports are running 1,29,30,39, and 60.



Knowing that all VLANs are coming from the ERX port, is there something I need to do on the port leading into the Brocade?

 

[kpfleming]

Yes, the port on the Brocade switch that connects to the ERX needs to have all of the tagged VLANs on it, otherwise the switch won't accept or forward any traffic for that VLAN on that port.

You also will not need any 'ip helper' configuration, your DHCP server and the DHCP clients will be talking to each other at layer 2, not layer 3.

 

[clix00]

Yes, the port on the Brocade switch that connects to the ERX needs to have all of the tagged VLANs on it, otherwise the switch won't accept or forward any traffic for that VLAN on that port.

You also will not need any 'ip helper' configuration, your DHCP server and the DHCP clients will be talking to each other at layer 2, not layer 3.

That would be it. I removed the ip helper as well.

If I bring all those over tagged on say 1/1/48, what do I do about getting vlan1 access--it seems tagging the inbound port kills that.

 

[kpfleming]

Just leave it untagged on that port.

 

[clix00]

When I tag the port for 29, it removes untagged vlan 1.

Spoiler: show vlan

telnet@ToroTheBull#show vlan
Total PORT-VLAN entries: 5
Maximum PORT-VLAN entries: 64

Legend: [Stk=Stack-Id, S=Slot]

PORT-VLAN 1, Name DEFAULT-VLAN, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1) 1 2 5 6 7 8 9 10 11 12 13 14
Untagged Ports: (U1/M1) 15 16 17 18 19 20 21 22 23 24 25 26
Untagged Ports: (U1/M1) 27 28 29 30 31 32 33 34 35 36 37 38
Untagged Ports: (U1/M1) 39 40 41 42 43 44 45 46 47
Untagged Ports: (U1/M2) 1 2
Tagged Ports: None
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

PORT-VLAN 29, Name [None], Priority level0, Spanning tree Off
Untagged Ports: (U1/M1) 4
Tagged Ports: (U1/M1) 3 48
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

EDIT: dual-mode wasn't enabled....

Thanks for the help!

 

[erik29gamer]

Recently adding some more 10gb connections to the ICX7150-24 I've been using for a few years. I previously had a proxmox host with a connectx-3 (no longer in use) and a synology nas, also with a connectx3 connected. Both worked great at full speed. I recently tried adding the spare CX3 to a new desktop, only to find it BSODs Windows 11. Instead, I purchased a Supermicro aoc-stgn-i2s. Unfortunately, having a ton of oddities with it.

Regardless of the sfp+ port I have it connected to, the link light on the switch is orange. When plugged into 3 of the 4 sfp+ ports, I can only get ~500mbps throughout in iperf, but while connected to one of them, close to what I expect (~8gbps). Additionally, one of the two ports on the card itself always gives me the same 500mbps, regardless of what port its attached to on the switch. I've tried two different dacs, as well as an ubuntu live usb, same result.

From what I've been able to find, orange link light isn't indicating full speed, but I can get the highest performance when both my NAS and desktop are lit orange on the switch...

I'm kind of at a loss as to what is happening really. Just a bad card?

 

[tillburn]

Addressing only the cable modem:
Shouldn't be a problem. Plug the cable modem into a port, set that port as untagged VLAN 50, make sure your trunk port to the other switch properly tagged, then if pfSense is physical, whatever port it's using for WAN, set to untagged VLAN 50. If you're trying to trunk everything into pfSense, can't help you there, I've never bothered with VLANs directly on pfSense. Mine's a VM, so one VNIC per interface instead of VLAN tagging and sub-interfaces.

I take an ethernet handoff from my ISPs ONT straight into a switch with a L2 untagged port, that gets handed to the VM cluster over a trunk port and pealed out as VNIC that pfSense sees as physical interface.

Using your VLANs and assuming you want to trunk all of them between both switches, here'res the basic commands for the 7250, where 1/2/1 is the 10G port you're using for uplink, and for the example, your cable modem is on 1/1/1:

vlan 10
tag e 1/2/1
vlan 20
tag e 1/2/1
vlan30
tag e 1/2/1
vlan 40
tag e 1/2/1
vlan 50
tag e 1/2/1
untag e 1/1/1

The 6610 side is probably the same unless you need tagged and untagged traffic on the same port, in which case there are extra steps that I am not conversant in.

You mention hypervisors, so if your pfSense is virtualized, then the ports to your VM hosts would also be tagged traffic, just like the inter-switch trunk ports. Then you'll setup VNICs using the relevant VLAN IDs.

For example, on my Hyper-V host, this is my opnSense VM:
View attachment 26710
VLAN 2 is the VLAN my fiber ONT is connected to. Each of those VNICs is tagged to a different VLAN. I personally prefer handling it at his level rather than inside opnSense.

Ok, I figured out where I was going wrong on the connection between the switches at least. I connected the switches using a dynamic lag (LACP) but I didn't know that I had to set a primary port and then "trunk deploy" command on the 6610 as it has older firmware.

So now both switches see each other and if I end up adding more power to the shop or whatever later, then I can dynamically add another 10g port to the LAG.

So how does the LAG effect traffic between the switches?
If I tag lag1 in vlan 10, 20, 30, 40, 50 that means only those vlans will traverse the lag1 right?

For now I need all traffic to traverse lag 1 so I can still remotely configure the 6610 switch and all the hypervisors on that side of the lag1 connection as I have yet to deploy pfsense on the designated hypervisor. Right now the cable modem and the temporary router is on the office side (7250) of the lag1, I am working on finishing that up, but it's going to take some doing with writing the correct rules and such for pfsense to be ready for the cable modem on vlan 50.

Any advice would be appreciated.

 

[kpfleming]

Yes, you will need to add (as untagged or tagged) all VLANs that you want to traverse any port or LAG in the switch; the ICX devices don't have a pure 'trunk' mode where a port/LAG is automatically a member of all VLANs on the device.

The LAG is part of VLAN 1 untagged by default, so if you are using that for management connections then you're all set. You'll just need to add all of the tagged VLANs as well (and set 'dual-mode' if it's an ICX 66xx I think).