Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[LodeRunner]

I need some config advice.

I have a 7250 flashed, licensed, racked up with UPS + cable modem, POE wifi AP + home office all wired up working fine.
I have a 6610 flashed, licensed, racked up with UPS + NAS, Disk shelf, Hypervisor machines, misc servers, Starlink and POE wifi AP in the shop.

Physically like this: Home office, 6610 1/3/1 mm-LC OM4 fiber--------------------------------------> Shop, 7250 1/2/1 mm-LC OM4 fiber.

I only have the single mm-LC 100 meter fiber cable linking the shop to the office.

7250/48p is 10.10.100.2
6610/48p is 10.10.100.3

Can access both switches via ssh on the network.

Both switches have vlans configured this way:

vlan 1 name DEFAULT-VLAN by port
router-interface ve 1

vlan 10 name management by port
vlan 20 name servers by port
vlan 30 name IOT by port
vlan 40 name sec_cam by port
vlan 50 name cablenet by port
vlan 51 name starlink by port
vlan 777 name native by port

I would like to use vlans to tunnel the cable modem to the rack in the shop, host either a physical or VM instance of pfsense and divide up the network with vlans to keep things more secure. Setting up pfsense with dual wan isn't a problem, but tunnelling the cable modem to the shop is a hurdle I need to clear.

Addressing only the cable modem:
Shouldn't be a problem. Plug the cable modem into a port, set that port as untagged VLAN 50, make sure your trunk port to the other switch properly tagged, then if pfSense is physical, whatever port it's using for WAN, set to untagged VLAN 50. If you're trying to trunk everything into pfSense, can't help you there, I've never bothered with VLANs directly on pfSense. Mine's a VM, so one VNIC per interface instead of VLAN tagging and sub-interfaces.

I take an ethernet handoff from my ISPs ONT straight into a switch with a L2 untagged port, that gets handed to the VM cluster over a trunk port and pealed out as VNIC that pfSense sees as physical interface.

Using your VLANs and assuming you want to trunk all of them between both switches, here'res the basic commands for the 7250, where 1/2/1 is the 10G port you're using for uplink, and for the example, your cable modem is on 1/1/1:

vlan 10
tag e 1/2/1
vlan 20
tag e 1/2/1
vlan30
tag e 1/2/1
vlan 40
tag e 1/2/1
vlan 50
tag e 1/2/1
untag e 1/1/1

The 6610 side is probably the same unless you need tagged and untagged traffic on the same port, in which case there are extra steps that I am not conversant in.

You mention hypervisors, so if your pfSense is virtualized, then the ports to your VM hosts would also be tagged traffic, just like the inter-switch trunk ports. Then you'll setup VNICs using the relevant VLAN IDs.

For example, on my Hyper-V host, this is my opnSense VM:

VLAN 2 is the VLAN my fiber ONT is connected to. Each of those VNICs is tagged to a different VLAN. I personally prefer handling it at his level rather than inside opnSense.

 

[LodeRunner]

Hi all, I've got a multigig question -- have searched this thread and found some partial answers, but nothing complete. Perhaps this will be of use to others.

I'm using an ICX7250-24p as my home switch. Love it. Got a server and two workstations on 10GbE, some APs on PoE, and then various other connections.

Now that faster Internet speeds are available, I'm looking at getting a DOCSIS modem with a 2.5GbE port. And that's where the trouble starts.

It appears that 2.5/5 GbE support was never added to the ICX7250, even in the latest 9.0.x releases. Is that correct? And if so, it strongly implies that using a "basic" multigig SFP+ adapter won't help, either.

I've seen references to one or more SFP+ to copper adapters which can handle the data rate conversion from 2.5/5.0 to 10 gbps. I've not been able to actually pin down model numbers, or find a search term that hits these.

So, is there a solution for getting 2.5 GbE into an ICX7250? I'd prefer to avoid a 2nd switch. The fallback is to put a 2.5GbE NIC into the server but that means losing all Internet connectivity if it goes down.

Thanks,
Allen

Several threads here, this is the most recent one I recall: https://forums.servethehome.com/ind...y-sfp-adapters-to-connect-2-5-gbe-rj45.38585/
I have no direct experience with these modules. From what I've read AQS-107 based SFPs (real ones, not knock off 'compatible' ones) and the Mikrotik S+RJ10 seem like best bets. Buy from somewhere with an easy return policy.

I know you want to avoid a second switch, but the TEG-3102WS seems fairly priced for a 8x 2.5Gbe + 2x SFP+ switch, or the TEG-S762 for a 4x 2.5Gbe + 2x 10Gbe switch (you'd need a plain 10G RJ45 SFP on the 7250). QNAP also has some low port count 2.5Gbe switches with combo 10G SFP+/RJ45 ports.

 

[AllenB]

Several threads here, this is the most recent one I recall: https://forums.servethehome.com/ind...y-sfp-adapters-to-connect-2-5-gbe-rj45.38585/
I have no direct experience with these modules. From what I've read AQS-107 based SFPs (real ones, not knock off 'compatible' ones) and the Mikrotik S+RJ10 seem like best bets. Buy from somewhere with an easy return policy.

I know you want to avoid a second switch, but the TEG-3102WS seems fairly priced for a 8x 2.5Gbe + 2x SFP+ switch, or the TEG-S762 for a 4x 2.5Gbe + 2x 10Gbe switch (you'd need a plain 10G RJ45 SFP on the 7250). QNAP also has some low port count 2.5Gbe switches with combo 10G SFP+/RJ45 ports.

Great, thanks for clarifying. I confess, I searched this huge thread but not the other forum threads.

If I needed more than one or two multi-gig ports, I'd definitely go with a switch like one of those. For a single port though, fitting it into the existing switch makes the most sense. Some of the AQS-107 stuff is expensive enough to trade off against a 2nd switch, but the Microtik and perhaps others are affordable. I'll have a look at buying one of those. Thanks!

 

[clix00]

Maybe someone can help tell me what I am doing wrong here.

What my end result is to have this become my primary switch to consolidate my rack. Current network config has the primary + 4 VLANS. However, I am having a hard time getting my Brocade (FCX648S-HPOE) to pass DHCP from any VLAN ports.

Currently, VLAN1, is working for DHCP, but I cannot get any other VLAN to work. Do I need create a VE for every VLAN? While writing this, I did try to ping the routers for the other VLANs with no response from the Brocade, however, I can ping it from my local computer that is ported through the same unmanaged switch as the Brocade.

Any suggestions on what I am doing wrong to get the VLANs set correctly?

Spoiler: sh run

telnet@ToroTheBull#sh run
Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 fcx-48-poe-port-management-module
module 2 fcx-cx4-2-port-16g-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 29 by port
tagged ethe 1/1/3
untagged ethe 1/1/4
router-interface ve 29
!
!
!
!
!
hostname ToroTheBull
ip dhcp-client disable
!
!
!
!
!
!
!
!
!
!
interface ve 1
ip address 10.29.27.87 255.255.255.0
!
interface ve 29
ip address 192.168.100.2 255.255.255.0
ip helper-address 1 192.168.100.1
!
!
!
!
!
!
!
!
!
end

 

[kpfleming]

Where is your DHCP server running? Is it connected to a trunk port with membership in all of the VLANs with DHCP clients, and configured to serve each of those VLANs?

This is very likely not a problem with the switch configuration at all, but more likely not having the proper type of configuration in place across the switch and DHCP server.

 

[clix00]

Where is your DHCP server running? Is it connected to a trunk port with membership in all of the VLANs with DHCP clients, and configured to serve each of those VLANs?

This is very likely not a problem with the switch configuration at all, but more likely not having the proper type of configuration in place across the switch and DHCP server.

It is running on my EdgeRouterX--all ports are running 1,29,30,39, and 60.



Knowing that all VLANs are coming from the ERX port, is there something I need to do on the port leading into the Brocade?

 

[kpfleming]

Yes, the port on the Brocade switch that connects to the ERX needs to have all of the tagged VLANs on it, otherwise the switch won't accept or forward any traffic for that VLAN on that port.

You also will not need any 'ip helper' configuration, your DHCP server and the DHCP clients will be talking to each other at layer 2, not layer 3.

 

[clix00]

Yes, the port on the Brocade switch that connects to the ERX needs to have all of the tagged VLANs on it, otherwise the switch won't accept or forward any traffic for that VLAN on that port.

You also will not need any 'ip helper' configuration, your DHCP server and the DHCP clients will be talking to each other at layer 2, not layer 3.

That would be it. I removed the ip helper as well.

If I bring all those over tagged on say 1/1/48, what do I do about getting vlan1 access--it seems tagging the inbound port kills that.

 

[kpfleming]

Just leave it untagged on that port.

 

[clix00]

When I tag the port for 29, it removes untagged vlan 1.

Spoiler: show vlan

telnet@ToroTheBull#show vlan
Total PORT-VLAN entries: 5
Maximum PORT-VLAN entries: 64

Legend: [Stk=Stack-Id, S=Slot]

PORT-VLAN 1, Name DEFAULT-VLAN, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1) 1 2 5 6 7 8 9 10 11 12 13 14
Untagged Ports: (U1/M1) 15 16 17 18 19 20 21 22 23 24 25 26
Untagged Ports: (U1/M1) 27 28 29 30 31 32 33 34 35 36 37 38
Untagged Ports: (U1/M1) 39 40 41 42 43 44 45 46 47
Untagged Ports: (U1/M2) 1 2
Tagged Ports: None
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

PORT-VLAN 29, Name [None], Priority level0, Spanning tree Off
Untagged Ports: (U1/M1) 4
Tagged Ports: (U1/M1) 3 48
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

EDIT: dual-mode wasn't enabled....

Thanks for the help!

 

[erik29gamer]

Recently adding some more 10gb connections to the ICX7150-24 I've been using for a few years. I previously had a proxmox host with a connectx-3 (no longer in use) and a synology nas, also with a connectx3 connected. Both worked great at full speed. I recently tried adding the spare CX3 to a new desktop, only to find it BSODs Windows 11. Instead, I purchased a Supermicro aoc-stgn-i2s. Unfortunately, having a ton of oddities with it.

Regardless of the sfp+ port I have it connected to, the link light on the switch is orange. When plugged into 3 of the 4 sfp+ ports, I can only get ~500mbps throughout in iperf, but while connected to one of them, close to what I expect (~8gbps). Additionally, one of the two ports on the card itself always gives me the same 500mbps, regardless of what port its attached to on the switch. I've tried two different dacs, as well as an ubuntu live usb, same result.

From what I've been able to find, orange link light isn't indicating full speed, but I can get the highest performance when both my NAS and desktop are lit orange on the switch...

I'm kind of at a loss as to what is happening really. Just a bad card?

 

[tillburn]

Addressing only the cable modem:
Shouldn't be a problem. Plug the cable modem into a port, set that port as untagged VLAN 50, make sure your trunk port to the other switch properly tagged, then if pfSense is physical, whatever port it's using for WAN, set to untagged VLAN 50. If you're trying to trunk everything into pfSense, can't help you there, I've never bothered with VLANs directly on pfSense. Mine's a VM, so one VNIC per interface instead of VLAN tagging and sub-interfaces.

I take an ethernet handoff from my ISPs ONT straight into a switch with a L2 untagged port, that gets handed to the VM cluster over a trunk port and pealed out as VNIC that pfSense sees as physical interface.

Using your VLANs and assuming you want to trunk all of them between both switches, here'res the basic commands for the 7250, where 1/2/1 is the 10G port you're using for uplink, and for the example, your cable modem is on 1/1/1:

vlan 10
tag e 1/2/1
vlan 20
tag e 1/2/1
vlan30
tag e 1/2/1
vlan 40
tag e 1/2/1
vlan 50
tag e 1/2/1
untag e 1/1/1

The 6610 side is probably the same unless you need tagged and untagged traffic on the same port, in which case there are extra steps that I am not conversant in.

You mention hypervisors, so if your pfSense is virtualized, then the ports to your VM hosts would also be tagged traffic, just like the inter-switch trunk ports. Then you'll setup VNICs using the relevant VLAN IDs.

For example, on my Hyper-V host, this is my opnSense VM:
View attachment 26710
VLAN 2 is the VLAN my fiber ONT is connected to. Each of those VNICs is tagged to a different VLAN. I personally prefer handling it at his level rather than inside opnSense.

Ok, I figured out where I was going wrong on the connection between the switches at least. I connected the switches using a dynamic lag (LACP) but I didn't know that I had to set a primary port and then "trunk deploy" command on the 6610 as it has older firmware.

So now both switches see each other and if I end up adding more power to the shop or whatever later, then I can dynamically add another 10g port to the LAG.

So how does the LAG effect traffic between the switches?
If I tag lag1 in vlan 10, 20, 30, 40, 50 that means only those vlans will traverse the lag1 right?

For now I need all traffic to traverse lag 1 so I can still remotely configure the 6610 switch and all the hypervisors on that side of the lag1 connection as I have yet to deploy pfsense on the designated hypervisor. Right now the cable modem and the temporary router is on the office side (7250) of the lag1, I am working on finishing that up, but it's going to take some doing with writing the correct rules and such for pfsense to be ready for the cable modem on vlan 50.

Any advice would be appreciated.

 

[kpfleming]

Yes, you will need to add (as untagged or tagged) all VLANs that you want to traverse any port or LAG in the switch; the ICX devices don't have a pure 'trunk' mode where a port/LAG is automatically a member of all VLANs on the device.

The LAG is part of VLAN 1 untagged by default, so if you are using that for management connections then you're all set. You'll just need to add all of the tagged VLANs as well (and set 'dual-mode' if it's an ICX 66xx I think).

 

[tillburn]

Yes, you will need to add (as untagged or tagged) all VLANs that you want to traverse any port or LAG in the switch; the ICX devices don't have a pure 'trunk' mode where a port/LAG is automatically a member of all VLANs on the device.

The LAG is part of VLAN 1 untagged by default, so if you are using that for management connections then you're all set. You'll just need to add all of the tagged VLANs as well (and set 'dual-mode' if it's an ICX 66xx I think).

When I enter conf t on the 7250 and enter "interface ?" I get a list of ethernet, group-ve, lag, loopback, management, tunnel, ve.
When I enter conf t on the 6610 the interface ? doesn't show "lag".

Would assuming that adding dual-mode to the interface eth 1/3/1 after tagging the vlans is the appropriate way to enter dual-mode for the lag 1 that I created? I added dual mode on the 6610 to eth 1/3/1 after adding 1/3/1 to vlan 10 and now lag 1 is working as expected. Thanks for the heads up on dual mode, I guess things a little more complicated by using two switches not of the same model series, but also on difference firmware revisions. At least I am learning, thanks again!!

Now I am off to the greater networking thread to read up on pfsense and inter-vlan routing so I can make the best decision for my hardware and needs.

 

[Krextyl]

I am new to the scene and just recently picked up an ICX 6610 - 48P largely in part due to this write-up. I have next to no experience but figured I'd use this as a learning opportunity/experiment to strengthen my knowledge of networking not to mention the performance benefits. I'm learning as I go but apparently I need baby steps .

I'm following the guide as well as supplementing it with additional sources for example learning what a management port is (see baby steps). At any rate, I've reached a point where I'm stumped, unfortunately, it's not far into the guide (timeout on copy of the flash files from the TFTP Server). I was wondering if someone might be able to help get beyond my hurdle.

I've got the TFTP (on windows client) from the main zip file provided and running as it was pre-configured. I've ensured the current directory is set to the unzipped TFTP-Content folder, and I've set the server interface to my connected PC's IP 192.168.50.25. If I show dir and browse I confirmed I can see the grz10100.bin file.

For the terminal window, I'm using Putty. Using serial connection with it set to COM3, 9600 Baud, 8 data bits, 1 stop bits, no parity and Xon/Xoff flow control (I've also tried none for flow control with same result).

COM3 is my USB com port that the rj45/usb cable is connected on. I only have one other COM port on this machine and that is COM1, I tried it for giggles but no response as expected.

In putty I am able to hit b to enter boot and set an ip per the guide, I used 192.168.50.199/24 in my case. When I attempt the next step "copy tftp flash 192.168.50.25 ICX6610-FCX/grz10100.bin boot" it seems to be working for a few moments but then it times out.

Given I'm getting responses and such through Putty I see no reason to think it's anything on that end but rather some issue with the TFTP config. I have closed it and relaunched it trying to go into the subfolder and omit that in the copy command and same issue. Not sure about things I started looking at other sources for possible issues and found a video somewhat related stating I may need to check the switch's space to see if it has room to copy the flashover, otherwise I may need to delete files to make space - that made sense but also nervous being a novice not knowing what could be safely deleted etc so I didn't do anything. I did look at the TFTP settings and have tried it with tftp bound to 192.168.50.25 to see if that would help but didn't make a difference.

I am stumped please help,

 

[tillburn]

I am new to the scene and just recently picked up an ICX 6610 - 48P largely in part due to this write-up. I have next to no experience but figured I'd use this as a learning opportunity/experiment to strengthen my knowledge of networking not to mention the performance benefits. I'm learning as I go but apparently I need baby steps .

I'm following the guide as well as supplementing it with additional sources for example learning what a management port is (see baby steps). At any rate, I've reached a point where I'm stumped, unfortunately, it's not far into the guide (timeout on copy of the flash files from the TFTP Server). I was wondering if someone might be able to help get beyond my hurdle.

I've got the TFTP (on windows client) from the main zip file provided and running as it was pre-configured. I've ensured the current directory is set to the unzipped TFTP-Content folder, and I've set the server interface to my connected PC's IP 192.168.50.25. If I show dir and browse I confirmed I can see the grz10100.bin file.

For the terminal window, I'm using Putty. Using serial connection with it set to COM3, 9600 Baud, 8 data bits, 1 stop bits, no parity and Xon/Xoff flow control (I've also tried none for flow control with same result).

COM3 is my USB com port that the rj45/usb cable is connected on. I only have one other COM port on this machine and that is COM1, I tried it for giggles but no response as expected.

In putty I am able to hit b to enter boot and set an ip per the guide, I used 192.168.50.199/24 in my case. When I attempt the next step "copy tftp flash 192.168.50.25 ICX6610-FCX/grz10100.bin boot" it seems to be working for a few moments but then it times out.

Given I'm getting responses and such through Putty I see no reason to think it's anything on that end but rather some issue with the TFTP config. I have closed it and relaunched it trying to go into the subfolder and omit that in the copy command and same issue. Not sure about things I started looking at other sources for possible issues and found a video somewhat related stating I may need to check the switch's space to see if it has room to copy the flashover, otherwise I may need to delete files to make space - that made sense but also nervous being a novice not knowing what could be safely deleted etc so I didn't do anything. I did look at the TFTP settings and have tried it with tftp bound to 192.168.50.25 to see if that would help but didn't make a difference.

I am stumped please help,

I am no expert lol, so take this with a grain of mega salt...

A sanity check would be to make sure if using windows that the firewall isn't blocking your tftp server application.



command would be: copy tftp flash IPHERE grz10100.bin bootrom (boot works too)

Can you sh ver , sh firmware ver , sh flash
You can use this video to help:

 

[kpfleming]

Would assuming that adding dual-mode to the interface eth 1/3/1 after tagging the vlans is the appropriate way to enter dual-mode for the lag 1 that I created?

I don't believe that is the correct thing to do; once you have created a LAG, you need to apply VLAN configuration and similar things to the LAG, not to the interfaces inside it.

At this point it would be really helpful if you could post your config files (passwords removed, if there are any), because otherwise we're just seeing snippets as you ask questions.

 

[Krextyl]

I am no expert lol, so take this with a grain of mega salt...

Thank you for the information, great point about possible windows firewall. I'll check these suggestions out when I'm back at the switch and able to fire it back up, etc.

 

[tillburn]

I don't believe that is the correct thing to do; once you have created a LAG, you need to apply VLAN configuration and similar things to the LAG, not to the interfaces inside it.

At this point it would be really helpful if you could post your config files (passwords removed, if there are any), because otherwise we're just seeing snippets as you ask questions.

I attached the configs for both if there is a more convenient way to post the configs I am happy to do that too, let me know.

Let me know how I can improve my config please and thank you for helping!

 

[LodeRunner]

Attachment is fine, you can also paste the full text in a code block wrapped in tags. [ spoiler ][ code ] config here [ /code ] [ /spoiler ] (but without the spaces between the words and the brackets).