Brocade ICX6610 10G network different VLANs

SuperMiguel

New Member
Jun 17, 2021
12
1
3
Hello all, I'm trying to troubleshoot a 10g network issue running as 1G. So I have 2 machines connected to the front 10G of my licensed icx6610, when I do show int brief, both interfaces show 10g, both machines show their connection as 10g, but when I run iperf3 I only get 1G speeds. Both machines are on a different VLANS, I have ve interfaces on each vlan, when I run show ip route I can see the route of each vlan there, but again im only getting 1G speeds. When i log into one machine and run traceroute to the other one on the other vlan, the 1st hop its my gateway (it has a 1g NIC on it ) so im wondering if intervlan traffic is going tru my gateway for some reason, is there a way to check and whats the fix if thats the case??

Note: If i move both machines to the same VLAN i get 10G speeds.

One fix would be to add a 10G nic to my router, but i guess i kinda prefer my switches doing L3 if they got the capabilities.
 
Last edited:

itronin

Well-Known Member
Nov 24, 2018
891
555
93
Denver, Colorado
Hello all, I'm trying to troubleshoot a 10g network issue running as 1G. So I have 2 machines connected to the front 10G of my licensed icx6610, when I do show int brief, both interfaces show 10g, both machines show their connection as 10g, but when I run iperf3 I only get 1G speeds.
Sadly seeing this issue is becoming a bit too common.

Are you two machines getting IP addresses and default gateway via DHCP?
Is the default gateway the ICX6610 VE address for the respective VLAN or is it your "gateway router" <--- my guess is the gateway router.
Are you running pfSense or OpnSense and using that to serve DHCP to your VLANs?

Both machines are on a different VLANS, I have ve interfaces on each vlan, when I run show ip route I can see the route of each vlan there, but again im only getting 1G speeds. When i log into one machine and run traceroute to the other one on the other vlan, the 1st hop its my gateway (it has a 1g NIC on it ) so im wondering if intervlan traffic is going tru my gateway for some reason, is there a way to check and whats the fix if thats the case??
You actually already answered this for yourself no reason to wonder.

Note: If i move both machines to the same VLAN i get 10G speeds.

One fix would be to add a 10G nic to my router, but i guess i kinda prefer my switches doing L3 if they got the capabilities.
It would only fix it if your router can handle 10G routing at wirespeed.

If you want your ICX6610 to handle vlan routing then the default gateway for hosts in any VLAN should be set to the VLAN;'s VE ip address.

It sounds like you have an interface (whether vlan tagged or physical) from your gateway router going to each VLAN. If you do then you will see asymmetric or split-path routing for Internet traffic when you configure 10g VLAN connected hosts to use the ICX 6610 VE as their default gateway.

There are multiple posts in the brocade megathread discussing (and possible options for_ the scenario I just outlined.

IMO best option is to create a transit VLAN between your ICX6610 and your gateway router and NOT have your gateway router connected to each VLAN. All internal traffic must use the ICX6610 for internal routing. If you are using pfSense or OpnSense and and using that for DHCP then you have another problem. The "senses" can only serve DHCP to directly connected networks.
 
  • Like
Reactions: kpfleming

SuperMiguel

New Member
Jun 17, 2021
12
1
3
Sadly seeing this issue is becoming a bit too common.

Are you two machines getting IP addresses and default gateway via DHCP?
Is the default gateway the ICX6610 VE address for the respective VLAN or is it your "gateway router" <--- my guess is the gateway router.
Are you running pfSense or OpnSense and using that to serve DHCP to your VLANs?



You actually already answered this for yourself no reason to wonder.



It would only fix it if your router can handle 10G routing at wirespeed.

If you want your ICX6610 to handle vlan routing then the default gateway for hosts in any VLAN should be set to the VLAN;'s VE ip address.

It sounds like you have an interface (whether vlan tagged or physical) from your gateway router going to each VLAN. If you do then you will see asymmetric or split-path routing for Internet traffic when you configure 10g VLAN connected hosts to use the ICX 6610 VE as their default gateway.

There are multiple posts in the brocade megathread discussing (and possible options for_ the scenario I just outlined.

IMO best option is to create a transit VLAN between your ICX6610 and your gateway router and NOT have your gateway router connected to each VLAN. All internal traffic must use the ICX6610 for internal routing. If you are using pfSense or OpnSense and and using that for DHCP then you have another problem. The "senses" can only serve DHCP to directly connected networks.
Correct the connection from my firewall to my ICx6610 is on port 1/3/1 and i have that port tagged on every single VLAN, I also tried to add a 10G card to my firewall (untangle) and i can only get about 4.9G of speed (better than the 900 i was getting before)
 

itronin

Well-Known Member
Nov 24, 2018
891
555
93
Denver, Colorado
Looks like you have the same question/discussion going on in the mega thread. I'll watch there. You have a reasonable recommendation there to use static routes, however if you have more than a few vlans then the static route path will be cumbersome and look at a transit vlan - however that will require you to use something other than pfSense DHCP service.
 

RobstarUSA

Active Member
Sep 15, 2016
200
63
28
Sadly seeing this issue is becoming a bit too common.

Are you two machines getting IP addresses and default gateway via DHCP?
Is the default gateway the ICX6610 VE address for the respective VLAN or is it your "gateway router" <--- my guess is the gateway router.
Are you running pfSense or OpnSense and using that to serve DHCP to your VLANs?



You actually already answered this for yourself no reason to wonder.



It would only fix it if your router can handle 10G routing at wirespeed.

If you want your ICX6610 to handle vlan routing then the default gateway for hosts in any VLAN should be set to the VLAN;'s VE ip address.

It sounds like you have an interface (whether vlan tagged or physical) from your gateway router going to each VLAN. If you do then you will see asymmetric or split-path routing for Internet traffic when you configure 10g VLAN connected hosts to use the ICX 6610 VE as their default gateway.

There are multiple posts in the brocade megathread discussing (and possible options for_ the scenario I just outlined.

IMO best option is to create a transit VLAN between your ICX6610 and your gateway router and NOT have your gateway router connected to each VLAN. All internal traffic must use the ICX6610 for internal routing. If you are using pfSense or OpnSense and and using that for DHCP then you have another problem. The "senses" can only serve DHCP to directly connected networks.
In cisco land we have "ip helper address" to forward dhcp requests to some other host on another vlan. I wonder if the icx6610 doesn't have something like this?
 

itronin

Well-Known Member
Nov 24, 2018
891
555
93
Denver, Colorado
In cisco land we have "ip helper address" to forward dhcp requests to some other host on another vlan. I wonder if the icx6610 doesn't have something like this?
It does.

a problem/feature is that pfsense (or opnsense) only supports dhcp serving subnets on directly connected interfaces.
 

RobstarUSA

Active Member
Sep 15, 2016
200
63
28
It does.

a problem/feature is that pfsense (or opnsense) only supports dhcp serving subnets on directly connected interfaces.
I'd think it would still work. IP Helper is just a DHCP relay.

Think of something like this:

Code:
                          ^
                          |
                        Vlan 20
                          |
OPNSense <-> VLan 10 <-> 6610 <-> Vlan 40
                          |
                        Vlan 30
                          |
                          V
Any request from Vlan 20/30/40 should be relayed to OPNSense on Vlan 10. If you go into opnsense, can't you define interaces for vlan 20/30/40 and then go to Services -> DHCPv4-> Enable DHCP Server and it will (just)work? If not, just do a "router on a stick" config (don't need a DHCP Relay) and do 802.1q between opnsense & the 6610 & it should work that way. You can still hand out the 6610 ve interface as the gateway. I currently have a 10Gbit/s interface with about 15 802.1q vlan interfaces on it & it works ok for me.

When I set this up at work Cisco/Windows dhcp server it "just worked". I don't think there was an special setup on the Windows side. I could of course be wrong. It happens quite a bit :)
 
Last edited:

kpfleming

Active Member
Dec 28, 2021
127
54
28
Pelham NY USA
I don't think it requires special setup, but the DHCP server does need to interpret some data injected into the DHCP requests as they pass through the relay (the relay does the injection), otherwise it will believe the requests came *from* the relay.
 

itronin

Well-Known Member
Nov 24, 2018
891
555
93
Denver, Colorado
I'd think it would still work. IP Helper is just a DHCP relay. When I set this up at work for a windows dhcp server it "just worked". I don't think there was an special setup on the windows side. I could of course be wrong. It happens quite a bit :)
search STH or google about dhcp relay and pfsense. As @kpfleming said "but the DHCP server does need to interpret some data injected into the DHCP" and that is exactly correct!

The Windows DHCP server is a VERY GOOD GUI DHCP server! It has known how to handle dhcp relay (helper) requests for quite a long while.
The lack of this functionality in the "senses" is a design/philosophical decision.
 
  • Like
Reactions: RobstarUSA

RobstarUSA

Active Member
Sep 15, 2016
200
63
28
search STH or google about dhcp relay and pfsense. As @kpfleming said "but the DHCP server does need to interpret some data injected into the DHCP" and that is exactly correct!

The Windows DHCP server is a VERY GOOD GUI DHCP server! It has known how to handle dhcp relay (helper) requests for quite a long while.
The lack of this functionality in the "senses" is a design/philosophical decision.
I see...then maybe he can do option 2 & do a bunch of trunks on a single link & run a dhcp server on each virtual interface on pfsense. I have that working on OPNSense and I'd imagine that should work on pfsense pretty good as well.
 

kpfleming

Active Member
Dec 28, 2021
127
54
28
Pelham NY USA
Which is likely the situation now, so the only real change is reconfiguring the DHCP server to tell clients to use the ICX as the default gateway.