Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

infoMatt

Active Member
Apr 16, 2019
222
99
28
Thank you for the response. I am not using ve on the Brocade side. I have issued enable command on the eth 1/3/1, assigned in an IP address and enabled OSPF on it. I figured that it is easier than messing with vlans and virtual interfaces. On the Edgerouter side I have not done any changes what so ever. It looks like the Edge router does not have the route back to Brocade for traffic destined for the switch. It is a point to point connection. So I am at a loss here. Not that I really need the switch to access the internet directly, but having the time sync would be good.

P.S. I have not configured loop back interface on the Brocade. Lowest VE interface would be ve 2. I'm not using vlan 1 at all. There is OSFP running on ve 2 interface. So it should be routing correctly...

There are all the VEs I have configured:

interface ve 2
ip address 192.168.0.11 255.255.255.0
ip ospf area 0
!
[..]
Well, you have more than one VE, so you're effectively using those :p
Ensure on the edgerouter that it knows where to route the packets for 192.168.0.11/24 via 192.168.29.1.
But please ensure that the multiple IPs that you've given to the switch belogs to different VLANs, otherwise some nasty things might occour...
Also, if you're lost, some PCAP is always useful :p
 

Hakujou

New Member
Apr 28, 2019
23
9
3
Hello,

Did anyone managed to get VRF working on ICX6610 ?
When I create one and try to enable address-family on it, it complains with:
Code:
SSH@sw-core-1(config-vrf-secure)#address-family ipv4
Error: has reached maximum system limit of maximum number of IPv4 routes
       available IPv4 routes for non-default VRF 0
Which is weird, because the limit of routes is set at 1024 ipv4/100 ipv6 and I have no other VRF or significant routes configured on it.
Code:
SSH@sw-core-1(config-vrf-SECURE)#show default values
sys log buffers:50         mac age time:300 sec       telnet sessions:5

ip arp age:10 min          bootp relay max hops:4     ip ttl:64 hops
ip addr per intf:24

when multicast enabled :
igmp group memb.:260 sec   igmp query:125 sec         hardware drop: enabled

when ospf enabled :
ospf dead:40 sec           ospf hello:10 sec          ospf retrans:5 sec
ospf transit delay:1 sec

when bgp enabled :
bgp local pref.:100        bgp keep alive:60 sec      bgp hold:180 sec
bgp metric:10              bgp local as:1             bgp cluster id:0
bgp ext. distance:20       bgp int. distance:200      bgp local distance:200

System Parameters    Default    Maximum    Current    Configured
ip-arp               4000       64000      4000       4000
ip-static-arp        512        6000       512        512
ip-cache             10000      32768      10000      10000
ip-filter-port       3066       3066       3066       3066
ip-filter-sys        2048       8192       2048       2048
l3-vlan              32         1024       32         32
ip-qos-session       1024       16000      1024       1024
mac                  32768      32768      32768      32768
ip-route             12000      15168      12000      12000
ip-static-route      64         2048       64         64
vlan                 64         4095       64         64
spanning-tree        32         254        32         32
mac-filter-port      16         256        16         16
mac-filter-sys       32         512        32         32
ip-subnet-port       24         128        24         24
session-limit        8192       16384      8192       8192
view                 10         65535      10         10
virtual-interface    255        512        255        255
hw-traffic-condition 896        896        896        896
rmon-entries         1024       32768      1024       1024
igmp-snoop-mcache    512        8192       512        512
mld-snoop-mcache     512        8192       512        512
ip6-route            908        2884       908        908
ip6-static-route     178        576        181        181
ip6-cache            908        2884       908        908
msdp-sa-cache        4096       8192       4096       4096
gre-tunnels          16         64         16         16
hw-ip-route-tcam     16384      16384      16384      16384
ip-vrf               16         16         16         16
ip-route-default-vrf 12000      15168      12000      12000
ip6-route-default-vr 908        2884       908        908
ip-route-vrf         1024       15168      1024       1024
ip6-route-vrf        100        2884       100        100
pim-hw-mcache        1024       6144       1024       1024
pim6-hw-mcache       512        1024       512        512
igmp-snoop-group-add 4096       8192       4096       4096
mld-snoop-group-addr 4096       8192       4096       4096
mac-notification-buf 4000       16000      4000       4000
traffic-policies-sys 1024       1024       1024       1024
dot1x-mka-policy-gro 8          8          8          8
openflow-flow-entrie 1024       12000      1024       1024
openflow-pvlan-entri 40         256        40         40
openflow-unprotected 40         256        40         40
openflow-group-selec 0          120        0          0
openflow-nexthop-ent 0          1024       0          0
max-dhcp-snoop-entri 1024       3072       1024       1024
max-static-inspect-a 512        1024       512        512
Is it a bug or am I missing something here ?

Thanks

EDIT: Figured it out. ip-route-default-vrf was set at the value of ip-route, making no left route availables for non-default vrf. Lowered it allowed vrf to enable address-family.
 
Last edited:

Ionitor

New Member
Jan 4, 2020
4
3
3
So, as a heads up: do not buy the DB9 to Mini USB cable linked to by vangoose above (current listing here, sold by "tidunkin2012"). I'm going to work with the seller and I'll report back what they say.
The seller said that they had no idea whether the reversed pins was an error or if that cable is not intended to work with the ICX 7250/7450. So, sounds like it's not the right option. I'm either going to make my own cable or order the double-adapter others linked to.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,587
2,776
113
31
fohdeesha.com
Hello,

Did anyone managed to get VRF working on ICX6610 ?
When I create one and try to enable address-family on it, it complains with:
Code:
SSH@sw-core-1(config-vrf-secure)#address-family ipv4
Error: has reached maximum system limit of maximum number of IPv4 routes
       available IPv4 routes for non-default VRF 0
Which is weird, because the limit of routes is set at 1024 ipv4/100 ipv6 and I have no other VRF or significant routes configured on it.
Code:
SSH@sw-core-1(config-vrf-SECURE)#show default values
sys log buffers:50         mac age time:300 sec       telnet sessions:5

ip arp age:10 min          bootp relay max hops:4     ip ttl:64 hops
ip addr per intf:24

when multicast enabled :
igmp group memb.:260 sec   igmp query:125 sec         hardware drop: enabled

when ospf enabled :
ospf dead:40 sec           ospf hello:10 sec          ospf retrans:5 sec
ospf transit delay:1 sec

when bgp enabled :
bgp local pref.:100        bgp keep alive:60 sec      bgp hold:180 sec
bgp metric:10              bgp local as:1             bgp cluster id:0
bgp ext. distance:20       bgp int. distance:200      bgp local distance:200

System Parameters    Default    Maximum    Current    Configured
ip-arp               4000       64000      4000       4000
ip-static-arp        512        6000       512        512
ip-cache             10000      32768      10000      10000
ip-filter-port       3066       3066       3066       3066
ip-filter-sys        2048       8192       2048       2048
l3-vlan              32         1024       32         32
ip-qos-session       1024       16000      1024       1024
mac                  32768      32768      32768      32768
ip-route             12000      15168      12000      12000
ip-static-route      64         2048       64         64
vlan                 64         4095       64         64
spanning-tree        32         254        32         32
mac-filter-port      16         256        16         16
mac-filter-sys       32         512        32         32
ip-subnet-port       24         128        24         24
session-limit        8192       16384      8192       8192
view                 10         65535      10         10
virtual-interface    255        512        255        255
hw-traffic-condition 896        896        896        896
rmon-entries         1024       32768      1024       1024
igmp-snoop-mcache    512        8192       512        512
mld-snoop-mcache     512        8192       512        512
ip6-route            908        2884       908        908
ip6-static-route     178        576        181        181
ip6-cache            908        2884       908        908
msdp-sa-cache        4096       8192       4096       4096
gre-tunnels          16         64         16         16
hw-ip-route-tcam     16384      16384      16384      16384
ip-vrf               16         16         16         16
ip-route-default-vrf 12000      15168      12000      12000
ip6-route-default-vr 908        2884       908        908
ip-route-vrf         1024       15168      1024       1024
ip6-route-vrf        100        2884       100        100
pim-hw-mcache        1024       6144       1024       1024
pim6-hw-mcache       512        1024       512        512
igmp-snoop-group-add 4096       8192       4096       4096
mld-snoop-group-addr 4096       8192       4096       4096
mac-notification-buf 4000       16000      4000       4000
traffic-policies-sys 1024       1024       1024       1024
dot1x-mka-policy-gro 8          8          8          8
openflow-flow-entrie 1024       12000      1024       1024
openflow-pvlan-entri 40         256        40         40
openflow-unprotected 40         256        40         40
openflow-group-selec 0          120        0          0
openflow-nexthop-ent 0          1024       0          0
max-dhcp-snoop-entri 1024       3072       1024       1024
max-static-inspect-a 512        1024       512        512
Is it a bug or am I missing something here ?

Thanks

EDIT: Figured it out. ip-route-default-vrf was set at the value of ip-route, making no left route availables for non-default vrf. Lowered it allowed vrf to enable address-family.
yep, as the very first page of the VRF configuration section of the manual states, you have to lower the amount of routes assigned to the default VRF so you have some to assign to non-defaults. I typically just run:

system-max ip-route-default-vrf 9000
system-max ip-route-vrf 128
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,587
2,776
113
31
fohdeesha.com
@sash you say you're not using VEs on the brocade side, and that you assigned an IP directly to port 1/3/1 (which I wouldn't recommend, especially in an l3 config, but it's up to you), but then post a big list of VEs ?

I would figure that out, and then try the ping command specifying a source interface to use, so you at least know what IP the ICX is using to try and ping with:

ping 8.8.8.8 source 192.168.1.1
 

cserve

New Member
Feb 17, 2019
2
0
1
So what I am going to do with mine is stack across the breakout ports, and use the 4 total ports across my 2x 6610's for 40g.
Hello all , I'm trying to do the same without success, using just one breakout port (an entire 4x 10G connect via QSFP DAC) for stacking across a pair of 6610 and leave the all 40G ports to connect to my servers as well as the other breakout port free.

I have tried to remove all stack-trunks but I cannot select 1/2/2 as a stack port. It errors out saying only 1/2/1 and 1/2/6 can be use as stacking ports which are the exact 40G port I was trying to leave free.

The stack works with the stack-trunk 1/2/1 to 1/2/2, stack-port 1/2/1 and only connects the breakout port 1/2/2 via QSFP DAC. However 1/2/1 is still tied up with the stack-trunk and cannot be used to connect to servers for the 40G connection.

Is this possible? I have searched the entire thread and did not see a configuration for this. I found an opposite configuration from fohdeesha where stacking is done on all the 40G ports and leave the breakout ports free.

Thanks
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,587
2,776
113
31
fohdeesha.com
Hello all , I'm trying to do the same without success, using just one breakout port (an entire 4x 10G connect via QSFP DAC) for stacking across a pair of 6610 and leave the all 40G ports to connect to my servers as well as the other breakout port free.

I have tried to remove all stack-trunks but I cannot select 1/2/2 as a stack port. It errors out saying only 1/2/1 and 1/2/6 can be use as stacking ports which are the exact 40G port I was trying to leave free.

The stack works with the stack-trunk 1/2/1 to 1/2/2, stack-port 1/2/1 and only connects the breakout port 1/2/2 via QSFP DAC. However 1/2/1 is still tied up with the stack-trunk and cannot be used to connect to servers for the 40G connection.

Is this possible? I have searched the entire thread and did not see a configuration for this. I found an opposite configuration where stacking is done on all the 40G ports and leave the breakout ports free.

I'm not sure if that's possible, but given that very specific error message, it looks like it's not. I'm running a bit of a mix of that and the 40g ports on my home stack, it's stacked via a trunk of 1 40g port and 1 breakout port, leaving me 2x 40gbE ports and 2x breakout ports

it's also worth noting stack traffic over the 4x 10gbE QSFP+ ports is hashed similar to standard LACP, eg by source IP / port. Meaning if you stack using only the breakout ports, you won't get single-stream speed above 10gbps between stuff connected to each stack member. so 40gbE host on switch 1 is not going to be able to do more than 10gbps for single stream stuff (eg something using only one source IP and one source port, like SMB shares etc (unless running SMB 3.0)) to a 40gbE device on switch 2, as it will get hashed to just 1 of the 10gbps stacking channels
 
Last edited:

sash

Member
Nov 22, 2019
32
4
8
@sash you say you're not using VEs on the brocade side, and that you assigned an IP directly to port 1/3/1 (which I wouldn't recommend, especially in an l3 config, but it's up to you), but then post a big list of VEs ?

I would figure that out, and then try the ping command specifying a source interface to use, so you at least know what IP the ICX is using to try and ping with:

ping 8.8.8.8 source 192.168.1.1
I tried to mimic the old Cisco switch config where I used a no switchport command on the interface to turn it to L3 interface and assigned IP address to it directly. What is the downside of assigning IP directly to the interface and not to virtual interface?

I have tried pinging with source address included and all vlans that are configured with ospf routing can access the internet without issues.

Code:
SSH@ICX6610-48P#ping 8.8.8.8 source 192.168.0.11
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 8.8.8.8         : bytes=16 time=15ms TTL=56
Success rate is 100 percent (1/1), round-trip min/avg/max=15/15/15 ms.
SSH@ICX6610-48P#ping 8.8.8.8 source 192.168.3.1
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 8.8.8.8         : bytes=16 time=14ms TTL=56
Success rate is 100 percent (1/1), round-trip min/avg/max=14/14/14 ms.
SSH@ICX6610-48P#ping 8.8.8.8 source 192.168.11.1
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.
SSH@ICX6610-48P#ping 8.8.8.8 source 192.168.12.1
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 8.8.8.8         : bytes=16 time=16ms TTL=55
Success rate is 100 percent (1/1), round-trip min/avg/max=16/16/16 ms.
SSH@ICX6610-48P#ping 8.8.8.8 source 192.168.35.1
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 8.8.8.8         : bytes=16 time=14ms TTL=56
Success rate is 100 percent (1/1), round-trip min/avg/max=14/14/14 ms.
SSH@ICX6610-48P#ping 8.8.8.8 source 10.0.0.1
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.
SSH@ICX6610-48P#ping 8.8.8.8 source 192.168.254.1
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.
Here is ospf configuration on the Edge router:
Code:
 protocols {
     ospf {
         area 0.0.0.0 {
             area-type {
                 normal
             }
             network 172.16.0.0/16
             network 192.168.29.0/24
         }
         log-adjacency-changes {
         }
         parameters {
             abr-type cisco
             router-id 91.200.xxx.xxx
         }
     }
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,587
2,776
113
31
fohdeesha.com
it will work (IP directly on switch port), but it's just much less flexible if you ever want to move anything around or create complex ACLs etc

judging from your results, it sounds like you haven't manually added static routes to the edge router for the non-ospf subnets on the ICX. that's why OSPF interfaces can get out to the internet, but the others can't. The ICX can get out to whatever, OSPF or not, as it has a default static route to the edgerouter (192.168.29.254), so the traffic is all being routed correctly from the ICX.

However when the edgerouter tries to send the ICMP response back to the ICX, like to address 192.168.11.1, it looks in its routing table to see what IP next-hop to send that packet to, and it has no entries, as they haven't been sent by the ICX via ospf, and you haven't entered one manually. If you have a default route on the edgerouter (like out to the internet), it will fall back to using this route as it couldn't find anything more specific, and the ICMP reply meant for the ICX will shoot back out to the internet. You can confirm this by running "show ip route 192.168.11.1" on the edgerouter, I bet it'll return your default internet gateway

The solution is to either have the ICX send the edgerouter the proper return routes via ospf by enabling ospf on these subnets as well, or entering static routes on the edgerouter for these non-ospf subnets. I'm not an ubiquiti user, but from a quick google, you would run the below to add the static routes for the non-ospf subnets

Code:
configure
set protocols static route 192.168.11.0/24 next-hop 192.168.29.1
set protocols static route 10.0.0.0/8 next-hop 192.168.29.1
etc
commit;save
 
Last edited:

sash

Member
Nov 22, 2019
32
4
8
(just fixed a couple typos on the edgerouter commands, if you copy/pasted them you might want to re-check)
My problem is different. I would like ICX itself to access the internet. Vlans 11 and 100 do not need to access internet by design. I'm routing everything through OSPF. I've added vlans to osfp routing and they work just fine. The routed interface on the ICX is also added to OSPF but it cannot access the internet. That is what I'm trying to figure out...

Code:
!
interface ethernet 1/3/1
 port-name Uplink to EdgeRoute6 on port eth5
 ip address 192.168.29.1 255.255.255.0
 ip ospf area 0
!
!
router ospf
 area 0
 neighbor 192.168.29.254
!
ip route 0.0.0.0/0 192.168.29.254
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,587
2,776
113
31
fohdeesha.com
My problem is different. I would like ICX itself to access the internet. Vlans 11 and 100 do not need to access internet by design
that doesn't change the issue, the ICX's IP address in those vlans is still in those subnets, and the edgerouter has no idea how to get packets back to those subnets. If you want to just allow traffic back to the ICX IPs themselves but not the rest of the subnet, it would be

Code:
configure
set protocols static route 192.168.11.1/32 next-hop 192.168.29.1
set protocols static route 10.0.0.1/32 next-hop 192.168.29.1
etc
commit;save
if you mean to say you ran "ping 8.8.8.8 source 192.168.29.1" (using the l3 interface connected to the edgerouter as the source) and it failed, that's another story, as you have indeed enabled ospf on that interface, but that shouldn't even matter, as the edgerouter has an interface directly in this subnet (192.168.29.254) so it will have a route to this subnet already automatically (unless it's netmask is misconfigured or something). If this ping command is indeed failing, post the output of "show ip route 192.168.29.1" from the edgerouter. I doubt that ping command will fail though, if the edgerouter didn't have a route back to 192.168.29.1, none of the ospf enabled subnets would work (and they do)
 

sash

Member
Nov 22, 2019
32
4
8
that doesn't change the issue, the ICX's IP address in those vlans is still in those subnets, and the edgerouter has no idea how to get packets back to those subnets. If you want to just allow traffic back to the ICX IPs themselves but not the rest of the subnet, it would be

Code:
configure
set protocols static route 192.168.11.1/32 next-hop 192.168.29.1
set protocols static route 10.0.0.1/32 next-hop 192.168.29.1
etc
commit;save
if you mean to say you ran "ping 8.8.8.8 source 192.168.29.1" (using the l3 interface connected to the edgerouter as the source) and it failed, that's another story, as you have indeed enabled ospf on that interface, but that shouldn't even matter, as the edgerouter has an interface directly in this subnet (192.168.29.254) so it will have a route to this subnet already automatically (unless it's netmask is misconfigured or something). If this ping command is indeed failing, post the output of "show ip route 192.168.29.1" from the edgerouter. I doubt that ping command will fail though, if the edgerouter didn't have a route back to 192.168.29.1, none of the ospf enabled subnets would work (and they do)
I don't need internet for those two subnets - they are for storage connections inside my lan, thus I did not include them into OSPF routing. Router does not need to know about them at all.
The only problem is internet access for ICX itself. ICX can access the router on the directly connected interface. It can also access remote VPN subnets without issues. But it cannot access the internet.

Code:
SSH@ICX6610-48P#ping 8.8.8.8
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.
SSH@ICX6610-48P#ping 192.168.29.254 (EDGE ROUTER)
Sending 1, 16-byte ICMP Echo to 192.168.29.254, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.29.254  : bytes=16 time<1ms TTL=64
Success rate is 100 percent (1/1), round-trip min/avg/max=0/0/0 ms.
SSH@ICX6610-48P#ping 192.168.7.1 (REMOVE SUBNET)
Sending 1, 16-byte ICMP Echo to 192.168.7.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.7.1     : bytes=16 time=14ms TTL=63
Success rate is 100 percent (1/1), round-trip min/avg/max=14/14/14 ms.
SSH@ICX6610-48P#ping 192.168.23.1 (REMOTE SUBNET)
Sending 1, 16-byte ICMP Echo to 192.168.23.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.23.1    : bytes=16 time=3ms TTL=63
Success rate is 100 percent (1/1), round-trip min/avg/max=3/3/3 ms.
 

CED6688

New Member
Dec 4, 2019
15
10
3
Looks like a NAT issue on the edgerouter. I seem to recall that the default edgerouter config only NATs the first LAN port.
 

dashpuppy

Member
Dec 16, 2018
47
17
8
Anyone know why the brocade switches (icx-6430-C12 ) would have issues displaying the web gui ? It's enable and have setup the user & aaa authentication login default local aaa authentication enable default local aaa authentication web default local commands but it only shows the port display. I can't get to any configuration menu's.

Bought a pair of them for Christmas, I have it up and running just having some little issues with vlans so i wanted to log into the GUI and see what i might be missing.

TIA !
 

infoMatt

Active Member
Apr 16, 2019
222
99
28
I don't need internet for those two subnets - they are for storage connections inside my lan, thus I did not include them into OSPF routing. Router does not need to know about them at all.
The only problem is internet access for ICX itself. ICX can access the router on the directly connected interface. It can also access remote VPN subnets without issues. But it cannot access the internet.
Understandable, but the problem relies on what I and @fohdeesha stated before: for its comunication the switch itself will choose the first address configured, and if it resides on an unknown network on the ER side, it can't route the traffic back for the response... you can define a IP range (that belongs to the OSPF area 0) for the loopback0, so the ER will be aware of the switch address and it can correctly route the traffic back...
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,587
2,776
113
31
fohdeesha.com
I don't need internet for those two subnets - they are for storage connections inside my lan, thus I did not include them into OSPF routing. Router does not need to know about them at all.
The only problem is internet access for ICX itself. ICX can access the router on the directly connected interface. It can also access remote VPN subnets without issues. But it cannot access the internet.

Code:
SSH@ICX6610-48P#ping 8.8.8.8
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.
SSH@ICX6610-48P#ping 192.168.29.254 (EDGE ROUTER)
Sending 1, 16-byte ICMP Echo to 192.168.29.254, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.29.254  : bytes=16 time<1ms TTL=64
Success rate is 100 percent (1/1), round-trip min/avg/max=0/0/0 ms.
SSH@ICX6610-48P#ping 192.168.7.1 (REMOVE SUBNET)
Sending 1, 16-byte ICMP Echo to 192.168.7.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.7.1     : bytes=16 time=14ms TTL=63
Success rate is 100 percent (1/1), round-trip min/avg/max=14/14/14 ms.
SSH@ICX6610-48P#ping 192.168.23.1 (REMOTE SUBNET)
Sending 1, 16-byte ICMP Echo to 192.168.23.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.23.1    : bytes=16 time=3ms TTL=63
Success rate is 100 percent (1/1), round-trip min/avg/max=3/3/3 ms.
You aren't understanding the issue, so I'll try and explain it a third and final time before moving on and letting you try and figure it out on your own. Your ICX does have internet access, as evidenced by the fact every IP address on the ICX that also has a proper return route on the edgerouter can ping out to 8.8.8.8. Which leaves me to assume what you actually want is not "the ICX needs to have internet access (because it already does, on like 6 of it's 8 addresses, but you're not happy about it) but you actually mean "I want the ICX to have internet access from EVERY IP it owns". For this, I've provided the commands to solve this on your edgerouter twice now.

The ping command (and a lot of other commands, when not supplied with a specific source interface to use) will default to just using the lowest owned IP as a traffic source. On your switch that means 10.0.0.1, which, again, has no return route on the edgerouter, so there's no way for internet traffic to get back to the ICX. To break it down even simpler, which seems to be required here, this means the command you keep running to verify "internet access", "ping 8.8.8.8", is the equivalent of running ping 8.8.8.8 source 10.0.0.1" - and as established 20 times now, the edgerouter has no return route for this ICX address, and I've provided the commands to fix it. If you REALLY don't want this address having internet access, then learn to specify the ICX source address in commands requiring internet access so it doesn't default to using 10.0.0.1, and choose an ICX source address that does have a return route. example: "ping 8.8.8.8 source 192.168.0.11" - like magic, your ICX will have internet access. Same with other protocols on the switch, for instance NTP can be told which source address to use with "source-interface ve 10"

This is not a NAT issue, it's a routing issue, your edge router does not have return routes for a handful of your ICX addresses. The addresses that do have return routes do have internet access, the addresses that do not have return routes do not have internet access. I'm not sure how else to make this any clearer besides painting pictures
 

dashpuppy

Member
Dec 16, 2018
47
17
8
You aren't understanding the issue, so I'll try and explain it a third and final time before moving on and letting you try and figure it out on your own. Your ICX does have internet access, as evidenced by the fact every IP address on the ICX that also has a proper return route on the edgerouter can ping out to 8.8.8.8. Which leaves me to assume what you actually want is not "the ICX needs to have internet access (because it already does, on like 6 of it's 8 addresses, but you're not happy about it) but you actually mean "I want the ICX to have internet access from EVERY IP it owns". For this, I've provided the commands to solve this on your edgerouter twice now.

The ping command (and a lot of other commands, when not supplied with a specific source interface to use) will default to just using the lowest owned IP as a traffic source. On your switch that means 10.0.0.1, which, again, has no return route on the edgerouter, so there's no way for internet traffic to get back to the ICX. To break it down even simpler, which seems to be required here, this means the command you keep running to verify "internet access", "ping 8.8.8.8", is the equivalent of running ping 8.8.8.8 source 10.0.0.1" - and as established 20 times now, the edgerouter has no return route for this ICX address, and I've provided the commands to fix it. If you REALLY don't want this address having internet access, then learn to specify the ICX source address in commands requiring internet access so it doesn't default to using 10.0.0.1, and choose an ICX source address that does have a return route. example: "ping 8.8.8.8 source 192.168.0.11" - like magic, your ICX will have internet access. Same with other protocols on the switch, for instance NTP can be told which source address to use with "source-interface ve 10"

This is not a NAT issue, it's a routing issue, your edge router does not have return routes for a handful of your ICX addresses. The addresses that do have return routes do have internet access, the addresses that do not have return routes do not have internet access. I'm not sure how else to make this any clearer besides painting pictures

Calm down man :p take a deep breath :) we appreciate your help.. Need a coffee ?
 

dashpuppy

Member
Dec 16, 2018
47
17
8
I guess I just have this weird tick where the first four times I have to explain the same thing it's no problem, but the fifth and above start to get frustrating. Weird, I know

It's ok man ! Sometimes it just needed to be worded differently OR example given :)

Example The car is red, with 4 tires but one tire is flat.. we know it is red and has flat tire BUT the other person doesn't see it that way.

Hey Your red car has a flat tire.. :)

All to do with perception :p