Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Wolfstar]

Ok, I'm out of ideas

Just for the sake of completeness: mgmt port is connected directly to the CPU, it is not part of the switched ports... maybe it causes some delay? I've onestly not tryed it yet... I might give it a shot tomorrow (now in Italy it's 1:30 AM...) on my 6450 24P, basically because now I'm curious
In any case, it doesn't explain why when you shut those ports the latencies go down...

Keep in mind that - in all likelihood - the ASIC driving those ports is the same ASIC. It's typical to have 4 ports to an ASIC on gigabit switches. @mythosmc if you've ever tried to apply QoS on four ports of a Layer 3 switch and seen it apply to three others, that's why.

If I had to guess, I'd say that the ASIC on those four ports is bad and it's causing issues, possibly flooding the TCAM with garbage. If you enable those ports, ping the switch, then disable them and check the logs (show log), what are you seeing between the port enable and disable?

It's entirely possible too that whatever fried the PoE board on that switch came through one of those four ports, and managed to kill the ASIC at the same time.

 

[infoMatt]

Keep in mind that - in all likelihood - the ASIC driving those ports is the same ASIC. It's typical to have 4 ports to an ASIC on gigabit switches. @mythosmc if you've ever tried to apply QoS on four ports of a Layer 3 switch and seen it apply to three others, that's why.

If I had to guess, I'd say that the ASIC on those four ports is bad and it's causing issues, possibly flooding the TCAM with garbage. If you enable those ports, ping the switch, then disable them and check the logs (show log), what are you seeing between the port enable and disable?

It's entirely possible too that whatever fried the PoE board on that switch came through one of those four ports, and managed to kill the ASIC at the same time.

Looking at the board, it seems that there are 6 chips (in case of the 24 port) near the ports, so your theory might be correct... Just looking, without a schematic, I cannot say however if those are switch chips or simply ethernet interfaces frontend to RGMII/SerDes connection to the main switch chip.

In every case, it can't explain the lag on the console configuration, that is managed by a completely different processor than the data plane.

 

[CorvetteGS]

Hi everyone, I'm looking to utilize my ICX 7150 as a layer 3 router and moving all inter-vlan routing off my edgerouter for my home network. I was wondering if anyone has any experience with this in regards to IPv6? My current configuration on the edgerouter uses DHCPv6-PD to request a /60 prefix from my ISP and distributes the prefixes to my various VLAN interfaces using SLAAC.

I assume I have to use DHCPv6-PD still on my edgerouter to request the /60 prefix from the ISP as it would be the only device with the WAN interface, but if I am using a transit VLAN to connect to the ICX, how would I distribute the various prefixes amongst my VLANs? Thanks in advance for any help provided!

 

[epicurean]

no special requirements when removing the PoE board, just remove it and plug the switch back in. I'm sure you could sell it to one of the members here who got fried PoE boards if you want to deal with shipping (would need to be the same port count switch)

Does that mean POE module for 24P wont work for 48p?

 

[fohdeesha]

Does that mean POE module for 24P wont work for 48p?

correct, they're totally different boards

 

[juey]

Hi everyone, I'm looking to utilize my ICX 7150 as a layer 3 router and moving all inter-vlan routing off my edgerouter for my home network. I was wondering if anyone has any experience with this in regards to IPv6? My current configuration on the edgerouter uses DHCPv6-PD to request a /60 prefix from my ISP and distributes the prefixes to my various VLAN interfaces using SLAAC.

I assume I have to use DHCPv6-PD still on my edgerouter to request the /60 prefix from the ISP as it would be the only device with the WAN interface, but if I am using a transit VLAN to connect to the ICX, how would I distribute the various prefixes amongst my VLANs? Thanks in advance for any help provided!

Try using the DHCPv6 Relay Agent in your switch, but im sure you have to configure your DHCPv6 Server by hand.

 

[Spiro]

Sorry for being new in all of this. Can this be used as a fan out cable for the back ports of the ICX 6610?

Or any resonable priced suggestions for something to breakout the rear 2 ports ? Prefereable a DAC since I do not need to go far.

Thank you

Just an FYI - ADI AKOA9N04ADLN0721 40G SR4 QSFP+ to 4x SFP+ 40Gb 850nm Infiniband 4x QDR 4m #ADI ADI AKOA9N04ADLN0721 40G SR4 QSFP+ to 4x SFP+ 40Gb 850nm Infiniband 4x QDR 4m | eBay
40gb - 4x 10gb dac.. working fine - $15.00 shipped. Built in QSFP/SFP+

 

[CorvetteGS]

Try using the DHCPv6 Relay Agent in your switch, but im sure you have to configure your DHCPv6 Server by hand.

I'm not using a DHCPv6 server. My ISP has a DHCPv6 server that hands out prefixes via DHCPv6-PD. All IPv6 addresses on my network are configured with SLAAC.

 

[fohdeesha]

Sorry for being new in all of this. Can this be used as a fan out cable for the back ports of the ICX 6610?

Or any resonable priced suggestions for something to breakout the rear 2 ports ? Prefereable a DAC since I do not need to go far.

Thank you

yes that will work fine

 

[Wolfstar]

I'm not using a DHCPv6 server. My ISP has a DHCPv6 server that hands out prefixes via DHCPv6-PD. All IPv6 addresses on my network are configured with SLAAC.

Yeah, problem is, you need a DHCPv6 server upstream of the 7150 in order to hand the 7150 prefix delegations. I think. I have had ZERO luck getting this done, because I don't know whether I need to tell the switch (in my case a 6450) that it should be asking for a prefix delegation, or if individual systems should be requesting, the relay happening, and the switch just picking up which IP blocks are for what VLAN.

To make matters worse, I'm using pfSense, and I don't think there's ACTUALLY a way for me to get its DHCP server - even DHCPv6 - to accept relay requests. Currently I'm using modified Pi Hole for my DHCPv4 because of that, but I can't exactly use it to assign IPv6 since the Pi Hole doesn't have the delegation itself.

 

[infoMatt]

To make matters worse, I'm using pfSense, and I don't think there's ACTUALLY a way for me to get its DHCP server - even DHCPv6 - to accept relay requests. Currently I'm using modified Pi Hole for my DHCPv4 because of that, but I can't exactly use it to assign IPv6 since the Pi Hole doesn't have the delegation itself.

I can't speak for certainty because I haven't tried going deep enough in the IPv6 world to set up a DHCP server (yet), but as far as I know, pfSense will lease out addresses only in the subnet defined in his configuration, ie. the ones which it is their gateway.
You can however install ISC DHCP server on your pihole box and configure it to lease out addresses on whathever subnet you want, even ones not directly connected to it. On the switch you must configure the DHCP forwarding to this address. If you don't want to mess up with the pihole confguration, you can run the DHCP server on a docker container, or bind it to a secondary IP address assigned to the pihole box itself.

 

[Wolfstar]

I can't speak for certainty because I haven't tried going deep enough in the IPv6 world to set up a DHCP server (yet), but as far as I know, pfSense will lease out addresses only in the subnet defined in his configuration, ie. the ones which it is their gateway.
You can however install ISC DHCP server on your pihole box and configure it to lease out addresses on whathever subnet you want, even ones not directly connected to it. On the switch you must configure the DHCP forwarding to this address. If you don't want to mess up with the pihole confguration, you can run the DHCP server on a docker container, or bind it to a secondary IP address assigned to the pihole box itself.

Yeah, pfSense refuses to allow assignment of IPs from subnets it has no record of/interface for. It's one of the things that drives me nuts about pfSense in general, and every request I've ever found (or made) to allow for DHCP on alternate subnets has been roundly ignored.

That said, Pi Hole is based on dnsmasq, which is what's handling DHCPv4 currently with multiple subnets configured via config files in /etc/dnsmasq.d/. It could also handle DHCPv6, including RA, if I really wanted to, but I don't want to do ULA/NAT and have no way to get the /60 prefix onto the Pi Hole box in the first place.

 

[Wolfstar]

I will have some actual (private) good news in a week or so, for anyone with an MLX/CER/VDX

Well, I know you said *A* week, but it's now *NEXT* week. So, close enough.

Let me know if you need a VDX guinea pig for anything.

 

[fohdeesha]

Well, I know you said *A* week, but it's now *NEXT* week. So, close enough.

Let me know if you need a VDX guinea pig for anything.

which VDX? I probably will

 

[Wolfstar]

which VDX? I probably will

It's a VDX6720-16-R, and has a POD license so all of them are enabled, but I have no problem losing 8 ports For Science. Especially if you happen to have a line on a newer NOS than 4.1.2ac. Some output below (with serial number masked):

sw0# show version

Network Operating System Software
Network Operating System Version: 4.1.2
Copyright (c) 1995-2014 Brocade Communications Systems, Inc.
Firmware name:      4.1.2ac
Build Time:         10:45:55 May 15, 2014
Install Time:       09:29:53 Jun 20, 2014
Kernel:             2.6.34.6
BootProm:           2.2.0
Control Processor:  e500v2 with 2048 MB of memory

Appl     Primary/Secondary Versions
------------------------------------------
NOS      4.1.2ac
         4.1.2ac

sw0# show license
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
       Ports on Demand license - additional 8 port upgrade license
       Feature name:PORTS_ON_DEMAND_1
       License is valid
sw0# show inventory


NAME:FAN 1              DESCR:Chassis Fan module
PN:N/A                  SN:N/A

NAME:FAN 2              DESCR:Chassis Fan module
PN:N/A                  SN:N/A

NAME:POWER SUPPLY 1     DESCR:Chassis PS module
PN:N/A                  SN:N/A

NAME:POWER SUPPLY 2     DESCR:Chassis PS module
PN:N/A                  SN:N/A

NAME: Chassis           DESCR:System Chassis
SID:BR-VDX6720-24       SwitchType:95
PN:40-1000505-15        SN:XXXXXXXXXXX

sw0# show env power

Power Supply #1 is OK
Airflow: Port Side Exhaust
Power Supply #2 is OK
Airflow: Port Side Exhaust

sw0# show env sensor
sensor  1: (Temperature) is Ok, value is 31 C
sensor  2: (Temperature) is Ok, value is 36 C
sensor  3: (Temperature) is Ok, value is 29 C
sensor  4: (Fan        ) is Ok,speed is 3413 RPM
sensor  5: (Fan        ) is Ok,speed is 3379 RPM
sensor  6: (Power Supply) is Ok
sensor  7: (Power Supply) is Ok

sw0# sho dpod
   24 ports are available in this switch
   1 POD license is installed
     Dynamic POD method is in use
  24 port assignments are provisioned for use in this switch:
        16 port assignments are provisioned by the base switch license
         8 port assignments are provisioned by the first POD license
      *  0 more assignments are added if the second POD license is installed
  24 ports are assigned to installed licenses:
        16 ports are assigned to the base switch license
         8 ports are assigned to the first POD license
  Ports assigned to the base switch license:
     Te 0/1, Te 0/2, Te 0/3, Te 0/4, Te 0/5, Te 0/6, Te 0/7, Te 0/8, Te 0/9, Te 0/10
     Te 0/11, Te 0/12, Te 0/13, Te 0/14, Te 0/15, Te 0/16
  Ports assigned to the first POD license:
     Te 0/17, Te 0/18, Te 0/19, Te 0/20, Te 0/21, Te 0/22, Te 0/23, Te 0/24
  Ports assigned to the second POD license:
     None
  Ports not assigned to a license:
     None

   0 license reservations are still available for use by unassigned ports
sw0#

 

[CorvetteGS]

Yeah, problem is, you need a DHCPv6 server upstream of the 7150 in order to hand the 7150 prefix delegations. I think. I have had ZERO luck getting this done, because I don't know whether I need to tell the switch (in my case a 6450) that it should be asking for a prefix delegation, or if individual systems should be requesting, the relay happening, and the switch just picking up which IP blocks are for what VLAN.

Perhaps I was a bit vague with my post, but what I mean is there's no DHCPv6 server running on my equipment. Of course the ISP has their own DHCPv6-PD server (to the best of my understanding, this is not the same as a DHCPv6 server which would actually assign IP, DNS, default gateway, etc. just like a DHCPv4 server) and my edgerouter is acting as a client(?) and requesting the /60 prefix from my ISP. Accordingly, I have a stanza in the edgerouter config that distributes the individual /64 subnets to my VLANs by the various VLAN interfaces on the router.

I think no matter what, my edgerouter has to request the prefix from my ISP as it is the only device with a link-local (fe80::/10) address that can talk to the ISP equipment. I then assume that I must configure an IPv6 point-to-point link from my edgerouter to my ICX 7150, of which I assume I'll just have to use a ULA subnet (fd00::/127). I've been reading this page from the Ruckus manual and I can't tell if it's the right application for it or not. For some reason I am getting the feeling that the application for that is if the ICX 7150 were in between my edgerouter and the ISP, for whatever reason, and the ICX 7150 had to forward on the prefix requests from my edgerouter to the ISP. But if that relay agent PD notification is able to be used to forward router advertisements to the various VLANs, I am not entirely sure how I would configure the edgerouter to send the appropriate prefix to the correct VLAN if it's not aware of the VLANs anymore since they are all routed at the ICX.

 

[Wolfstar]

I am not entirely sure how I would configure the edgerouter to send the appropriate prefix to the correct VLAN if it's not aware of the VLANs anymore since they are all routed at the ICX.

Yeah, that's what you need a DHCPv6 server on your EdgeRouter for. You can't do Prefix Delegation via SLAAC - it's a function of DHCPv6. So if you want PD to be handed down from the EdgeRouter to the 7150, the EdgeRouter has to be running DHCPv6.

What I'm *not* sure of is whether or not you can assign the entire /60 to the DHCPv6 server to hand out /64s from, or if some of them need to be reserved, or what. And I can't figure out how (at least with my 6450) to get the ICX to request prefixes for ve interfaces.

 

[CorvetteGS]

Yeah, that's what you need a DHCPv6 server on your EdgeRouter for. You can't do Prefix Delegation via SLAAC - it's a function of DHCPv6. So if you want PD to be handed down from the EdgeRouter to the 7150, the EdgeRouter has to be running DHCPv6.

What I'm *not* sure of is whether or not you can assign the entire /60 to the DHCPv6 server to hand out /64s from, or if some of them need to be reserved, or what. And I can't figure out how (at least with my 6450) to get the ICX to request prefixes for ve interfaces.

I think there is a misunderstanding, I am requesting a prefix with my edgerouter and hosts are configured from that prefix via SLAAC currently. I do not host a DHCPv6 server on my edgerouter, it simply requests a /60 prefix (16 IPv6 subnets) from my ISP and the edgerouter advertises one subnet per VLAN interface with router adverts for SLAAC configuration using the delegated prefix. I can explicitly see in my current edgerouter config where the DHCPv6 server is disabled, as I had previously wanted to use DHCPv6 on each individual subnet to allocate IPv6 addresses to all hosts before I understood the benefits of SLAAC.

I agree completely on not understanding how the ICX 7150 can request prefixes for the VE interfaces, but at the same time I don't know if the ICX 7150 is supposed to request the prefixes, or if it should just pass the information from the VLANs to my edgerouter.

 

[Wolfstar]

No, you are doing ADDRESS delegation via SLAAC right now. You are currently getting a /60 from your ISP, and assigning /64s from that block to individual VLANs on the EdgeRouter. If you are not routing between VLANs on the ER, you should not be passing the VLANs from the 7150 to the ER. It should be a point-to-point Layer 3 link between the two, with the ER end providing the default gateway route for the 7150.

So if you're moving to the 7150 being the core and doing inter-VLAN routing, the 7150 assigns IPs via SLAAC to individual clients, which means that the individual Layer 3 VLAN interfaces on the 7150 need to obtain a /64 prefix from somewhere else. That somewhere else would be the upstream device - your EdgeRouter - and because of THAT, the EdgeRouter needs to run a DHCPv6 Server ONLY for the purpose of handing out prefixes it has available to the 7150's VLAN interfaces. DHCPv6 is not being used to assign individual addresses to clients, but only to assign entire blocks of IPs to "router" interfaces - in this case, the ICX "ve" interfaces.

 

[CorvetteGS]

No, you are doing ADDRESS delegation via SLAAC right now. You are currently getting a /60 from your ISP, and assigning /64s from that block to individual VLANs on the EdgeRouter. If you are not routing between VLANs on the ER, you should not be passing the VLANs from the 7150 to the ER. It should be a point-to-point Layer 3 link between the two, with the ER end providing the default gateway route for the 7150.

So if you're moving to the 7150 being the core and doing inter-VLAN routing, the 7150 assigns IPs via SLAAC to individual clients, which means that the individual Layer 3 VLAN interfaces on the 7150 need to obtain a /64 prefix from somewhere else. That somewhere else would be the upstream device - your EdgeRouter - and because of THAT, the EdgeRouter needs to run a DHCPv6 Server ONLY for the purpose of handing out prefixes it has available to the 7150's VLAN interfaces. DHCPv6 is not being used to assign individual addresses to clients, but only to assign entire blocks of IPs to "router" interfaces - in this case, the ICX "ve" interfaces.

Okay, I am starting to understand better. I was thinking more like the ICX 7150 having a function similar to 'IPv4 helper address' where the ICX 7150 would forward router adverts and other IPv6 info from the edgerouter to it's associated VLANs via a transit link (fd00::/127).But

I see what your saying, so it's like the edgerouter receives the /60 prefix from the ISP, probably has to take one /64 subnet for itself, but then has its own prefix delegation server to delegate the remaining 15 /64s to the ICX 7150, and it would assign those to the VE's accordingly.

I guess now I need to research if the edgerouter can delegate prefixes.