Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[bpye]

I know the ICX6450 has pretty limited IPv6 support, but is the DHCPv6 relay expected to work fully? I was wondering if I could use it to have a routed WAN -> ICX6450 -> Gateway by relaying the DHCP requests from the gateway to the WAN - but it looks like these packets are getting swallowed?

1/1/37 is the connection to the gateway, and 1/2/1 is the WAN. Would I need to add some additional route to make this work, or is the issue that PD requests are getting filtered? I see that 'Prefix Delegation Information' is included in 'show ipv6 dhcp-relay' - and I see a number of PD related DHCP relay commands in the command reference, however these commands don't actually appear to exist, for example 'ipv6 dhcp-relay maximum-delegated-prefixes'.

SSH@ICX6450-48P Router(config-if-e1000-1/1/37)#show ipv6 dhcp-relay                                                   
Current DHCPv6 Relay Agent State: Enabled
DHCPv6 enabled interface(s): e 1/1/37 
DHCPv6 Relay Agent Statistics:
 Total DHCPv6 Packets, Received:12, Transmitted:0 
 Received DHCPv6 Packets: RELEASE:0,RELAY_FORWARD:0,RELAY_REPLY:0 
                          OtherServertoClient:0,OtherClinettoServer:12
DHCPv6 Relay Prefix Delegation Notification:
 Maximum Delegated Prefixes Allowed:0 Current Delegated Prefixes:0
SSH@ICX6450-48P Router(config-if-e1000-1/1/37)#show ipv6 dhcp-relay interface ethernet 1/1/37
DHCPv6 Relay Information for interface e 1/1/37: 
Destinations:
 Destination                    OutgoingInterface 
 ff05::1:3                      e 1/2/1
 ff02::1:2                      e 1/2/1
Options:
 Interface-Id: No     Remote-Id:No
Prefix Delegation Information:
 Current:0 Maximum:0 AdminDistance:10

 

[fohdeesha]

I know the ICX6450 has pretty limited IPv6 support, but is the DHCPv6 relay expected to work fully? I was wondering if I could use it to have a routed WAN -> ICX6450 -> Gateway by relaying the DHCP requests from the gateway to the WAN - but it looks like these packets are getting swallowed?

1/1/37 is the connection to the gateway, and 1/2/1 is the WAN. Would I need to add some additional route to make this work, or is the issue that PD requests are getting filtered? I see that 'Prefix Delegation Information' is included in 'show ipv6 dhcp-relay' - and I see a number of PD related DHCP relay commands in the command reference, however these commands don't actually appear to exist, for example 'ipv6 dhcp-relay maximum-delegated-prefixes'.

SSH@ICX6450-48P Router(config-if-e1000-1/1/37)#show ipv6 dhcp-relay                                                 
Current DHCPv6 Relay Agent State: Enabled
DHCPv6 enabled interface(s): e 1/1/37
DHCPv6 Relay Agent Statistics:
Total DHCPv6 Packets, Received:12, Transmitted:0
Received DHCPv6 Packets: RELEASE:0,RELAY_FORWARD:0,RELAY_REPLY:0
                          OtherServertoClient:0,OtherClinettoServer:12
DHCPv6 Relay Prefix Delegation Notification:
Maximum Delegated Prefixes Allowed:0 Current Delegated Prefixes:0
SSH@ICX6450-48P Router(config-if-e1000-1/1/37)#show ipv6 dhcp-relay interface ethernet 1/1/37
DHCPv6 Relay Information for interface e 1/1/37:
Destinations:
Destination                    OutgoingInterface
ff05::1:3                      e 1/2/1
ff02::1:2                      e 1/2/1
Options:
Interface-Id: No     Remote-Id:No
Prefix Delegation Information:
Current:0 Maximum:0 AdminDistance:10

Is there a reason you're doing it on raw l2 ports and not on an addressed VE? How have you specified the relay destination? can't help much unless you share the config. I've been using dhcpv6 relay on a 6 series for years (to redundant dhcpd servers as well) without issue, but I'm not doing prefix delegation:

interface ve 10
ip address 192.168.1.1 255.255.255.0
ip helper-address 1 172.16.110.2
ip helper-address 2 172.16.110.3
ipv6 address xxx:xxx/64
ipv6 enable
ipv6 dhcp-relay destination zzz:xxx::2
ipv6 dhcp-relay destination zzz:xxx::3
ipv6 dhcp-relay include-options interface-id remote-id
ipv6 nd managed-config-flag

you're also showing 0 transmitted dhcpv6 packets, I dunno if this is part of the cause, or just a symptom of your ICX getting PD requests and intentionally not forwarding them for some config issue reason

telnet@ICX1(config-vif-10)#show ipv6 dhcp-rela
Current DHCPv6 Relay Agent State: Enabled
DHCPv6 enabled interface(s): ve 10
DHCPv6 Relay Agent Statistics:
Total DHCPv6 Packets, Received:23352, Transmitted:32847
Received DHCPv6 Packets: RELEASE:3,RELAY_FORWARD:0,RELAY_REPLY:11607
                          OtherServertoClient:0,OtherClinettoServer:11742
DHCPv6 Relay Prefix Delegation Notification:
Maximum Delegated Prefixes Allowed:512 Current Delegated Prefixes:0

on my 6610 the maximum delegated seems to default to 512, but yeah I can't change it at first glance, from any config level:

telnet@ICX1(config)#ipv6 dhcp-relay maximum-delegated-prefixes 50
Invalid input -> dhcp-relay maximum-delegated-prefixes 50
Type ? for a list

my guess is the command was only introduced to 7 series switches so the command won't work on the 6 series fw builds, the limit seems hardcoded (but indeed strange it would be limited to 0 on the 6450, not sure why they would include it as a configurable feature in that case)

 

[fohdeesha]

OK yeah, it's fastiron 6 series bug ****ery. The command exists and works on my lab FCX, which runs **the exact same bootloader and fw image as the 6610**:

telnet@ORAN-FCX(config)#ipv6 dhcp-relay maximum-delegated-prefixes
  DECIMAL   Maximum delegated prefixes, <0-512>
telnet@ORAN-FCX(config)#ipv6 dhcp-relay maximum-delegated-prefixes 50
telnet@ORAN-FCX(config)#

On a 6610 next to it with the bit-identical fw image booted:

telnet@ICX1(config)#ipv6 dhcp-relay maximum-delegated-prefixes 50
Invalid input -> dhcp-relay maximum-delegated-prefixes 50
Type ? for a list

probably worth reporting to ruckus on their user forum at least so it's documented, but who knows if it'll ever be addressed (I think they have one more bug & security release in them before these are officially End-Of-Support in Nov 2023). At the very least their documentation should be updated to state the command only works on the FCX (I've ran into a couple other commands like this, that exist in the base firmware but only become active when booted on an FCX, namely "ip ssh key-exchange-method dh-group14-sha1", which is completely nonsensical to me as the FCX runs an identical management plane vs the 6610, exact PPC part number, although less RAM I suppose )

 

[bpye]

My bad - relevant config:

vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
!
vlan 100 name wan by port
 untagged ethe 1/2/1 
!
vlan 101 by port
 untagged ethe 1/1/37 
!
ipv6 unicast-routing
interface ethernet 1/1/1
 inline power
!
interface ethernet 1/1/37
 ipv6 enable
 ipv6 dhcp-relay destination ff02::1:2 outgoing-interface ethernet 1/2/1
 ipv6 dhcp-relay destination ff05::1:3 outgoing-interface ethernet 1/2/1
 ipv6 dhcp-relay include-options interface-id remote-id           
 ipv6 nd managed-config-flag
!
interface ethernet 1/2/1
 ipv6 enable
 ipv6 nd suppress-ra
 speed-duplex 1000-full-master
!
interface ve 1
 ip address 192.168.1.10 255.255.255.0

No real reason I'm using the L2 interfaces...

 

[fohdeesha]

My bad - relevant config:

vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 100 name wan by port
untagged ethe 1/2/1
!
vlan 101 by port
untagged ethe 1/1/37
!
ipv6 unicast-routing
interface ethernet 1/1/1
inline power
!
interface ethernet 1/1/37
ipv6 enable
ipv6 dhcp-relay destination ff02::1:2 outgoing-interface ethernet 1/2/1
ipv6 dhcp-relay destination ff05::1:3 outgoing-interface ethernet 1/2/1
ipv6 dhcp-relay include-options interface-id remote-id          
ipv6 nd managed-config-flag
!
interface ethernet 1/2/1
ipv6 enable
ipv6 nd suppress-ra
speed-duplex 1000-full-master
!
interface ve 1
ip address 192.168.1.10 255.255.255.0

No real reason I'm using the L2 interfaces...

Does your icx6450 have an ipv6 address it can source traffic from to these ff02::1:2/3 destinations? eg can you run a v6 ping from the 6450 to those addresses and get a reply? guess in the dark here, but maybe because you're doing dhcpv6 relay on an interface with no IPv6 address? (well, other than the local link). maybe try assigning an ipv6 address to 1/1/37 just to test, then se if the delegation limit increases, if you're relying on link-local addresses for relay source and destination perhaps it doesn't like that (although technically in a perfect world it shouldn't matter)

 

[bpye]

Does your icx6450 have an ipv6 address it can source traffic from to these ff02::1:2/3 destinations? eg can you run a v6 ping from the 6450 to those addresses and get a reply? guess in the dark here, but maybe because you're doing dhcpv6 relay on an interface with no IPv6 address? (well, other than the local link). maybe try assigning an ipv6 address to 1/1/37 just to test, then se if the delegation limit increases, if you're relying on link-local addresses for relay source and destination perhaps it doesn't like that (although technically in a perfect world it shouldn't matter)

I'll give that a try later today after work, I am thinking I should maybe try and find a 7250, or ideally a 7150 for the fanless mode but I don't see many. The 6450 was cheap and does do the basic stuff I need, and whilst I knew it didn't have full IPv6 support I missed that it didn't support PBR, which is more annoying...

 

[tochnia]

Not sure if anyone have tested noise levels before, but I got calibrated Audio Analyser and measured noise from my 6610 with B revision Power supplies.
With only one Power Supply connected to power noise measurement are:

56dB at 1 meter distance
50dB at 2 meter distance

This is too much noise for me to stay permanently in the same room with switch..

66dB at 1 meter distance - Boot noise
60dB at 2 meter distance - Boot noise

Also Power Consumption measured on my 6610-48P is between 100-116W with no PoE enabled ports.

 

[Rain]

I've still been playing around and trying to get stacking working with 10GBASE-T transceivers between two 7250-48p switches. Nothing I've tried seems to work. ...

I meant to follow up on my attempts to stack with 10GBASE-T transceivers a while back but I've been busy.

To anyone else thinking of stacking with 10GBASE-T transceivers: It doesn't appear to be possible on the ICX 7250 switches (and likely all other ICX switches as well). I imagine the desire to stack over CAT6 is rather limited, but hopefully this saves someone a headache.

It isn't a design limitation with the switches, it's an issue with the transceivers. The 10GBASE-T transceivers can only send/receive ethernet data/frames (IEEE 802.3) whereas fiber transceivers and/or DACs are raw data connections that can be used for other, non-ethernet connections (Fiber Channel, for example). As far as I can tell, this is true of every current 10GBASE-T transceiver on the market (understandably; they're RJ45 adapters, after all).

The ICX switches appear to negotiate via ethernet and then switch to communicating between stack members via a raw connection. This is why the interfaces appear to flap when attempting to stack with 10GBASE-T transceivers. The switches bring the stacking interfaces up, figure out who's on the other end, and then switch to a raw data connection. Since the transceivers don't support non-802.3 operation, they promptly disconnect. The switch the resets the interfaces and the cycle repeats.

One of my searches while trying to figure this out rendered this internal Brocade slide-deck on stacking: stack-unit - Brocade Community Forums (edit: Despite the title, this isn't a link to the Brocade forums; it's to "studylib.net") -- Notable slides that support this theory are 30 & 31.

 

[MelnorMelvin]

Does anyone know what the difference between these AWF-2DC-1000W-E part numbers is? I was thinking at first it might tie into the hardware revision, but that didn't pan out.

23-0000142-01
23-0000142-02
23-0000142-04

More or less I'm trying to find the best deal for a Rev B/C power supply for an ICX6610-48P-PE.

 

[MelnorMelvin]

One more question. I am needing to pick up 2x ICX-FAN10-E for my ICX6610-48P-PE, and was hoping not to spend a fortune on fans.

Pretty much as cheap as I can find for an item actually tagged as ICX-FAN10-E is ~$49.50, some with a P/N of 80-1008308-02 on the factory box.

However, I did come across an item which looks physically identical with part number 60-1002274-03 (sticker on the bottom of the fans) at a much more reasonable ~$14.99, with bulk discounts (qty 2 at $13.49/ea).

Does anyone know if these two items are actually the same?

 

[klui]

Better hurry. Only 1 left.

 

[MelnorMelvin]

I’ll be honest, I already bought two because I was afraid of missing out on the price. I’m just hoping they will work for the ICX6610.

 

[eduncan911]

One more question. I am needing to pick up 2x ICX-FAN10-E for my ICX6610-48P-PE, and was hoping not to spend a fortune on fans.

Pretty much as cheap as I can find for an item actually tagged as ICX-FAN10-E is ~$49.50, some with a P/N of 80-1008308-02 on the factory box.

However, I did come across an item which looks physically identical with part number 60-1002274-03 (sticker on the bottom of the fans) at a much more reasonable ~$14.99, with bulk discounts (qty 2 at $13.49/ea).

Does anyone know if these two items are actually the same?

If you could remember to ping, ping us (or me) when you get the fans and let us know. I am in need of 1 or 2 as well.

 

[Mithril]

For the 6450 I seem to recall someone having instructions on how to change the PWM values used for low and high fan speed, but I can't find the post again

 

[sdyoung]

Assign one. That's the way I set up my switch, with a desktop plugged directly into the management port of the switch.

In Windows 11, type the start button and search for "view network connections". Right click Ethernet and double-click "Internet Protocol Version 4 (TCP/IPv4). You can then "Use the following IP address" and set whatever address you want there. Say 10.1.1.50 with a subnet mask of 255.255.255.0. Then follow fodheesas guide EXACTLY, setting a compatible IP:

factory set-default
setenv ipaddr 10.1.1.99
setenv netmask 255.255.255.0

#tell the switch the IP of your tftp server:
setenv serverip 10.1.1.50
setenv image_name ICX64xx/ICX64R08
...

Once you are done following the guide, set Win 11 back to "Obtain an IP Address automatically" and plug it back into your network.

@Lone Wolf, plugging the switch directly into my desktop, instead of another switch on my network, and manually assigning a new IP to the desktop did the trick. Was able to run update_uboot with no issues and am now on my way through the rest of the guide! Thanks again for the help man!

 

[Lone Wolf]

Awesome, I'm glad I was able to help! Enjoy your new switch!

 

[Pitchfork]

Greetings from sunny Brisbane

Before I start let me say that despite working hand-on in the IT industry for over 40 years, my experience of networking previously has been limited to plugging RJ45 cables into dumb unmanaged switches. If I sound like a total noob it's because, well, I am.

Having got that out of the way, we recently bought 4 x ICX6610s and I am currently working my way through learning about them. We have "just plugged one in" to connect our servers temporarily and am working through another one of them using the guide on fohdeesha.com.

My first issue is that we need to transfer a lot of data (umpteen TB) from one server to another as we transition hardware and at the moment everything plugged into the switch is running by default at 1gbE. I understand that the 8 ports on the left hand side of the front of the switch are SPF+ and can run at 10gbE. I'm guessing that I need some kind of adapter to plug RJ45 cables into these. So my first question is, what do I look for when I go searching for them? Are there any that work better than others? Any that should be avoided?

The second question is regarding the breakout ports at the back of the unit. The diagram I am looking at says that the rightmost pair of 40gbE ports can break out into 8 x 10gbE. Again, what do I need to do this.

Third question is about the uplink ports (the left most two on the back of the unit). I have cables that came with the units. Is there any special way that these need to be connected together or are they just daisy chained from one switch to another?

Apologies for all these dumb questions - I've been reading and playing (and going deaf from the noise) for a week or so and I could really use the input.

Thanks and Regards

Nigel.

 

[Rand__]

My first issue is that we need to transfer a lot of data (umpteen TB) from one server to another as we transition hardware and at the moment everything plugged into the switch is running by default at 1gbE. I understand that the 8 ports on the left hand side of the front of the switch are SPF+ and can run at 10gbE. I'm guessing that I need some kind of adapter to plug RJ45 cables into these. So my first question is, what do I look for when I go searching for them? Are there any that work better than others? Any that should be avoided?

Look at these (other models might be available too) https://www.servethehome.com/sfp-to-10gbase-t-adapter-module-buyers-guide/

The second question is regarding the breakout ports at the back of the unit. The diagram I am looking at says that the rightmost pair of 40gbE ports can break out into 8 x 10gbE. Again, what do I need to do this.

40GB -> 4x10 GB needs a breakout QSFP cable (and potentially configuration at interface level, have not done it on the brocades)

Third question is about the uplink ports (the left most two on the back of the unit). I have cables that came with the units. Is there any special way that these need to be connected together or are they just daisy chained from one switch to another?

Check the manual to get all the information you need
Look for " Dedicated stacking ports and trunks in slot 2 on an ICX 6610 "

 

[kpfleming]

Look at these (other models might be available too) https://www.servethehome.com/sfp-to-10gbase-t-adapter-module-buyers-guide/

Also, keep in mind that 'plugging RJ45 cables into these' may not be the actual question you want to ask

If the devices you are using have 10GbE copper (RJ-45) NICs, or NICs with SFP+ that are equivalent, then using Cat-6 RJ-45 cables to connect them to the switch could provide 10GbE links. If those devices do not have 10GbE NICs, or they don't have copper ports, then you'll need to upgrade both ends (device and switch) in order to obtain 10GbE speeds.

 

[Pitchfork]

Also, keep in mind that 'plugging RJ45 cables into these' may not be the actual question you want to ask

If the devices you are using have 10GbE copper (RJ-45) NICs, or NICs with SFP+ that are equivalent, then using Cat-6 RJ-45 cables to connect them to the switch could provide 10GbE links. If those devices do not have 10GbE NICs, or they don't have copper ports, then you'll need to upgrade both ends (device and switch) in order to obtain 10GbE speeds.

Thanks for the response - yes the ones in question have 10GbE NICs.

I should have lifted the veil a little higher. It's all a complex (to me) jigsaw - we are connecting together 3 x Dell R730's, a stack of 3 x Isilon X210's (with a Mellanox IS4022 switch) and a rack mounted QNAP plus the ICX-6610s (we have two 6610 and 2 x 6610P to pick from) . The idea is that the 3 x R730's will form a VMWare cluster and will use the Isilons for main storage (they have none of their own). The main QNAP will us used for backups and for copying the whole lot up into the cloud for a secondary backup.

Two of the R730's have network connections that look like this. The two RJ-45 NICs are 10/100/1000. I have no idea what the other connections are (like I said - this is my first networking rodeo).



The third R730 looks a little different. This one has 4 x RJ45 NICs but I have no idea what the heck the second interfaces are.


So far as the Isilon X210s are concerned, I believe that the three of them are connected to the Mellanox switch via their own fibre connections to form their own "internal" network for the Isilons to talk to each other. How the Isilons (or the Mellanox switch) connect into the main network is (so far) beyond me.


The main QNAP in question is a new QNAP TS-1273AU-RP-8G with a QXG-10G2T-107 Dual Port Expansion card (10GbE / 5GbE / 2.5GbE / 1GbE / 100MbE)

We have a network engineer coming out from Dell / Unisys early next week to recommission the Isilons and I am hoping to get his advice on optimal connections of all the parts in this jigsaw but I was hoping that the collected wisdom here might be able to make some suggestions as to "how do we connect all these bits of disparate kit together to get the best network throughput speed overall at the minimum (of course) cost".

If anyone with any suggestions happens to be here in or close to Bris-Vegas I'd be happy to pour vast quantities of beer down your throat and stuff you full of BBQ in exchange for advice, knowledge, suggestions, wisdom etc.

Apologies for the length of the post and the multitude of pictures. I hope I'm not inadvertently breaking any of the forum rules.

Thanks again

Nigel.