Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[LodeRunner]

Hi all, I configured NTP in client-only mode with Google's time servers according to @fohdeesha's ICX 7150 guide, but all four of them are being tagged as "reject" ("This peer is rejected by the selection algorithm.") and stratum 16. Currently running 08.0.95ca on a 7150-48ZP.
Would greatly appreciate any help or guidance anyone can offer.

The official documentation's NTP Client Mode Configuration Example shows adding a peer in addition to multiple server entries. Is that strictly required? Am I missing something else? I'd really love the datetimes on the logs to correspond with reality...

NTP Status Output Reference
NTP Association Output Reference
NTP Association Details Output Reference

Relevant outputs below:

SW-CORE>show ntp status
Clock is unsynchronized, no reference clock
NTP server mode is disabled, NTP client mode is enabled
NTP master mode is disabled, NTP master stratum is 8
NTP is not in panic mode

SW-CORE>show ntp associations
address                                   Domain name                             Reference Clock  st  when  poll  Reach delay  offset   disp
~ 216.239.35.0                            None                                     INIT            16     -    64     0   0.00    0.000 15937.
~ 216.239.35.4                            None                                     INIT            16     -    64     0   0.00    0.000 15937.
~ 216.239.35.8                            None                                     INIT            16     -    64     0   0.00    0.000 15937.
~ 216.239.35.12                           None                                     INIT            16     -    64     0   0.00    0.000 15937.
* synced, # selected, + candidate, - outlayer, x falseticker, ~ configured, **More characters in domain name

SW-CORE>ping 216.239.35.0
Sending 1, 16-byte ICMP Echo to 216.239.35.0, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 216.239.35.0    : bytes=16 time=22ms TTL=107
Success rate is 100 percent (1/1), round-trip min/avg/max=22/22/22 ms.

SW-CORE>show ntp association detail
216.239.35.0 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93802499
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         576       118       185       249       313       380       444       511

216.239.35.4 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93802499
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         576       117       181       247       312       379       446       511

216.239.35.8 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93801000
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         577       119       184       250       314       380       444       510

216.239.35.12 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93799500
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         578       117       184       249       314       381       448       513

What is output of "sh clock"?

Edit: I just checked the NTP config on my 7450, v8092, and I only used server declarations, no peer declarations.

ntp
disable serve
server 0.pool.ntp.org minpoll 10 burst
server 1.pool.ntp.org minpoll 10 burst
server 2.pool.ntp.org minpoll 10 burst
server 3.pool.ntp.org minpoll 10 burst

SSH@core#sh ntp associations
address         Domain name     Reference Clock  st  when  poll  Reach delay  offset   disp 
+~ 70.35.196.28  0.pool.ntp.org   132.163.96.4     2   438  1024   373  8.662 -11.7717 23.113
*~ 108.62.122.57 1.pool.ntp.org   249.224.99.213   2   189  1024   177  3.440 -29.2227 20.330
+~ 67.205.162.81 2.pool.ntp.org   216.218.192.202  2   419  1024   377 14.880 -12.0084 22.009
+~ 69.1.1.251    3.pool.ntp.org   35.73.197.144    2   629  1024   337 49.003  -5.5585 23.003

I manually set the clock on my switch to local time before configuring NTP. Unsure if there's a maximum deviation where NTP won't sync, hence my original post just asking for output of "sh clock".

 

[hmw]

Can anyone with a ICX-7650 ZP comment on how loud the switch is? The data sheets show the 7650ZP at 56 dBA, whereas the ICX-6610 is rated at 49 dBA. Wondering if the 7650-ZP is quieter with single PSU and if it's as quiet as the 6610 ...

 

[scot1297]

I might have missed it, or the videos were over my head, because this is all new to me. I just setup a 6610 and it is plugged into my pfsense box. I was able to use a serial cable do all the updates and all ports are setup as vlan 1 default. I followed setting up the vlan 1 default route and ip. I can plug things into the switch and it works and can get to the internet just fine.

I am trying to add vlan 20 to my switch. It is defined in pfsense and this vlan is going to be used for my wifi network. I have a unified AP point that is plugged into port 3. It works just fine and can connect to the wifi when I am just using the default Vlan 1. I tried adding vlan 20 to the switch and tagging it for e 1/1/3. I also updated my unifi ap to be on vlan 20, but it is a no go.

I am missing something, but not sure what yet. I tried adding a interface ve 20 to the vlan, but it also didn't work. So if you can point me in the right direction that would be great.

Thanks again for all this information on here, it really is good, but I might be a little dense on the vlan setup for a port for a unifi ap.

 

[dennisp]

Try setting 1/1/3 to untagged. Since you're using the default vlan, I think you'll need dual mode on that port also.

 

[klui]

I am trying to add vlan 20 to my switch. It is defined in pfsense and this vlan is going to be used for my wifi network. I have a unified AP point that is plugged into port 3. It works just fine and can connect to the wifi when I am just using the default Vlan 1. I tried adding vlan 20 to the switch and tagging it for e 1/1/3. I also updated my unifi ap to be on vlan 20, but it is a no go.

You need to ensure vlan 20 defined in pfSense is brought to your switch through its interconnect. Use sh interfaces brief/sh vlan/sh int ve 20/sh mac-address.

 

[Jason Antes]

Also does anybody have a spare 6610's fan that could sell me in a reasonable price? $69.99 plus tax, shipping and custom duties ($95-110 USD) are way more that I'd like to spend on ebay.

Port or rear exhaust?

 

[Jason Antes]

You need to ensure vlan 20 defined in pfSense is brought to your switch through its interconnect. Use sh interfaces brief/sh vlan/sh int ve 20/sh mac-address.

To add to this, since I just went through this (vpadro look for my thread in this forum), yes make sure that you have the VE interface designated on the switch for the VLAN and that it matches the VLAN ID (20 in this case). Put the port you are using as the uplink to the AP as dual port. If you don't you won't be able to manage it. I set up VLANs 10, 20, 30, and 40. VLAN 10 is dual port and only does AP management, 20 is my normal wireless and is a tagged port only as well as set up in the controller as a tagged network. 30 for my IOT devices, again tagged and set up as a separate network in the controller. 40 is my DMZ and same as the others. All have the VE port set that corresponds to the VLAN ID. It took me a bit to realize that one of the VLANs I had setup in the switch didn't have that field populated so it didn't work. Double check.

 

[Jason Antes]

Question for the 6610 owners out there:

The material here says the 6610 is 16" deep, no matter the number of ports.

1. Does that depth figure include the fan and power supply handles in the back, or is it just for the chassis itself (not including the handles)?

View attachment 18311

2. I noticed in one of the pictures from an ebay seller that you can mount the rack ears a bit further to the back than normal, like this:

View attachment 18312

Does anybody have the measurement of the distance represented by the red line above? How many inches is that?

I'm trying to see if I can fit a 6610 in the rack I have on order. I'm considering if it's worth going with a bigger one just for this.

Thanks!

Sadly, I only have the 2 post ears as a full kit (I have the back mounts for a full rack rail kit but not the switch mount). Anyway, this is what came with my 6610-48 POE model.

Here is the rack mount part of the guide: Brocade ICX 6610 Stackable Switch Hardware Installation Guide - Manual (Page 31)

 

[richtj99]

This brings up a question for me too - if I am doing a LACP / LAG - I am using the 10gb fiber and have three multimode fiber (total of 6 strands):

Is the point for extra redundancy, extra speed, or both?
Is there a command to check current speed usage?
If I set up two of my three fiber pairs as a LAG - will it stop working if only one pair is available?
-two fiber SFP+ in use, then one does down or is removed - will this bring down the network?
Any issue LAG'ing a 6450 and a 7250?

Thanks,
Rich

There is some good stuff here: NetAdmin.us Website

The ICX Configuration Tool (GUI) can generate L2 switch config based on your input.

Terry Henry also has alot of good snippets on different things: https://www.youtube.com/c/TerryHenry/videos

 

[RealJamesDean]

There is some good stuff here: NetAdmin.us Website

The ICX Configuration Tool (GUI) can generate L2 switch config based on your input.

Terry Henry also has alot of good snippets on different things: https://www.youtube.com/c/TerryHenry/videos

The site looks mostly like what I was looking for. Unfortunately I'm not a big Windows user, so the Config tool isn't much use to me. (I don't even maintain a Windows VM.) Videos are not my favorite medium to learn from, but I'll give those a look too. Thank You!

 

[eduncan911]

Question for all the networking experts like @kapone : any reason why i couldn’t use pfsense as the dedicated firewall with pihole on a rpi4 doing DNS and DHCP for several VLANs, using the 6610 to do all the layer 3 routing? My understanding is that pihole 5 does serve DHCP to non-connected subnets.

does anyone have direct experience with this particular setup? I’m a VLAN noob here and a bit hesitant to take the leap,

There's a few issues in this design. I keep setting up and tearing down different VLAN concepts myself, using my HomeLAN as my HomeLAB[tm].

First of all, DHCP requests are generally kept on the same subnet. That is, whatever device you use as your gateway (either InterVLAN Virtual IP within the switch, or the Router-on-a-Stick concept of just trunking all VLANs to the pfSense device) will need to be able to listen for DHCP requests (e.g. the pfSense box with multiple VLANs) or forward the DHCP requests (within the switch). I haven't done the forwarding yet, but that's my next experiment to keep DHCP on the router and do intervlan setups within the switch.

That was a lot to unpack. But, it's a lot to search and learn on first as it will save you a LOT of headaches (DHCP, VLAN, InterVLAN, DHCP Forwarding, etc). I think the Brocade commands are (don't quote me on this, as I haven't used it yet):

interface ve 21
ip address 10.11.0.1 255.255.255.0
ip helper-address 1 10.3.0.125
ip helper-address 2 10.1.0.125

The advantage I see here is that you could potentially setup two DHCP servers (on two different sub-nets) and have redundancy.

With that said... If you want to use the ASIC routing within the switches (high-performance InterVLAN hardware-routing within the switches), then you'll want to document a set of VLANs up front to configure within the switch, along with the VIP within the switch to handle all internal VLAN routing. Then it's a matter of just forwarding requests/broadcasts to your upstream devices (DNS, DHCP, Chromecast/Bonjour, etc).

I'm currently trunking all VLAN traffic to my pfSense VM, as I run it on my Proxmox cluster. It has dual 10G fiber in LAG, so I'm not worried about bandwidth for my home across the VLANs. I did this as it was the fastest to get setup and going, and I really like to monitor the VLAN traffic - especially across the air-gapped IoT and Management VLANs. InterVLAN is my next experiment, whenever I have time.

 

[eduncan911]

Duck-Typing...

I'm having trouble affording a second ICX7250-48P to stack on fleebay (I need more 10G ports, and for redundancy/LAG/failover experiments, especially with Proxmox HA). Everyone wants at least $310+ now (with shipping factored in) with the lowest counter-offers I've had. There's been a few auctions, and they went for about $290 with shipping. That's just out my laid-off budget.

I've been considering switching over to a couple of 6610-24s to stack. A lot cheaper, and I could sell my one ICX7250-48P to cover the costs.

However, my concerns are the costs of the breakout cables/components as well as the idle/power consumption (not because of power cost, but in terms of heat generation of long-time use in a small tight closet with just a little ventilation/air-control). Noise level is a little issue, but since it's going into an enclosed rack eventually I think I'll be ok setting them up and getting rid of them shortly after SSH is enabled.

Seems the most inexpensive option I've found is:
* Brocade 57-1000267-01 (64G Fiber-to-QSFP) $10 (thanks @fohdeesha )
* MPO/MTP to 8x LC breakout cable, like this for a whooping $170 each.

So those breakout cables exceed the cost savings... Not even sure if they would work either.

There are much cheaper MPO-to-LCs from China, but I don't want to take a chance (nor wait).

Looking for the most inexpensive option for 10G LC fiber connections to breakout of the QSFP ports. I mean, I could get away with just the 8x 10G ports on the front of the 6610-24s for now (for 16x ports). But I'd be maxed out.

 

[itronin]

That was a lot to unpack. But, it's a lot to search and learn on first as it will save you a LOT of headaches (DHCP, VLAN, InterVLAN, DHCP Forwarding, etc). I think the Brocade commands are (don't quote me on this, as I haven't used it yet):

interface ve 21
ip address 10.11.0.1 255.255.255.0
ip helper-address 1 10.3.0.125
ip helper-address 2 10.1.0.125

Yes.

The advantage I see here is that you could potentially setup two DHCP servers (on two different sub-nets) and have redundancy.

yeppir.

What I'm about to say should be obvious but gonna say it anyway as sometimes folks forget (including me with blind cut/n/paste across servers ):

If you are using two ip helpers for the same subnet please use non-overlapping IP address ranges in the defined scopes for each subet (vlan) on each server.

 

[itronin]

Looking for the most inexpensive option for 10G LC fiber connections to breakout of the QSFP ports.

Any reason you can't DAC or AOC? 3.28feet is pretty short well within tolerance. IIRC DAC is lowest power/heat, AOC is next or equiv to Xver and cables, then 10G-base-t which you aren't interested in.

 

[eduncan911]

Any reason you can't DAC or AOC? 3.28feet is pretty short well within tolerance. IIRC DAC is lowest power/heat, AOC is next or equiv to Xver and cables, then 10G-base-t which you aren't interested in.

Not really, no. However, the Dell ones linked to on the front page are still $95 each.

It's just the latency factor and remaining 100% fiber everywhere - for the hell of it. I mean, I ran a 75ft fiber just to connect my little-used desktop on the other side of the house - as I don't have any Cat6a cables, ends, etc. Everything is fiber through the switches and DMZ, all the way to the pfsense router via IOMMU passthrough - and then goes to RJ45 to the ISP (ugh). Pinging in gaming servers are still quiet low!

But you are right. Those DACs are a bit cheaper at about 1/2 the cost. And again, there are $35 ones in China... But then again, it's China.

 

[eduncan911]

Any reason you can't DAC or AOC? 3.28feet is pretty short well within tolerance. IIRC DAC is lowest power/heat, AOC is next or equiv to Xver and cables, then 10G-base-t which you aren't interested in.

Actually, could you link to a post about AOC/Xver cables? I haven't read about this option yet.

 

[itronin]

Here's a used AOC breakout.

and some new ones.

I'm using this to go from one 6610 down to my servers from the 10Gbe breakouts.
I've used short NetAPP QSFP cables to stack 6610's.
I'm currently using short 40Gbe AOC (QSFP) to go to a TNC server with an HPE reflashed to Mellanox firmware 40Gbe adapter. works great. I think the AOC cables were like $15 each used.

You can certainly do the Xver's and fiber runs off the front longer individual patches and think you'll find that is not too bad $$ ... $8 per xver and $15-$25 for the cable?
If you are doing short runs off the front in your rack the used Cisco CUCM - length etc. tend to be between 9 and 15 each used and I've not had a problem with them.

 

[eduncan911]

Here's a used AOC breakout.
and some new ones.

I'm using this to go from one 6610 down to my servers from the 10Gbe breakouts.
I've used short NetAPP QSFP cables to stack 6610's.
I'm currently using short 40Gbe AOC (QSFP) to go to a TNC server with an HPE reflashed to Mellanox firmware 40Gbe adapter. works great. I think the AOC cables were like $15 each used.

You can certainly do the Xver's and fiber runs off the front longer individual patches and think you'll find that is not too bad $$ ... $8 per xver and $15-$25 for the cable?
If you are doing short runs off the front in your rack the used Cisco CUCM - length etc. tend to be between 9 and 15 each used and I've not had a problem with them.

Ah, those are the perfect cables and price! Perhaps these need to be added to that 3rd or 4th post on the first page, where it talks about the 6610 breakout.

 

[kapone]

Lot 14 DELL Force 10 Copper Breakout Cable QSFP+ 4X SFP Network 3M 27GG5 | eBay

Less than $10 per cable, although gotta buy 14 of them. And they work perfectly, I'm using them right now.

 

[neonclash]

I've got a strange mystery with stacking a pair of these I have finally been able to get my hands on. I have a 7150-C12P and a 7150-24P, and I can stack them just fine with some twinax in the SFP+ ports. However, I haven't been able to coax them into linking over 10g ethernet to get a stack across the apartment. When the SFP isn't set as a stack-port on the 24P, they will negotiate a link, and when neither is set, I can use that as a trunk. But the moment I set them both as stack ports, they start alternating between down and up on both ends every few seconds before settling into a state where the 12P thinks the link is up, but the 24p sees it as down.

This happens regardless of the combination of SFP+ module (from FLYPROFiber, I have one marked "for brocade" and two "for cisco"), stacking port (either of the two on both ends) and cable (I have tried all of the CAT6 and CAT7 I have). Attaching the 24p to itself (stacking 1/3/1 to nonstacking 1/3/2) forms a link, and so does connecting the 24p to itself. Do I need to purchase another transceiver, find another used switch, or resign myself to only being able to trunk them?