Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Bladerunner · Apr 19, 2021

Hi all, I configured NTP in client-only mode with Google's time servers according to @fohdeesha's ICX 7150 guide, but all four of them are being tagged as "reject" ("This peer is rejected by the selection algorithm.") and stratum 16. Currently running 08.0.95ca on a 7150-48ZP.
Would greatly appreciate any help or guidance anyone can offer.

The official documentation's NTP Client Mode Configuration Example shows adding a peer in addition to multiple server entries. Is that strictly required? Am I missing something else? I'd really love the datetimes on the logs to correspond with reality...

NTP Status Output Reference
NTP Association Output Reference
NTP Association Details Output Reference

Relevant outputs below:

Code:

SW-CORE>show ntp status
Clock is unsynchronized, no reference clock
NTP server mode is disabled, NTP client mode is enabled
NTP master mode is disabled, NTP master stratum is 8
NTP is not in panic mode

SW-CORE>show ntp associations
address                                   Domain name                             Reference Clock  st  when  poll  Reach delay  offset   disp
~ 216.239.35.0                            None                                     INIT            16     -    64     0   0.00    0.000 15937.
~ 216.239.35.4                            None                                     INIT            16     -    64     0   0.00    0.000 15937.
~ 216.239.35.8                            None                                     INIT            16     -    64     0   0.00    0.000 15937.
~ 216.239.35.12                           None                                     INIT            16     -    64     0   0.00    0.000 15937.
* synced, # selected, + candidate, - outlayer, x falseticker, ~ configured, **More characters in domain name

SW-CORE>ping 216.239.35.0
Sending 1, 16-byte ICMP Echo to 216.239.35.0, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 216.239.35.0    : bytes=16 time=22ms TTL=107
Success rate is 100 percent (1/1), round-trip min/avg/max=22/22/22 ms.

SW-CORE>show ntp association detail
216.239.35.0 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93802499
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         576       118       185       249       313       380       444       511

216.239.35.4 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93802499
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         576       117       181       247       312       379       446       511

216.239.35.8 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93801000
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         577       119       184       250       314       380       444       510

216.239.35.12 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93799500
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         578       117       184       249       314       381       448       513

Archos · Apr 19, 2021

RealJamesDean said:
Is there anything like a 6450 basic admin guide for dummies out there? I'm not trying to do much beyond L2 stuff at the moment. Especially looking for guidance in setting up SFP and SFP+ modules and bonding ports with LACP.

There is some good stuff here: NetAdmin.us Website

The ICX Configuration Tool (GUI) can generate L2 switch config based on your input.

Terry Henry also has alot of good snippets on different things: https://www.youtube.com/c/TerryHenry/videos

pod · Apr 19, 2021

Bladerunner said:

Hi all, I configured NTP in client-only mode with Google's time servers according to @fohdeesha's ICX 7150 guide, but all four of them are being tagged as "reject" ("This peer is rejected by the selection algorithm.") and stratum 16. Currently running 08.0.95ca on a 7150-48ZP.
Would greatly appreciate any help or guidance anyone can offer.

The official documentation's NTP Client Mode Configuration Example shows adding a peer in addition to multiple server entries. Is that strictly required? Am I missing something else? I'd really love the datetimes on the logs to correspond with reality...

NTP Status Output Reference
NTP Association Output Reference
NTP Association Details Output Reference

Relevant outputs below:

Code:

SW-CORE>show ntp status
Clock is unsynchronized, no reference clock
NTP server mode is disabled, NTP client mode is enabled
NTP master mode is disabled, NTP master stratum is 8
NTP is not in panic mode

SW-CORE>show ntp associations
address                                   Domain name                             Reference Clock  st  when  poll  Reach delay  offset   disp
~ 216.239.35.0                            None                                     INIT            16     -    64     0   0.00    0.000 15937.
~ 216.239.35.4                            None                                     INIT            16     -    64     0   0.00    0.000 15937.
~ 216.239.35.8                            None                                     INIT            16     -    64     0   0.00    0.000 15937.
~ 216.239.35.12                           None                                     INIT            16     -    64     0   0.00    0.000 15937.
* synced, # selected, + candidate, - outlayer, x falseticker, ~ configured, **More characters in domain name

SW-CORE>ping 216.239.35.0
Sending 1, 16-byte ICMP Echo to 216.239.35.0, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 216.239.35.0    : bytes=16 time=22ms TTL=107
Success rate is 100 percent (1/1), round-trip min/avg/max=22/22/22 ms.

SW-CORE>show ntp association detail
216.239.35.0 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93802499
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         576       118       185       249       313       380       444       511

216.239.35.4 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93802499
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         576       117       181       247       312       379       446       511

216.239.35.8 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93801000
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         577       119       184       250       314       380       444       510

216.239.35.12 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93799500
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         578       117       184       249       314       381       448       513

Are you using the dns names time1.google.com, time2.google.com...? I would guess they do the Cloudflare no direct access thing.

kapone · Apr 19, 2021

darthray said:
Question for the 6610 owners out there:

The material here says the 6610 is 16" deep, no matter the number of ports.

1. Does that depth figure include the fan and power supply handles in the back, or is it just for the chassis itself (not including the handles)?

View attachment 18311

2. I noticed in one of the pictures from an ebay seller that you can mount the rack ears a bit further to the back than normal, like this:

View attachment 18312

Does anybody have the measurement of the distance represented by the red line above? How many inches is that?

I'm trying to see if I can fit a 6610 in the rack I have on order. I'm considering if it's worth going with a bigger one just for this.

Thanks!

I did a rough measurement, without de-racking mine at home, and the red line should be ~6.5".

LodeRunner · Apr 19, 2021

Bladerunner said:

Hi all, I configured NTP in client-only mode with Google's time servers according to @fohdeesha's ICX 7150 guide, but all four of them are being tagged as "reject" ("This peer is rejected by the selection algorithm.") and stratum 16. Currently running 08.0.95ca on a 7150-48ZP.
Would greatly appreciate any help or guidance anyone can offer.

The official documentation's NTP Client Mode Configuration Example shows adding a peer in addition to multiple server entries. Is that strictly required? Am I missing something else? I'd really love the datetimes on the logs to correspond with reality...

NTP Status Output Reference
NTP Association Output Reference
NTP Association Details Output Reference

Relevant outputs below:

Code:

SW-CORE>show ntp status
Clock is unsynchronized, no reference clock
NTP server mode is disabled, NTP client mode is enabled
NTP master mode is disabled, NTP master stratum is 8
NTP is not in panic mode

SW-CORE>show ntp associations
address                                   Domain name                             Reference Clock  st  when  poll  Reach delay  offset   disp
~ 216.239.35.0                            None                                     INIT            16     -    64     0   0.00    0.000 15937.
~ 216.239.35.4                            None                                     INIT            16     -    64     0   0.00    0.000 15937.
~ 216.239.35.8                            None                                     INIT            16     -    64     0   0.00    0.000 15937.
~ 216.239.35.12                           None                                     INIT            16     -    64     0   0.00    0.000 15937.
* synced, # selected, + candidate, - outlayer, x falseticker, ~ configured, **More characters in domain name

SW-CORE>ping 216.239.35.0
Sending 1, 16-byte ICMP Echo to 216.239.35.0, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 216.239.35.0    : bytes=16 time=22ms TTL=107
Success rate is 100 percent (1/1), round-trip min/avg/max=22/22/22 ms.

SW-CORE>show ntp association detail
216.239.35.0 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93802499
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         576       118       185       249       313       380       444       511

216.239.35.4 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93802499
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         576       117       181       247       312       379       446       511

216.239.35.8 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93801000
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         577       119       184       250       314       380       444       510

216.239.35.12 configured server, reject,  stratum 16
ref ID INIT, time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
our mode client, peer mode server, our poll intvl 16, peer poll intvl 64,
root delay 0 msec, root disp 0, root dist 15.93799500
delay 0 msec, offset 0 msec, dispersion 15937.,
precision 2**-16, version 4
org time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
rcv time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
xmt time 0.0 (19:00:00.0 GMT-05 Sun Dec 31 1899)
filter delay       0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter offset      0.000     0.000     0.000     0.000     0.000     0.000     0.000     0.000
filter disp    16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000 16000.000
filter epoch         578       117       184       249       314       381       448       513

What is output of "sh clock"?

Edit: I just checked the NTP config on my 7450, v8092, and I only used server declarations, no peer declarations.

Code:

ntp
disable serve
server 0.pool.ntp.org minpoll 10 burst
server 1.pool.ntp.org minpoll 10 burst
server 2.pool.ntp.org minpoll 10 burst
server 3.pool.ntp.org minpoll 10 burst

Code:

SSH@core#sh ntp associations
address         Domain name     Reference Clock  st  when  poll  Reach delay  offset   disp 
+~ 70.35.196.28  0.pool.ntp.org   132.163.96.4     2   438  1024   373  8.662 -11.7717 23.113
*~ 108.62.122.57 1.pool.ntp.org   249.224.99.213   2   189  1024   177  3.440 -29.2227 20.330
+~ 67.205.162.81 2.pool.ntp.org   216.218.192.202  2   419  1024   377 14.880 -12.0084 22.009
+~ 69.1.1.251    3.pool.ntp.org   35.73.197.144    2   629  1024   337 49.003  -5.5585 23.003

I manually set the clock on my switch to local time before configuring NTP. Unsure if there's a maximum deviation where NTP won't sync, hence my original post just asking for output of "sh clock".

hmw · Apr 19, 2021

Can anyone with a ICX-7650 ZP comment on how loud the switch is? The data sheets show the 7650ZP at 56 dBA, whereas the ICX-6610 is rated at 49 dBA. Wondering if the 7650-ZP is quieter with single PSU and if it's as quiet as the 6610 ...

scot1297 · Apr 19, 2021

I might have missed it, or the videos were over my head, because this is all new to me. I just setup a 6610 and it is plugged into my pfsense box. I was able to use a serial cable do all the updates and all ports are setup as vlan 1 default. I followed setting up the vlan 1 default route and ip. I can plug things into the switch and it works and can get to the internet just fine.

I am trying to add vlan 20 to my switch. It is defined in pfsense and this vlan is going to be used for my wifi network. I have a unified AP point that is plugged into port 3. It works just fine and can connect to the wifi when I am just using the default Vlan 1. I tried adding vlan 20 to the switch and tagging it for e 1/1/3. I also updated my unifi ap to be on vlan 20, but it is a no go.

I am missing something, but not sure what yet. I tried adding a interface ve 20 to the vlan, but it also didn't work. So if you can point me in the right direction that would be great.

Thanks again for all this information on here, it really is good, but I might be a little dense on the vlan setup for a port for a unifi ap.

dennisp · Apr 19, 2021

Try setting 1/1/3 to untagged. Since you're using the default vlan, I think you'll need dual mode on that port also.

klui · Apr 19, 2021

scot1297 said:
I am trying to add vlan 20 to my switch. It is defined in pfsense and this vlan is going to be used for my wifi network. I have a unified AP point that is plugged into port 3. It works just fine and can connect to the wifi when I am just using the default Vlan 1. I tried adding vlan 20 to the switch and tagging it for e 1/1/3. I also updated my unifi ap to be on vlan 20, but it is a no go.

You need to ensure vlan 20 defined in pfSense is brought to your switch through its interconnect. Use sh interfaces brief/sh vlan/sh int ve 20/sh mac-address.

Jason Antes · Apr 19, 2021

vpadro said:
Also does anybody have a spare 6610's fan that could sell me in a reasonable price? $69.99 plus tax, shipping and custom duties ($95-110 USD) are way more that I'd like to spend on ebay.

Port or rear exhaust?

Jason Antes · Apr 19, 2021

klui said:
You need to ensure vlan 20 defined in pfSense is brought to your switch through its interconnect. Use sh interfaces brief/sh vlan/sh int ve 20/sh mac-address.

To add to this, since I just went through this (vpadro look for my thread in this forum), yes make sure that you have the VE interface designated on the switch for the VLAN and that it matches the VLAN ID (20 in this case). Put the port you are using as the uplink to the AP as dual port. If you don't you won't be able to manage it. I set up VLANs 10, 20, 30, and 40. VLAN 10 is dual port and only does AP management, 20 is my normal wireless and is a tagged port only as well as set up in the controller as a tagged network. 30 for my IOT devices, again tagged and set up as a separate network in the controller. 40 is my DMZ and same as the others. All have the VE port set that corresponds to the VLAN ID. It took me a bit to realize that one of the VLANs I had setup in the switch didn't have that field populated so it didn't work. Double check.

Jason Antes · Apr 19, 2021

darthray said:
Question for the 6610 owners out there:

The material here says the 6610 is 16" deep, no matter the number of ports.

1. Does that depth figure include the fan and power supply handles in the back, or is it just for the chassis itself (not including the handles)?

View attachment 18311

2. I noticed in one of the pictures from an ebay seller that you can mount the rack ears a bit further to the back than normal, like this:

View attachment 18312

Does anybody have the measurement of the distance represented by the red line above? How many inches is that?

I'm trying to see if I can fit a 6610 in the rack I have on order. I'm considering if it's worth going with a bigger one just for this.

Thanks!

Sadly, I only have the 2 post ears as a full kit (I have the back mounts for a full rack rail kit but not the switch mount). Anyway, this is what came with my 6610-48 POE model.

Here is the rack mount part of the guide: Brocade ICX 6610 Stackable Switch Hardware Installation Guide - Manual (Page 31)

richtj99 · Apr 19, 2021

This brings up a question for me too - if I am doing a LACP / LAG - I am using the 10gb fiber and have three multimode fiber (total of 6 strands):

Is the point for extra redundancy, extra speed, or both?
Is there a command to check current speed usage?
If I set up two of my three fiber pairs as a LAG - will it stop working if only one pair is available?
-two fiber SFP+ in use, then one does down or is removed - will this bring down the network?
Any issue LAG'ing a 6450 and a 7250?

Thanks,
Rich

Archos said:
There is some good stuff here: NetAdmin.us Website

The ICX Configuration Tool (GUI) can generate L2 switch config based on your input.

Terry Henry also has alot of good snippets on different things: https://www.youtube.com/c/TerryHenry/videos

RealJamesDean · Apr 20, 2021

Archos said:
There is some good stuff here: NetAdmin.us Website

The ICX Configuration Tool (GUI) can generate L2 switch config based on your input.

Terry Henry also has alot of good snippets on different things: https://www.youtube.com/c/TerryHenry/videos

The site looks mostly like what I was looking for. Unfortunately I'm not a big Windows user, so the Config tool isn't much use to me. (I don't even maintain a Windows VM.) Videos are not my favorite medium to learn from, but I'll give those a look too. Thank You!

eduncan911 · Apr 21, 2021

seatrope said:
Question for all the networking experts like @kapone : any reason why i couldn’t use pfsense as the dedicated firewall with pihole on a rpi4 doing DNS and DHCP for several VLANs, using the 6610 to do all the layer 3 routing? My understanding is that pihole 5 does serve DHCP to non-connected subnets.

does anyone have direct experience with this particular setup? I’m a VLAN noob here and a bit hesitant to take the leap,

There's a few issues in this design. I keep setting up and tearing down different VLAN concepts myself, using my HomeLAN as my HomeLAB[tm].

First of all, DHCP requests are generally kept on the same subnet. That is, whatever device you use as your gateway (either InterVLAN Virtual IP within the switch, or the Router-on-a-Stick concept of just trunking all VLANs to the pfSense device) will need to be able to listen for DHCP requests (e.g. the pfSense box with multiple VLANs) or forward the DHCP requests (within the switch). I haven't done the forwarding yet, but that's my next experiment to keep DHCP on the router and do intervlan setups within the switch.

That was a lot to unpack. But, it's a lot to search and learn on first as it will save you a LOT of headaches (DHCP, VLAN, InterVLAN, DHCP Forwarding, etc). I think the Brocade commands are (don't quote me on this, as I haven't used it yet):

Code:

interface ve 21
ip address 10.11.0.1 255.255.255.0
ip helper-address 1 10.3.0.125
ip helper-address 2 10.1.0.125

The advantage I see here is that you could potentially setup two DHCP servers (on two different sub-nets) and have redundancy.

With that said... If you want to use the ASIC routing within the switches (high-performance InterVLAN hardware-routing within the switches), then you'll want to document a set of VLANs up front to configure within the switch, along with the VIP within the switch to handle all internal VLAN routing. Then it's a matter of just forwarding requests/broadcasts to your upstream devices (DNS, DHCP, Chromecast/Bonjour, etc).

I'm currently trunking all VLAN traffic to my pfSense VM, as I run it on my Proxmox cluster. It has dual 10G fiber in LAG, so I'm not worried about bandwidth for my home across the VLANs. I did this as it was the fastest to get setup and going, and I really like to monitor the VLAN traffic - especially across the air-gapped IoT and Management VLANs. InterVLAN is my next experiment, whenever I have time.

eduncan911 · Apr 21, 2021

Duck-Typing...

I'm having trouble affording a second ICX7250-48P to stack on fleebay (I need more 10G ports, and for redundancy/LAG/failover experiments, especially with Proxmox HA). Everyone wants at least $310+ now (with shipping factored in) with the lowest counter-offers I've had. There's been a few auctions, and they went for about $290 with shipping. That's just out my laid-off budget.

I've been considering switching over to a couple of 6610-24s to stack. A lot cheaper, and I could sell my one ICX7250-48P to cover the costs.

However, my concerns are the costs of the breakout cables/components as well as the idle/power consumption (not because of power cost, but in terms of heat generation of long-time use in a small tight closet with just a little ventilation/air-control). Noise level is a little issue, but since it's going into an enclosed rack eventually I think I'll be ok setting them up and getting rid of them shortly after SSH is enabled.

Seems the most inexpensive option I've found is:
* Brocade 57-1000267-01 (64G Fiber-to-QSFP) $10 (thanks @fohdeesha )
* MPO/MTP to 8x LC breakout cable, like this for a whooping $170 each.

So those breakout cables exceed the cost savings... Not even sure if they would work either.

There are much cheaper MPO-to-LCs from China, but I don't want to take a chance (nor wait).

Looking for the most inexpensive option for 10G LC fiber connections to breakout of the QSFP ports. I mean, I could get away with just the 8x 10G ports on the front of the 6610-24s for now (for 16x ports). But I'd be maxed out.

itronin · Apr 21, 2021

eduncan911 said:
That was a lot to unpack. But, it's a lot to search and learn on first as it will save you a LOT of headaches (DHCP, VLAN, InterVLAN, DHCP Forwarding, etc). I think the Brocade commands are (don't quote me on this, as I haven't used it yet):

Code:

interface ve 21 ip address 10.11.0.1 255.255.255.0 ip helper-address 1 10.3.0.125 ip helper-address 2 10.1.0.125

Yes.

The advantage I see here is that you could potentially setup two DHCP servers (on two different sub-nets) and have redundancy.

yeppir.

What I'm about to say should be obvious but gonna say it anyway as sometimes folks forget (including me with blind cut/n/paste across servers

):

If you are using two ip helpers for the same subnet please use non-overlapping IP address ranges in the defined scopes for each subet (vlan) on each server.

itronin · Apr 21, 2021

eduncan911 said:
Looking for the most inexpensive option for 10G LC fiber connections to breakout of the QSFP ports.

Any reason you can't DAC or AOC? 3.28feet is pretty short well within tolerance. IIRC DAC is lowest power/heat, AOC is next or equiv to Xver and cables, then 10G-base-t which you aren't interested in.

eduncan911 · Apr 21, 2021

itronin said:
Any reason you can't DAC or AOC? 3.28feet is pretty short well within tolerance. IIRC DAC is lowest power/heat, AOC is next or equiv to Xver and cables, then 10G-base-t which you aren't interested in.

Not really, no. However, the Dell ones linked to on the front page are still $95 each.

It's just the latency factor and remaining 100% fiber everywhere - for the hell of it. I mean, I ran a 75ft fiber just to connect my little-used desktop on the other side of the house - as I don't have any Cat6a cables, ends, etc. Everything is fiber through the switches and DMZ, all the way to the pfsense router via IOMMU passthrough - and then goes to RJ45 to the ISP (ugh). Pinging in gaming servers are still quiet low!

But you are right. Those DACs are a bit cheaper at about 1/2 the cost. And again, there are $35 ones in China... But then again, it's China.

eduncan911 · Apr 21, 2021

itronin said:
Any reason you can't DAC or AOC? 3.28feet is pretty short well within tolerance. IIRC DAC is lowest power/heat, AOC is next or equiv to Xver and cables, then 10G-base-t which you aren't interested in.

Actually, could you link to a post about AOC/Xver cables? I haven't read about this option yet.

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

New Member

New Member

New Member

Well-Known Member

Active Member

Active Member

New Member

New Member

Well-Known Member

Active Member

Active Member

Active Member

Attachments

Member

New Member

The New James Dean

The New James Dean

Well-Known Member

Well-Known Member

The New James Dean

The New James Dean