Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[itronin]

@LodeRunner beat me to the reply. Teach me to not refresh before posting.

I attached the configs for both if there is a more convenient way to post the configs I am happy to do that too, let me know.

Let me know how I can improve my config please and thank you for helping!

A quicky for your configs.

a code block for each config and then wrap a spoiler block around it all.
the spoiler keeps the page size down for those folks uninterested. the code block keeps the formatting reasonable.

me, I'd download your attachments and open each one in a window so I can flip back and forth - so attaching is also good.
Just as easy to cut-n-paste as to download the files though.

Spoiler: myconfigs

config 6610

ver 08.0.30uT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
stack disable
stack mac 748e.f8cf.f71a
!
global-stp
!
!
lag uplink1 dynamic id 1
ports ethernet 1/3/1
primary-port 1/3/1
deploy
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
spanning-tree priority 26000
!
vlan 10 name Management by port
tagged ethe 1/3/1
!
vlan 20 name Servers_Data by port
tagged ethe 1/3/1
!
vlan 30 name IOT by port
tagged ethe 1/3/1
!
vlan 40 name Sec_Cam by port
tagged ethe 1/3/1
!
vlan 50 name Sparklight by port
tagged ethe 1/3/1
untagged ethe 1/1/1
!
vlan 51 name StarLink by port
!
vlan 60 name Additive by port
tagged ethe 1/3/1
!
vlan 777 name Native by port
tagged ethe 1/3/1
!
optical-monitor
aaa authentication web-server default local
aaa authentication login default local
enable telnet authentication
enable super-user-password .....
hostname SW01
ip dhcp-client disable
ip dns server-address 10.10.100.1
ip route 0.0.0.0/0 10.10.100.1
!
username  password .....
!
!
clock summer-time
clock timezone gmt GMT-08
!
!
ntp
disable serve
server 216.239.35.0
server 216.239.35.4
!
!
web-management https
hitless-failover enable

interface ethernet 1/3/1
dual-mode
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
speed-duplex 10G-full
!
interface ethernet 1/3/5
speed-duplex 10G-full
!
interface ethernet 1/3/6
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
speed-duplex 10G-full
!
interface ve 1
ip address 10.100.10.3 255.255.255.0
!

ip ssh  idle-time 240

config 7250

ver 08.0.95hT213
!
stack unit 1
  module 1 icx7250-48p-poe-port-management-module
  module 2 icx7250-sfp-plus-8port-80g-module
stack mac 609c.9f07.21a8
!
!
!
lag uplink1 dynamic id 1
ports ethe 1/2/1
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 10 name Management by port
tagged lag 1
!
vlan 20 name Servers_Data by port
tagged lag 1
!
vlan 30 name IOT by port
tagged lag 1
!
vlan 40 name Sec_Cam by port
tagged lag 1
!
vlan 50 name Sparklight by port
tagged lag 1
untagged ethe 1/1/1
!
vlan 51 name Starlink by port
!
vlan 60 name Additive by port
tagged lag 1
!
vlan 777 name Native by port
tagged lag 1
!
optical-monitor
optical-monitor non-ruckus-optic-enable
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable telnet authentication
hostname SW00
ip dhcp-client disable
ip dns server-address 10.10.100.1
ip route 0.0.0.0/0 10.10.100.1
!
username  password .....
!
snmp-server community ..... ro
!
clock summer-time
clock timezone gmt GMT-08
!
ntp
disable serve
server 216.239.35.0
server 216.239.35.4
!
hitless-failover enable
!
manager disable
!
manager port-list 987
!
interface ve 1
ip address 10.100.10.2 255.255.255.0
!
no lldp run
!
ip ssh  idle-time 240

 

[kpfleming]

I attached the configs for both if there is a more convenient way to post the configs I am happy to do that too, let me know.

For what it's worth the 7250 config looks correct based on what you have described as your desired configuration. I can't comment on the 6610 config as I've never used 08.0.30 or anything older and the configuration structure is quite different.

 

[tillburn]

For what it's worth the 7250 config looks correct based on what you have described as your desired configuration. I can't comment on the 6610 config as I've never used 08.0.30 or anything older and the configuration structure is quite different.

I did try and set up the 6610 like I did the 7250, but lag 1 was not an option on the 6610. Maybe someone else has some input and knowledge they want to drop

 

[kpfleming]

According to the docs for 08.0.30 you've done it correctly; the lowest-numbered port in the LAG (also called a 'trunk group') is the 'lead port' and any VLAN configuration applied to that port applies to all ports in the LAG.

 

[kpfleming]

According to the docs for 08.0.30 you've done it correctly; the lowest-numbered port in the LAG (also called a 'trunk group') is the 'lead port' and any VLAN configuration applied to that port applies to all ports in the LAG.

So based on the configs you've posted, you should have a working configuration. All of the VLANs extend across the switches, and you've got an untagged port on VLAN 50 on both switches, so devices on those ports should be able to talk to each other.

What problems are you experiencing now?

 

[tillburn]

So based on the configs you've posted, you should have a working configuration. All of the VLANs extend across the switches, and you've got an untagged port on VLAN 50 on both switches, so devices on those ports should be able to talk to each other.

What problems are you experiencing now?

None at all thank you for your help!

I do need to plan out the rest of the network conversion phase, mostly around taking the network more vertical with the vlans, using the proxmox to host the vm's/services and using pfsense or something else to handle all the intervlan routing. I think I can piece those things together and make an adaptive plan.

But if you had any advice for network layout given the unique situation I am all ears! Again I appreciate the help I have received!

 

[itronin]

Let me know how I can improve my config please and thank you for helping!

Before you get too deep with configuration tweaking and now that you have it working.

My suggestion is to rework things now and avoid using the default vlan (1) in your configurations.

While in a home lab you are quite unlikely to get compromised by using it - it is a good practice to have and IMO it is good to understand why you should not use it.

 

[selta]

Having a good deal of trouble with the 40GbE ports on an ICX-6610, hoping for some help/sanity checks here. Server is a Dell R640 (can post full specs if needed).

First, sh flash and lic:

Spoiler: flash, lic, run

SSH@RackSwitch(config-if-e40000-1/2/6)#sh flash
Stack unit 1:
Compressed Pri Code size = 10545591, Version:08.0.30uT7f3 (ICX6610-FCX/FCXR08030u.bin)
Compressed Sec Code size = 7762230, Version:08.0.30nT7f1 (FCXS08030n.bin)
Compressed Boot-Monitor Image size = 370695, Version:10.1.00T7f5
Code Flash Free Space = 46399488
SSH@RackSwitch(config-if-e40000-1/2/6)#sh lic
Index Lic Mode Lic Name Lid/Serial No Lic Type Status Lic Period Lic Capacity
Stack unit 1:
1 Node Lock ICX6610-PREM-LIC-SW redacted Normal Invalid Unlimited 1
2 Node Lock ICX6610-10G-LIC-POD H4CKTH3PLN8 Normal Active Unlimited 8
3 Node Lock ICX-MACSEC-LIC H4CKTH3PLN8 Normal Active Unlimited 1
4 Node Lock ICX6610-ADV-LIC-SW H4CKTH3PLN8 Normal Active Unlimited 1
SSH@RackSwitch(config-if-e40000-1/2/6)#sh run
Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
no legacy-inline-power
stack disable
!
!

Next, here's sh int output on the ICX:

Spoiler: int et 1/2/6 config

SSH@RackSwitch(config-if-e40000-1/2/6)#sh int e 1/2/6
40GigabitEthernet1/2/6 is up, line protocol is up
Port up for 4 minute(s) 30 second(s)
Hardware is 40GigabitEthernet, address is 748e.f8dd.c154 (bia 748e.f8dd.c18a)
Interface type is 40Gig Fiber
Configured speed 40Gbit, actual 40Gbit, configured duplex fdx, actual fdx
Configured mdi mode AUTO, actual none
Member of L2 VLAN ID 2, port is untagged, port state is FORWARDING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Openflow is Disabled, Openflow Hybrid mode is Disabled, Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
Port name is Rear 40G
MTU 1500 bytes, encapsulation ethernet
300 second input rate: 80 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 8536 bits/sec, 11 packets/sec, 0.00% utilization
10746 packets input, 11471566 bytes, 0 no buffer
Received 159 broadcasts, 425 multicasts, 10162 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
461499 packets output, 35119251 bytes, 0 underruns
Transmitted 16789 broadcasts, 52196 multicasts, 392514 unicasts
0 output errors, 0 collisions

And finally, from the server (R640 w/ Intel XL710-Q2, Ubuntu 22.04.1):

Spoiler: Server OS info.

selta@server2:~$ uname -r
5.15.0-58-generic

selta@server2:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy

3b:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
Subsystem: Intel Corporation Ethernet Converged Network Adapter XL710-Q2
Kernel driver in use: i40e
Kernel modules: i40e

selta@server2:~$ ifconfig enp59s0f1
enp59s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.3 netmask 255.255.252.0 broadcast 192.168.3.255
inet6 fe80::3efd:feff:fed0:2841 prefixlen 64 scopeid 0x20<link>
ether 3c:fd:fe:d0:28:41 txqueuelen 1000 (Ethernet)
RX packets 396762 bytes 26947129 (26.9 MB)
RX errors 25681 dropped 1689 overruns 0 frame 25680
TX packets 9977 bytes 11336141 (11.3 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

selta@server2:~$ ping -I enp59s0f1 4.2.2.1
PING 4.2.2.1 (4.2.2.1) from 192.168.1.3 enp59s0f1: 56(84) bytes of data.
64 bytes from 4.2.2.1: icmp_seq=1 ttl=57 time=29.3 ms
64 bytes from 4.2.2.1: icmp_seq=2 ttl=57 time=36.9 ms
64 bytes from 4.2.2.1: icmp_seq=3 ttl=57 time=29.8 ms

selta@server2:~$ sudo ethtool enp59s0f1
Settings for enp59s0f1:
Supported ports: [ FIBRE ]
Supported link modes: 40000baseCR4/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 40000baseCR4/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 40000Mb/s
Duplex: Full
Auto-negotiation: off
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x0000000f (15)
drv probe link timer
Link detected: yes

So my issue is that... this thing is just barely operational. As you can see, I can ping out from it even to a WAN address, so basic connectivity is there. However,ssh and any other type of connectivity beyond a basic ping just do not work (as in, no connection). ssh connections simply time out, nslookup fails due to timeout, apt update works but shows a throughput of a whopping 2KB/s.

To me, everything looks OK in dmesg:

Spoiler: dmesg

[ 9.743896] i40e 0000:3b:00.1 eth0: NIC Link is Up, 40 Gbps Full Duplex, Flow Control: RX/TX
[ 10.074942] i40e 0000:3b:00.1: PCI-Express: Speed 8.0GT/s Width x8
[ 10.086993] i40e 0000:3b:00.1: Features: PF-id[1] VFs: 64 VSIs: 66 QP: 56 RSS FD_ATR FD_SB NTUPLE CloudF DCB VxLAN Geneve NVGRE PTP VEPA
[ 10.090389] i40e 0000:3b:00.0 enp59s0f0: renamed from eth1
[ 10.139769] i40e 0000:3b:00.1 enp59s0f1: renamed from eth0
[ 2132.001293] i40e 0000:3b:00.1 enp59s0f1: NIC Link is Down
[ 2132.815259] i40e 0000:3b:00.1 enp59s0f1: NIC Link is Up, 40 Gbps Full Duplex, Flow Control: RX/TX

And here's an iperf3 test to another server on the same subnet and VLAN, connected to the same switch as another example of the issue:

Spoiler: iperf3

└─(21:24:10)──> iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.3, port 56676
[ 5] local 192.168.1.0 port 5201 connected to 192.168.1.3 port 56692
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 7.48 MBytes 62.7 Mbits/sec
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -

I tried all of the same tests on 1/2/1, the other "stacking" 40GbE port with identical results. This is with a brand new QSFP cable (Arista brand, if that matters).
I tried with the i40e and the ixl drivers.

Kind of at a loss as to what else to check. I could buy another brand new QSFP if that's the suspect, but, I'd expect different issues if it were (CRC errors at least). Hopefully someone here can help - let me know if I missed providing any needed information.

 

[selta]

Of course, after I post all of that, I see all the RX errors... gonna try a new cable.

 

[itronin]

Of course, after I post all of that, I see all the RX errors... gonna try a new cable.

see this post (and previous posts by OP). TLDR: While the 710 did not work, a low cost (used) mellanox CX3 did. Testing done using DAC, maybe AOC will work as it was not tested.

 

[selta]

Interesting. I see at least 2-3 people here have the XL710 and the same issue I am having. I'm not going to sink more money into 40GbE here though -- my R640 came with the XL710, and was the only reason I was going to try to use it. I'll just use 10GbE and call it good.

 

[tillburn]

So based on the configs you've posted, you should have a working configuration. All of the VLANs extend across the switches, and you've got an untagged port on VLAN 50 on both switches, so devices on those ports should be able to talk to each other.

What problems are you experiencing now?

I am reading up on spanning tree and rapid spanning tree, is there any primer you recommend? I feel like I am having some congestion on the network, specifically on the vlan carrying the cable modem as after about 6 hours the download speeds seems to be very slow, but if I reset both routers and the wireless AP the speeds for download are then just fine.

Physically it's laid out like this:

7250 port 1/1/1 vlan 50 ----->cable modem
7250 port 1/1/2 ------> unmanaged switch -----> 3x PC
7250 port 1/1/48 POE ----->wap (Vlans unimplemented yet).
7250 port 1/2/7 ----->10g Computer
7250 port 1/2/1 ----->10g LAG 1 all vlans -----> 6610

6610 port 1/1/1 vlan 50 ----->Pfsense VM
6610 port 1/1/2 Hypervisor 00
6610 port 1/1/3 Hypervisor 01
6610 port 1/1/48 POE ----> wap (Vlans unimplemented yet).
6610 port 1/3/1 -----> 10g LAG 1 all vlans------> 7250
6610 port 1/3/7 -----> 10g VM NAS
6610 port 1/3/8 -----> 10g VM Pfsense LAN

Any advice on spanning tree/rstp or if there potentially something else plaguing the network that I should be aware of please let me know.

 

[Coach43]

Not a network guy, but LOVE computers and what they can do!
Thank you Fohdeesha - I have TWO ICX6610's fully licensed and trunked via a 40Gb fiber optic link!
Instructions worked like a charm, but did not instill all layer 2 and layer 3 knowledge in me (I've been reading a lot of this 408 page forum etc, but still...)

The remote unit is a 6610P and I am now setting up security cameras. I'd like to isolate them onto their own vlan so they can't phone home to china or get accessed by the internet, but so they can still connect to my other internal systems. My firewall and DHCP are hosted upstream from the routers and that all works fine. I can easily add this new network there, I just need to make sure my 6610's know what's going on.

I think it's a few lines for those of you that know what you are doing.

My default network 192.168.1.0 and I'd like the new one to be 192.168.10.0.
I'm running router code.

Thanks in advance!


P.S. I'm interested in a 'Router Programming in a Nutshell' reference. Open to recommendations.

 

[Vilmalith]

Would the icx6450 poe be a noticable upgrade from some Netgear multigig switches, even with some stuff like aps running multigig?

 

[msg7086]

I wonder if there are other branded switches that are as affordable, power efficient, noise friendly, multi functional as those?

Looking for a new switch that's going to the furnace/tech room with 5 PoE and 8 non-PoE cat6 in a new construction home. My ICX6450-48P sounds a bit too hot, noisy, and overkill in that scenario, but I couldn't find much cheap alternative that can do some PoE, cool and quiet, and with a 10G SFP port. Considering mikrotik 8P-2S+ but it's $200+. Maybe I should just throw the 6450 in and call it a day.

Sorry for the derailment.

 

[kpfleming]

I am reading up on spanning tree and rapid spanning tree, is there any primer you recommend? I feel like I am having some congestion on the network, specifically on the vlan carrying the cable modem as after about 6 hours the download speeds seems to be very slow, but if I reset both routers and the wireless AP the speeds for download are then just fine.

Your connection configuration looks fine. You've got a 10G link between the ICX devices which should be plenty, and STP/RSTP should not be a concern if there aren't any switching loops. Since you only have two switches it's unlikely you have any loops.

You shouldn't need two physical links from the 6610 to the pfSense VM; one should be plenty, with two VLANs (one tagged and one untagged, until you setup the other VLANs) on it. Still, it's not a problem to have two links.

When you say 'reset both routers' are you referring to the ICX boxes?

 

[Mark]

Picked up an ICX-7650-48ZP on ebay and it has not arrived yet. Wondering if anyone can confirm actually how the breakouts work or don't? The documentation says the backports can be used for uplink but then that will not allow the front 4x10gb ports to work and there is no indication about the ability to actually breakout the back ports.

 

[tillburn]

Your connection configuration looks fine. You've got a 10G link between the ICX devices which should be plenty, and STP/RSTP should not be a concern if there aren't any switching loops. Since you only have two switches it's unlikely you have any loops.

You shouldn't need two physical links from the 6610 to the pfSense VM; one should be plenty, with two VLANs (one tagged and one untagged, until you setup the other VLANs) on it. Still, it's not a problem to have two links.

When you say 'reset both routers' are you referring to the ICX boxes?

Yes that was a typo, I did mean to say both switches (ICX boxes).

 

[TonyArrr]

Hi all, this thread is amazing, so much knowledge.
At 400 pages, I kind of wonder if it should be a subsection all of its own

I’ve been looking for a switch for home in advance of wiring up the apartment, and am very much sold on a 24 port 7250, or 7450 if I could find one functioning cheap enough.

I’ve read a bunch about fan mods, exploring moving enough air quietly enough, adding fans to the ASIC etc, and I’m wondering if anyone has explored installing larger heatsinks (or smaller ones with integrated fans), or more conductive thermal interface material, as a way to make it easier for heat to move out of the PCB and into the path of the airflow?

I haven’t seen anything going that way, aside from the fan mod attaching a fan on the existing ASIC heatsinks, or installing fans on the top casing blowing into the heatsinks.

Or are the heatsinks not removable? Obviously not gonna get better thermal conductivity than if they’re soldered to the ICs, but I haven’t found any info either way

 

[fleeball]

Hi All

Have had the ICX6610-48P running fine for ages. Decided to upgrade to the latest firmware(8030u) and bootloader as I was having problems with PoE. Firmware upgrade went all ok, but now I can not log into the web gui. I can get to the web gui via an ip address but it does not accept the username and password that was setup before the firmware upgrade. Previous username was root and password root also. I don't have a console cable lying around at the moment and need to enable PoE. Does anyone know what the default username and password are for the latest firmware? Or have any ideas as to how I can enable PoE without a console cable( I know close to impossible).