Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

mrzarquon

New Member
Aug 24, 2021
1
2
3
Been lurking a while, wanted to thank @fohdeesha for this thread (and docs page) as it introduced me to Brocade hardware and let me dump my Unifi stuff entirely. I picked up two ICX-7150 12's and a 7150-24P for my setup.

I wanted to share some of my findings:

7150-24p Fan Mod:
It appears to be very happy with the fans replaced with Noctua NF-A4x20 PWM (while I could run it in fanless mode, I leave the fans on and still keep my POE draw below the fanless threshold). These are 4 wire fans that have board connectors that just work. I doubt they'll work in the 7250's, but if you end up with a 7150-24p that you want to keep quiet but want more airflow than fanless, these seem to be a good solution (they don't stall at low voltage either).

Multi Subnet/Domain DHCP (since that appears to be the next step after "i got vlan routing and a transit network working"):
For anyone doing multiple vlans/subnets who want dhcp dns working with different domains (ie, subnet A with *.mgmt.foo.lan, subnet B with *.bfs.foo.lan) but don't want to go so far as to roll your own, you can resort to using Pi-Hole. It is dnsmasq with some special settings and helpers, and most importantly, it still supports additional conf-dir directives. This lets you drop in additional settings ontop of the existing pi-hole tunings, as files in a configuration directory. When you do this it is best to also use the command sudo pihole-FTL dnsmasq-test to ensure you don't have conflicting entries / didn't mess up formatting - it attempts to load all your configuration fragments and will report errors if it finds any - saves you the risk of reloading the service and it crashing at start.

In my setup that looks like:
- PiHole (at 10.10.10.7) installed and running on the mgmt.foo.lan / 10.10.10.0/24 / vlan 10, with DHCP enabled (nothing fancy here), ICX providing routing at 10.10.10.1
- ICX doing DHCP relay for bfs.foo.lan / 10.10.20.0/24 / vlan 20, sending to 10.10.10.7:

Code:
interface ve 20
 ip address 10.10.20.1 255.255.255.0
 ip helper-address 1 10.10.10.7
- I've added this file to my Pi-Hole: /etc/dnsmasq.d/09-git-configs.conf
Code:
# this is a folder checked out of git for version control
# the .../conf,*.conf means it will only load files that end in .conf
# yes this is technically a security issue that i have my dns/dhcp
# configuration pointed to load files from a folder owned by a 
# generic user
# this also means if pi-hole *does* blow away /etc/dnsmasq.q/*
# restoring configurations after an update is just adding this file

conf-dir=/home/pi/dnsmasq-options/conf,*.conf
This lets me have a per subnet / per file configuration:
So this is a heavily commented file I use (for my future self) dnsmasq-options/conf/01-bfs-dhcp.conf
Code:
# I want my local static dns entries to be in git also
addn-hosts=/home/pi/dnsmasq-options/hosts/

# I want to configure DHCP settings for a specific subnet / range
# I can set a tag once (set:bfs) and then add settings to it by using
# it in the next line with tag:bfs, this works for all dhcp-* settings
# https://thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html

dhcp-range=set:bfs,10.10.20.100,10.10.20.199,255.255.255.0,12h
dhcp-option=tag:bfs,option:router,10.10.20.1

# this tells dnsmasq that it will be getting DHCP relay's from 10.10.10.1
# for the 10.10.20.0/24 subnet
shared-network=10.10.10.1,10.10.20.0

# this is the important bit for dual subnet, pi-hole will just set mgm.foo.lan
# but because this is more specific (with 10.10.20.0/24) it will be allowed
# letting us assign dns names via dhcp hosts for the 10.10.20.0 subnet
domain=bfs.foo.lan,10.10.20.0/24
One recent TIL: you can add static dhcp entries outside of your DHCP reservation pool. Something I never thought much about in the last 20 years since most interfaces to create a static entry grab the devices existing reservation to work with (which is from the pool). I've used that to throw all my dumb IOT things into the 200-249 range (force of habit has anything with a manually assigned IP < .100).
 

RoachedCoach

Member
Feb 4, 2020
47
54
18
Hi all, I'm new to STH Forums but have been lurking this thread for quite some time! I picked up a ICX 7250-24 recently, and am looking forward to using it in my homelab!

Like so many homelabbers, I find the stock Foxconn PIA040H12P fan to be much too loud for where my lab is located. So I've been on a venture to find a fan that keeps the 7250 cool while also being nice and quiet.

@rootwyrm I tried your suggestion of the Delta EFB0412VHD-F00 but the temp just keeps creeping up. After an hour of running idle, the switch is at 93 deg C and continues to climb. This seems a little toasty to me since there's only about 10 deg C headroom until thermal shutdown. I also want to avoid oscillating between fan speed 1 and 2.

I took a look and found a few other fans that seem to move a bit more air, and was wondering what your thoughts might be on them:

Mfr/model​
Size​
Air flow​
Static Pressure​
Noise​
Fan speed​
Price​
Delta FFB0412VHN-F0040mm x 28mm D15.8 cfm0.509 in H2O45 dBA9500 rpm$14.31
Sanyo Denki 109P0412G301340mm x 28mm D14.8 cfm0.719 in H2O42 dBA11500 rpm$10.51
Sanyo Denki 109P0412B301340mm x 28mm D13.4 cfm0.574 in H2O40 dBA10300 rpm$10.24
[COMPARE] Delta EFB0412VHD-F0040mm x 20mm D10.1 cfm0.416 in H2O32.5 dBA9000 rpm$12.57

It seems like the 109P0412G3013 might be the most promising, as it has the highest static pressure of the bunch and moves a bit more air per minute, but I am curious to hear what you think.

Thanks so much!
This is what I did. The secret sauce is slapping a fan on the ASIC and changing out the fans.

 

Attachments

nlj

Member
Sep 8, 2021
8
27
13
Thank you - I read about your solution and it seems novel, but I would like to stick with soft-modding the switch if at all possible. I don’t mind swapping out a fan but I would like to avoid screwing a fan into the ASIC heatsink if I can!

I went ahead and ordered the 3 fans I mentioned in my post - I will give each a try and update this thread with my findings!
 
  • Like
Reactions: dswartz

deeceesth

New Member
Jul 30, 2021
17
4
3
I just got a second ICX7250 and it came with an old bootloader (10.1.06T215) and was running 8030t. I can't get it to boot anything newer than 8030. I tried stepping through each version back from 8090 and it just spits out "wrong image format for bootm command" after flashing and rebooting.

Is there a separate bootloader update process? the latest guide update says 8090 should be updateable from any version but I can't get anything to boot.
 

up3up4

Member
Jun 10, 2018
89
33
18
I just got a second ICX7250 and it came with an old bootloader (10.1.06T215) and was running 8030t. I can't get it to boot anything newer than 8030. I tried stepping through each version back from 8090 and it just spits out "wrong image format for bootm command" after flashing and rebooting.

Is there a separate bootloader update process? the latest guide update says 8090 should be updateable from any version but I can't get anything to boot.
#by usb
device# copy disk0 flash spz10114.bin bootrom

#by tftp
device# copy tftp flash 10.176.220.51 spz10114.bin bootrom
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,914
3,440
113
34
fohdeesha.com
I just got a second ICX7250 and it came with an old bootloader (10.1.06T215) and was running 8030t. I can't get it to boot anything newer than 8030. I tried stepping through each version back from 8090 and it just spits out "wrong image format for bootm command" after flashing and rebooting.

Is there a separate bootloader update process? the latest guide update says 8090 should be updateable from any version but I can't get anything to boot.
yeah sorry about that, another user PMd me with the same issue, looks like I didn't test old enough bootloaders. but yeah, turns out 8030 and earlier bootloaders can't flash the 8090m, the easy fix is just flash a newer bootloader first, you can do it from the existing bootloader. Will be adding it to the guide soon but if you find yourself with that error, in the bootloader just:

Code:
setenv ipaddr 192.168.1.50
setenv netmask 255.255.255.0

#tell the switch the IP of your tftp server:
setenv serverip 192.168.1.8
setenv uboot ICX7xxx/spz10118.bin
update_uboot
reset
#smash b to get back into new bootloader
then you can follow the guide
 
Last edited:
  • Like
Reactions: deeceesth

Nnyan

Active Member
Mar 5, 2012
178
71
28
Sacramento CA
I'm sure you've figured out I've spent (and am still spending) a lot of time with the IOS CLI. ;)
I don't mind going to the CLI when I need to but junos just feels like unneeded pain. I'm by far not an expert but I've always managed to get a switch up and running pretty quickly, the EX3300? not so much.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,914
3,440
113
34
fohdeesha.com
I don't mind going to the CLI when I need to but junos just feels like unneeded pain. I'm by far not an expert but I've always managed to get a switch up and running pretty quickly, the EX3300? not so much.
JunOS is...verbose. Which is amazing in a production environment where I need to create nested lists that feed into BGP filters etc, but at home, yeah you're probably gunna feel like you're wasting time if you don't already have a bunch of junOS experience. Wanna trade for an ICX6450? lmao
 

klui

༺༻
Feb 3, 2019
977
571
93
I very much like commit confirmed exists in JunOS even though I typically don't use it. I normally use commit check. I've been bit before where issuing a change in conf t immediately borked my config forcing me to use the console or power cycle.

Aside from that, if one doesn't use it regularly it's difficult to remember the higher level node names and once you do see the config scrolling up is necessary to see them--a pain in the ass. For that show config|display set displays the full command in a non-structured format.
 
  • Like
Reactions: fohdeesha

jahsoul

Active Member
Dec 13, 2013
262
34
28
War Eagle Country
JunOS is...verbose. Which is amazing in a production environment where I need to create nested lists that feed into BGP filters etc, but at home, yeah you're probably gunna feel like you're wasting time if you don't already have a bunch of junOS experience. Wanna trade for an ICX6450? lmao
I swear you just described my experience with my EX2300. lol. The initial configuration was fine but dang, creating Firewall Filters are like writing a novel. I can create Cisco ACLs in my sleep but these firewall filters have me stuck.
 
  • Like
Reactions: fohdeesha

aaroneaton

New Member
Jan 15, 2021
12
0
1
www.rfehosting.com
Ok ive got a weird issue going on.
So i had issues getting breakout working, so ended up getting a new 6610. I was able to get the breakouts working on the new one.
I then setup stacking between the new and old. And am now not able to get breakout working on the new one. And it only works on the old one.

I used the directions above. So its like the breakout port fixed itself on the old and broke on the new. So it has to do with stacking somehow.

I have 2 diff breakout cables. one is FS, and one is Brocade from FS.
I have the exact same config/setup as you used here.

Any other thoughts on how to make breakout work on unit 1? It was working before stacking.
ID Type Role Mac Address Pri State Comment
1 S ICX6610-48P active 748e.f8fb.b8b6 128 local Ready
2 S ICX6610-48P standby cc4e.2413.2d44 0 remote Ready

active standby
+---+ +---+
-2/6| 1 |2/1--2/1| 2 |2/6-
| +---+ +---+ |
| |
|------------------------|
Standby u2 - protocols ready, can failover

stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
priority 128
stack-port 1/2/1 1/2/6
stack unit 2
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack-port 2/2/1 2/2/6
stack enable


Unit 1 is the new one, unit 2 is the old. Breakout never worked on the old one pre stacking. It worked fine on the new one prestacking. Now it is reversed, and does not work on the new one, but works on the old.

Thoughts?




OK - considering the rear ports are not intended to be used for data ports, you won't find any information on this in the manuals etc. Using them as just data ports is easy enough, but splitting them up and using some for stacking and some for data ports requires some finangling to make the stacking code not try and take over all 4 ports.

However it's possible and stable across reboots etc. basically follow this, it might take you a couple times. This will enable stacking between two units just using the 40gbE qsfp ports, leaving 4x breakout QSFPs total available for data use


Code:
#unplug all stacking cables except for 1, going between port 1/2/1 on unit 1 and 1/2/1 on unit two
#don't plug anything else into unit 2 or you'll get a loop
#first you have to unstack all the units and remove any existing stack configuration:
enable
stack unconfigure all
write mem
reload
#both switches should reload, wait till they come back up

#when they're back up, on unit 1:
enable
conf t
stack unit 1
no stack-trunk 1/2/1 to 1/2/2
stack-port 1/2/1 1/2/6
#ignore the warning that pops up about not showing up in "show run"

#still running the below on unit 1!
stack unit 2
no stack-trunk 2/2/1 to 2/2/2
stack-port 2/2/1 2/2/6
#ignore the warning that pops up about not showing up in "show run"
exit
stack enable
write mem
exit
stack secure-setup

#When the setup is done, do "show run"
#stack unit 2 section probably has "stack-trunk 2/2/1 to 2/2/2" added back to it

#the second switch is currently rebooting to rejoin the stack, wait for it to do so
#once it comes back online, do the following to remove the stack unit 2 trunk:
#still on unit 1:
conf t
stack unit 2
no stack-trunk 2/2/1 to 2/2/2
#will probably reboot unit 2 again
#wait for it to come back online, then connect the second pair of 40gbE ports
Once you connect the second pair of 40gbe ports (1/2/6 on unit 1 to 1/2/6 on unit 2) you should be done. Check by running "show stack", you should see both units "ready" with the connection diagram showing both 40gbE ports linked up like below:

Code:
ICX1#sh stack
T=6m28.6: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-48P   active  cc4e.24b8.d9d0 128 local   Ready
2  S ICX6610-48P   standby cc4e.243e.aa74   0 remote  Ready

    active       standby
     +---+        +---+
-2/6| 1 |2/1--2/1| 2 |2/6-
|   +---+        +---+   |
|                        |
|------------------------|
Standby u2 - protocols ready, can failover
Current stack management MAC is cc4e.24b8.d9d0
If you have the same output, do a "write mem" then a "reload" to reload the whole stack. It should come back up fresh in the correct state and running "show stack" again should have the same output as before.

Your final stacking config should look like this:

Code:
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  priority 128
  stack-port 1/2/1 1/2/6
stack unit 2
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-port 2/2/1 2/2/6
stack enable
 

ZFSZealot

New Member
Aug 16, 2021
29
7
3
Having trouble using the breakout stack ports on a 6610 to a server. Stacking is not enabled. Using a decent quality breakout DAC cable.

I'm sure I'm missing something stupid.

I configured a static LAG to go to an ESXi server using 1/2/9 and 1/2/10, primary port 1/2/9. Deployed. Links were up, lights on the back of the server were lit. Then I added 1/2/9 as tagged to one of the vlans and the link went down on both ports. I have not been able to get them to come back up. I know the cable/nic/switchports are physically fine because I saw link before. The link dropped the instant that I did the "tagged ethe 1/2/9" in the vlan config. I saw it out of the corner of my eye because I was tailing the vmkernel.log on the server. I have since completely removed the ports from all vlans, the lag itself, added the two ports to my main vlan (happens to be 101) and rebooted the ESXi. There is no chance of a loop as the NICs are "unused adapters" on the esxi side. STP appears to be enabled on the port but not on any of my vlans.

I cannot for the life of me get these links to come back up. I have not reloaded the switch because it is passing other traffic right now but I can try that if necessary. It doesn't seem like this is the sort of thing that should require reboot of the switch. What stupid thing am I missing?

Code:
  10GigabitEthernet 1/2/9 is down, line protocol is down      
  Port down for 11 minute(s) 48 second(s)                       
  Hardware is   10GigabitEthernet , address is 748e.f8e9.a6d0 (bia 748e.f8e9.a709)
  Configured speed 10Gbit, actual unknown, configured duplex fdx, actual unknown
  Configured mdi mode AUTO, actual unknown                     
  Member of L2 VLAN ID 101, port is untagged, port state is BLOCKING
    BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled                             
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled                             
  Mac-notification is disabled                                 
  Not member of any active trunks                               
  Not member of any configured trunks                           
  Port name is Trunk vSphere Umbra 10G                         
  MTU 1500 bytes, encapsulation ethernet                       
  300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  0 packets input, 0 bytes, 0 no buffer                         
  Received 0 broadcasts, 0 multicasts, 0 unicasts               
  0 input errors, 0 CRC, 0 frame, 0 ignored                     
  0 runts, 0 giants                                             
  0 packets output, 0 bytes, 0 underruns                       
  Transmitted 0 broadcasts, 0 multicasts, 0 unicasts           
  0 output errors, 0 collisions                                 
  Relay Agent Information option: Disabled
 
Last edited:

nickf1227

Active Member
Sep 23, 2015
197
129
43
34
Protip:
If you upgrade/migrate from the "switch mode" firmware to the "router mode" firmware in @fohdeesha 's guides, you will break your management interface and have to console into the switch ;)