Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

LodeRunner

Active Member
Apr 27, 2019
540
227
43
When you remove a LAG on an ICX it disables the ports to prevent loops. You'll have to go to each affected port and issue 'enable'. Note in each interface definition it has 'disable'. A disabled port will not link, hence no lights.
 

tubs-ffm

Active Member
Sep 1, 2013
171
57
28
I exchanged on my 7250-24P the two fans to Delta EFB0412VHD-F00. This was a huge improvement for noise but now I got a temperature issue. I am waiting for the Sunon MF60101V3-1000U-A99 fan I ordered to put on top of the ASIC. Unfortunately, I could not find any other alternative with thin height and have to be patient due to long delivery time.
To follow-up on this topic.

Additionally, to the two Delta EFB0412VHD-F00 in the chassis today I installed the Sunon MF60101V3-1000U-A99 on top of the ASIC. I connected the Sunon in parallel to one of the chassis fans so it also will slow down in fan mode 1. After running a couple of hours in idle mode at room temperature I get these stable temperatures. Look OK to me. Unfortunately, I did not note the temperatures in the original setup with Foxconn fans.

Code:
Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 62.4 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 42.5 deg-C
Just in case someone is asking. I am not planning to use heavy PoE load. Two devices only.
 

dswartz

Active Member
Jul 14, 2011
610
79
28
When you remove a LAG on an ICX it disables the ports to prevent loops. You'll have to go to each affected port and issue 'enable'. Note in each interface definition it has 'disable'. A disabled port will not link, hence no lights.
Yeah, that confused me the first time I had this happen :)
 

plexisaurus

New Member
Jan 14, 2021
6
2
3
To follow-up on this topic.

Additionally, to the two Delta EFB0412VHD-F00 in the chassis today I installed the Sunon MF60101V3-1000U-A99 on top of the ASIC. I connected the Sunon in parallel to one of the chassis fans so it also will slow down in fan mode 1. After running a couple of hours in idle mode at room temperature I get these stable temperatures. Look OK to me. Unfortunately, I did not note the temperatures in the original setup with Foxconn fans.

Code:
Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 62.4 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 42.5 deg-C
Just in case someone is asking. I am not planning to use heavy PoE load. Two devices only.
The Delta I bought from Mouser. The Sunon I ordered from RS.



My plan is first to connect this fan in parallel to the housing fans. So, it will run at fan level 1 anyhow slower and at a different noise level. If this is not enough, I will connect it to permanently 12 V. This is my plan. No idea if it will work. I can report later.

For me the housing fans were expensive as I could not find anything at local shops and I had to order oversee with high shipping costs. But this Sunon I could find locally and it was not more than the value of a beer.

If all will not work, the 7250-24P will go back to eBay where it comes from. Too bad, I really like the spec. But this was part of my plan.
Just installed the MF60101V1-1000U-G99 today in parallel with the 3 sunon housing fans in my 7250-48. ASIC temps are about 2-3c better than stock fans at 52.4c and noise level hasn't increased over the sunon mf40201vx-1000u-g99 40mm fans. Very quiet at level 1.

I also installed a temporary cover made of thin ( mostly transparent to thermal) packing tape to check component/psu temps with my thermal camera. Psu and hotter ICs were at most low 40s.

Overall, very happy with result. All 4 fans were like $28+shipping, but it will save that or more in reduced energy use over life of switch.

Update: switch installed in wiring rack under stairs. ASIC temp has settled at 60c after 24hours under light load
 

Attachments

Last edited:

mrizzo

New Member
Feb 6, 2021
10
1
3
Hi all, I am looking at getting a Brocade ICX6610 for a rack that will have a 10Gb uplink. All incoming traffic is going to be tunneled via GRE from a third party DDoS protection service.

Can the Brocade handle a tunnel (just regular GRE, no encryption) that will have inbound traffic peaking at a few Gbps? I was looking at the Mikrotik CRS354 at first, but from my research none of the tunneling is offloaded from the CPU so the performance is bad.
 

tommybackeast

Active Member
Jun 10, 2018
286
105
43
Brocade firmware Question: 7250 and 7150 currently on 8080 - if i wish to go to 8092; can I upgrade from 8080 to 8092 -or- must I upgrade to 8090 and then 8092 ? thanks
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,075
113
33
fohdeesha.com
Hi all, I am looking at getting a Brocade ICX6610 for a rack that will have a 10Gb uplink. All incoming traffic is going to be tunneled via GRE from a third party DDoS protection service.

Can the Brocade handle a tunnel (just regular GRE, no encryption) that will have inbound traffic peaking at a few Gbps? I was looking at the Mikrotik CRS354 at first, but from my research none of the tunneling is offloaded from the CPU so the performance is bad.
yes, I actually have an icx6610 doing exactly this in NYC (terminating a GRE tunnel from a ddos provider). it's all done in hardware at line rate. note than enabling gre tunnels disables a couple counter features like ipv6 ACLs. full details starting on page 103 of fastiron-08030mb-l3guide.pdf
 

mrizzo

New Member
Feb 6, 2021
10
1
3
yes, I actually have an icx6610 doing exactly this in NYC (terminating a GRE tunnel from a ddos provider). it's all done in hardware at line rate. note than enabling gre tunnels disables a couple counter features like ipv6 ACLs. full details starting on page 103 of fastiron-08030mb-l3guide.pdf
Thank you so much! Glad to hear that I can do this without issue.

Question about the licensing - I just bought a BNIB one on eBay ( ICX6610-48-PI ) which has the Premium license. I saw somebody selling a license for the 10G upgrade on ebay so I bought that as well, but the listing says that the advanced license features are now included in the premium license. Can you confirm that or will I need to find an advances license to use GRE?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,075
113
33
fohdeesha.com
Thank you so much! Glad to hear that I can do this without issue.

Question about the licensing - I just bought a BNIB one on eBay ( ICX6610-48-PI ) which has the Premium license. I saw somebody selling a license for the 10G upgrade on ebay so I bought that as well, but the listing says that the advanced license features are now included in the premium license. Can you confirm that or will I need to find an advances license to use GRE?
you don't need an advanced license, it's merged into premium, and as the first sentence of this thread says, the port license you just spent $$$ on are free :)
 

tubs-ffm

Active Member
Sep 1, 2013
171
57
28
I must do something fundamental wrong in the L3 routing. I placed my question in a separate thread (see below). But maybe here in this Brocade thread I can find more guys that are familiar with FastIron.

What I want to achieve is to move the L3 routing between two networks (LAN and DMZ) from the firewall to the brocade switch. In theory this should work, but not with my config as linked below.

Any Idea where the mistake is?


I have a basic question to inter (V)LAN routing and the related network topology with two routers in it.

I am a home user and today my set-up is a router-on-a-stick configuration. One L2 switch to manage VLANs. All ACLs and routing done on the firewall. So far, all OK. For performance reasons I would like to move the routing between the networks called "LAN" and "DMZ" to a L3 switch. For the other VLANs this is not required as there is no routing to or from other subnets (Guest_WLAN, IoT).

Is the routing including ACLs between LAN and DMZ possible on the L3 switch with a network topology as shown on the picture?
Or must I move the DMZ and LAN network completely to the L3 switch and create "transport networks" and static routes between router and L3 switch?

View attachment 17392
Any idea what's wrong with my config?

My real config is using different networks than my illustration in the post above:

LAN: 192.168.2.0/24
DMZ: 192.168.10.0/24

Code:
Current configuration:
!
ver 08.0.92eT213
!
stack unit 1
  module 1 icx7250-24p-poe-port-management-module
  module 2 icx7250-sfp-plus-8port-80g-module
  stack-port 1/2/1
  stack-port 1/2/3
!
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 10 name DMZ by port
tagged ethe 1/1/9 ethe 1/2/6 ethe 1/2/8
untagged ethe 1/1/3 ethe 1/1/11 to 1/1/12 ethe 1/2/5 ethe 1/2/7
router-interface ve 10
!
vlan 20 name IoT by port
tagged ethe 1/1/5 ethe 1/1/7 ethe 1/1/9
!
vlan 30 name Guest by port
tagged ethe 1/1/5 ethe 1/1/7 ethe 1/1/9
untagged ethe 1/1/10
!
!
!
!
!
!
!
!
!
!
!
!
!
!
optical-monitor
optical-monitor non-ruckus-optic-enable
aaa authentication web-server default local
aaa authentication login default local
enable telnet authentication
enable aaa console
hostname icx7250
ip dhcp-client disable
ip dns domain-list test.lan
ip dns server-address 192.168.2.1 192.168.2.15
ip route 0.0.0.0/0 192.168.2.1
ip route 0.0.0.0/0 192.168.10.1
!
telnet timeout 10
no telnet server
username admin password .....
!
!
snmp-server contact Administrator
snmp-server location Server Room
!
!
clock timezone gmt GMT+09
!
!
ntp
disable serve
server 192.168.2.1
!
!
web-management https
web-management frame bottom
web-management page-menu
web-management session-timeout 1200
!
!
manager registrar
!
!
!
!
!
!
!
!
!
interface ethernet 1/1/1
port-name OPNsense-2
!
interface ethernet 1/1/2
disable
!
interface ethernet 1/1/3
port-name OPNsense-3
!
interface ethernet 1/1/4
disable
!
interface ethernet 1/1/5
port-name OPNsense-4
!
interface ethernet 1/1/6
disable
!
interface ethernet 1/1/7
port-name WLAN-AP
!
interface ethernet 1/1/8
disable
!
interface ethernet 1/1/9
port-name Trunk-Office
!
interface ethernet 1/1/10
port-name Work-PC
!
interface ethernet 1/1/11
port-name Server-DMZ
!
interface ethernet 1/1/12
port-name PC-DMZ
!
interface ethernet 1/2/1
port-name PC-LAN
!
interface ethernet 1/2/3
port-name Server-LAN
!
interface ethernet 1/2/5
port-name PC-DMZ
!
interface ethernet 1/2/6
port-name PC-Trunk
!
interface ethernet 1/2/7
port-name Server-DMZ
!
interface ethernet 1/2/8
port-name Server-Trunk
!
interface ve 1
ip access-group lan_out in
ip address 192.168.2.2 255.255.255.0
!
interface ve 10
ip access-group dmz_out in
ip address 192.168.10.2 255.255.255.0
!
!
ip access-list extended lan_out
remark allow LAN to switch management
sequence 10 permit tcp 192.168.2.0 0.0.0.255 host 192.168.2.2 eq ssh
sequence 20 permit tcp 192.168.2.0 0.0.0.255 host 192.168.2.2 eq http
sequence 30 permit tcp 192.168.2.0 0.0.0.255 host 192.168.2.2 eq ssl
remark allow LAN to DMZ
sequence 40 permit icmp 192.168.2.0 0.0.0.255 192.168.10.0 0.0.0.255
sequence 50 permit tcp 192.168.2.0 0.0.0.255 192.168.10.0 0.0.0.255 eq http
sequence 60 permit tcp 192.168.2.0 0.0.0.255 192.168.10.0 0.0.0.255 eq ssl
sequence 70 permit tcp 192.168.2.0 0.0.0.255 192.168.10.0 0.0.0.255 eq ssh
sequence 80 permit tcp 192.168.2.0 0.0.0.255 192.168.10.0 0.0.0.255 eq ftp
sequence 90 permit tcp 192.168.2.0 0.0.0.255 host 192.168.10.20 eq 8006
sequence 100 permit tcp 192.168.2.0 0.0.0.255 host 192.168.10.20 eq 26
sequence 110 permit tcp 192.168.2.0 0.0.0.255 host 192.168.10.40 eq smtp
sequence 120 permit tcp 192.168.2.0 0.0.0.255 192.168.10.0 0.0.0.255 eq 8083
sequence 130 permit tcp 192.168.2.0 0.0.0.255 192.168.10.0 0.0.0.255 eq 9090
sequence 140 permit tcp 192.168.2.0 0.0.0.255 host 192.168.10.60 eq 5001
sequence 150 permit tcp 192.168.2.0 0.0.0.255 host 192.168.10.60
sequence 160 permit udp 192.168.2.0 0.0.0.255 host 192.168.10.60
remark deny all other to DMZ
sequence 170 deny ip 192.168.2.0 0.0.0.255 192.168.10.0 0.0.0.255
remark allow every else to everywhere
sequence 180 permit ip any any
!
ip access-list extended dmz_out
remark allow DMZ to LAN
sequence 10 permit tcp 192.168.10.0 0.0.0.255 host 192.168.2.15 eq dns
sequence 20 permit udp 192.168.10.0 0.0.0.255 host 192.168.2.15 eq dns
sequence 30 permit tcp host 192.168.10.10 host 192.168.2.15 eq ldap
sequence 40 permit tcp host 192.168.10.20 host 192.168.2.15 eq ldap
sequence 50 permit tcp host 192.168.10.10 host 192.168.2.15 eq ldaps
sequence 60 permit tcp host 192.168.10.20 host 192.168.2.15 eq ldaps
sequence 70 permit tcp host 192.168.10.10 host 192.168.2.15 eq microsoft-ds
sequence 80 permit tcp host 192.168.10.20 host 192.168.2.30 eq 2525
sequence 90 permit tcp host 192.168.10.40 host 192.168.2.30 eq 2525
sequence 100 permit tcp host 192.168.10.40 host 192.168.2.30 eq smtp
remark deny all other to LAN
sequence 110 deny ip 192.168.10.0 0.0.0.255 192.168.2.0 0.0.0.255
remark allow every else to everywhere
sequence 120 permit ip any any
!
!
!
no lldp run
!
!
ip ssh  idle-time 0
!
!
!
!
!
end
 
  • Like
Reactions: tommybackeast

Roelf Zomerman

Active Member
Jan 10, 2019
147
27
28
blog.azureinfra.com
did you configure your DHCP to provide 10.2 and 2.2 as the default gateway for your clients (per DHCP scope) ?

and you do not need 2x 0.0.0.0 routes as your brocade switch should be the router itself.. it will forward the traffic either to 192.168.2.1 (over interface ve1) or to 192.168.10.1 (over interface ve10) / choose one ..


(Client to Client example)
Client [10.x]--->>Brocade[192.168.10.2]--->>[192.168.2.2]--->>client2[192.168.2.x]

(internet example 1 - if you wish to keep the 10.1 route and drop the 2.1 route)
Client [10.x]--->>Brocade[192.168.10.2]--->>router[192.168.10.1]--->>internet

(internet example 2 - if you wish to keep the 2.1 route and drop the 10.1 route)
Client [10.x]--->>Brocade[192.168.10.2]--->>[192.168.2.2]-->>router[192.168.2.1]--->>internet

also make sure your router knows the route back to both interfaces.... either through the route on an interface or the static added route..

let's take 192.168.2.0/24 as the primary subnet (internet example 2) -.. then you'd need to add: 192.168.10.0/24 --> next-hop - 192.168.2.2 and your Brocade only has 1 default route of 0.0.0.0/0 --> 192.168.2.2

you cannot create an asynchronous routing - where a packet from 192.168.10.x to internet is routed through the Brocade to 192.168.2.2 - to 192.168.2.1 to internet.. and back via internet->192.168.10.1--> 192.168.10.x , bypassing the brocade router completely - most firewalls will block this and a lot of protocols cannot handle this.. so make sure your routing outbound passes the same routers as the return traffic

if you want to retain dual routing tables (also possible) - your clients will essentially have 192.168.2.1 and 192.168.10.1 as their default gateway (for 0.0.0.0/0 traffic) and you will manually have to add a route for 192.168.10.0/24 next-hop 192.168.2.2 on the 192.168.2.0/24 clients and 192.168.2.0/24 next-hop 192.168.10.2 for the 10.x clients..
 
Last edited:
  • Like
Reactions: tommybackeast

tubs-ffm

Active Member
Sep 1, 2013
171
57
28
did you configure your DHCP to provide 10.2 and 2.2 as the default gateway for your clients (per DHCP scope) ?

and you do not need 2x 0.0.0.0 routes as your brocade switch should be the router itself.. it will forward the traffic either to 192.168.2.1 (over interface ve1) or to 192.168.10.1 (over interface ve10) / choose one ..
Thank you. I will go through your text in detail the next days.


The idea behind the two routes was that I wanted to use the bandwidth of three available ports at the OPNsense firewall by avoiding VLAN and LAG on firewall side. So, I connected LAN and DMZ network to separate interfaces on the firewall (untacked port at Bracade). This basically is the same configuration I am running with brocade as L2 switch and the firewall as only router. What I did for testing was adding ve on Brocade side to both VLAN, creating of ACLs, adding the routes mentioned above and manually change the default gateway on the clients to the brocade IP. Traffic to and from DMZ was supposed to take the ve10 and traffic two and from the LAN was supposed to take the other ve1. But this seems to be not possible in this way.

But for sure. If my idea was not good or wrong, any advice for best practice is welcome.
 

ipmifreely

New Member
Jan 16, 2021
5
0
1
When you remove a LAG on an ICX it disables the ports to prevent loops. You'll have to go to each affected port and issue 'enable'. Note in each interface definition it has 'disable'. A disabled port will not link, hence no lights.
ah ha! thank you. duh. now i can get rid of my hacky workarounds.
 

tubs-ffm

Active Member
Sep 1, 2013
171
57
28
if you want to retain dual routing tables (also possible) - your clients will essentially have 192.168.2.1 and 192.168.10.1 as their default gateway (for 0.0.0.0/0 traffic) and you will manually have to add a route for 192.168.10.0/24 next-hop 192.168.2.2 on the 192.168.2.0/24 clients and 192.168.2.0/24 next-hop 192.168.10.2 for the 10.x clients..
Sorry, to answer by two posts. But I first needed some time to understand your answer. This was very helpful for me.


What you described with dual routing tables seems to be exactly what I wanted to achieve. Now I know how to name it. Now it also is clear to me what I did wrong: Default gateway for clients to switch (.10.2 and 2.2) instead to firewall (10.1 and 2.1) and no next hop to the switch on the clients.

The need for a manual configuration of the next hop on the clients would be possible for me but seems to be not the "elegant way". The more common approach, if my understanding is correct, would be a set-up like shown below:

Networks 192.168.2.0/24 and 192.168.10.0/24 are present only on the L3 switch and the the L3 switch is default gateway for the clients. Default route on the switch for 0.0.0.0/0 trough tunnel network 192.168.1.0/30 to the firewall. For VLAN 20, 30 and 40 the switch is doing L2 switching only. No ve and no routing for these.

Or any better idea?

Network2.PNG
 

Roelf Zomerman

Active Member
Jan 10, 2019
147
27
28
blog.azureinfra.com
yups that works too.. all clients point to your switch, and your switch points to the router with the default route in the virtual router on the 192.168.1.0/30 subnet..

by not giving the 20/30/40 VLAN's an interface, you essentially take them out of the L3 option of the switch.. and they would be normal untagged or tagged traffic inside the switch..
 

Wronglebowski

New Member
Jun 18, 2018
5
0
1
Is there a good source for newer firmware? The fohdeesha website for the ICX 7250 has 2019 Firmware. If I can find newer firmware is there any reason not to upgrade?
 

neb50

Member
Aug 28, 2018
73
23
18
FYI. I am on 8.0.92dc and it looks like they have a 8.0.92e out now. I did not jump to the .95 train since I don't need any of those features and figured that .92 was out long enough to be stable.