Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

tubs-ffm

Active Member
Sep 1, 2013
122
39
28
So here's some fans you can buy that I'd recommend.
  • Delta EFB0412VHD-F00 - $12.40/ea @ Digikey
    There's a REASON Delta's a top pick for ODM and OEM. 40x40x20mm, 10.1CFM, 0.416in H2O (which is insane,) 32.5dBA @ 1m, rated for 70k hours at 50C. Make sure it's THAT part number and not the Rev C, which is a vastly inferior part.
I am very satisfied with the noise level of two Delta EFB0412VHD-F00 in comparison to the original two Foxconn PIA040H12P n my ICX 7250-14P. But unfortunately, the cooling performance is not sufficient. For testing the switch is placed free on a desk. It is wintertime and the ambient temperature is somewhere 20 - 22 °C. Without any load and only 1 cable connected for ssh access, after about 15 min the temperature limit of 93 °C is exceeded and the level 2 of fan power starts. It runs only for 30 sec in level 2. But this cycle repeats about every 5 to 10 min.

I will give it one more trial by adding an additional fan over the cooling rips of the CPU. First, I have to search the correct one. Sunon MF60101V3-1000U-A99 I was reading somewhere. Otherwise, this device and me we will go separate ways again.
 

craig5571

Member
May 31, 2020
60
6
8
I don't know anything about the esxi side (don't use it), but switch side is easy.

Code:
vlan 1
tag eth 1/2/1
vlan 2
tag eth 1/2/1
So on and so forth. If you want to set the native untagged clan on a trunk port, you can just directly do

Code:
vlan 3
untagged eth 1/2/1
now, no more fooling around with dual-mode. You don't have to worry about any of that switchport mode trunk, switchport encapsulation, switchport trunk allowed vlan add, nonsense.

Unless I'm missing something, your implementation is really simple, but yeah vlan assignment is one of the biggest differences from IOS.
thank you so much , will this allow intervlan routing as well for example three vlans 10 20 and 30
all on different subnets 192.168.10.0 , 20.0, 30.0 etc..

will a device on 20.0 be able to ping 10.0 and 30.0 and vice versa

i had thought i read something about having to have a ve interface

BROCADE CAMPUS FEATURE EXPLAINER SERIES - VLAN ROUTING WITH ROUTER-INTERFACE CONFIGURATION


I should know this all by heart, but I been doing Wi-Fi so long.. alot of simple wired stuff has slipped my mind..

thanks again
 

ipmifreely

New Member
Jan 16, 2021
5
0
1
inter vlan routing WITHOUT doing it on the switch (icx6610)

hi all,

I've just upgraded from a HP procurve with layer 2 vlan support and my intervlan routing is not working - it was working fine with the HP. I'm seeing a lot of folks in this thread with similar problems but (i think) they are mostly trying to solve the problem IN switch and i want to keep my nice opnsense GUI....

my setup: I have a hand full of vlans and Opnsense running as a VM on proxmox. Proxmox has and LACP/LAG bonded connection to the switch. so

the setup of the switch is

vlan 3000 - wan
vlan 1 - management
vlan 2 - lan
vlan 4 - wifi
vlan 10 - defualt_vlan (i reassigned it)

for the wan iput port i set it as dual mode becuase my ISP is GPON and sets the services on speific vlans (yes i probably don't need this - but i don't want that port part of the default vlan and don't know how else to do it)
for the LACP ports for proxmox i set vlan y as dual mode (because i couldn't get untagged to work with LACP) and the rest of the vlans to tagged.
for the rest of the 1st 22 ports i set vlan 1 as untagged
for ports 22&24 i set vlan 4 (wifi) as untaggged
for ports 25-48 i set vlan 2 as untagged.

this is more or less and exact replication of what i was doing with my procurve (the new switch just has more ports so i've spread things about a bit)

The setup of Proxmox is:

i create the bond
i setup all the vlans i need.
i create bridge interfaces for all the vlans
i share the bridge interfaces for all the vlans with my VM and to them they look like separate adapters...

The setup inside the opnsense VM is:
is sees all the bridge interfaces from poxmox as network adapters and applys firewall and routing between them.

yes i know that the icx is layer 3 - I got the ICX6610 for more ports, 10 & 40 Gig, and POE and not so much for Layer 3 at it is overwhelming for me.

am i being thick ? i searched the forum already but didn't find anything that seemed to pertain to my sistuation (routing from a router that is a VM). if my problem is easily googilable please just point me in the right direction and accept my appologies in advance :)
 

drtaru

New Member
Jan 10, 2021
5
3
3
I ordered sunon fans and when they are connected i get pretty terrible coil whine sounding noise when they ramp down to 4v, is there any way to force a ICX 7250-48 to run the fans at speed 2 always?
 

acpatel

New Member
Sep 8, 2020
5
0
1
OK. Powered up my ICX7250-48P. It is LOUD. That said, it'll live in the basement, so it's not an urgent issue.
I got it to boot once and it let me factory set-default, reset. OK.
Then I got it to let me do enable, configure terminal, ip dhcp-client disable -- which didn't work because it has layer 2 firmware on it. Fine. Restart to load new firmware. I want to load the new firmware off of usb, however ever since then, every time I enable then configure terminal, it core dumps. I have no idea why. What all do you need from me to help troubleshoot? Thanks.

Also, is there still separate PoE firmware in v8.0.92? Or do the UFI images include that?
 

rocketpanda40

Member
Dec 12, 2019
39
24
8
OK. Powered up my ICX7250-48P. It is LOUD. That said, it'll live in the basement, so it's not an urgent issue.
I got it to boot once and it let me factory set-default, reset. OK.
Then I got it to let me do enable, configure terminal, ip dhcp-client disable -- which didn't work because it has layer 2 firmware on it. Fine. Restart to load new firmware. I want to load the new firmware off of usb, however ever since then, every time I enable then configure terminal, it core dumps. I have no idea why. What all do you need from me to help troubleshoot? Thanks.

Also, is there still separate PoE firmware in v8.0.92? Or do the UFI images include that?
The UFI images automatically flash the PoE firmware - watch the console output, you'll see it. But when you download the firmware, you'll get the separate poe image as well if you want to manually flash it for assurance.
 

rocketpanda40

Member
Dec 12, 2019
39
24
8
thank you so much , will this allow intervlan routing as well for example three vlans 10 20 and 30
all on different subnets 192.168.10.0 , 20.0, 30.0 etc..

will a device on 20.0 be able to ping 10.0 and 30.0 and vice versa

i had thought i read something about having to have a ve interface

BROCADE CAMPUS FEATURE EXPLAINER SERIES - VLAN ROUTING WITH ROUTER-INTERFACE CONFIGURATION


I should know this all by heart, but I been doing Wi-Fi so long.. alot of simple wired stuff has slipped my mind..

thanks again
If you want the switch to do inter-vlan routing, you'll need the virtual ethernet interfaces, yeah.

That's as simple as

Code:
vlan 1
router-interface ve 1
int ve 1
ip add 10.1.1.1/24

vlan 2
router-interface ve 2
int ve 2
ip add 10.2.2.1/24
Now the switch will route between 10.1.1.0/24 and 10.2.2.0/24.

Here's one of my complete, working configs of an ICX7250-48P acting as my home's core switch. There's a mix of vlans being routed by the switch and others in L2 only, being routed and firewalled by vyos upstream. Also present is OSPF, LACP LAGs, etc. This is before I added a few simple ACLs though, but should help in translating the Cisco IOS knowledge to Fastiron.

It will probably help understanding to know that:

  • I've connected my wan straight to the switch on a vlan, and have my firewall's wan interface also on that vlan
  • Port with name "B2" goes to another switch I have connecting my workstation (with its own lab), an access point, and a printer, so that switch is also running L3 firmware
  • "wlc" is my wireless controller
  • "vyos" is my firewall
  • OSPF exchanges routes between all of the aforementioned
  • The LAG interfaces are all 2x1Gb links to my proxmox servers. I'm not currently using the 10Gb interfaces because after massively downscaling my lab, I now have like 30 10Gb DACs and 0 10Gb PCIe cards (I had a whole bunch of dual 10Gb OCP cards that I sold with my old servers)
 
  • Like
Reactions: craig5571

acpatel

New Member
Sep 8, 2020
5
0
1
The UFI images automatically flash the PoE firmware - watch the console output, you'll see it. But when you download the firmware, you'll get the separate poe image as well if you want to manually flash it for assurance.
Thoughts on the seg fault?
 

craig5571

Member
May 31, 2020
60
6
8
If you want the switch to do inter-vlan routing, you'll need the virtual ethernet interfaces, yeah.

That's as simple as

Code:
vlan 1
router-interface ve 1
int ve 1
ip add 10.1.1.1/24

vlan 2
router-interface ve 2
int ve 2
ip add 10.2.2.1/24
Now the switch will route between 10.1.1.0/24 and 10.2.2.0/24.

Here's one of my complete, working configs of an ICX7250-48P acting as my home's core switch. There's a mix of vlans being routed by the switch and others in L2 only, being routed and firewalled by vyos upstream. Also present is OSPF, LACP LAGs, etc. This is before I added a few simple ACLs though, but should help in translating the Cisco IOS knowledge to Fastiron.

It will probably help understanding to know that:

  • I've connected my wan straight to the switch on a vlan, and have my firewall's wan interface also on that vlan
  • Port with name "B2" goes to another switch I have connecting my workstation (with its own lab), an access point, and a printer, so that switch is also running L3 firmware
  • "wlc" is my wireless controller
  • "vyos" is my firewall
  • OSPF exchanges routes between all of the aforementioned
  • The LAG interfaces are all 2x1Gb links to my proxmox servers. I'm not currently using the 10Gb interfaces because after massively downscaling my lab, I now have like 30 10Gb DACs and 0 10Gb PCIe cards (I had a whole bunch of dual 10Gb OCP cards that I sold with my old servers)
major thank you!! , have you modified the fans at all? I can totally live with the 7250P stock , but if I could make it quieter and keep it cool ( the most important thing ) that would be nice.. ( the switch is on the shelf about six feet to my left at approximately ear height.. )

I was thinking of replacing the two stock fans
https://forums.servethehome.com/proxy.php?image=https%3A%2F%2Fi.imgur.com%2Fr8Kd6dH.png&hash=451a90f836a49a4837fa8bbc33a7eb36

with three of these

  • Comair Rotron "Gryphon" GDA4028-12BB - discontinued, alas
    If you can find these? BUY THESE. 40x40x28mm, but 11CFM, 0.34in H2O, 8800RPM, but just 31.4dBA! I am still mad they discontinued them. No, I am not selling any of my spares.
I got 3 of them.. but they came with bare wires.. so I ordered ends to put on them..

pic of the fan
3 Wires - (Red-positive, Black-negative, White-Tachometer)

I'm looking for the wire diagram for the original fans to know which wire is which wire.. the original fans three wires green yellow and black ( from left to right) but I dont know what green= or yellow= or black= , I assume positive, negative and tachometer.


ICX7250-24P Internals is the one I have..

in post #4275 fan options are discussed very in depth
" If you want to quiet down these units, the BIGGEST change you can make is to remove the stamped grill with a Dremel. It's all sharp edges, and significantly blocks the hub as well. Turbulence from things like grills are what generate serious noise. If you still need protection, tack on a wire grill on the OUTSIDE of the chassis "

the Stamped Grill is the circular holes where the air comes out on the back ? ( at least thats how I think...)


and if all that made the system quieter.. but not cool enough...
I was thinking of adding a fractal design 140 mm fan ( i have an extra one) on the top of the case blowing down in the case ,( i test this out by getting a sheet of plastic from lowes cutting that to verify the idea works before , modding the case ) the other 3 fans would blow the air out..

I haven't done any of this fan work yet.. just brain storming.. I sometimes get things wrong.. but alot of times I get it right.
 
Last edited:

tubs-ffm

Active Member
Sep 1, 2013
122
39
28
I'm looking for the wire diagram for the original fans to know which wire is which wire.. the original fans three wires green yellow and black ( from left to right) but I dont know what green= or yellow= or black= , I assume positive, negative and tachometer.
I replaced the original Foxconn PIA040H12P in my 7250-24P yesterday and was facing the same issue. I used google and found some standard wiring definitions that fit:

black --> -
yellow --> +
green --> frequency

Only the position on the connector of the Ruckus was not matching to the standard definition. I wired the new fan to match to the cable position of the original fan.

Edit: typos corrected
 
Last edited:
  • Like
Reactions: craig5571

ipmifreely

New Member
Jan 16, 2021
5
0
1
inter vlan routing WITHOUT doing it on the switch (icx6610)

hi all,

I've just upgraded from a HP procurve with layer 2 vlan support and my intervlan routing is not working - it was working fine with the HP. I'm seeing a lot of folks in this thread with similar problems but (i think) they are mostly trying to solve the problem IN switch and i want to keep my nice opnsense GUI....

my setup: I have a hand full of vlans and Opnsense running as a VM on proxmox. Proxmox has and LACP/LAG bonded connection to the switch. so

the setup of the switch is

vlan 3000 - wan
vlan 1 - management
vlan 2 - lan
vlan 4 - wifi
vlan 10 - defualt_vlan (i reassigned it)

for the wan iput port i set it as dual mode becuase my ISP is GPON and sets the services on speific vlans (yes i probably don't need this - but i don't want that port part of the default vlan and don't know how else to do it)
for the LACP ports for proxmox i set vlan y as dual mode (because i couldn't get untagged to work with LACP) and the rest of the vlans to tagged.
for the rest of the 1st 22 ports i set vlan 1 as untagged
for ports 22&24 i set vlan 4 (wifi) as untaggged
for ports 25-48 i set vlan 2 as untagged.

this is more or less and exact replication of what i was doing with my procurve (the new switch just has more ports so i've spread things about a bit)

The setup of Proxmox is:

i create the bond
i setup all the vlans i need.
i create bridge interfaces for all the vlans
i share the bridge interfaces for all the vlans with my VM and to them they look like separate adapters...

The setup inside the opnsense VM is:
is sees all the bridge interfaces from poxmox as network adapters and applys firewall and routing between them.

yes i know that the icx is layer 3 - I got the ICX6610 for more ports, 10 & 40 Gig, and POE and not so much for Layer 3 at it is overwhelming for me.

am i being thick ? i searched the forum already but didn't find anything that seemed to pertain to my sistuation (routing from a router that is a VM). if my problem is easily googilable please just point me in the right direction and accept my appologies in advance :)
i thought i might have messed up earlier in my config by following fodesha's acl guide but show running-config dosn't list any ACLs so thats not it...
 

-MoNsTeRRR

New Member
May 24, 2020
5
1
3
Hello,

I've tried today to configure IPv6 ACL on my ICX 6650 and i've faced this issue when i've applyed an ACL :
Code:
Insufficient hardware resources to apply the V6 ACL. Please remove already applied ACL(s) and/or Security features and try again.
I've 6 ACL for IPv4 and 6 ACL for IPv6. I've applyed all IPv4 ACL and 5 IPv6 ACL and can't apply the last.

I've tried to increase system-max values but it seems ip-filter-sys & ip-filter-port doesn't resolve the issue with max values.

I've deleted all the rules in the ACL, i've added the ACL to the ve interface and tried to reapply the rules but I faced this error :
Code:
ERROR: Insufficient hardware resource for binding the ACL to interface v10.
Here is the IPv6 & IPv4 ACL applyed to ve10
Code:
ipv6 access-list DMZ_IN
remark "ALLOW ANY to ANY ESTABLISHED"
permit tcp any any established
remark "ALLOW ANY to ANY ICMP ECHO-REPLY"
permit icmp any any echo-reply
remark "ALLOW DHCP1-REPLY to ANY"
permit udp host 2a0c:b641:2c0:110::23 any eq 68
remark "ALLOW DHCP2-REPLY to ANY"
permit udp host 2a0c:b641:2c0:110::24 any eq 68
remark "ALLOW DNS1/UDP to ANY"
permit udp host 2a0c:b641:2c0:110::21 eq 53 any
remark "ALLOW DNS2/UDP to ANY"
permit udp host 2a0c:b641:2c0:110::22 eq 53 any
remark "ALLOW VM-MONITORING to VLAN-MANAGEMENBT SNMP"
permit udp host 2a0c:b641:2c0:110::101 2a0c:b641:2c0:140::/64 eq 161
remark "ALLOW VM-MONITORING to LAB NODE-EXPORTER"
permit tcp host 2a0c:b641:2c0:110::101 2a0c:b641:2c0:120::/64 eq 9100
remark "ALLOW VM-MONITORING to LAB VMWARE-EXPORTER"
permit tcp host 2a0c:b641:2c0:110::101 host 2a0c:b641:2c0:140::10 eq 9272
remark "ALLOW VM-MONITORING to UNIFI-POLLER"
permit tcp host 2a0c:b641:2c0:110::101 host 2a0c:b641:2c0:120::50 eq 8443
remark "ALLOW VM-MONITORING to NAS-SNMP"
permit udp host 2a0c:b641:2c0:110::101 host 2a0c:b641:2c0:120::42 eq 161
remark "ALLOW VM-MONITORING to HS110-RACK1"
permit tcp host 2a0c:b641:2c0:110::101 host 2a0c:b641:2c0:150::101 eq 9233
remark "ALLOW VM-MONITORING to HS110-RACK2"
permit tcp host 2a0c:b641:2c0:110::101 host 2a0c:b641:2c0:150::102 eq 9233
remark "ALLOW VM-MONITORING to HS110-CHAMBRE1"
permit tcp host 2a0c:b641:2c0:110::101 host 2a0c:b641:2c0:150::103 eq 9233
remark "ALLOW VM-MONITORING to WIREGUARD-EXPORTER"
permit tcp host 2a0c:b641:2c0:110::101 host 2a0c:b641:2c0:170::29 eq 9998
remark "DENY ROUTING"
deny ipv6 any 2a0c:b641:2c0:105::/64
remark "DENY LOOPBACK"
deny ipv6 any 2a0c:b641:2c0:106::/64
remark "DENY VLAN-LAB"
deny ipv6 any 2a0c:b641:2c0:120::/64
remark "DENY VLAN-ADMIN"
deny ipv6 any 2a0c:b641:2c0:130::/64
remark "DENY VLAN-MANAGEMENT"
deny ipv6 any 2a0c:b641:2c0:140::/64
remark "DENY VLAN-HOME"
deny ipv6 any 2a0c:b641:2c0:150::/64
remark "DENY VLAN-GUEST"
deny ipv6 any 2a0c:b641:2c0:160::/64
remark "ALLOW ANY to ANY"
remark "DENY VLAN-VPN"
deny ipv6 any 2a0c:b641:2c0:170::/64
permit ipv6 any any


ip access-list extended DMZ_IN
remark "ALLOW ANY to ANY ESTABLISHED"
permit tcp any any established
remark "ALLOW ANY to ANY ICMP ECHO-REPLY"
permit icmp any any echo-reply
remark "ALLOW DHCP1-REPLY to ANY"
permit udp host 192.168.10.23 any eq 68
remark "ALLOW DHCP2-REPLY to ANY"
permit udp host 192.168.10.24 any eq 68
remark "ALLOW DNS1/UDP to ANY"
permit udp host 192.168.10.21 eq 53 any
remark "ALLOW DNS2/UDP to ANY"
permit udp host 192.168.10.22 eq 53 any
remark "ALLOW VM-MONITORING to VLAN-MANAGEMENBT SNMP"
permit udp host 192.168.10.101 192.168.40.0 0.0.0.255 eq 161
remark "ALLOW VM-MONITORING to LAB NODE-EXPORTER"
permit tcp host 192.168.10.101 192.168.20.0 0.0.0.255 eq 9100
remark "ALLOW VM-MONITORING to LAB VMWARE-EXPORTER"
permit tcp host 192.168.10.101 host 192.168.40.10 eq 9272
remark "ALLOW VM-MONITORING to UNIFI-POLLER"
permit tcp host 192.168.10.101 host 192.168.20.50 eq 8443
remark "ALLOW VM-MONITORING to NAS-SNMP"
permit udp host 192.168.10.101 host 192.168.20.42 eq 161
remark "ALLOW VM-MONITORING to HS110-RACK1"
permit tcp host 192.168.10.101 host 192.168.50.101 eq 9233
remark "ALLOW VM-MONITORING to HS110-RACK2"
permit tcp host 192.168.10.101 host 192.168.50.102 eq 9233
remark "ALLOW VM-MONITORING to HS110-CHAMBRE1"
permit tcp host 192.168.10.101 host 192.168.50.103 eq 9233
remark "ALLOW VM-MONITORING to WIREGUARD-EXPORTER"
permit tcp host 192.168.10.101 host 192.168.70.29 eq 9998
remark "DENY ROUTING"
deny ip any 192.168.5.0 0.0.0.255
remark "DENY LOOPBACK"
deny ip any 192.168.6.0 0.0.0.255
remark "DENY VLAN-LAB"
deny ip any 192.168.20.0 0.0.0.255
remark "DENY VLAN-ADMIN"
deny ip any 192.168.30.0 0.0.0.255
remark "DENY VLAN-MANAGEMENT"
deny ip any 192.168.40.0 0.0.0.255
remark "DENY VLAN-HOME"
deny ip any 192.168.50.0 0.0.0.255
remark "DENY VLAN-GUEST"
deny ip any 192.168.60.0 0.0.0.255
remark "ALLOW ANY to ANY"
remark "DENY VLAN-VPN"
deny ip any 192.168.70.0 0.0.0.255
permit ip any any

interface ve 10
 ip access-group DMZ_IN in
 ip address 192.168.10.2 255.255.255.0
 ip helper-address 1 192.168.10.23
 ip helper-address 2 192.168.10.24
 ipv6 address 2a0c:b641:2c0:110::2/64
 ipv6 enable
 ipv6 traffic-filter DMZ_IN in
 ipv6 dhcp-relay destination 2a0c:b641:2c0:110::23
 ipv6 dhcp-relay destination 2a0c:b641:2c0:110::24
Is it a limitation or am I doing wrong ?

Thanks :)
 

SIlviu

Member
May 27, 2016
73
8
8
Got a 6450 48 port on ebay with POE dead, I want to use it without POE, I removed the POE board, no more errors in CLI but the fans are 100% all the time. Is there a way to disable the POE on the device or to slow down the fans ?
 

Originalus

New Member
Dec 22, 2020
28
9
3
i am interested in ICX7150-C12P, but have two main questions.
1. can it be used as router? looking at spec sheets and posts looks like can. maybe someone is running it like router.
2. can sfp+ and rj45 uplink ports used as simple ports for wiring lan devices?

one extra:
i have gpon fiber at home. Would it be possible to use gpon sfp transceiver on sfp ports?
 

Wronglebowski

New Member
Jun 18, 2018
5
0
1
There seems to be multiple variants of the "Brocade Official SFP+". Is there any particular model I need to get the added benefits?


 

rocketpanda40

Member
Dec 12, 2019
39
24
8
i am interested in ICX7150-C12P, but have two main questions.
1. can it be used as router? looking at spec sheets and posts looks like can. maybe someone is running it like router.
2. can sfp+ and rj45 uplink ports used as simple ports for wiring lan devices?

one extra:
i have gpon fiber at home. Would it be possible to use gpon sfp transceiver on sfp ports?
1. Yes, in the sense that you can run L3 firmware. It won't do NAT and extended ACLs are a pain compared to 'normal' firewall rules so it won't replace your home router/firewall if that's what you mean.
2. Yup
 
Last edited:

rocketpanda40

Member
Dec 12, 2019
39
24
8
There seems to be multiple variants of the "Brocade Official SFP+". Is there any particular model I need to get the added benefits?


The first link are 8Gb Fiber channel transceivers. You want the second link.
 

plexisaurus

New Member
Jan 14, 2021
6
2
3
I recently purchased a new icx 7250-48 off ebay. I'm experimenting with fan modding because stock ones are too loud for me. Can anyone hazard a educated guess what a safe 24/7 temp is? it was 65-70c at stock idle, now it is around 77c after 100minutes idle and not quite reached steady state. I know these have a limit of 100-105, but my experience with other silicon/laptops/pcs is that staying just below peak limit still isn't great long term.

FYI I've swapped the two stock fans with two Sunon MF40201vx-1000u-g99 MagLev ones. I'm considering adding a 3rd, adding a slim fan on top of asic, or go full nuclear and cut out holes for mounting dual top-down 140-180mm fans