Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Spearfoot

Member
Apr 22, 2015
68
26
18
62
Thanks for your reply itronin.

Code:
SSH@ICX6610-24 Switch#sh run
Current configuration:
!
ver 08.0.30tT7f1
!
stack unit 1
  module 1 icx6610-24-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  priority 128
  stack-port 1/2/1 1/2/6
stack unit 2
  module 1 icx6610-24-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-port 2/2/1 2/2/6
stack enable
stack mac xxxx.xxxx.xx74
!
!
!
lag xxxx dynamic id 1
ports ethernet 1/3/8 ethernet 2/3/8
primary-port 1/3/8
  deploy
port-name xxxx ethernet 1/3/8
port-name xxxx ethernet 2/3/8
!
lag xxxx dynamic id 2
ports ethernet 1/3/3 ethernet 2/3/3
primary-port 1/3/3
deploy
!
lag xxxx dynamic id 3
ports ethernet 1/3/4 ethernet 2/3/4
primary-port 1/3/4
deploy
!
!
vlan 1 name DEFAULT-VLAN by port
!
vlan xxxx name xxxx by port
tagged ethe 1/3/1 ethe 1/3/8 ethe 2/3/1 ethe 2/3/8
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/3/1 ethe 1/3/8 ethe 2/3/1 ethe 2/3/8
  spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
spanning-tree 802-1w
!
!
!
!
!
aaa authentication enable default local
aaa authentication login default local
boot sys fl sec
jumbo
enable telnet authentication
ip address x.x.x.x x.x.x.x
ip dns server-address 8.8.8.8
no ip dhcp-client enable
ip default-gateway x.x.x.x
!
username xxxx password .....
!
!
clock timezone gmt GMT+03
!
!
ntp
disable serve
server 192.36.143.130
server 213.251.52.185
!
!
hitless-failover enable
interface ethernet 1/1/17
dual-mode  xxxx
!
interface ethernet 1/2/1
port-name xxxx
!
interface ethernet 1/3/1
port-name xxxx
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
speed-duplex 10G-full
!
interface ethernet 1/3/5
dual-mode  xxxx
speed-duplex 10G-full
!
interface ethernet 1/3/6
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
port-name xxxx
speed-duplex 10G-full
!
interface ethernet 2/2/4
no spanning-tree
!
interface ethernet 2/3/1
speed-duplex 10G-full
!
interface ethernet 2/3/2
speed-duplex 10G-full
!
interface ethernet 2/3/5
speed-duplex 10G-full
!
interface ethernet 2/3/6
speed-duplex 10G-full
!
interface ethernet 2/3/7
speed-duplex 10G-full
!
!
!
!
!
!
ip ssh  timeout 60
!
!
end
Looks like you have all of your VLANs tagged on just about every port on these switches. Is that what you mean to do?

I only configure ports that way when I need to use them for trunking to another switch, or to pfSense. Most of my ports are set up with a single, untagged VLAN. Here's a snippet from my 6610-48 config, where you can see that ports 1/1/1, 1/1/2, 1/3/,1, and 1/3/2 are 'trunk' ports for connecting switches and everything else is for connecting edge devices:
Code:
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
!
vlan 10 name Secure by port
 tagged ethe 1/1/1 to 1/1/2 ethe 1/3/1 to 1/3/2 
 untagged ethe 1/1/3 to 1/1/5 ethe 1/1/7 ethe 1/1/9 to 1/1/11 ethe 1/1/13 to 1/1/36 ethe 1/2/2 to 1/2/5 ethe 1/3/3 to 1/3/5 
 router-interface ve 10
!
vlan 20 name Gizmos by port
 tagged ethe 1/1/1 to 1/1/2 ethe 1/3/1 to 1/3/2 
 untagged ethe 1/1/37 to 1/1/38 
 router-interface ve 20
!
vlan 30 name WiFi by port
 tagged ethe 1/1/1 to 1/1/2 ethe 1/3/1 to 1/3/2 
 untagged ethe 1/1/45 to 1/1/48 
 router-interface ve 30
!
vlan 40 name Cameras by port
 tagged ethe 1/1/1 to 1/1/2 ethe 1/3/1 to 1/3/2 
 untagged ethe 1/1/39 to 1/1/44 
 router-interface ve 40
!
vlan 1000 name Storage by port
 untagged ethe 1/1/6 ethe 1/1/8 ethe 1/1/12 ethe 1/2/7 to 1/2/10 ethe 1/3/6 to 1/3/8 
 router-interface ve 1000
!
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,824
113
29
fohdeesha.com
Code:
All fans in Low speed (0x83)
This is with only 2/3 fans connected to the 6450-48p btw. Will see how temps look once it actually has work to do.

BTW, my power meter between the switch and outlet reports only 97-99V, any idea how sensitive it is for undervoltage? Old house with likely shitty wiring.
spec sheet for the OEM psu model number I have for the 6450 says 100vac - 130vac but it's probably a bit more lenient than that. That's really bad voltage drop though, is that circuit in your house really loaded down? Even with poor wiring you shouldn't be dropping nearly 30 volts between the breaker panel and the outlet unless you're doing like 20 amps over it. Can you measure the incoming power at your breaker panel or meter? If it's that low there that's usually bad enough that the power company will do something about it. you could be at the end of a very long run after the nearest stepdown transformer or something
 

3nodeproblem

Member
Jun 28, 2020
42
11
8
spec sheet for the OEM psu model number I have for the 6450 says 100vac - 130vac but it's probably a bit more lenient than that. That's really bad voltage drop though, is that circuit in your house really loaded down? Even with poor wiring you shouldn't be dropping nearly 30 volts between the breaker panel and the outlet unless you're doing like 20 amps over it. Can you measure the incoming power at your breaker panel or meter? If it's that low there that's usually bad enough that the power company will do something about it. you could be at the end of a very long run after the nearest stepdown transformer or something
That's a good recommendation, will try that!

I'm in Japan though, where 100V is standard, so the drop is not *that* drastic.
 

karpuzvepeynir

New Member
Aug 19, 2020
4
0
1
Looks like you have all of your VLANs tagged on just about every port on these switches. Is that what you mean to do?

I only configure ports that way when I need to use them for trunking to another switch, or to pfSense. Most of my ports are set up with a single, untagged VLAN. Here's a snippet from my 6610-48 config, where you can see that ports 1/1/1, 1/1/2, 1/3/,1, and 1/3/2 are 'trunk' ports for connecting switches and everything else is for connecting edge devices:
Code:
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 10 name Secure by port
tagged ethe 1/1/1 to 1/1/2 ethe 1/3/1 to 1/3/2
untagged ethe 1/1/3 to 1/1/5 ethe 1/1/7 ethe 1/1/9 to 1/1/11 ethe 1/1/13 to 1/1/36 ethe 1/2/2 to 1/2/5 ethe 1/3/3 to 1/3/5
router-interface ve 10
!
vlan 20 name Gizmos by port
tagged ethe 1/1/1 to 1/1/2 ethe 1/3/1 to 1/3/2
untagged ethe 1/1/37 to 1/1/38
router-interface ve 20
!
vlan 30 name WiFi by port
tagged ethe 1/1/1 to 1/1/2 ethe 1/3/1 to 1/3/2
untagged ethe 1/1/45 to 1/1/48
router-interface ve 30
!
vlan 40 name Cameras by port
tagged ethe 1/1/1 to 1/1/2 ethe 1/3/1 to 1/3/2
untagged ethe 1/1/39 to 1/1/44
router-interface ve 40
!
vlan 1000 name Storage by port
untagged ethe 1/1/6 ethe 1/1/8 ethe 1/1/12 ethe 1/2/7 to 1/2/10 ethe 1/3/6 to 1/3/8
router-interface ve 1000
!
> Looks like you have all of your VLANs tagged on just about every port on these switches. Is that what you mean to do?
No.

After I've made the switches stack, some rear ports didn't work:

1/2/2 not working
1/2/3 working
1/2/4 working
1/2/5 not working
1/2/7 not working
1/2/8 not working
1/2/9 working
1/2/10 working

2/2/2 not working
2/2/3 not working
2/2/4 not working
2/2/5 not working
2/2/7 not working
2/2/8 working
2/2/9 working
2/2/10 working

This log shows what happens about not working ports:

Code:
Aug 19 18:45:51:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:45:51:I:STP: VLAN 1 Port 2/2/4 STP State -> DISABLED (PortDown)
Aug 19 18:45:51:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (PortDown)
Aug 19 18:45:50:I:System: Interface ethernet 2/2/4, state up
Aug 19 18:45:50:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (MakeFwding)
Aug 19 18:44:31:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:44:31:I:STP: VLAN 1 Port 2/2/4 STP State -> DISABLED (PortDown)
Aug 19 18:44:31:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (PortDown)
Aug 19 18:44:30:I:System: Interface ethernet 2/2/4, state up
Aug 19 18:44:30:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (MakeFwding)
This time STP is disabled on the port:
Code:
Aug 19 18:51:38:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:51:37:I:System: Interface ethernet 2/2/4, state up
Aug 19 18:51:22:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:51:22:I:System: Interface ethernet 2/2/4, state up
 

dodgy route

Member
Aug 12, 2020
9
26
13
Australia
So I finally registered after lurking for a bit, as I am getting into 10G networking at home.
This thread has been an amazing source of information! Thanks fohdeesha for sharing the information with us!
Managed to pick up a good condition ICX6610-48P-E for $300 Australia dollary doos off of eBay which I was impressed with.

I have just finished the multi vlan configuration and tested successfully, just PoE section left, and 10G stuff, very happy so far.

Before I put it in actual full time use considering modding the fans to 120/140mm versions with the use of some fan speed holes up top.
Have seen already a few attempts and they require the fan speed trickery so the switch even boots, frustrating but not a problem!

Is there any quicker solution or is ordering the parts and waiting the name of the game?
Asking just in case I have missed something, despite reading what must be almost all of the pages in this mega thread over the past few weeks since I decided to pull the trigger on 10G

Cheers!
 
  • Like
Reactions: tommybackeast

dodgy route

Member
Aug 12, 2020
9
26
13
Australia
So I finally registered after lurking for a bit, as I am getting into 10G networking at home.
This thread has been an amazing source of information! Thanks fohdeesha for sharing the information with us!
Managed to pick up a good condition ICX6610-48P-E for $300 Australia dollary doos off of eBay which I was impressed with.

I have just finished the multi vlan configuration and tested successfully, just PoE section left, and 10G stuff, very happy so far.

Before I put it in actual full time use considering modding the fans to 120/140mm versions with the use of some fan speed holes up top.
Have seen already a few attempts and they require the fan speed trickery so the switch even boots, frustrating but not a problem!

Is there any quicker solution or is ordering the parts and waiting the name of the game?
Asking just in case I have missed something, despite reading what must be almost all of the pages in this mega thread over the past few weeks since I decided to pull the trigger on 10G

Cheers!
Straight after I finished posting, I remembered I bought 2 Arduino Nano's about 2 or 3 years ago that I forgot about, could be the ticket to quickly get this sorted im thinking
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,824
113
29
fohdeesha.com

dodgy route

Member
Aug 12, 2020
9
26
13
Australia
  • Like
Reactions: fohdeesha

liberty

New Member
Nov 4, 2019
4
2
3
Any ideas on getting a 7250-48P cool enough so the fans run at low speed?
It's easy to disable the PSU temp sensor on a 7250-48P.

There is a 2 wire jumper on the main switch board clearly labeled 'PSU T_Sen'. When disconnected, the PSU sensor reports a constant 10.0c reading. ASIC temps are stable at 65c and the fans are running at low speed. I am guessing PSU temp is also in the 60's based on the readings I saw with a single fan at high speed. 3 fans on low is moving about the same air as a single fan on high. It is running 10-15c cooler than the stock 7250-24 in the same rack.
 
  • Like
Reactions: fohdeesha

Pestx

New Member
Aug 21, 2020
3
3
3
I have a dumb question: is the boot time of the ICX affected by the number of Lan declared inside the switch ? I have had at home a TP Link TL-SG3424. One day after a power failure, my switch didn't boot. I thought it was dead, but in fact it eventually boot but in 40 minutes. I had made trunks to my Proxmox cluster with all vlan, and created 100 Vlans for testing purpose and my switch didn't like it. I know this is a dump question as I had never seen things like that witch Cisco or HPE switch I used to work with...
I didn't receive my unit yet, but I'll definitively test that if no one ever did here before ;)
 

infoMatt

Active Member
Apr 16, 2019
184
79
28
I have a dumb question: is the boot time of the ICX affected by the number of Lan declared inside the switch ?
Honestly, I haven't tried or measured, but I don't think that any configuration possible could make a "real" difference for the boot time, maybe one or two seconds, but it's a tiny margin.
40 minutes is absolutely out of any stetch of imagination ;)
 
Last edited:
  • Like
Reactions: Pestx

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,824
113
29
fohdeesha.com
It's easy to disable the PSU temp sensor on a 7250-48P.

There is a 2 wire jumper on the main switch board clearly labeled 'PSU T_Sen'. When disconnected, the PSU sensor reports a constant 10.0c reading. ASIC temps are stable at 65c and the fans are running at low speed. I am guessing PSU temp is also in the 60's based on the readings I saw with a single fan at high speed. 3 fans on low is moving about the same air as a single fan on high. It is running 10-15c cooler than the stock 7250-24 in the same rack.
Good find. I'm assuming you mean this guy?



I was wondering what that's for, A while back I saw the psu_sense silkscreen on the PCB and figured it was just a simple presence sense for the optional external power supply or something. Missed the T. Interestingly it's not present at all or even populated on the PCB on the non-PoE models, I guess they don't monitor PSU temps at all (they'd have no way to now that I look at the internals, there's only 2 wires coming off the PSU itself, 12V DC)
 
  • Like
Reactions: liberty

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,824
113
29
fohdeesha.com
I have a dumb question: is the boot time of the ICX affected by the number of Lan declared inside the switch ? I have had at home a TP Link TL-SG3424. One day after a power failure, my switch didn't boot. I thought it was dead, but in fact it eventually boot but in 40 minutes. I had made trunks to my Proxmox cluster with all vlan, and created 100 Vlans for testing purpose and my switch didn't like it. I know this is a dump question as I had never seen things like that witch Cisco or HPE switch I used to work with...
I didn't receive my unit yet, but I'll definitively test that if no one ever did here before ;)

they definitely have no effect on boot time. When I've used some of these as colo routers I think the largest config I've gotten to was 3000 lines and it booted the same as all the others. The longest process during bootup is reading the firmware image off of flash and uncompressing it into RAM (like any other switch)
 
  • Like
Reactions: Pestx

infoMatt

Active Member
Apr 16, 2019
184
79
28
Interestingly it's not present at all or even populated on the PCB on the non-PoE models, I guess they don't monitor PSU temps at all (they'd have no way to now that I look at the internals, there's only 2 wires coming off the PSU itself, 12V DC)
Mainly because (IMHO) the power consumption of the switch itself is fairly low, so no need of any special treatment of the PSU itself... the problems arise when in the same space (more or less) it has to deliver some 500-ish W of PoE power instead of the 50W for the switch alone.
 
Last edited:
  • Like
Reactions: fohdeesha

rootwyrm

Member
Mar 25, 2017
44
48
18
www.rootwyrm.com
Search has failed me or nobody's posted about it, but has anyone had any experience buying these from Andover CG DBA Network Tigers via eBay? I know there were a few vendors known to routinely be selling known bad units, so when the price looks this good I'm a little hesitant.

They have a HUGE batch of pulls up for sale currently.
 
  • Like
Reactions: 3nodeproblem

blinkenlights

Active Member
May 24, 2019
143
64
28
Mainly because (IMHO) the power consumption of the switch itself is fairly low, so no need of any special treatment of the PSU itself... the problems arise when in the same space (more or less) it has to deliver some 500-ish W of PoE power instead of the 50W for the switch alone.
Reminds me of a question.. about a month ago while playing musical PSUs with my Brocade gear, I decided I wanted an extra exhaust (-E) power supply for a non-PoE 7450. The 250-watt versions (RPS15) were in short supply and priced too high, so I began looking at the PoE 1,000-watt versions (RPS16).

I can't remember exactly where I read it... a post stating the 1kW power supplies are really 250W (switch) + 750W (PoE), and not 1kW (shared). Does that sound right? And will the RPS16 will work just fine in a non-PoE switch?