Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Looks like it was actually a bad PSU I was using. I had two Rev. A PSUs sitting in my shelf and I knew one of them was a dud. I assumed it was the one with the bent handle (you know, the physically damaged one...) and tried the other one. On a whim today during my lunch break I tried the other one, and it started booting!
Although it does seem to be getting stuck a little ways into the boot process. This is what I get:
Code:
$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 71f1
BOOT INFO: verify flash files..........
BOOT INFO: load image from primary copy...
platform type = 12
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
.............................
Firmware integrity checksum passed
.......
Starting Main Task ...CPSS DxCh Version: cpss3.4p1 release
Pre Parsing Config Data ...Edit: apparently I just needed to be more patient. It continued startup pretty much as soon as I posted my reply.
yeah I dunno what it is but the 6610's boot sloooowwww. The FCX has the same CPU and has to unpack the exact same firmware to memory and it boots like 5x faster
Is it unpacking with the same settings (all cores/threads vs single thread)?
Is the boot device connected in the same way?
Is the boot device on pair?
the management plane between the two is identical (that's why they run the same firmware and even the same bootloader image). Both are single core single thread PPC procs booting from CFI flash, the flash part number is even the same between them (S29GL512P). the main difference is the ASIC of course, perhaps the newer marvell in the 6610 takes longer to initialize
I did my brocade switches stack like I wish with fohdeesha's description below.
https://forums.servethehome.com/ind...p-powerful-10gbe-40gbe-switching.21107/page-7
But I faced a strange issue. Some rear ports 1/2/2 to 1/2/5, 1/2/7 to 1/2/10 and same ports on the second switch don’t work when they connected to servers. There is no problem with network cards of servers.
I would be very happy if you suggest anything about the issue.
https://forums.servethehome.com/ind...p-powerful-10gbe-40gbe-switching.21107/page-7
But I faced a strange issue. Some rear ports 1/2/2 to 1/2/5, 1/2/7 to 1/2/10 and same ports on the second switch don’t work when they connected to servers. There is no problem with network cards of servers.
I would be very happy if you suggest anything about the issue.
This log shows what happens about not working ports:
This time STP is disabled on the port:
Code:
Aug 19 18:45:51:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:45:51:I:STP: VLAN 1 Port 2/2/4 STP State -> DISABLED (PortDown)
Aug 19 18:45:51:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (PortDown)
Aug 19 18:45:50:I:System: Interface ethernet 2/2/4, state up
Aug 19 18:45:50:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (MakeFwding)
Aug 19 18:44:31:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:44:31:I:STP: VLAN 1 Port 2/2/4 STP State -> DISABLED (PortDown)
Aug 19 18:44:31:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (PortDown)
Aug 19 18:44:30:I:System: Interface ethernet 2/2/4, state up
Aug 19 18:44:30:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (MakeFwding)
Code:
Aug 19 18:51:38:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:51:37:I:System: Interface ethernet 2/2/4, state up
Aug 19 18:51:22:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:51:22:I:System: Interface ethernet 2/2/4, state upThanks for your reply itronin.
Code:
SSH@ICX6610-24 Switch#sh run
Current configuration:
!
ver 08.0.30tT7f1
!
stack unit 1
module 1 icx6610-24-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
priority 128
stack-port 1/2/1 1/2/6
stack unit 2
module 1 icx6610-24-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack-port 2/2/1 2/2/6
stack enable
stack mac xxxx.xxxx.xx74
!
!
!
lag xxxx dynamic id 1
ports ethernet 1/3/8 ethernet 2/3/8
primary-port 1/3/8
deploy
port-name xxxx ethernet 1/3/8
port-name xxxx ethernet 2/3/8
!
lag xxxx dynamic id 2
ports ethernet 1/3/3 ethernet 2/3/3
primary-port 1/3/3
deploy
!
lag xxxx dynamic id 3
ports ethernet 1/3/4 ethernet 2/3/4
primary-port 1/3/4
deploy
!
!
vlan 1 name DEFAULT-VLAN by port
!
vlan xxxx name xxxx by port
tagged ethe 1/3/1 ethe 1/3/8 ethe 2/3/1 ethe 2/3/8
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/3/1 ethe 1/3/8 ethe 2/3/1 ethe 2/3/8
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
spanning-tree 802-1w
!
!
!
!
!
aaa authentication enable default local
aaa authentication login default local
boot sys fl sec
jumbo
enable telnet authentication
ip address x.x.x.x x.x.x.x
ip dns server-address 8.8.8.8
no ip dhcp-client enable
ip default-gateway x.x.x.x
!
username xxxx password .....
!
!
clock timezone gmt GMT+03
!
!
ntp
disable serve
server 192.36.143.130
server 213.251.52.185
!
!
hitless-failover enable
interface ethernet 1/1/17
dual-mode xxxx
!
interface ethernet 1/2/1
port-name xxxx
!
interface ethernet 1/3/1
port-name xxxx
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
speed-duplex 10G-full
!
interface ethernet 1/3/5
dual-mode xxxx
speed-duplex 10G-full
!
interface ethernet 1/3/6
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
port-name xxxx
speed-duplex 10G-full
!
interface ethernet 2/2/4
no spanning-tree
!
interface ethernet 2/3/1
speed-duplex 10G-full
!
interface ethernet 2/3/2
speed-duplex 10G-full
!
interface ethernet 2/3/5
speed-duplex 10G-full
!
interface ethernet 2/3/6
speed-duplex 10G-full
!
interface ethernet 2/3/7
speed-duplex 10G-full
!
!
!
!
!
!
ip ssh timeout 60
!
!
endI have implemented the @RoachedCoach Mechatronics and Sunon fan mods discussed here on a ICX7250-48P. The new fans are definitely quieter, but I am not getting the results that others have. From a cold boot, the ASIC temps rise to mid-60's which is no problem, however the air outlet temps rise to the speed 2 threshold, the fans go to high and the temp drops to exactly 34c, the threshold for dropping back to speed 1. So once speed 2 becomes active, it never drops back. I tried just running with 1 fan, air outlet temps go into the 60's, but still have the one fan running at high speed. With the fan(s) running at high speed, the 7250-48P is louder than a 6610-24F with 2 Rev. A power supplies.
Due to a comedy of seller errors, I also have a 7250-24 and 7250-24P with dead PoE. The 7250-24 with its single OEM fan runs happily in the same rack with the fan at low speed. But I am not fond of the idea of going back to PoE power bricks.
Any ideas on getting a 7250-48P cool enough so the fans run at low speed?
Code:
Fan 1 ok, speed (auto): 1<->[[2]]
Fan 2 ok, speed (auto): 1<->[[2]]
Fan 3 ok, speed (auto): 1<->[[2]]
Fan controlled temperature:
Rule 1/2 (MGMT THERMAL PLANE): 51.7 deg-C
Rule 2/2 (AIR OUTLET NEAR PSU): 34.0 deg-C
Fan speed switching temperature thresholds:
Rule 1/2 (MGMT THERMAL PLANE):
Speed 1: NM<-----> 95 deg-C
Speed 2: 85<----->105 deg-C (shutdown)
Rule 2/2 (AIR OUTLET NEAR PSU)***active***:
Speed 1: NM<-----> 41 deg-C
Speed 2: 34<----->105 deg-C (shutdown)
Fan 1 Air Flow Direction: Front to Back
Fan 2 Air Flow Direction: Front to Back
Fan 3 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 51.7 deg-C (Sensor 1), 34.0 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-CDue to a comedy of seller errors, I also have a 7250-24 and 7250-24P with dead PoE. The 7250-24 with its single OEM fan runs happily in the same rack with the fan at low speed. But I am not fond of the idea of going back to PoE power bricks.
Any ideas on getting a 7250-48P cool enough so the fans run at low speed?
I just finished doing the exact same modification this afternoon on my 7250-48P but instead of the Sunon fan I replaced it with an EFB0612LA-R00 because Digi-Key was out of stock on the Sunon.
Mods and temps:
24P - 2 fans
48P - 3 fans + ASIC
I'd love to be able to pull the fan speed RPMs via SNMP but can't find an OID
Last edited:
Having replaces 2/3 fans with Noctua NF-A4x20 FLX, reqiring the pins as others suggested (black -> black, red -> yellow, yellow -> green), the fans show as "failed" - is this all good? The fans are spinning at a very low RPM but temps look OK (haven't gone further than idle);
Code:
ICX6450-48P Switch#show chassis
The stack unit 1 chassis info:
Power supply 1 (NA - AC - Regular) present, status ok
Power supply 2 not present
Power supply 3 not present
Fan 1 failed
Fan 2 failed
Fan 3 failed
Fan controlled temperature: 41.5 deg-C
Fan speed switching temperature thresholds:
Speed 1: NM<----->61 deg-C
Speed 2: 56<-----> 79 deg-C (shutdown)
Sensor B Temperature Readings:
Current temperature : 37.0 deg-C
Sensor A Temperature Readings:
Current temperature : 41.5 deg-C
Warning level.......: 69.0 deg-C
Shutdown level......: 79.0 deg-Cit's probably showing failed because the fans are spinning slower than the "failed" threshold RPM. to check if it's still getting RPM data just run "dm fan-speed". Either way on the 6450 it doesn't matter, as long as your temps are good. if it were a 7 series it'd be refusing to boot because of it but the 6 series doesn't give a shiza about no dang ole fans
Code:
All fans in Low speed (0x83)BTW, my power meter between the switch and outlet reports only 97-99V, any idea how sensitive it is for undervoltage? Old house with likely shitty wiring.
Looks like you have all of your VLANs tagged on just about every port on these switches. Is that what you mean to do?
I only configure ports that way when I need to use them for trunking to another switch, or to pfSense. Most of my ports are set up with a single, untagged VLAN. Here's a snippet from my 6610-48 config, where you can see that ports 1/1/1, 1/1/2, 1/3/,1, and 1/3/2 are 'trunk' ports for connecting switches and everything else is for connecting edge devices:
Code:
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 10 name Secure by port
tagged ethe 1/1/1 to 1/1/2 ethe 1/3/1 to 1/3/2
untagged ethe 1/1/3 to 1/1/5 ethe 1/1/7 ethe 1/1/9 to 1/1/11 ethe 1/1/13 to 1/1/36 ethe 1/2/2 to 1/2/5 ethe 1/3/3 to 1/3/5
router-interface ve 10
!
vlan 20 name Gizmos by port
tagged ethe 1/1/1 to 1/1/2 ethe 1/3/1 to 1/3/2
untagged ethe 1/1/37 to 1/1/38
router-interface ve 20
!
vlan 30 name WiFi by port
tagged ethe 1/1/1 to 1/1/2 ethe 1/3/1 to 1/3/2
untagged ethe 1/1/45 to 1/1/48
router-interface ve 30
!
vlan 40 name Cameras by port
tagged ethe 1/1/1 to 1/1/2 ethe 1/3/1 to 1/3/2
untagged ethe 1/1/39 to 1/1/44
router-interface ve 40
!
vlan 1000 name Storage by port
untagged ethe 1/1/6 ethe 1/1/8 ethe 1/1/12 ethe 1/2/7 to 1/2/10 ethe 1/3/6 to 1/3/8
router-interface ve 1000
!spec sheet for the OEM psu model number I have for the 6450 says 100vac - 130vac but it's probably a bit more lenient than that. That's really bad voltage drop though, is that circuit in your house really loaded down? Even with poor wiring you shouldn't be dropping nearly 30 volts between the breaker panel and the outlet unless you're doing like 20 amps over it. Can you measure the incoming power at your breaker panel or meter? If it's that low there that's usually bad enough that the power company will do something about it. you could be at the end of a very long run after the nearest stepdown transformer or something
That's a good recommendation, will try that!
I'm in Japan though, where 100V is standard, so the drop is not *that* drastic.
> Looks like you have all of your VLANs tagged on just about every port on these switches. Is that what you mean to do?
No.
After I've made the switches stack, some rear ports didn't work:
1/2/2 not working
1/2/3 working
1/2/4 working
1/2/5 not working
1/2/7 not working
1/2/8 not working
1/2/9 working
1/2/10 working
2/2/2 not working
2/2/3 not working
2/2/4 not working
2/2/5 not working
2/2/7 not working
2/2/8 working
2/2/9 working
2/2/10 working
This log shows what happens about not working ports:
Code:
Aug 19 18:45:51:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:45:51:I:STP: VLAN 1 Port 2/2/4 STP State -> DISABLED (PortDown)
Aug 19 18:45:51:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (PortDown)
Aug 19 18:45:50:I:System: Interface ethernet 2/2/4, state up
Aug 19 18:45:50:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (MakeFwding)
Aug 19 18:44:31:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:44:31:I:STP: VLAN 1 Port 2/2/4 STP State -> DISABLED (PortDown)
Aug 19 18:44:31:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (PortDown)
Aug 19 18:44:30:I:System: Interface ethernet 2/2/4, state up
Aug 19 18:44:30:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (MakeFwding)
Code:
Aug 19 18:51:38:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:51:37:I:System: Interface ethernet 2/2/4, state up
Aug 19 18:51:22:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:51:22:I:System: Interface ethernet 2/2/4, state upSo I finally registered after lurking for a bit, as I am getting into 10G networking at home.
This thread has been an amazing source of information! Thanks fohdeesha for sharing the information with us!
Managed to pick up a good condition ICX6610-48P-E for $300 Australia dollary doos off of eBay which I was impressed with.
I have just finished the multi vlan configuration and tested successfully, just PoE section left, and 10G stuff, very happy so far.
Before I put it in actual full time use considering modding the fans to 120/140mm versions with the use of some fan speed holes up top.
Have seen already a few attempts and they require the fan speed trickery so the switch even boots, frustrating but not a problem!
Is there any quicker solution or is ordering the parts and waiting the name of the game?
Asking just in case I have missed something, despite reading what must be almost all of the pages in this mega thread over the past few weeks since I decided to pull the trigger on 10G
Cheers!
This thread has been an amazing source of information! Thanks fohdeesha for sharing the information with us!
Managed to pick up a good condition ICX6610-48P-E for $300 Australia dollary doos off of eBay which I was impressed with.
I have just finished the multi vlan configuration and tested successfully, just PoE section left, and 10G stuff, very happy so far.
Before I put it in actual full time use considering modding the fans to 120/140mm versions with the use of some fan speed holes up top.
Have seen already a few attempts and they require the fan speed trickery so the switch even boots, frustrating but not a problem!
Is there any quicker solution or is ordering the parts and waiting the name of the game?
Asking just in case I have missed something, despite reading what must be almost all of the pages in this mega thread over the past few weeks since I decided to pull the trigger on 10G
Cheers!