Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

fohdeesha

Kaini Industries
Nov 20, 2016
1,950
1,777
113
29
fohdeesha.com
yeah that's pretty crazy, I originally thought you just managed to get something bent or got a broken product, but when I ordered one and received it, it did the exact same thing. took a screwdriver to get it out. Looks like monoprice needs to work on their manufacturing tolerances, at least when it comes to QSFP's
 

PGlover

Active Member
Nov 8, 2014
468
55
28
54
I purchased the QSFP and Fan-out patch cable below. The QSFP does not seems to fit snug in the ICX 6610 QSFP slot. You can pull it out very easily. Additionally, the fan-out-patch cable does not stay in the QSFP. You can easily pull it out as well. What am I doing wrong?

Brocade XBR-000232 QSFP 57-1000267-01 4*16Gb SWL 100m for DCX8510 EMC | eBay

MPO Fan-out Patch Cable, Multimode Fiber, MPO to LC Uniboot, 3-meter 10FT 603149398665 | eBay


View attachment 9007
For some reason, ports 1/2/2 - 1/2/5 is not active using the following parts.. If there something special I need to do?

 

PGlover

Active Member
Nov 8, 2014
468
55
28
54
@PGlover - Dude...you're struggling so much with that port 1/2/2 that I'm inclined to just ship you a breakout cable (That I KNOW works), just to eliminate any cable nonsense. Seriously.

PM me your address.
What cable are you using? Send me a link to the cable you are using.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,950
1,777
113
29
fohdeesha.com
yeah I really don't get it at this point - it's worth re-iterating, I bought the exact same monoprice dac he had (the one that didn't work for him) and it worked first go in my breakout ports, regardless of config etc. I'm also using the exact optic and breakout cable he just linked, those work perfect here too - so I think he may have some other issue going on. As we talked about in PM, if you give me teamviewer access @PGlover to a serial console window to the switch I can poke around at the low level ASIC/OS level and see exactly wtf is going on
 
  • Like
Reactions: SRussell

kapone

Well-Known Member
May 23, 2015
796
388
63
@PGlover - At this point, all I can say is, what I said before. Sorry, out of ideas.

Well, I guess we have no choice but back to basics, if we want to troubleshoot this. But you have a live environment...so, your call.

- NO ROUTER in this test. Use static IPs.
- NO VLANS for this test, we're just testing. Port connection testing does not require VLANs
- Take one 6610, reset it to factory default. Connect breakout cable to 1/2/2 and connect at least two 10g devices. Confirm that both work at 10g.
- Take a second 6610, reset it to factory default. Connect breakout cable to 1/2/2 and connect at least two 10g devices. Confirm that both work at 10g.
- Stack both of them on the 40G ports. Connect the two devices to 1/2/2 and see what happens. Connect them to 2/2/2 and see what happens.
 
  • Like
Reactions: fohdeesha

billc.cn

Member
Oct 6, 2017
41
7
8
I am quite tempted by the L3 features of the 6450, but the only supplier I can find with a reasonable price is this eBay listing.

I contacted the seller, and he said: "this item comes without any special license", so I'd have to pay separately for the 2x 10G ports and the L3 features, neither of which are available at a reasonable price (<£1000) in Europe.

Now I think I will have to count on @fohdeesha still having some licenses available and they don't come in any physical form that will require mailing :S.

Alternatively, I have plans to travel to the States later this year and pick one up on the way; however, I wonder what's the likelihood that a random one on eBay would come with both licenses?

Additional question: does the routing feature support any source routing/basic firewall? I want to have the switch do inter-VLan routing, but only in certain directions
 
Last edited:

SantaSCSI

New Member
May 22, 2011
19
3
3
Small question onn the matter of ICX6450 10Gb licensing. It seems that by default ports 2 and 4 are not licensed for 10Gbit. Any way one can get his hands on a license for those 2 additional ports?
 

scurrier

New Member
Aug 4, 2018
18
23
3
Illinois, USA
In the ICX 6450 guide here one of the first steps is factory defaulting the switch at the boot prompt. But it doesn't seem to recognize this command. Any ideas?

Code:
ICX64XX-boot>> factory set-default
Unknown command 'factory' - try 'help'
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,950
1,777
113
29
fohdeesha.com
the VERY old bootloaders did not have that command yet, so alternatively in the bootloader you can run "no password" then "boot"

it'll boot and bypass any password the previous config has

when it boots fully run:

enable
erase startup-config
reload

it'll reboot into a clean empty config, same result as factory set-default but a little more work
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,950
1,777
113
29
fohdeesha.com
IAlternatively, I have plans to travel to the States later this year and pick one up on the way; however, I wonder what's the likelihood that a random one on eBay would come with both licenses?

Additional question: does the routing feature support any source routing/basic firewall? I want to have the switch do inter-VLan routing, but only in certain directions
re: licenses just send me a PM, it's a digital license file you import onto the switch. the chance of an ICX6450 from ebay coming with a license is basically slim to none

if by source routing you mean policy based routing, the ICX6450 does not have it (the 6610 does though). However it does have all the normal inter-vlan routing and static routes and ospf and vrrp and all that. It also has pretty thorough layer 3 & layer 4 ACL's if that's what you mean by firewall (it will NOT do NAT), you can block traffic by destination, source, udp/tcp/ port number etc. If you want inter-vlan routing but only in one direction, you can just use a couple simple ACL rules to drop everything in the other direction
 
Last edited:

AlphaG

Member
Jun 8, 2017
84
16
8
50
the VERY old bootloaders did not have that command yet, so alternatively in the bootloader you can run "no password" then "boot"

it'll boot and bypass any password the previous config has

when it boots fully run:

enable
erase startup-config
reload

it'll reboot into a clean empty config, same result as factory set-default but a little more work
Thanks. Looks like my bootloader is also very old. I was able to erase the startup config with this help and reload but the flash fails with a "TFTP session timed out. TFTP to Flash Error - code 5". Part of the problem may be that I am unable to disable the dhcp client as "disable" is not an available command and it is enabled by default. I thought I avoided this by assigning an ip after reload, but still get the flash error. I am running tftpd64 pointed at a root temp directory and the ethernet adapter for my Surface attached directly to the switch, which is wired directly to another switch on my network. The Surface is attached to port 6 and the linking ethernet cable is on port 5.

Also, when I "show ip address" on the switch, it is shown as 192.168.1.55, but the assignment is not visible on the DHCP leases on the pfsense router, not sure why. The address can be pinged with success however.

EDIT: Nevermind ... Amateur mistake, I'm sure. Not my laptop so forgot to turn off windows firewall. The flash is proceeding normally now.
 
Last edited:

aliasxneo

New Member
Aug 4, 2018
8
1
3
Replaced the fans in the 6450 (PoE) with 40mm Sunon's.

Code:
ICX6450-24P Router(config)#sh chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present

Fan 1 failed
Fan 2 failed

Fan controlled temperature: 61.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->64       deg-C
                Speed 2:       59<-----> 88 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 58.5 deg-C
Sensor A Temperature Readings:
        Current temperature : 61.5 deg-C
        Warning level.......: 78.0 deg-C
        Shutdown level......: 88.0 deg-C
Boot Prom MAC : 748e.f8d1.2fa0
Management MAC: 748e.f8d1.2fa0
The fans did run on first boot, but halfway through when the fans normally ramp down they just shut off completely. As seen above, they show as failed because they are of course not spinning. I haven't been able to get the temperature up to the speed 2 setpoint yet, but I'm concerned that they don't run at all.

Anyone else had this issue with replacements?

* Edit *

Got it to speed 2 set point and they did finally kick on. Still showed as failed even though they were actually running.

Also tried it with a Noctua I had laying around and had the same issue where the fan will not run when the switch is in the low speed setting.
 
Last edited:

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
I tried to get a similar Noctua work with both of my 24's (one standard, one PoE) and never had any luck. I probably shouldn't modify electronics after staying awake all night, I roasted a Noctua after getting the wrong stock fan spec sheet and reversing wire polarity. Oops.

The 6.3CFM 5200RPM Sunon I bought worked on the first try. I also bought one of these (10.2CFM 8200RPM) but haven't needed to use it as the 6.3CFM does a fine job in a stock -24.

I wonder if the fan headers have a minimum draw requirement before they switch on?
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
1,950
1,777
113
29
fohdeesha.com
Thanks. Looks like my bootloader is also very old. I was able to erase the startup config with this help and reload but the flash fails with a "TFTP session timed out. TFTP to Flash Error - code 5". Part of the problem may be that I am unable to disable the dhcp client as "disable" is not an available command and it is enabled by default. I thought I avoided this by assigning an ip after reload, but still get the flash error. I am running tftpd64 pointed at a root temp directory and the ethernet adapter for my Surface attached directly to the switch, which is wired directly to another switch on my network. The Surface is attached to port 6 and the linking ethernet cable is on port 5.

Also, when I "show ip address" on the switch, it is shown as 192.168.1.55, but the assignment is not visible on the DHCP leases on the pfsense router, not sure why. The address can be pinged with success however.

EDIT: Nevermind ... Amateur mistake, I'm sure. Not my laptop so forgot to turn off windows firewall. The flash is proceeding normally now.
glad you got it figured out - I was going to say, it must have came with VERY old firmware if "ip dhcp-client disable" was not a command. Don't forget to run it and everything else once you're on the new firmware

The fans did run on first boot, but halfway through when the fans normally ramp down they just shut off completely. As seen above, they show as failed because they are of course not spinning. I haven't been able to get the temperature up to the speed 2 setpoint yet, but I'm concerned that they don't run at all.
To run the fans slower, it just outputs a lower voltage. Speed 2 is the normal full 12v the fans are designed to run at, but at speed one, I'd imagine it's only providing 6 or 7 volts to the fan header (via PWM I'd imagine). For fans not designed to run at slower speeds, this is usually not even enough voltage to start them spinning. If you look up the datasheet for the stock fans on these switches, you'll see the "operating voltage" does extend down from 12v to 6 or 7. Another point to keep in mind when trying to outsmart the original engineers of this switch :)

The sunon models that @arglebargle mentioned work fine, because as you can see from their datasheet, they're designed to run all the way down to 4.5v. Some fan datasheets (like for this sunon for instance) are even kind enough to list the "minimum starting voltage". This number for your fan needs to be lower than whatever the header is putting out on speed 1
 

aliasxneo

New Member
Aug 4, 2018
8
1
3
I tried to get a similar Noctua work with both of my 24's (one standard, one PoE) and never had any luck. I probably shouldn't modify electronics after staying awake all night, I roasted a Noctua after getting the wrong stock fan spec sheet and reversing wire polarity. Oops.

The 6.3CFM 5200RPM Sunon I bought worked on the first try. I also bought one of these (10.2CFM 8200RPM) but haven't needed to use it as the 6.3CFM does a fine job in a stock -24.

I wonder if the fan headers have a minimum draw requirement before they switch on?
That second one you listed is the one I can't get to work on the low speed setting. I did look at the 6 CFM one but was concerned it wouldn't be enough airflow (my rack tends to run ~30c ambient).

I figured it had something to do with the rated voltage. This data sheet has the model number on it, but I don't see anything showing the full operating voltage. I'd be curious if you run into the same issue with it.

@fohdeesha, can you link the datasheet you were using? I have the second Sunon he listed and that's the one not working at the lower voltage.