Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[|DSI|]

Thanks to this thread (=fohdeesha), I've managed to buy 3 ICX6450-24P units, which will be replacing my current mixed setup of access switches.
Being located in EU it wasn't that cheap (180 EUR/ea inc. postage and taxes), but judging from early experience it was well worth.

I'm now already looking for a pair of 6610-24P to replace my ageing core switches and upgrade to 40G, but it seems that they are hard to find in EU.

 

[DRAGONKZ]

@fohdeesha Since mine is officially dead (new powersupply didn't fix it, still shows no sign of life other than raging fans)

Have an image https://i.imgur.com/xPiiEMk.jpg
Any other chips you need a high res shot of? I think imgur is downscaling them, and until the replacement switch is in on monday, I won't have a place to host the full res ones.

Want to sell some of the heatsink pins?

My heatsink with a broken one either appears to be using a tim pad, or some form of epoxy as it won’t budge (without a lot of force)

Did your heatsinks just pull off easily once the pins were out?

 

[zanechua]

Bought 2 x ICX7150-C12P-2X1G from RUCKUS Switch ICX7150-C12P-2X1G w/ 12×10 PoE+ 2×1 uplink-ports and 2×1 GbE SFP 98379122171 | eBay for $250 each.

Not sure if they'll go lower. That was my first offer.

 

[Zombielinux]

Want to sell some of the heatsink pins?

My heatsink with a broken one either appears to be using a tim pad, or some form of epoxy as it won’t budge (without a lot of force)

Did your heatsinks just pull off easily once the pins were out?

Sure! Send me a PM and let me know where they're going. They kinda did, the trick with heatsinks like those is to twist rather than pry, you're less likely to ruin the chips underneath.

 

[Todd F]

First off, I'd like to thank all those traveling this road before me. Lots of useful information. Additionally, shout out to Fohdeesha for the amazing resource and help.

I wanted to add some information about getting the console to work on an ICX7250. I did not own a 'Cisco console cable', e.g. USB -> RJ45 serial. I had a couple RS232 to USB adapters I'd used with microcontrollers, etc. Those did not work. All I got was screens of gibberish text no matter what terminal settings I used.

What worked was:
- 'Cisco console cable' - just a generic one linked many pages ago.
- Mini-USB to 5-pin motherboard adapter. Basically just a cable that already is pinned out to a connector.
- RJ45 breakout board
- Header pins to solder to RJ45 breakout.

Simply repined the USB connector to match the RJ45 console cable et voila, working console!

I have a 24 port model, so I should probably just cut a hole in the front panel and install my breakout there.

 

[Smbaker]

Can an icx6450 do 5gbe with a copper module in the sfp+ slot?

Thanks,
Scott

 

[DRAGONKZ]

I’ve tracked down some “new old stock” 4 post mounting kits for the 6610 in Australia, if anyone in AUS is after some.

They aren’t necessarily cheap though!

 

[fohdeesha]

Can an icx6450 do 5gbe with a copper module in the sfp+ slot?

Thanks,
Scott

no

 

[juey]

1.) Is it possible to stack the 6610 using one of the 40G ports and one breakout port ?
tried that before, but i'm only getting the stack-trunk work with one 40G and one of the 10G breakout ports (e.g. 1/2/1 and 1/2/2, not 1/2/1 & 1/2/2 to 1/2/5).

2.) What does the led "MS" mean ? On one of my stack members, its green, and led on the other one is orange.
nvmd. i read the hardware installation guide, its the stacking role, green -> master, orange -> standby​

3.) My 2nd psu from my newly arrived switch is revving up and down, i tried to switch the psus but nothing changed. the 2nd fan tray (the left one, looking from behind) is also not working (no green light), is my board damaged ?

Its the left side, left psu revving up and down, left fan not working, anything damaged ? Can it be debugged ?

Here is the show chassis cmd (i deleted the mac addresses and X'd out the serials), stack unit 1 is my perfectly working 24p-e, stack unit 2 the new 48-e, i've plugged the second fan tray in the 24p-e, so currently the 48-e is only running one fan tray:

SSH@sw-core#sh chassis
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
        Model Number:   23-0000142-02
        Serial Number:  XXX
        Firmware Ver:    A
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 46.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->78       deg-C
                Speed 2:       73<-----> 87 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
        Current temperature : 41.5 deg-C
CPU Temperature Readings:
        Current temperature : 40.0 deg-C
sensor A Temperature Readings:
        Current temperature : 26.5 deg-C
sensor B Temperature Readings:
        Current temperature : 39.0 deg-C
sensor C Temperature Readings:
        Current temperature : 20.0 deg-C
sensor D Temperature Readings:
        Current temperature : 18.0 deg-C
stacking card Temperature Readings:
        Current temperature : 46.0 deg-C
        Warning level.......: 84.0 deg-C
        Shutdown level......: 87.0 deg-C
Boot Prom MAC :
Management MAC:

The stack unit 2 chassis info:

Power supply 1 (AC - Regular) present, status ok
        Model Number:   23-0000144-01
        Serial Number:  XXX
        Firmware Ver:    B
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 (AC - Regular) present, status ok
        Model Number:   23-0000144-01
        Serial Number:  XXX
        Firmware Ver:    B
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 not present

Fan controlled temperature: 58.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->80       deg-C
                Speed 2:       75<-----> 85 deg-C (shutdown)

Fan 1 Air Flow Direction:Front to Back
MAC 1 Temperature Readings:
        Current temperature : 44.5 deg-C
MAC 2 Temperature Readings:
        Current temperature : 58.0 deg-C
CPU Temperature Readings:
        Current temperature : 57.5 deg-C
sensor A Temperature Readings:
        Current temperature : 50.5 deg-C
sensor B Temperature Readings:
        Current temperature : 48.5 deg-C
sensor C Temperature Readings:
        Current temperature : 37.5 deg-C
stacking card Temperature Readings:
        Current temperature : 55.0 deg-C
        Warning level.......: 81.0 deg-C
        Shutdown level......: 85.0 deg-C

 

[fohdeesha]

yes you can stack with the 40gbe port and breakout port, that's exactly how I have mine running at home. You only need to specify the first breakout port (1/2/2) and it automatically uses the entire 4 ports of that breakout. Your config should look like this:

Current configuration:
!
ver 08.0.30tT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  priority 128
  stack-trunk 1/2/1 to 1/2/2
  stack-port 1/2/1
stack unit 2
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-trunk 2/2/1 to 2/2/2
  stack-port 2/2/1

To verify it's using all 5 links (1 40gbe and 4 breakouts) run "show stack con", you should see 5 links total (the diagram won't show it, because they probably did not have this kind of custom stack connection in mind when making those diagrams):

Probing the topology. Please wait ...
telnet@ICX1>
    standby      active
     +---+        +---+
     | 2 |2/1==2/1| 1 |
     +---+        +---+

trunk probe results: 1 links
Link 1: u1 -- u2, num=5
  1: 1/2/1 (T0) <---> 2/2/1 (T0)
  2: 1/2/2 (T0) <---> 2/2/2 (T0)
  3: 1/2/3 (T0) <---> 2/2/3 (T0)
  4: 1/2/4 (T0) <---> 2/2/4 (T0)
  5: 1/2/5 (T0) <---> 2/2/5 (T0)
CPU to CPU packets are fine between 2 units.

no clue about your fan/hardware situation

 

[Smbaker]

no

Hmm, that's kind of what I was afraid the answer was going to be. I had visions of potentially using some QNA-UC5G1T USB3 5gbpe adapters, but sounds like that's not going to work. Wish that QNAP adapter had an SFP+ instead of copper and was physical at 10gbe even if it could still only sustain about 4gbe throughput on the usb side.

Going to have to decide whether I really want a 6450 or if I should go with something like a Mikrotik, which is a less capable switch and costs more money (new), but I think it can do the 1/2.5/5/10 copper SFPs.

 

[infoMatt]

Hmm, that's kind of what I was afraid the answer was going to be. I had visions of potentially using some QNA-UC5G1T USB3 5gbpe adapters, but sounds like that's not going to work.

What NAS do you have? Most of the times, you can get pretty much the maximum speed of the 2-3 bay models by LAGGing the two ethernet interfaces... The only scenario where 2Gbps of bandwidth could potentially not max out the drives is a 3+ bay with a RAID0 (so you'll have about 150-180MB/s per drive of burst write speed, assuming you are using "pro" drives @7200 rpm or more), but I'd bet that those processors cannot sustain a 2+Gbps stream to them via Eth on PCI... let alone via an USB adapter with the resource overhead of the USB bus...

 

[Smbaker]

What NAS do you have? Most of the times, you can get pretty much the maximum speed of the 2-3 bay models by LAGGing the two ethernet interfaces... The only scenario where 2Gbps of bandwidth could potentially not max out the drives is a 3+ bay with a RAID0 (so you'll have about 150-180MB/s per drive of burst write speed, assuming you are using "pro" drives @7200 rpm or more), but I'd bet that those processors cannot sustain a 2+Gbps stream to them via Eth on PCI... let alone via an USB adapter with the resource overhead of the USB bus...

Synology DS1019+. If I'd have spent a hundred bucks more and bought the DXS1618+ instead, it would have come with a PCIE slot and I could have thrown a 10G NIC in it, but I didn't... Live and learn. Running RAID6 on five WD RED NAS drives. Also has a SSD read cache in it. Using this to serve NFS, iSCSI to ESXi on a couple NUCs as well as SMB to my desktop. Performance to the NUCs is fine, but I would like to optimize performance to my desktop.

There is a guy who ported the driver for that QNAP 5Gbe USB3 adapter to the Synology, and tried it on a DS918+ (one less bay than mine) and was reporting ~ 434MB/s sustained transfers to and from it. I don't recall if he said what his CPU utilization was, but it sounded like at ~ 434MB/s it was limited by the USB3 link speed, not CPU. These were iperf numbers, so not sure what happens when you throw in physical disk performance. Then you have to figure how much of the working set fits into the NAS's RAM, and how much can be served by the SSD read cache. (Could do SSD write cache as well on this NAS, but too many people report write cache errors crashing their volume for me to be comfortable with that)

Once the 6450 arrives, I am going to try to utilize the NAS's two 1G interfaces, either bonding or SMB multichannel.

 

[Freebsd1976]

some sellers list icx7750-26q as is (no fan try no psu) and return is not acceptable , price 160usd, want give it a try.
but my question how to got fan tray and psu , what the price will be?
edit：
find another seller list new icx7750 also new but no fan trays no psu , only 110 usd

 

[ewer0012]

Howdy,

I just picked up a 6610 off of eBay from a random "Make Offer" of $100. Had 2 PSUs and 1 fan. It seems to be booting up fine into the boot monitor to update stuff, but when I follow the instructions Fohdeesha has to update it through TFTP and then go to boot it up, I get:

Monitor>ip address 10.0.1.73
  IP address = 10.0.1.73
  IP subnet mask = 255.255.255.0
Monitor>copy tftp flash 10.0.1.19 grz10100.bin boot
Loading image from Tftp server.................................................................Done
Programming boot flash, please wait..
Erasing....
Writing
Done
Monitor>copy tftp flash 10.0.1.19 FCXR08030t.bin primary
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done
.Monitor>factory set-default
This command will remove configuration and keys detail.
Do you want to continue? (Y/N) Y
Done.
Monitor>reset
$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 71f1
BOOT INFO: verify flash files........
BOOT INFO: load image from primary copy...

platform type = 12
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
..............................
Firmware integrity checksum passed
.......
Starting Main Task .Applying factory defaults..
INFO: startup config data is not available, try to read from backup
INFO: startup config data in the backup area is not available
..CPSS DxCh Version: cpss3.4p1 release

And then it stalls there.

Any help is appreciated. Hopefully, I didn't receive a dud.

 

[Wolfstar]

Was it booting before you did the firmware flashing? Have you tried booting from the secondary flash?

 

[ewer0012]

Was it booting before you did the firmware flashing? Have you tried booting from the secondary flash?

It wasn't, which is why I tried to update it. It doesn't boot from the secondary, either.

 

[Wolfstar]

Was it being sold as for-parts? If not I'd hit up the seller since it doesn't work. But yeah, sounds like it might be toast - maybe fohdeesha will know of something to work around there.

 

[fohdeesha]

definitely sounds broke to me

 

[ewer0012]

Nope, was sold as "Used".

Looks like I need to put in for a return No wonder they accepted the offer of $100 for it. Too good to be true.

Thanks for the ideas, I appreciate the help.