Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[itronin]

Here's a used AOC breakout.

and some new ones.

I'm using this to go from one 6610 down to my servers from the 10Gbe breakouts.
I've used short NetAPP QSFP cables to stack 6610's.
I'm currently using short 40Gbe AOC (QSFP) to go to a TNC server with an HPE reflashed to Mellanox firmware 40Gbe adapter. works great. I think the AOC cables were like $15 each used.

You can certainly do the Xver's and fiber runs off the front longer individual patches and think you'll find that is not too bad $$ ... $8 per xver and $15-$25 for the cable?
If you are doing short runs off the front in your rack the used Cisco CUCM - length etc. tend to be between 9 and 15 each used and I've not had a problem with them.

 

[eduncan911]

Here's a used AOC breakout.
and some new ones.

I'm using this to go from one 6610 down to my servers from the 10Gbe breakouts.
I've used short NetAPP QSFP cables to stack 6610's.
I'm currently using short 40Gbe AOC (QSFP) to go to a TNC server with an HPE reflashed to Mellanox firmware 40Gbe adapter. works great. I think the AOC cables were like $15 each used.

You can certainly do the Xver's and fiber runs off the front longer individual patches and think you'll find that is not too bad $$ ... $8 per xver and $15-$25 for the cable?
If you are doing short runs off the front in your rack the used Cisco CUCM - length etc. tend to be between 9 and 15 each used and I've not had a problem with them.

Ah, those are the perfect cables and price! Perhaps these need to be added to that 3rd or 4th post on the first page, where it talks about the 6610 breakout.

 

[kapone]

Lot 14 DELL Force 10 Copper Breakout Cable QSFP+ 4X SFP Network 3M 27GG5 | eBay

Less than $10 per cable, although gotta buy 14 of them. And they work perfectly, I'm using them right now.

 

[neonclash]

I've got a strange mystery with stacking a pair of these I have finally been able to get my hands on. I have a 7150-C12P and a 7150-24P, and I can stack them just fine with some twinax in the SFP+ ports. However, I haven't been able to coax them into linking over 10g ethernet to get a stack across the apartment. When the SFP isn't set as a stack-port on the 24P, they will negotiate a link, and when neither is set, I can use that as a trunk. But the moment I set them both as stack ports, they start alternating between down and up on both ends every few seconds before settling into a state where the 12P thinks the link is up, but the 24p sees it as down.

This happens regardless of the combination of SFP+ module (from FLYPROFiber, I have one marked "for brocade" and two "for cisco"), stacking port (either of the two on both ends) and cable (I have tried all of the CAT6 and CAT7 I have). Attaching the 24p to itself (stacking 1/3/1 to nonstacking 1/3/2) forms a link, and so does connecting the 24p to itself. Do I need to purchase another transceiver, find another used switch, or resign myself to only being able to trunk them?

 

[BlackCat]

Does ICX 6450-48 (non-POE version) have to speed down fan after booting or not? I cannot hear any far RPM changes after boot (relative loud).

show chassis output:
The stack unit 1 chassis info:

Power supply 1 (NA - AC - Regular) present, status ok
Power supply 2 not present

Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
1 -> 2 @ 69 deg-C
1 <- 2 @ 64 deg-C

Sensor B Temperature Readings:
Current temperature : 38.0 deg-C
Sensor A Temperature Readings:
Current temperature : 45.5 deg-C
Warning level.......: 66.0 deg-C
Shutdown level......: 76.0 deg-C

 

[fohdeesha]

Does ICX 6450-48 (non-POE version) have to speed down fan after booting or not? I cannot hear any far RPM changes after boot (relative loud).

show chassis output:
The stack unit 1 chassis info:

Power supply 1 (NA - AC - Regular) present, status ok
Power supply 2 not present

Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
1 -> 2 @ 69 deg-C
1 <- 2 @ 64 deg-C

Sensor B Temperature Readings:
Current temperature : 38.0 deg-C
Sensor A Temperature Readings:
Current temperature : 45.5 deg-C
Warning level.......: 66.0 deg-C
Shutdown level......: 76.0 deg-C

fan machine broke

 

[richtj99]

Hi,

I bought a 7250-48p & did the setup. I have one question because I am paranoid on reboot I keep seeing this message:

Unmounting the External USB

I am using a serial connection (mini-usb to serial) with nothing else (I can see) plugged in. My ICX6450 doesnt have that sort of message & I dont see anything related to the above command.

Is this something that the ebay seller left behind on the unit or nothing?

ICX7250-48P Router#reload
Are you sure? (enter 'y' or 'n'): y
ICX7250-48P Router#Unmounting the External USB
The system is going down NOW!
Sent SIGTERM to all processes
[  568.590068] [BrcdSoftlockup]: sim_softwatchdog thread is detached on core=0
Sent SIGKILL to all processes
Requesting system reboot
[  570.200419] reboot: Restarting system
[  570.244267] Asserting GHPIO[7] reset sequence from Kernel ...
[  570.320073] Syetem Going Down ..

EDIT: Adding a bit more:

The Cntr +Y on the 7250 gives:

OS>show remote
  IP address      : 10.14.234.82
  subnet mask     : 255.255.255.0
  default gateway : 10.14.234.1
OS>

Starting the FastIron.
USER=root
HOME=/
GCOV_PREFIX=/fast_iron
ethaddr=78:a6:e1:0c:e5:a7?
TERM=vt102
PATH=/sbin:/usr/sbin:/bin:/usr/bin
ip=10.14.234.82:10.14.202.36:10.14.234.1:255.255.255.0:SI:eth0:off
SHELL=/bin/sh

NOTE: I did not put that IP in nor is the switch connected to anything other than a serial port - that IP range isnt something I use.

Doing the same thing on the 6450 gives:

OS>show remote
  IP address      : 0.0.0.0
  subnet mask     : 0.0.0.0
  default gateway : 0.0.0.0
OS>

Is there leftover stuff on the 7250 somehow? I factory reset it a second time & am a bit confused.

 

[fohdeesha]

Hi,

I bought a 7250-48p & did the setup. I have one question because I am paranoid on reboot I keep seeing this message:

Unmounting the External USB

I am using a serial connection (mini-usb to serial) with nothing else (I can see) plugged in. My ICX6450 doesnt have that sort of message & I dont see anything related to the above command.

Is this something that the ebay seller left behind on the unit or nothing?

ICX7250-48P Router#reload
Are you sure? (enter 'y' or 'n'): y
ICX7250-48P Router#Unmounting the External USB
The system is going down NOW!
Sent SIGTERM to all processes
[  568.590068] [BrcdSoftlockup]: sim_softwatchdog thread is detached on core=0
Sent SIGKILL to all processes
Requesting system reboot
[  570.200419] reboot: Restarting system
[  570.244267] Asserting GHPIO[7] reset sequence from Kernel ...
[  570.320073] Syetem Going Down ..

EDIT: Adding a bit more:

The Cntr +Y on the 7250 gives:

OS>show remote
  IP address      : 10.14.234.82
  subnet mask     : 255.255.255.0
  default gateway : 10.14.234.1
OS>

Starting the FastIron.
USER=root
HOME=/
GCOV_PREFIX=/fast_iron
ethaddr=78:a6:e1:0c:e5:a7?
TERM=vt102
PATH=/sbin:/usr/sbin:/bin:/usr/bin
ip=10.14.234.82:10.14.202.36:10.14.234.1:255.255.255.0:SI:eth0:off
SHELL=/bin/sh

NOTE: I did not put that IP in nor is the switch connected to anything other than a serial port - that IP range isnt something I use.

Doing the same thing on the 6450 gives:

OS>show remote
  IP address      : 0.0.0.0
  subnet mask     : 0.0.0.0
  default gateway : 0.0.0.0
OS>

Is there leftover stuff on the 7250 somehow? I factory reset it a second time & am a bit confused.

that's all standard stuff, the internal flash on the 7 series is eUSB (soldered), it's referring to how it's connected to the ARM CPU, not like a thumb drive like you're thinking

 

[richtj99]

OK Good to hear - the preset remote IP address is OK also? I edited the post with my boot/remote which I did not input.

OS>show remote
  IP address      : 10.14.234.82
  subnet mask     : 255.255.255.0
  default gateway : 10.14.234.1
OS>

Starting the FastIron.
USER=root
HOME=/
GCOV_PREFIX=/fast_iron
ethaddr=78:a6:e1:0c:e5:a7?
TERM=vt102
PATH=/sbin:/usr/sbin:/bin:/usr/bin
ip=10.14.234.82:10.14.202.36:10.14.234.1:255.255.255.0:SI:eth0:off
SHELL=/bin/sh

 

[LodeRunner]

My 7450 is showing this series of events every now and then:

Apr 20 17:30:13:I:System: Stack unit 1 POE  Power supply 2  with 748000 mwatts capacity is up  
Apr 20 17:30:12:I:System: Stack unit 1 have AC and DC PSUs, this is not supported 
Apr 20 17:30:11:I:System: Stack unit 1 have AC and DC PSUs, this is not supported 
Apr 20 17:30:10:I:System: Stack unit 1 have AC and DC PSUs, this is not supported 
Apr 20 17:30:09:C:System: Stack unit 1 POE  Power supply 2  with 748000 mwatts capacity is down

Is this a sign PSU2 is dying or a possible board/internal power distribution issue?
There's no PoE load on this switch right now as I haven't re-run the cables from my APs.
PSU 2 is fed from a UPS protected PDU, the cord is secure at both ends, and nothing else on the PDU is reporting any power issues.

 

[eduncan911]

PSU 2 is fed from a UPS protected PDU(snip)

OT: I've never considered this. Are you running the primary PSUs across your devices directly off of your line AC or another PDU connected to AC outlets (switches, servers, etc)? IOW, is there no UPS to the primary PSUs?

If so, what's the reason? To save the UPS battery? Never even considered this but could that potentially lengthen the life of UPS batteries?

Unfortunately here in our home, we have dirty (and low) voltage. So I have to use a sign wave UPS to get a clean signal to everything. Then again, I could get a PSU that cleans up the signal with caps.

 

[eduncan911]

So... I am joining the ICX6610 team. Picked up a couple of ICX6610-24P-Es from ebay, those $79+$40 s&h for sale right now. Made a deal to get two @ $80 for $50 shipping, $210 total. Seemed really inexpensive so I hope they are fine, and that the seller is reputable.

The ones he's auctioning have rack ears, a single PSU (unknown Rev), and a single fan unit, both E units. From what I read on this thread, that seems to be ok - as long as they are in the LEFT modules (viewed from the back).

I'll post up my 7250-48P for sale whenever I get these stacked, configured, and installed.

Thanks to everyone who suggested the inexpensive QSFP+ breakout cables and NetApp direct ones.

 

[LodeRunner]

OT: I've never considered this. Are you running the primary PSUs across your devices directly off of your line AC or another PDU connected to AC outlets (switches, servers, etc)? IOW, is there no UPS to the primary PSUs?

If so, what's the reason? To save the UPS battery? Never even considered this but could that potentially lengthen the life of UPS batteries?

Unfortunately here in our home, we have dirty (and low) voltage. So I have to use a sign wave UPS to get a clean signal to everything. Then again, I could get a PSU that cleans up the signal with caps.

I should have clarified, both PSUs are fed from PDUs on battery; each PDU has a separate UPS. Line voltage is typically 109 to 113 and AVR is boosting to 120. My local utility is typically rock steady at 60 Hz, I very rarely see it dip to 59.9.

Edit: Now that I think about it though, one UPS is an older BR1500G model and the other UPS is a SRT1500RMXLA; I will be getting a second SRT1500 to replace the BR1500G this month. Both units have AVR.

Further edit: incoming voltage today is 105.5 to 106; the SRT1500 is in AVR mode boosting to 120; the BR1500G is passing it through untouched. Which makes sense because the SRT is a true double-conversion topology and the BR1500G is an online topology.

 

[richtj99]

im poking around the linux portion of the switch - where would i find where this is hard coded? It sticks after a factory reset:

OS>show remote
IP address : 10.14.234.82
subnet mask : 255.255.255.0
default gateway : 10.14.234.1
OS>

 

[LodeRunner]

im poking around the linux portion of the switch - where would i find where this is hard coded? It sticks after a factory reset:

OS>show remote
IP address : 10.14.234.82
subnet mask : 255.255.255.0
default gateway : 10.14.234.1
OS>

That's probably being read from the uBoot environment. In uBoot, what's the output of printenv?

 

[NablaSquaredG]

I'm just about to buy two ICX6610-48P-E - They're totally overkill for the project, but I don't care.

The two 6610s will be connected to two SX6036 - SX6036 are the core network, Brocades provide network to all the other devices.
Redundancy and fault tolerance is extremely important.

My first plan was to connect the two 6610s with two SFP+ lines (different ports on 6610 and the SX6036 to account for port failure) to each switch. So each ICX6610 switch would have 4 SFP+ links (2 per SX6036) in total.
But I've seen that the Brocade also has 4 QSFP+ ports - Are two of them only for stacking? Because if they worked just like generic ports, I could save on the Mellanox QSA Adaptors and use QSFP+ link instead of SFP+ link.

If I use SFP+ links - Would stacking the 6610s make sense?

What would you recommend?

What are the best beginners tips I should follow?
What is the best source of information for the really nasty stuff (like Layer 3 routing)?

 

[eduncan911]

Searched through the thread on several QSFP+ cables, but couldn't find a direct answer.

Would this cable work for stacking two 6610s? New HP Mellanox 670760-B23 SPS-CA Active Optical Fiber 40GE QSFP 7M 670857-001 | eBay

Could someone stack across the QSFP+ 4x10G Breakout ports (found fohdeesha's saying how to do that across two 6610s, but not 6610<->6450).

And, could one use the 40Gbps ports strictly for server/workstation connections? I read where several 40G QSFP cards have issues with 4x10G links? Found on fohdeesha's website.

 

[eduncan911]

well, he's asking about using the breakout ports for a MIXED stack (eg to a ICX6450) and I'm about 99% sure it won't let you do that. There's some mixed stacking port commands that I'm pretty sure won't even let you use the breakout port numbers as an argument

keep in mind mixed stacking with an ICX6450 and a 6610, the 6450 basically just acts as a port extender for a 6610. Everything works great and you get the unified CLI, but the 6450 offloads all the l3 work to the 6610, and if you lose the 6610, the 6450 will no longer do much if anything. It's not an HA thing like two 6610s

(emphasis mine)

I plan on stacking two ICX6610s in some form of extended/ring mode (not failover). As I understand it, I can only use PoE ports on one device if not in failover/master-slave mode.

Could someone use two ICX6610 in this configuration, with a single ICX6450 as part of the stack?

Had the idea that I could use an ICX6450 in the office, and possibly stack it across 2x10G ports - one to each ICX6610. But if one 6610 reboots, I wouldn't want everything go offline. Would the ICX6450 use the other 6610?

 

[itronin]

(emphasis mine)

I plan on stacking two ICX6610s in some form of extended/ring mode (not failover). As I understand it, I can only use PoE ports on one device if not in failover/master-slave mode.

Could someone use two ICX6610 in this configuration, with a single ICX6450 as part of the stack?

Had the idea that I could use an ICX6450 in the office, and possibly stack it across 2x10G ports - one to each ICX6610. But if one 6610 reboots, I wouldn't want everything go offline. Would the ICX6450 use the other 6610?

that AOC you linked should work. Cheapest is just a pair of NETAPP QSFP (so copper DAC basically) at about $10.00 each shipped.
also lowest power and heat... just saying... How far apart will the switches be?

You can do something like 1x40Gbe interface for stacking, with a 1x4x10Gbe interface for stacking on each switch and then you'd have a total of 2 x40Gbe (one on each switch) for a server with redundancy or two servers, and then 2 x 4x10Gbe breakouts off the back for for say SFP+ to servers and then 2 x 8 10Gbe SFP+ off the front. That config is a little fiddly as I recall with a switch reboot or two to get things correct...

 

[Rex Manning]

I purchased a 6610 recently and apparently in it's previous life it was id 2 in a stack. Is there a way to change this stack id to 1? I tried stack secure-setup but it just reports there are no neighbors and exits. More of an annoyance than impactful but I thought I'd ask.

I flashed bootloader and os per fohdeesha's guide, factory reset, reprovision modules to slot 1-3 (denied, switch is active), unconfigure and stack secure-setup.

TIA