Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[richtj99]

Hi,

I bought a 7250-48p & did the setup. I have one question because I am paranoid on reboot I keep seeing this message:

Unmounting the External USB

I am using a serial connection (mini-usb to serial) with nothing else (I can see) plugged in. My ICX6450 doesnt have that sort of message & I dont see anything related to the above command.

Is this something that the ebay seller left behind on the unit or nothing?

ICX7250-48P Router#reload
Are you sure? (enter 'y' or 'n'): y
ICX7250-48P Router#Unmounting the External USB
The system is going down NOW!
Sent SIGTERM to all processes
[  568.590068] [BrcdSoftlockup]: sim_softwatchdog thread is detached on core=0
Sent SIGKILL to all processes
Requesting system reboot
[  570.200419] reboot: Restarting system
[  570.244267] Asserting GHPIO[7] reset sequence from Kernel ...
[  570.320073] Syetem Going Down ..

EDIT: Adding a bit more:

The Cntr +Y on the 7250 gives:

OS>show remote
  IP address      : 10.14.234.82
  subnet mask     : 255.255.255.0
  default gateway : 10.14.234.1
OS>

Starting the FastIron.
USER=root
HOME=/
GCOV_PREFIX=/fast_iron
ethaddr=78:a6:e1:0c:e5:a7?
TERM=vt102
PATH=/sbin:/usr/sbin:/bin:/usr/bin
ip=10.14.234.82:10.14.202.36:10.14.234.1:255.255.255.0:SI:eth0:off
SHELL=/bin/sh

NOTE: I did not put that IP in nor is the switch connected to anything other than a serial port - that IP range isnt something I use.

Doing the same thing on the 6450 gives:

OS>show remote
  IP address      : 0.0.0.0
  subnet mask     : 0.0.0.0
  default gateway : 0.0.0.0
OS>

Is there leftover stuff on the 7250 somehow? I factory reset it a second time & am a bit confused.

 

[fohdeesha]

Hi,

I bought a 7250-48p & did the setup. I have one question because I am paranoid on reboot I keep seeing this message:

Unmounting the External USB

I am using a serial connection (mini-usb to serial) with nothing else (I can see) plugged in. My ICX6450 doesnt have that sort of message & I dont see anything related to the above command.

Is this something that the ebay seller left behind on the unit or nothing?

ICX7250-48P Router#reload
Are you sure? (enter 'y' or 'n'): y
ICX7250-48P Router#Unmounting the External USB
The system is going down NOW!
Sent SIGTERM to all processes
[  568.590068] [BrcdSoftlockup]: sim_softwatchdog thread is detached on core=0
Sent SIGKILL to all processes
Requesting system reboot
[  570.200419] reboot: Restarting system
[  570.244267] Asserting GHPIO[7] reset sequence from Kernel ...
[  570.320073] Syetem Going Down ..

EDIT: Adding a bit more:

The Cntr +Y on the 7250 gives:

OS>show remote
  IP address      : 10.14.234.82
  subnet mask     : 255.255.255.0
  default gateway : 10.14.234.1
OS>

Starting the FastIron.
USER=root
HOME=/
GCOV_PREFIX=/fast_iron
ethaddr=78:a6:e1:0c:e5:a7?
TERM=vt102
PATH=/sbin:/usr/sbin:/bin:/usr/bin
ip=10.14.234.82:10.14.202.36:10.14.234.1:255.255.255.0:SI:eth0:off
SHELL=/bin/sh

NOTE: I did not put that IP in nor is the switch connected to anything other than a serial port - that IP range isnt something I use.

Doing the same thing on the 6450 gives:

OS>show remote
  IP address      : 0.0.0.0
  subnet mask     : 0.0.0.0
  default gateway : 0.0.0.0
OS>

Is there leftover stuff on the 7250 somehow? I factory reset it a second time & am a bit confused.

that's all standard stuff, the internal flash on the 7 series is eUSB (soldered), it's referring to how it's connected to the ARM CPU, not like a thumb drive like you're thinking

 

[richtj99]

OK Good to hear - the preset remote IP address is OK also? I edited the post with my boot/remote which I did not input.

OS>show remote
  IP address      : 10.14.234.82
  subnet mask     : 255.255.255.0
  default gateway : 10.14.234.1
OS>

Starting the FastIron.
USER=root
HOME=/
GCOV_PREFIX=/fast_iron
ethaddr=78:a6:e1:0c:e5:a7?
TERM=vt102
PATH=/sbin:/usr/sbin:/bin:/usr/bin
ip=10.14.234.82:10.14.202.36:10.14.234.1:255.255.255.0:SI:eth0:off
SHELL=/bin/sh

 

[LodeRunner]

My 7450 is showing this series of events every now and then:

Apr 20 17:30:13:I:System: Stack unit 1 POE  Power supply 2  with 748000 mwatts capacity is up  
Apr 20 17:30:12:I:System: Stack unit 1 have AC and DC PSUs, this is not supported 
Apr 20 17:30:11:I:System: Stack unit 1 have AC and DC PSUs, this is not supported 
Apr 20 17:30:10:I:System: Stack unit 1 have AC and DC PSUs, this is not supported 
Apr 20 17:30:09:C:System: Stack unit 1 POE  Power supply 2  with 748000 mwatts capacity is down

Is this a sign PSU2 is dying or a possible board/internal power distribution issue?
There's no PoE load on this switch right now as I haven't re-run the cables from my APs.
PSU 2 is fed from a UPS protected PDU, the cord is secure at both ends, and nothing else on the PDU is reporting any power issues.

 

[eduncan911]

PSU 2 is fed from a UPS protected PDU(snip)

OT: I've never considered this. Are you running the primary PSUs across your devices directly off of your line AC or another PDU connected to AC outlets (switches, servers, etc)? IOW, is there no UPS to the primary PSUs?

If so, what's the reason? To save the UPS battery? Never even considered this but could that potentially lengthen the life of UPS batteries?

Unfortunately here in our home, we have dirty (and low) voltage. So I have to use a sign wave UPS to get a clean signal to everything. Then again, I could get a PSU that cleans up the signal with caps.

 

[eduncan911]

So... I am joining the ICX6610 team. Picked up a couple of ICX6610-24P-Es from ebay, those $79+$40 s&h for sale right now. Made a deal to get two @ $80 for $50 shipping, $210 total. Seemed really inexpensive so I hope they are fine, and that the seller is reputable.

The ones he's auctioning have rack ears, a single PSU (unknown Rev), and a single fan unit, both E units. From what I read on this thread, that seems to be ok - as long as they are in the LEFT modules (viewed from the back).

I'll post up my 7250-48P for sale whenever I get these stacked, configured, and installed.

Thanks to everyone who suggested the inexpensive QSFP+ breakout cables and NetApp direct ones.

 

[LodeRunner]

OT: I've never considered this. Are you running the primary PSUs across your devices directly off of your line AC or another PDU connected to AC outlets (switches, servers, etc)? IOW, is there no UPS to the primary PSUs?

If so, what's the reason? To save the UPS battery? Never even considered this but could that potentially lengthen the life of UPS batteries?

Unfortunately here in our home, we have dirty (and low) voltage. So I have to use a sign wave UPS to get a clean signal to everything. Then again, I could get a PSU that cleans up the signal with caps.

I should have clarified, both PSUs are fed from PDUs on battery; each PDU has a separate UPS. Line voltage is typically 109 to 113 and AVR is boosting to 120. My local utility is typically rock steady at 60 Hz, I very rarely see it dip to 59.9.

Edit: Now that I think about it though, one UPS is an older BR1500G model and the other UPS is a SRT1500RMXLA; I will be getting a second SRT1500 to replace the BR1500G this month. Both units have AVR.

Further edit: incoming voltage today is 105.5 to 106; the SRT1500 is in AVR mode boosting to 120; the BR1500G is passing it through untouched. Which makes sense because the SRT is a true double-conversion topology and the BR1500G is an online topology.

 

[richtj99]

im poking around the linux portion of the switch - where would i find where this is hard coded? It sticks after a factory reset:

OS>show remote
IP address : 10.14.234.82
subnet mask : 255.255.255.0
default gateway : 10.14.234.1
OS>

 

[LodeRunner]

im poking around the linux portion of the switch - where would i find where this is hard coded? It sticks after a factory reset:

OS>show remote
IP address : 10.14.234.82
subnet mask : 255.255.255.0
default gateway : 10.14.234.1
OS>

That's probably being read from the uBoot environment. In uBoot, what's the output of printenv?

 

[NablaSquaredG]

I'm just about to buy two ICX6610-48P-E - They're totally overkill for the project, but I don't care.

The two 6610s will be connected to two SX6036 - SX6036 are the core network, Brocades provide network to all the other devices.
Redundancy and fault tolerance is extremely important.

My first plan was to connect the two 6610s with two SFP+ lines (different ports on 6610 and the SX6036 to account for port failure) to each switch. So each ICX6610 switch would have 4 SFP+ links (2 per SX6036) in total.
But I've seen that the Brocade also has 4 QSFP+ ports - Are two of them only for stacking? Because if they worked just like generic ports, I could save on the Mellanox QSA Adaptors and use QSFP+ link instead of SFP+ link.

If I use SFP+ links - Would stacking the 6610s make sense?

What would you recommend?

What are the best beginners tips I should follow?
What is the best source of information for the really nasty stuff (like Layer 3 routing)?

 

[eduncan911]

Searched through the thread on several QSFP+ cables, but couldn't find a direct answer.

Would this cable work for stacking two 6610s? New HP Mellanox 670760-B23 SPS-CA Active Optical Fiber 40GE QSFP 7M 670857-001 | eBay

Could someone stack across the QSFP+ 4x10G Breakout ports (found fohdeesha's saying how to do that across two 6610s, but not 6610<->6450).

And, could one use the 40Gbps ports strictly for server/workstation connections? I read where several 40G QSFP cards have issues with 4x10G links? Found on fohdeesha's website.

 

[eduncan911]

well, he's asking about using the breakout ports for a MIXED stack (eg to a ICX6450) and I'm about 99% sure it won't let you do that. There's some mixed stacking port commands that I'm pretty sure won't even let you use the breakout port numbers as an argument

keep in mind mixed stacking with an ICX6450 and a 6610, the 6450 basically just acts as a port extender for a 6610. Everything works great and you get the unified CLI, but the 6450 offloads all the l3 work to the 6610, and if you lose the 6610, the 6450 will no longer do much if anything. It's not an HA thing like two 6610s

(emphasis mine)

I plan on stacking two ICX6610s in some form of extended/ring mode (not failover). As I understand it, I can only use PoE ports on one device if not in failover/master-slave mode.

Could someone use two ICX6610 in this configuration, with a single ICX6450 as part of the stack?

Had the idea that I could use an ICX6450 in the office, and possibly stack it across 2x10G ports - one to each ICX6610. But if one 6610 reboots, I wouldn't want everything go offline. Would the ICX6450 use the other 6610?

 

[itronin]

(emphasis mine)

I plan on stacking two ICX6610s in some form of extended/ring mode (not failover). As I understand it, I can only use PoE ports on one device if not in failover/master-slave mode.

Could someone use two ICX6610 in this configuration, with a single ICX6450 as part of the stack?

Had the idea that I could use an ICX6450 in the office, and possibly stack it across 2x10G ports - one to each ICX6610. But if one 6610 reboots, I wouldn't want everything go offline. Would the ICX6450 use the other 6610?

that AOC you linked should work. Cheapest is just a pair of NETAPP QSFP (so copper DAC basically) at about $10.00 each shipped.
also lowest power and heat... just saying... How far apart will the switches be?

You can do something like 1x40Gbe interface for stacking, with a 1x4x10Gbe interface for stacking on each switch and then you'd have a total of 2 x40Gbe (one on each switch) for a server with redundancy or two servers, and then 2 x 4x10Gbe breakouts off the back for for say SFP+ to servers and then 2 x 8 10Gbe SFP+ off the front. That config is a little fiddly as I recall with a switch reboot or two to get things correct...

 

[Rex Manning]

I purchased a 6610 recently and apparently in it's previous life it was id 2 in a stack. Is there a way to change this stack id to 1? I tried stack secure-setup but it just reports there are no neighbors and exits. More of an annoyance than impactful but I thought I'd ask.

I flashed bootloader and os per fohdeesha's guide, factory reset, reprovision modules to slot 1-3 (denied, switch is active), unconfigure and stack secure-setup.

TIA

 

[fohdeesha]

(emphasis mine)

I plan on stacking two ICX6610s in some form of extended/ring mode (not failover). As I understand it, I can only use PoE ports on one device if not in failover/master-slave mode.

Could someone use two ICX6610 in this configuration, with a single ICX6450 as part of the stack?

Had the idea that I could use an ICX6450 in the office, and possibly stack it across 2x10G ports - one to each ICX6610. But if one 6610 reboots, I wouldn't want everything go offline. Would the ICX6450 use the other 6610?

just stack them like they're supposed to be using the 40gb ports on the back, and you'll have seamless failover. You can stack/connect the 6450 to the two 6610s, and yes the 6610s will be the "brains" for the 6450 routing wise. However if you lose the master 6610, the second one will take over the master role and still perform the routing brains for the 6450

also not sure what you mean about not being able to use poe ports on other units, poe functionality is unchanged during stacking and you can use poe on any stack member

 

[fohdeesha]

im poking around the linux portion of the switch - where would i find where this is hard coded? It sticks after a factory reset:

OS>show remote
IP address : 10.14.234.82
subnet mask : 255.255.255.0
default gateway : 10.14.234.1
OS>

yes these are just debug variables that aren't even used unless you go in and enable remote GDB debugging among other things, they are normal to be populated from factory

 

[fohdeesha]

OT: I've never considered this. Are you running the primary PSUs across your devices directly off of your line AC or another PDU connected to AC outlets (switches, servers, etc)? IOW, is there no UPS to the primary PSUs?

If so, what's the reason? To save the UPS battery? Never even considered this but could that potentially lengthen the life of UPS batteries?

Unfortunately here in our home, we have dirty (and low) voltage. So I have to use a sign wave UPS to get a clean signal to everything. Then again, I could get a PSU that cleans up the signal with caps.

That's the pretty typical setup (one feed directly on utility power, one feed off UPS). This way if/when the UPS fails or is replaced, the devices are unaffected and remain powered. if it's devices (like our switches) that support load splitting between PSUs, then it does take some load off the UPS transformer (or inverter & rectifier if it's an online double conversion UPS). Won't make any change for battery lifespan, the only time the batteries are pulled from is when utility power is lost, so there's not going to be any load sharing on the PSUs during that event

 

[fohdeesha]

My 7450 is showing this series of events every now and then:

Apr 20 17:30:13:I:System: Stack unit 1 POE  Power supply 2  with 748000 mwatts capacity is up 
Apr 20 17:30:12:I:System: Stack unit 1 have AC and DC PSUs, this is not supported
Apr 20 17:30:11:I:System: Stack unit 1 have AC and DC PSUs, this is not supported
Apr 20 17:30:10:I:System: Stack unit 1 have AC and DC PSUs, this is not supported
Apr 20 17:30:09:C:System: Stack unit 1 POE  Power supply 2  with 748000 mwatts capacity is down

Is this a sign PSU2 is dying or a possible board/internal power distribution issue?
There's no PoE load on this switch right now as I haven't re-run the cables from my APs.
PSU 2 is fed from a UPS protected PDU, the cord is secure at both ends, and nothing else on the PDU is reporting any power issues.

I was about to say (looks like you already pinpointed it), to rule out the PSU itself, connect both PSUs to the "good" UPS. If the PSU2 faults stop, then it's not getting along with the older UPS

 

[fohdeesha]

I purchased a 6610 recently and apparently in it's previous life it was id 2 in a stack. Is there a way to change this stack id to 1? I tried stack secure-setup but it just reports there are no neighbors and exits. More of an annoyance than impactful but I thought I'd ask.

I flashed bootloader and os per fohdeesha's guide, factory reset, reprovision modules to slot 1-3 (denied, switch is active), unconfigure and stack secure-setup.

TIA

So you did run the following? " stack unconfigure clean " and hit yes at the prompt?
to nuke it you can do over serial:

press ctrl+y then let go, then press m, then enter. you should drop to an "OS>" prompt. run the following commands

del stacking.boot
del startup-config
del startup-config.backup

then type "dir" and on the output you should *not* see those three files now. now you can run "reset" to reboot it

 

[richtj99]

That's probably being read from the uBoot environment. In uBoot, what's the output of printenv?

Here it is - I get I am being paranoid but I want to make sure I am not putting something on my network that can phone home to the last owner. I thought a factory reset would get rid of this but it seems hard coded?

Thanks!

Ruckus Wireless Bootloader: 10.1.14T215 (Nov 15 2018 - 04:59:10)

Validate Shmoo parameters stored in flash ..... OK
Restoring Shmoo parameters from flash .....
Running simple memory test ..... OK

Uboot is running in Diagnostic mode...

 soc_pcie_hw_init : port->reg_base = 0x18012000 , its value = 0x1
PCIe port 402726912 in End-Point mode - ignored

**************
 port 0 is not active!!
**************
In:    serial
Out:   serial
Err:   serial
UBOOT: BE Mode
ICX7250-48 Copper (POE), PVT1
SYS CPLD VER: 0x6, Released Ver: 0
arm_clk=1000MHz, axi_clk=499MHz, apb_clk=124MHz, arm_periph_clk=500MHz
Access set to PRIMARY FL..
SF: Detected MX25L6405D with page size 4 KiB, total 8 MiB
Put the USB HUB in Reset.
phy ctrl 01d36440
Initial usb2h clock is: 000003fc
Initial usb2h clock is: 000003fd
iClk = 00000078, USBClk = 01536440

Bring usb2h_out of reset.......
usb2_reset_state is set and now it is: 00000000
Take the USB HUB out of Reset.
Net:   Registering eth
Broadcom BCM IPROC Ethernet driver 0.1
Using GMAC0 (0x18022000)
et0: ethHw_chipAttach: Chip ID: 0xdc14; phyaddr: 0x1
et0: gmac_serdes_init() serdes_status0: 0xf100ff00; serdes_status1: 0xf00
et0: gmac_serdes_init() PLL ready brought up exit
serdes_reset_core pbyaddr(0x1) id2(0xf)
ethHw_portLinkUp: Retrying link status
ethHw_portLinkUp: Retrying link status
ethHw_portLinkUp: Retrying link status
ethHw_portLinkUp: Retrying link status
ETH LINK UP: 1000FD
bcmiproc_eth-0
Access set to PRIMARY FL..
SF: Detected MX25L6405D with page size 4 KiB, total 8 MiB
Access set to PRIMARY FL..
SF: Detected MX25L6405D with page size 4 KiB, total 8 MiB
Erasing SPI flash...Writing to SPI flash...Flash set to SECONDARY FL
SF: Detected MX25L6405D with page size 4 KiB, total 8 MiB
done
Valid environment: 1
Booted From: Uboot partition 1
Enter 'b' to stop at boot monitor:  0
ICX7250-Boot> printenv
baudrate=9600
bootargs=console=ttyS0,9600n8 maxcpus=1 mem=1930M
bootcmd=run defbootcmd
bootdelay=3
bootfile=uImage
config_name=conf@3
cons_log=true
consoledev=ttyS0
defbootcmd=u_imi; sf probe 0;  setenv bootargs ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}:${netdev}:off console=${consoledev},${baudrate},${extra_bootargs},${fips_reset} maxcpus=1 mem=2021M ethaddr=${ethaddr}? ${quiet}; nand read 0x70000000 0x0 0x2000000;bootm 70000200#${config_name}
diag_mode_enable=1
ethact=bcmiproc_eth-0
ethaddr=78:a6:e1:0c:e5:a7
fdt_high=0xffffffff
fileaddr=1000000
filesize=1B620C4
flash_addr=200
gatewayip=10.14.234.1
hostname=SI
image_name=SPS08030h.bin
initrd_high=0xffffffff
ipaddr=10.14.234.82
loadaddr=0x61007dc0
maxcpus=1
netdev=eth0
netmask=255.255.255.0
quiet=quiet
serverip=10.14.202.36
stderr=serial
stdin=serial
stdout=serial
uboot=spz10114
uboot_1_ver=10.1.14T215 (Nov 15 2018 - 04:59:15)
uboot_2_ver=10.1.14T215
uboot_size=786432
ver=10.1.14T215 (Nov 15 2018 - 04:59:15)

Environment size: 1063/16379 bytes
ICX7250-Boot>