Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

French Chamallow · Nov 6, 2019

Thank you Wolfstar!
It's much clearer
I'm going to watch it tonight

UTC +1 here.

nerdalertdk · Nov 6, 2019

Hi

Can you use a breakout cable on this ICX7400-1X40GQ module for the icx7400 serie ?

Wolfstar · Nov 6, 2019

nerdalertdk said:
Hi

Can you use a breakout cable on this ICX7400-1X40GQ module for the icx7400 serie ?

Not so far as I'm aware. You would have to replace it with an ICX7400-4x10G* module instead. (GF for Fiber, GC for copper/10GBaseT.)

fohdeesha · Nov 6, 2019

Wolfstar said:
Not so far as I'm aware. You would have to replace it with an ICX7400-4x10G* module instead. (GF for Fiber, GC for copper/10GBaseT.)

correct AFAIK

umass1966 · Nov 6, 2019

Can anybody help me with this question. i have icx7150-c12p switch and want to setup a 10gbe network like freenas server - switch - computer. want suggestions for network card , optic sfp+ and fiber cable . thanks.

French Chamallow · Nov 6, 2019

did not read this message unless you want to laugh ....
I forgot to add the firewall rules in my LANTEST

still sorry to annoy you but I assure you that I look before posting!

I did some tests tonight and My icx does not ping the ip 192.168.2.1
my pfsense ping well the ip of ve 100 in 192.168.2.2
I see a continuous traffic on the LANTEST interface ( 120B)

I added a transit vlan 100
I added the port 1/1/24
gave it to him ip 192.168.2.2

I went in my pfsense and I added a LANTEST on one of the free physical ports of my pfsense in 192.168.2.1

I added a gateway in System / Routing / Gateways
name ICX6450
LANTEST interface
Gateway 192.168.2.2

I specify that I left all my config of origin and thus my main lan works on another physical inerface of my pfsense ( 192.168.0.1)

SSH @ ICX6450 (config) #show ip route
Total number of IP routes: 3
Type Codes - B: BGP D: Connected O: OSPF R: RIP S: Static; Cost - Dist / Metric
BGP Codes - i: iBGP e: eBGP
OSPF Codes - i: Inter Area 1: External Type 1 2: External Type 2
Destination Gateway Port Cost Type Uptime
1 0.0.0.0/0 192.168.0.1 ve 1 1/1 S 4d3h
2 192.168.0.0/24 DIRECT ve 1 0/0 D 4d3h
3 192.168.2.0/24 DIRECT ve 100 0/0 D 1h14m

information on my 1/1/24
GigabitEthernet1/1/24 is up, line protocol is up
Port up for 1 hour(s) 38 minute(s) 28 second(s)
Hardware is GigabitEthernet, address is redacted (bia redacted)
Configured speed auto, actual 1Gbit, configured duplex fdx, actual fdx
Configured mdi mode AUTO, actual MDIX
Member of L2 VLAN ID 100, port is untagged, port state is FORWARDING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is config enabled, oper enabled, negotiation disabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
Inter-Packet Gap (IPG) is 96 bit times
MTU 1500 bytes, encapsulation ethernet
300 second input rate: 1000 bits/sec, 1 packets/sec, 0.00% utilization
300 second output rate: 1008 bits/sec, 1 packets/sec, 0.00% utilization
7470 packets input, 478194 bytes, 0 no buffer
Received 1 broadcasts, 0 multicasts, 7469 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
7489 packets output, 479410 bytes, 0 underruns
Transmitted 22 broadcasts, 0 multicasts, 7467 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

my ping

SSH@ICX6450#ping 192.168.0.1
Sending 1, 16-byte ICMP Echo to 192.168.0.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.0.1 : bytes=16 time=1ms TTL=64
Success rate is 100 percent (1/1), round-trip min/avg/max=1/1/1 ms.
SSH@ICX6450#ping 192.168.2.1
Sending 1, 16-byte ICMP Echo to 192.168.2.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.

microserf · Nov 7, 2019

So I picked up an ICX 7150-C12P and updated the firmware. On boot, I'm seeing this:

Code:

Ruckus Wireless Bootloader: 10.1.14T225 (Nov 15 2018 - 04:59:18 -0800)
Booted from partition 2
DRAM:  Validate Shmoo parameters stored in flash ..... OK
ICX7150-12 (POE), PVT1
SYS CPLD VER: 0x4 Released Ver: 0xa
device 0 offset 0x0, size 0xc0000
Enter 'b' to stop at boot monitor:  0
device 0 offset 0x0, size 0xc0000
bootdelay: ===
Booting image from Primary
NAND read: device 0 offset 0x0, size 0x2000000
Skipping bad block 0x00a00000
 33554432 bytes read: OK

The second last line, "Skipping bad block 0x00a00000", concerned me so I flashed the firmware again but there was no change. The switch appears to boot fine. Comments or suggestions?

fohdeesha · Nov 8, 2019

I would reflash the bootloader as well (it's on the same SPI flash chip) but it looks like it's detected the block and marked it off which is pretty normal, you should be fine. When you need to worry is when the number of bad blocks starts rising over time, then your flash is failing

microserf · Nov 8, 2019

fohdeesha said:
I would reflash the bootloader as well (it's on the same SPI flash chip) but it looks like it's detected the block and marked it off which is pretty normal, you should be fine. When you need to worry is when the number of bad blocks starts rising over time, then your flash is failing

But it's all shiny and new

. I re-flashed the bootloader but no change. What's the mailing address for Fohdeesha's SMD Solderworks?

aflow · Nov 8, 2019

Hi,
Is it possible to made this modification with the ICX6450?.

Thanks.

fohdeesha said:
I noticed that with the monoprice DAC breakout, it was showing up as absolutely nothing with the "show media" command. Even though it doesn't affect the behavior of the links, I figured something was up with the I2C EEPROM in the DAC cable (this is where vendor/serial/cable type info is stored for optics and DACs).

You can see the show media command can't find any vendor or cable type info for the monoprice DAC, so it shows EMPTY for the 4 breakout slots:

Code:

telnet@ICX3#show media | include 1/2/[2-5] Port 1/2/2: Type : EMPTY Port 1/2/3: Type : EMPTY Port 1/2/4: Type : EMPTY Port 1/2/5: Type : EMPTY

So I dropped down to the debug console (over serial, press ctrl+y, let go, then press m, then enter) to use some I2C read and write commands.

First, let's read the EEPROM from a proper official brocade DAC cable:

Code:

OS>i2c read 3f 0 256 0000: 0d 00 06 00 00 00 00 00-00 00 00 00 00 00 00 00 ................ 0010: 00 00 00 00 00 00 01 81-00 00 00 00 00 00 00 00 ................ 0020: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................ 0030: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................ 0040: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................ 0050: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................ 0060: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................ 0070: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................ 0080: 0d 00 23 08 00 00 00 00-00 00 00 00 67 00 00 00 ..#.........g... 0090: 00 00 01 a0 42 52 4f 43-41 44 45 20 20 20 20 20 ....BROCADE 00a0: 20 20 20 20 07 00 05 1e-35 38 2d 30 30 30 30 30 ....58-00000 00b0: 33 33 2d 30 31 20 20 20-41 20 04 05 00 00 46 08 33-01 A ....F. 00c0: 00 00 00 00 50 41 45 31-31 34 31 30 33 30 30 30 ....PAE114103000 00d0: 38 37 36 20 31 34 30 33-30 35 20 20 00 00 00 c2 876 140305 .... 00e0: 31 31 31 30 34 30 39 30-37 31 20 20 20 20 20 20 1110409071 00f0: 20 31 20 20 20 20 20 20-20 20 00 00 00 00 00 00 1 ......

You can see the first half is mostly empty, and the vendor information does not begin until the second half of the EEPROM, where you get the byte specifying the cable type, the vendor string, a serial number, etc. This is as it should be.

Now let's read the Monoprice DAC that doesn't show up properly:

Code:

OS>i2c read 41 0 256 0000: 0c 00 21 00 00 00 00 00-04 80 00 06 67 00 00 00 ..!.........g... 0010: 00 00 01 a0 4d 4f 4e 4f-50 52 49 43 45 20 04 20 ....MONOPRICE . 0020: 31 32 30 36 39 20 2e 2e-2e 2e 2e 2e 2e 2e 2e 2e 12069 .......... 0030: 77 77 77 2e 6d 6f 6e 6f-70 72 69 63 65 2e 63 6f www.monoprice.co 0040: 6d f4 31 32 30 36 39 31-34 31 30 30 30 39 36 43 m.1206914100096C 0050: 48 49 4e 41 20 20 20 00-00 00 f4 ff ff ff ff ff HINA ......... 0060: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................ 0070: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................ 0080: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................ 0090: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................ 00a0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................ 00b0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................ 00c0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................ 00d0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................ 00e0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................ 00f0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................

You can see they must have been in a rush programming these, as they stuck all the vendor and serial information in the incorrect section. No wonder it shows up as empty/unknown media type! Upon closer inspection, they programmed it like an SFP+ module. This first half of the EEPROM should be populated with these values like this in an SFP+, but for a QSFP+ module, it should be in the second half.

So I used a series of i2c write commands to copy byte for byte the contents of the official brocade DAC, to the monoprice DAC, overwriting the incorrectly placed vendor info with properly placed brocade info.

The monoprice DAC EEPROM contents now looks like this:

Code:

OS>i2c read 41 0 256 0000: 0d 00 06 00 00 00 00 00-00 00 00 00 00 00 00 00 ................ 0010: 00 00 00 00 00 00 01 81-00 00 00 00 00 00 00 00 ................ 0020: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................ 0030: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................ 0040: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................ 0050: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................ 0060: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................ 0070: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................ 0080: 0d 00 23 08 00 00 00 00-00 00 00 00 67 00 00 00 ..#.........g... 0090: 00 00 01 a0 42 52 4f 43-41 44 45 20 20 20 20 20 ....BROCADE 00a0: 20 20 20 20 07 00 05 1e-35 38 2d 30 30 30 30 30 ....58-00000 00b0: 33 33 2d 30 31 20 20 20-41 20 04 05 00 00 46 08 33-01 A ....F. 00c0: 00 00 00 00 50 41 45 31-31 34 31 30 33 30 30 30 ....PAE114103000 00d0: 38 37 36 20 31 34 30 33-30 35 20 20 00 00 00 c2 876 140305 .... 00e0: 31 31 31 30 34 30 39 30-37 31 20 20 20 20 20 20 1110409071 00f0: 20 31 20 20 20 20 20 20-20 20 00 00 00 00 00 00 1 ......

Look familiar? Our monoprice DAC now has identical EEPROM contents, compared to our actual official brocade DAC.

And as expected, after unplugging and replugging the cable, the switch now ID's it as a DAC, and thinks it's an official brocade part:

Code:

telnet@ICX3#show media | include 1/2/[2-5] Port 1/2/2: Type : 40GE-Passive Copper Port 1/2/3: Type : 40GE-Passive Copper Port 1/2/4: Type : 40GE-Passive Copper Port 1/2/5: Type : 40GE-Passive Copper ##even more details - still the monoprice DAC telnet@ICX3#show media e 1/2/2 Port 1/2/2:Type : 40GBASE-Passive Copper Vendor Name: BROCADE Serial Num: PAE114103000876 Revision: A

I wouldn't think this has any effect on functionality, as I said the links worked perfectly fine beforehand. Re-coding it into a brocade part like this was only possible because the monoprice DAC did not have the EEPROM write protect bit set. This exact same procedure can be used to re-code optics to different vendors to bypass vendor locks, but many non-OEM optics have a bit set that prevents EEPROM writing without special unlock strings

A little more information:

The SFP+ EEPROM data location layout, you can see the identity and vendor etc strings are at the beginning (like our QSFP+ monoprice incorrectly had) - http://files.siemon.com/sis/application-guide/sfp-plus-to-sfp-plus_eeprom_summary.pdf

The QSFP+ EEPROM layout, you can see that the vendor and cable type etc now does not start until around the middle. This is where the ICX was trying to read to figure out the cable type, and of course on the monoprice there was nothing there - http://files.siemon.com/sis/application-guide/qsfpplus_to_qsfpplus_eeprom_summary.pdf

Upon closer inspection, this could have potentially caused it refusing to link up, as it could not even read the ident bit to figure out what kind of cable it was. For QSFP modules, it reads byte number 128 (first byte on line 80 in the dump above). This byte is set to "0d" hex to tell the host it's a QSFP+ device. On the monoprice dac, when it read that byte, it was completely empty. You can see in the brocade dump, that bit is correctly set to 0d.

here's a table of possible values for that byte:

https://i.imgur.com/2Id6xp9.png

You can see that if we set that byte to 03 instead, we could make the switch think it's an SFP module. It would probably not like this, given it's a QSFP+ slot (or on second thought, it would just assume it's an SFP+ optic in an SFP+ to QSFP+ adapter)

And upon even further digging, the EEPROM also contains signal attenuation values for the given cable at both 2.5ghz and 5ghz (bytes 186 and 187 respectively), and the host (the switch) uses these values to set up it's transmitter and receiver EQ appropriately. Without it being able to read these on the stock monoprice DAC, the transmitter and receiver probably defaulted to "fallback" values, which could have also resulted in the not quite working links.

Again, copying over the brocade DAC contents should fix this, as it contains these values (4db and 5db, respectively). I'd imagine the attenuation values there are very close to what the monoprice DAC is supposed to have, given they are both the same length, and both 30AWG twinax. Given how much stuff the monoprices come with wrong from factory, I'm surprised they ever worked, especially with cisco equipment (which is what they're advertised for). Thankfully it's not write-locked, so all this is easily fixed with a few commands

fohdeesha · Nov 8, 2019

aflow said:
Hi,
Is it possible to made this modification with the ICX6450?.

Thanks.

That modification is for as QSFP breakout cable and the ICX6450 has no QSFP ports, so no

If you want to rewrite an SFP+ optic, 99% of them have the eeprom locked so you won't be able to write anything. The ICX6450 runs linux so it does not have the same commands, the only I2C write commands I remember are "dm i2c-write" but it only takes one byte at a time so you'll have to run that ~100 times and you'll need to find out the I2C ID of the optic as well. Don't have one powered on at the moment to test

am45931472 · Nov 8, 2019

Is it possible to manually change the fan speed on the Brocade Ruckus 7150-24p? the ruckus website seems to say that fans can only be automatically controlled. I've tried fan-speed via CLI and it is not recognized as a command. I've swapped out the stock fans for some noctua ones going from an audible but quiet switch to a basically silent switch. Temps are good-ok, between 65-75c, but I think even at full speed these noctua fans would be near silent. 72c seems to be the temp threshold for high speed.

microserf · Nov 9, 2019

Code:

ICX7150-Boot>nand info

Device 0: nand0, sector size 1024 KiB, Micron NAND 2GiB
  Page size       4096 b
  OOB size         224 b
  Erase size   1048576 b
  subpagesize     4096 b
  options     0x   10200
  bbt options 0x       0
ICX7150-Boot>nand bad

Device 0 bad blocks:
  00a00000
  05a00000
  05b00000

Gah.

fohdeesha · Nov 9, 2019

microserf said:

Code:

ICX7150-Boot>nand info

Device 0: nand0, sector size 1024 KiB, Micron NAND 2GiB
  Page size       4096 b
  OOB size         224 b
  Erase size   1048576 b
  subpagesize     4096 b
  options     0x   10200
  bbt options 0x       0
ICX7150-Boot>nand bad

Device 0 bad blocks:
  00a00000
  05a00000
  05b00000

Gah.

you'll be aight

fohdeesha · Nov 9, 2019

am45931472 said:
Is it possible to manually change the fan speed on the Brocade Ruckus 7150-24p? the ruckus website seems to say that fans can only be automatically controlled. I've tried fan-speed via CLI and it is not recognized as a command. I've swapped out the stock fans for some noctua ones going from an audible but quiet switch to a basically silent switch. Temps are good-ok, between 65-75c, but I think even at full speed these noctua fans would be near silent. 72c seems to be the temp threshold for high speed.

nope. if it gets too hot it'll ramp the fans up no need to worry about it

levi danzer · Nov 11, 2019

Just a heads up to any one looking. There is a icx6650-56-e-adv on ebay right now. The seller countered with $300.

Debated on picking it up... but I grabbed an sx6036 for my 40gbe needs...

-Levi

French Chamallow · Nov 12, 2019

I move slowly but I learn a lot thanks to my ICX-6450 to you all!
My vlans are mounted and I have my ip ranges in each vlan as desired.
intervlan work in the switch.

I start moving vlan 1 devices to all the vlans I've mounted.

I'm having a problem, I have a raspberry with domoticz on vlan 1 and I have a Xiaomi gateway ( 192.168.60.3 ) to handle zigbee devices.

The xiaomi gateway is connected in wifi on a Nano Hd on the IOT vlan.

From pi I can reach the port of the xiaomi Gateway ansd the port is open.

pi@raspberrypi:~ $ sudo nmap -sU 192.168.60.3 -p 9898

Starting Nmap 7.40 ( Nmap: the Network Mapper - Free Security Scanner ) at 2019-11-12 22:36 CET
Nmap scan report for 192.168.60.3
Host is up (0.0021s latency).
PORT STATE SERVICE
9898/udp open monkeycom

Pi don"t receive update information and I can't control zigbee device

I think I have a problem with multicast but I do not know anything about it.
How do I know or make sure that one vlan's multicast arrives on another vlan? sorry if I express myself badly, but I never played with that!

infoMatt · Nov 12, 2019

I don't know exactly how the Xiaomi zigbee hub works, but as far as I know, multicast in the "consumer" domotic/IoT devices is used mainly for the discovery of the devices, as those communicate using some sort of mDNS query (Chromecast and Apple AirPlay/Bonjour are some notable example of these applications). If that's the case, you cannot route those requests, as those are sent to the multicast address 224.0.0.251 with a TTL=1; if you need to make those devices discoverable between VLANs you'd have to install an mDNS repeater software (say Avahi for example) on a VM/device with an interface on each network you want to listen from or repeat messages to (obviously you can trunk 'em on a single NIC, and make logical tagged interfaces on it).

Proper multicast routing is a whole different beast, and it involves pretty solid knowledge of IGMP, PIM routers and rendez-vous... knowledge that I don't have, unfourtunately, so on this topic I cannot help you much

.

Googling a bit I've found this document: http://www.netadmin.us/docs/Multicast.pdf
I'm pretty sure that inside the beefy documentation that @fohdeesha has put togheter there are the exact commands and examples that you might need (yea, I've not read it entirely - yet).

Octopuss · Nov 13, 2019

How much does 7150-24 (the plain one, not PoE) usually go, 2nd hand?

lec668 · Nov 13, 2019

Hello from a new frenchy 7250/48 owner.
160 bucks for a 2 years old swith was (almost) indecent.

@fohdeesha : many thanks for sharing such a massive amout of info with us ! Rare to read such a complete HowTo guide.
Champagne will be offered if you ever come to Paris.

Already thinking about a 40g upgrade !

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

New Member

Fleet Admiral

Active Member

Kaini Industries

New Member

New Member

New Member

Kaini Industries

New Member

New Member

Kaini Industries

Member

New Member

Kaini Industries

Kaini Industries

New Member

New Member

Active Member

Active Member

New Member