Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

magi

New Member
Aug 22, 2019
9
3
3
United States
nope, and even if you could you wouldn't want to. You wanna let it get even hotter than 85C before ramping up? hehe
Ah, forgot to mention that I added the 120mm fans and the temp is well below threshold (40C). Just want to further push down on the thermal equilibrium :)
 

French Chamallow

New Member
Mar 15, 2019
17
3
3
Can you tell me if I'm going on the right road :)
my original language is not english but you must have noticed it ...
Actually it's my pfsense that manages all the routing.

that look like that in pfsense :
Capture


I have a nano HD AP with several different ssd and 1 vlan per ssid
I did my first vlan 2 weeks ago to explain my level!

for me all this is new and I try to read and learn.
I update my 6450, active the poe for the AP successfully all run.

Next step :
As I have an ICX-6450 with the firmware R I want to do the routing by the ICX.
if it's a bad idea for a beginner like me tell me :)


WAN 1000baseT <full-duplex> redacted
LAN 1000baseT <full-duplex> 192.168.0.1
VLAN50_WIFI_ENFANT 192.168.50.1
VLAN_IOT 192.168.60.1
VLAN40_STEVE 192.168.40.1
VLAN30_HELENE_AUTRE 192.168.30.1

I have my pfsense in 1/1/1 with this vlan : 30 - 40 - 50 - 60
I have a Nano HD wifi in 1/1/2 with this vlan : 30 - 40 - 50 - 60
1/1/1 is in dual mode
1/1/2 is in dual mode


From what I read it takes a dhcp server separate from the switch and my pfsense and do some ve on the switch.
---------------------------------------------------
here's what I'm thinking of doing but I'm scared to get started

1 install isc-dhcp-server on a raspberry pi 3
apt install isc-dhcp-server
nano /etc/default/isc-dhcp-server
INTERFACESv4="interface of the Pi"

nano /etc/dhcp/dhcpd.conf
option domain-name "$yourdomainhere";
option domain-name-servers 192.168.0.1 ( my pfsense )
default-lease-time 3600;
max-lease-time 7200;
authoritative;

# exemple VLAN 50 wifi enfant, VE Int 50
subnet 192.168.50.0 netmask 255.255.255.0 {
option routers 192.168.50.1;
option subnet-mask 255.255.255.0;
range 192.168.50.1 192.168.50.20;
}

# exemple VLAN40_STEVE, VE Int 40
subnet 192.168.40.0 netmask 255.255.255.0 {
option routers 192.168.40.1;
option subnet-mask 255.255.255.0;
range 192.168.40.1 192.168.40.100;


#exemple static ip
host plex {
hardware ethernet xx:xx:xx:xx:xx:xx;
fixed-address 192.168.40.200;
option host-name "plex";
}

# exemple dhcp for transport pfsense -> ICX
subnet 192.168.0.0 netmask 255.255.255.248 {
option routers 192.168.0.1;
option subnet-mask 255.255.255.248;
range 192.168.0.1 192.168.0.7;
}



2 configure icx6450
( what I read "you almost never assign an IP directly to a physical interface in layer 3 firmware/switches"

#"add a vlan for the route to pfsense port 1/1/1"
vlan 100
untagged e 1/1/1
router-interface ve 100
exit
int ve 100
ip addr 192.168.0.2/29

#"add ve for vlan 50"
vlan 50
router-interface ve 50
exit
int ve 50
ip addr 192.168.50.1

ip helper-adress n xxx.xxx.xxx.xxx

- where n is a number starting from 1. If this is the only "helper", use 1.
- xxx.... is the IP address of the DHCP server.

3 add a route ( I do not know how to do... I see this link but I don't understand)
dont forget that pfsense will also need a route to know how to reach all the vlans, see the diagram here: https://forums.servethehome.com/ind...ware-trouble-routing-vlans.21113/#post-196527

question in pfsense /system/Routing/static route/
I see destination network
I see Gateway, find what to use ?

4 learn ACL
 

Wolfstar

Active Member
Nov 28, 2015
155
81
28
45
Okay, I'm going to lay out a couple of things here that I see that are wrong with your examples. One, your dhcpd.conf examples are configuring to hand out your router IP address. You want the "range" option to be something more like, for example on VLAN50, "range 192.168.50.50 192.168.50.200". If done that way it will issue IPs only from between 50.50 and 50.200 to the VLAN 50 clients.

Second, you NEVER want to use DHCP to configure IPv4 addresses on a router or switch. Leave out your DHCP for transport section. I would simply configure 192.168.0.1 for the pfSense LAN interface, and 192.168.0.2 for the ve100 interface.

For your ICX configurations, there's some things you need to do first - specifically, you would need to remove 1/1/1 from the tagged VLANs before it will allow you to add the untagged one:

Code:
config t
interface ethe 1/1/1
no dual-mode
!
vlan 30 40 50 60
no tagged ethe 1/1/1
!
vlan 100
untagged ethe 1/1/1
!
end
Alternately, since your current setup is working, I would strongly recommend you use a different port for the new configuration - say, 1/1/3, or 1/1/48, just something else. Then configure THAT port for the new setup on VLAN 100. That will mean you can configure it, move the cable to the new port, and if it doesn't work right away you can move back.

For your routes, you want Destination Network on pfSense to be 192.168.0.0/16 (or 192.168.0.0 netmask 255.255.0.0) and Gateway to be the ve100 interface IP (192.168.0.2). On the ICX, you need to configure "ip route 0.0.0.0/0 192.168.0.1" in order for traffic to get past the switch in the first place. Remember that routing will always use the most specific (smallest IP range) route first, so even though 192.168.0.0/16 covers all of your VLANs, it will prefer the routes for /24 ranges over the larger one. This also means you need to pick a VLAN to test from and disable its interface on the pfSense before you switch over, or it will not work.
 
  • Like
Reactions: French Chamallow

umass1966

New Member
Nov 2, 2019
6
1
3
Can anybody help me with this question. i have icx7150-c12p switch and want to setup a 10gbe network like freenas server - switch - computer. want suggestions for network card , optic sfp+ and fiber cable . thanks.
 

French Chamallow

New Member
Mar 15, 2019
17
3
3
did not read this message unless you want to laugh ....
I forgot to add the firewall rules in my LANTEST :)


still sorry to annoy you but I assure you that I look before posting!

I did some tests tonight and My icx does not ping the ip 192.168.2.1
my pfsense ping well the ip of ve 100 in 192.168.2.2
I see a continuous traffic on the LANTEST interface ( 120B)

I added a transit vlan 100
I added the port 1/1/24
gave it to him ip 192.168.2.2

I went in my pfsense and I added a LANTEST on one of the free physical ports of my pfsense in 192.168.2.1

I added a gateway in System / Routing / Gateways
name ICX6450
LANTEST interface
Gateway 192.168.2.2


I specify that I left all my config of origin and thus my main lan works on another physical inerface of my pfsense ( 192.168.0.1)

SSH @ ICX6450 (config) #show ip route
Total number of IP routes: 3
Type Codes - B: BGP D: Connected O: OSPF R: RIP S: Static; Cost - Dist / Metric
BGP Codes - i: iBGP e: eBGP
OSPF Codes - i: Inter Area 1: External Type 1 2: External Type 2
Destination Gateway Port Cost Type Uptime
1 0.0.0.0/0 192.168.0.1 ve 1 1/1 S 4d3h
2 192.168.0.0/24 DIRECT ve 1 0/0 D 4d3h
3 192.168.2.0/24 DIRECT ve 100 0/0 D 1h14m


information on my 1/1/24
GigabitEthernet1/1/24 is up, line protocol is up
Port up for 1 hour(s) 38 minute(s) 28 second(s)
Hardware is GigabitEthernet, address is redacted (bia redacted)
Configured speed auto, actual 1Gbit, configured duplex fdx, actual fdx
Configured mdi mode AUTO, actual MDIX
Member of L2 VLAN ID 100, port is untagged, port state is FORWARDING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is config enabled, oper enabled, negotiation disabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
Inter-Packet Gap (IPG) is 96 bit times
MTU 1500 bytes, encapsulation ethernet
300 second input rate: 1000 bits/sec, 1 packets/sec, 0.00% utilization
300 second output rate: 1008 bits/sec, 1 packets/sec, 0.00% utilization
7470 packets input, 478194 bytes, 0 no buffer
Received 1 broadcasts, 0 multicasts, 7469 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
7489 packets output, 479410 bytes, 0 underruns
Transmitted 22 broadcasts, 0 multicasts, 7467 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled


my ping

SSH@ICX6450#ping 192.168.0.1
Sending 1, 16-byte ICMP Echo to 192.168.0.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.0.1 : bytes=16 time=1ms TTL=64
Success rate is 100 percent (1/1), round-trip min/avg/max=1/1/1 ms.
SSH@ICX6450#ping 192.168.2.1
Sending 1, 16-byte ICMP Echo to 192.168.2.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.
 
Last edited:

microserf

New Member
May 20, 2019
12
4
3
So I picked up an ICX 7150-C12P and updated the firmware. On boot, I'm seeing this:
Code:
Ruckus Wireless Bootloader: 10.1.14T225 (Nov 15 2018 - 04:59:18 -0800)
Booted from partition 2
DRAM:  Validate Shmoo parameters stored in flash ..... OK
ICX7150-12 (POE), PVT1
SYS CPLD VER: 0x4 Released Ver: 0xa
device 0 offset 0x0, size 0xc0000
Enter 'b' to stop at boot monitor:  0
device 0 offset 0x0, size 0xc0000
bootdelay: ===
Booting image from Primary
NAND read: device 0 offset 0x0, size 0x2000000
Skipping bad block 0x00a00000
 33554432 bytes read: OK
The second last line, "Skipping bad block 0x00a00000", concerned me so I flashed the firmware again but there was no change. The switch appears to boot fine. Comments or suggestions?
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,942
1,773
113
29
fohdeesha.com
I would reflash the bootloader as well (it's on the same SPI flash chip) but it looks like it's detected the block and marked it off which is pretty normal, you should be fine. When you need to worry is when the number of bad blocks starts rising over time, then your flash is failing
 
  • Like
Reactions: microserf

microserf

New Member
May 20, 2019
12
4
3
I would reflash the bootloader as well (it's on the same SPI flash chip) but it looks like it's detected the block and marked it off which is pretty normal, you should be fine. When you need to worry is when the number of bad blocks starts rising over time, then your flash is failing
But it's all shiny and new :(. I re-flashed the bootloader but no change. What's the mailing address for Fohdeesha's SMD Solderworks?
 

aflow

New Member
Apr 13, 2016
2
0
1
40
Hi,
Is it possible to made this modification with the ICX6450?.

Thanks.

I noticed that with the monoprice DAC breakout, it was showing up as absolutely nothing with the "show media" command. Even though it doesn't affect the behavior of the links, I figured something was up with the I2C EEPROM in the DAC cable (this is where vendor/serial/cable type info is stored for optics and DACs).

You can see the show media command can't find any vendor or cable type info for the monoprice DAC, so it shows EMPTY for the 4 breakout slots:

Code:
telnet@ICX3#show media | include 1/2/[2-5]
Port 1/2/2:  Type : EMPTY
Port 1/2/3:  Type : EMPTY
Port 1/2/4:  Type : EMPTY
Port 1/2/5:  Type : EMPTY
So I dropped down to the debug console (over serial, press ctrl+y, let go, then press m, then enter) to use some I2C read and write commands.

First, let's read the EEPROM from a proper official brocade DAC cable:

Code:
OS>i2c read 3f 0 256
0000: 0d 00 06 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0010: 00 00 00 00 00 00 01 81-00 00 00 00 00 00 00 00 ................
0020: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0030: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0040: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0050: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0060: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0070: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0080: 0d 00 23 08 00 00 00 00-00 00 00 00 67 00 00 00 ..#.........g...
0090: 00 00 01 a0 42 52 4f 43-41 44 45 20 20 20 20 20 ....BROCADE
00a0: 20 20 20 20 07 00 05 1e-35 38 2d 30 30 30 30 30     ....58-00000
00b0: 33 33 2d 30 31 20 20 20-41 20 04 05 00 00 46 08 33-01   A ....F.
00c0: 00 00 00 00 50 41 45 31-31 34 31 30 33 30 30 30 ....PAE114103000
00d0: 38 37 36 20 31 34 30 33-30 35 20 20 00 00 00 c2 876 140305  ....
00e0: 31 31 31 30 34 30 39 30-37 31 20 20 20 20 20 20 1110409071
00f0: 20 31 20 20 20 20 20 20-20 20 00 00 00 00 00 00  1        ......
You can see the first half is mostly empty, and the vendor information does not begin until the second half of the EEPROM, where you get the byte specifying the cable type, the vendor string, a serial number, etc. This is as it should be.

Now let's read the Monoprice DAC that doesn't show up properly:

Code:
OS>i2c read 41 0 256
0000: 0c 00 21 00 00 00 00 00-04 80 00 06 67 00 00 00 ..!.........g...
0010: 00 00 01 a0 4d 4f 4e 4f-50 52 49 43 45 20 04 20 ....MONOPRICE .
0020: 31 32 30 36 39 20 2e 2e-2e 2e 2e 2e 2e 2e 2e 2e 12069 ..........
0030: 77 77 77 2e 6d 6f 6e 6f-70 72 69 63 65 2e 63 6f www.monoprice.co
0040: 6d f4 31 32 30 36 39 31-34 31 30 30 30 39 36 43 m.1206914100096C
0050: 48 49 4e 41 20 20 20 00-00 00 f4 ff ff ff ff ff HINA   .........
0060: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0070: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0080: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0090: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00a0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00b0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00c0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00d0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00e0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00f0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
You can see they must have been in a rush programming these, as they stuck all the vendor and serial information in the incorrect section. No wonder it shows up as empty/unknown media type! Upon closer inspection, they programmed it like an SFP+ module. This first half of the EEPROM should be populated with these values like this in an SFP+, but for a QSFP+ module, it should be in the second half.

So I used a series of i2c write commands to copy byte for byte the contents of the official brocade DAC, to the monoprice DAC, overwriting the incorrectly placed vendor info with properly placed brocade info.

The monoprice DAC EEPROM contents now looks like this:
Code:
OS>i2c read 41 0 256
0000: 0d 00 06 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0010: 00 00 00 00 00 00 01 81-00 00 00 00 00 00 00 00 ................
0020: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0030: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0040: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0050: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0060: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0070: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0080: 0d 00 23 08 00 00 00 00-00 00 00 00 67 00 00 00 ..#.........g...
0090: 00 00 01 a0 42 52 4f 43-41 44 45 20 20 20 20 20 ....BROCADE
00a0: 20 20 20 20 07 00 05 1e-35 38 2d 30 30 30 30 30     ....58-00000
00b0: 33 33 2d 30 31 20 20 20-41 20 04 05 00 00 46 08 33-01   A ....F.
00c0: 00 00 00 00 50 41 45 31-31 34 31 30 33 30 30 30 ....PAE114103000
00d0: 38 37 36 20 31 34 30 33-30 35 20 20 00 00 00 c2 876 140305  ....
00e0: 31 31 31 30 34 30 39 30-37 31 20 20 20 20 20 20 1110409071
00f0: 20 31 20 20 20 20 20 20-20 20 00 00 00 00 00 00  1        ......
Look familiar? Our monoprice DAC now has identical EEPROM contents, compared to our actual official brocade DAC.

And as expected, after unplugging and replugging the cable, the switch now ID's it as a DAC, and thinks it's an official brocade part:

Code:
telnet@ICX3#show media | include 1/2/[2-5]
Port 1/2/2:  Type : 40GE-Passive Copper
Port 1/2/3:  Type : 40GE-Passive Copper
Port 1/2/4:  Type : 40GE-Passive Copper
Port 1/2/5:  Type : 40GE-Passive Copper

##even more details - still the monoprice DAC
telnet@ICX3#show media e 1/2/2
Port   1/2/2:Type  : 40GBASE-Passive Copper
Vendor Name: BROCADE          Serial Num: PAE114103000876 Revision: A
I wouldn't think this has any effect on functionality, as I said the links worked perfectly fine beforehand. Re-coding it into a brocade part like this was only possible because the monoprice DAC did not have the EEPROM write protect bit set. This exact same procedure can be used to re-code optics to different vendors to bypass vendor locks, but many non-OEM optics have a bit set that prevents EEPROM writing without special unlock strings

A little more information:

The SFP+ EEPROM data location layout, you can see the identity and vendor etc strings are at the beginning (like our QSFP+ monoprice incorrectly had) - http://files.siemon.com/sis/application-guide/sfp-plus-to-sfp-plus_eeprom_summary.pdf

The QSFP+ EEPROM layout, you can see that the vendor and cable type etc now does not start until around the middle. This is where the ICX was trying to read to figure out the cable type, and of course on the monoprice there was nothing there - http://files.siemon.com/sis/application-guide/qsfpplus_to_qsfpplus_eeprom_summary.pdf

Upon closer inspection, this could have potentially caused it refusing to link up, as it could not even read the ident bit to figure out what kind of cable it was. For QSFP modules, it reads byte number 128 (first byte on line 80 in the dump above). This byte is set to "0d" hex to tell the host it's a QSFP+ device. On the monoprice dac, when it read that byte, it was completely empty. You can see in the brocade dump, that bit is correctly set to 0d.

here's a table of possible values for that byte:

https://i.imgur.com/2Id6xp9.png

You can see that if we set that byte to 03 instead, we could make the switch think it's an SFP module. It would probably not like this, given it's a QSFP+ slot (or on second thought, it would just assume it's an SFP+ optic in an SFP+ to QSFP+ adapter)

And upon even further digging, the EEPROM also contains signal attenuation values for the given cable at both 2.5ghz and 5ghz (bytes 186 and 187 respectively), and the host (the switch) uses these values to set up it's transmitter and receiver EQ appropriately. Without it being able to read these on the stock monoprice DAC, the transmitter and receiver probably defaulted to "fallback" values, which could have also resulted in the not quite working links.

Again, copying over the brocade DAC contents should fix this, as it contains these values (4db and 5db, respectively). I'd imagine the attenuation values there are very close to what the monoprice DAC is supposed to have, given they are both the same length, and both 30AWG twinax. Given how much stuff the monoprices come with wrong from factory, I'm surprised they ever worked, especially with cisco equipment (which is what they're advertised for). Thankfully it's not write-locked, so all this is easily fixed with a few commands
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,942
1,773
113
29
fohdeesha.com
Hi,
Is it possible to made this modification with the ICX6450?.

Thanks.
That modification is for as QSFP breakout cable and the ICX6450 has no QSFP ports, so no :) If you want to rewrite an SFP+ optic, 99% of them have the eeprom locked so you won't be able to write anything. The ICX6450 runs linux so it does not have the same commands, the only I2C write commands I remember are "dm i2c-write" but it only takes one byte at a time so you'll have to run that ~100 times and you'll need to find out the I2C ID of the optic as well. Don't have one powered on at the moment to test
 

am45931472

Member
Feb 26, 2019
75
17
8
Is it possible to manually change the fan speed on the Brocade Ruckus 7150-24p? the ruckus website seems to say that fans can only be automatically controlled. I've tried fan-speed via CLI and it is not recognized as a command. I've swapped out the stock fans for some noctua ones going from an audible but quiet switch to a basically silent switch. Temps are good-ok, between 65-75c, but I think even at full speed these noctua fans would be near silent. 72c seems to be the temp threshold for high speed.
 

microserf

New Member
May 20, 2019
12
4
3
Code:
ICX7150-Boot>nand info

Device 0: nand0, sector size 1024 KiB, Micron NAND 2GiB
  Page size       4096 b
  OOB size         224 b
  Erase size   1048576 b
  subpagesize     4096 b
  options     0x   10200
  bbt options 0x       0
ICX7150-Boot>nand bad

Device 0 bad blocks:
  00a00000
  05a00000
  05b00000
Gah.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,942
1,773
113
29
fohdeesha.com
Code:
ICX7150-Boot>nand info

Device 0: nand0, sector size 1024 KiB, Micron NAND 2GiB
  Page size       4096 b
  OOB size         224 b
  Erase size   1048576 b
  subpagesize     4096 b
  options     0x   10200
  bbt options 0x       0
ICX7150-Boot>nand bad

Device 0 bad blocks:
  00a00000
  05a00000
  05b00000
Gah.

you'll be aight
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,942
1,773
113
29
fohdeesha.com
Is it possible to manually change the fan speed on the Brocade Ruckus 7150-24p? the ruckus website seems to say that fans can only be automatically controlled. I've tried fan-speed via CLI and it is not recognized as a command. I've swapped out the stock fans for some noctua ones going from an audible but quiet switch to a basically silent switch. Temps are good-ok, between 65-75c, but I think even at full speed these noctua fans would be near silent. 72c seems to be the temp threshold for high speed.
nope. if it gets too hot it'll ramp the fans up no need to worry about it
 

levi danzer

New Member
Sep 29, 2019
3
1
3
Just a heads up to any one looking. There is a icx6650-56-e-adv on ebay right now. The seller countered with $300.

Debated on picking it up... but I grabbed an sx6036 for my 40gbe needs...

-Levi
 
  • Like
Reactions: fohdeesha