Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

So I've sent @fohdeesha a couple PM's on the subject of trying to convert/re-code my ipolex 10G SFP+ RJ45 copper transceivers to official Brocade in my ICX6450-24P. While I await his reply I figured I'd post here in case anyone else can help. These are the optics I am working with:

https://www.amazon.com/gp/product/B08NW1VBX6/

Here is what I get at the terminal when I show media:

SSH@switch(config)#show media ethernet 1/2/1
Port 1/2/1: Type : 10GE SR 300m ((SFP+))
Vendor: OEM Version: 02
Part# : SFP-10G-SR Serial#: CSF101L33816

I suspect that the eeprom may be unlocked because it looks like ipolex just copied the eeprom flash from a Cisco SFP-10G-SR and substituted Cicsco with "OEM." Obviously they are not Cisco, are not 300m, and are not 10GBASE SFP+. They are ipolex 10GBASE-T modules with RJ-45.

I'm trying to figure out if I can write to them or not using i2c. So far I have not been able to figure out the context. Firstly I looked at the post where he figured out how to recode the monoprice passive copper cables here and also several other posts.


Unfortunately, on the 6450 at the serial debug console "i2c read" is not a recognized command. In fact there are very few i2c commands here on the 6450 as far as I can tell using help or ?. That seems to be a dead end.

So next I looked at @fohdeesha's hidden dm menu inside enable config terminal. There are also some i2c commands their, but none of the seem helpful either.

Finally I used @fohdeesha's guide on "Hidden Brocade Dev Stuff" and then used "Hidden Bootloader Modes." This has proven to provide some answers. However, I cannot figure out the correct context. This is what I got so far inside the hidden bootloader.

ICX64XX-boot>> help i2cprobe
i2cprobe <device>
- probe special i2c device id
device : Valid devices are <pd69000|info_eeprom|sfp_port1|sfp_port2|sfp_port3|sfp_port4s
|cpld|rtc|pca9535_sfp|pca9535_led|pca9535_led_stack|pca9535_id|hwm>
ICX64XX-boot>> i2cprobe sfp_port1
I2C has probe the SFP Port 1.(Reg0=0x03)
loop: 1
i2cprobe PASS
ICX64XX-boot>>

We can see that the optics show up in SFP Port 1 and 3 using this command. In this hidden bootloader there is indeed an i2c read function, but this is where I am not sure about the correct syntax.

ICX64XX-boot>> help i2cread
i2cread <devAddr> <reg_addr> <addrlen> <get_len>
- Get special i2c device id
devAddr : I2C device address
reg_addr : I2c device register
addrlen : I2C device address size, [0/1/2] byte
get_len : Get data bytes

I have tried many variables for all of the switches above, but not to my surprise, nothing has worked. Here are some examples of my trials. Many don't make a lot of sense, but I was throwing the kitchen sink at it:

ICX64XX-boot>> i2cread 0=0x03 1 0 256
i2c read length fail (getLen=256)

ICX64XX-boot>> i2cread 0=0x03 1/2/1 0 FF
TWSI: mvTwsiRead: 977: mvTwsiAddrSet failed
i2c_halRead fail (ret=-200)

ICX64XX-boot>> i2cread 0 0x03 1/2/1 0 256
i2c_halRead fail (ret=-200)

ICX64XX-boot>> i2cread 0 0x03 0 256
i2c_halRead fail (ret=-200)

ICX64XX-boot>> i2cread 0 0x03 0 0 256
i2c_halRead fail (ret=-200)

ICX64XX-boot>> i2cread 0 0x03 1 0 256
i2c_halRead fail (ret=-200)

ICX64XX-boot>> i2cread 0 0x03 2 0 256
i2c_halRead fail (ret=-200)

ICX64XX-boot>> i2cread 0=0x03 1/2/1 0 256
i2c read length fail (getLen=256)

So what exactly is the syntax for devAddr (I2C device address), reg_addr (I2c device register), addrlen (I2C device address size) or do I even need to specify this one, and of course get_len (Get data bytes)? It was super late and I was really tired when I tried all this so it's a bit fuzzy already... I may have also been a bit inebriated o_O

Assuming I can figure out how to read the eeprom, after I back it up, I would then like to try and write one byte to see if it's unlocked. Anybody know the commands to write?

Thanks all,
craigr
 
There was also this I think from the default bootloader:

ICX64XX-boot>> i2cprobe
I2C has probe the PD69000.(Reg0=0x03)
I2C has probe the EEPROM.(Reg0=0x0000)
Read device SFP Port 4.0 fail
I2C has probe the SFP Port 3.(Reg0=0x03)
Read device SFP Port 2.0 fail
I2C has probe the SFP Port 1.(Reg0=0x03)
I2C has probe the CPLD Interrupt Mask Control.(Reg3=0xff)
I2C has probe the RTC.(Reg0=0x50)
I2C has probe the PCA9535 SFP.(Reg0=0x8d)
I2C has probe the PCA9535 LED.(Reg0=0xff)
I2C has probe the PCA9535 STACK LED.(Reg0=0xff)
I2C has probe the Hardware Monitor.(Regfd=0x50)
loop: 1
i2cprobe FAIL
ICX64XX-boot>>

craigr
 

vpadro

New Member
Jul 16, 2015
14
2
3
45
So I'm trying to figure out what went wrong trying to replicate kapone's post but can't seem to find the culprit of it.

First of all, you'll have to forgive me if I don't make too much sense, since English is not my first language but always try to do my best to communicate with it.

Right now my home gear network consists of the following:

PC Engines APU2 - PfSense
Cisco SG300-52 L3 enabled
ICX7250-48 L3 10Gb license (Thanks @fohdeesha)
ICX6610-48 Fully licensed too (1 PSU rev3, 1 FAN) - Not in use, too loud after a few minutes (maybe will sell it since PSU and fan will cost me as much as another unit)

What I am trying to achive?

Have a native L3 network after a while since FW was handling the VLANs as a Router on a Stick approach, right now it can't handle inter-vlan 1Gb network traffic after doing so for a "some time", since I have the gear that can do all the L3 at its core.

My core networking/services/servers (more APU2s) are on a 12U startech rack is hangin on my apartment's entrance corner and my office is a few meters away cabled with 6 CAT6 ethernet drops, maybe more, but don't tell my wife.

My first approach was to use the ICX7250 as a Core switch for my place on the aforementioned startech rack, and the ICX6610 for my 24U rack with 6 SM servers all with 10Gb NICs and a 40Gbps NIC on my main ESX/NAS server, short long story, as I mentioned the switch is too loud to have it 24/7 on my apartment I didn't even setup up correctly on both ends. - Currently discarded until further notice or until christmas bonus. lol

Then I tried to use the SG300 as my Core L3 switch and the ICX7250 as my rack switch with inter-VLAN routing on my main LAN, this worked "well" can access the SVIs, setup the firewall rules, static routes both the switches and FW can see and communicate, everything was ok but then I realized L3 routing was performed at the Cisco so 10Gb traffic was limited to 1Gb as you might guessed. - Discarded for the time being, maybe will get back to this if could find the routing issue on the ICX.

So third attempt, since the APU has 3 ethernet ports (WAN, LAN, OPT1) tried the Kapone's post guideline, using the OPT1 which was unused to connect one of the cable drops to my office directly to the ICX, created a /30 transit VLAN, gateway, static route on FW, static route on SW, FW rules and what not, but can't communicate from my main home network to the VLANs associated on the ICX, I'm still using the SG300 on my LAN, nothing has done yet in there (No L3 switching, VLANs SVIs, nothing really, just a dumb SW ATM), was thinking to use it as an access L2 SW for the VLANs needed for the APU2s VMs and LXC containers using a second drop back from my rack to the startech rack.

So with all this, which approach will be the best to execute, and more important, am I missing something on my config, steps that might be overlooked?

Basically TDLR;

Need to setup a L3 network using Cisco SG300, ICX7250 and pfsense, but have failed doing so.


Here's the precious data if needed:

VLANs: 40 (Transit), 51-54, 60, 65, 70, 80, 90-92

PfSense
LAN IP: 192.168.50.1/24
Transit IP: 192.168.40.1/30

ICX7250
VLAN/VE 1 IP: 192.168.50.254/24
VLAN/VE 40 (Transit) IP: 192.168.40.2/30

ICX7250 sh run:

Code:
Current configuration:
!
ver 08.0.80eT213
!
stack unit 1
  module 1 icx7250-48-port-management-module
  module 2 icx7250-sfp-plus-8port-80g-module
!
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
 loop-detection
!
vlan 40 name Transit by port
 tagged ethe 1/1/1
 router-interface ve 40
!
vlan 51 name IPMI by port
 tagged ethe 1/1/1
 untagged ethe 1/1/3 to 1/1/12
 router-interface ve 51
!
vlan 52 name VoIP by port
 tagged ethe 1/1/1 ethe 1/1/13 to 1/1/36
 router-interface ve 52
!
vlan 53 name "Guest WiFi" by port
 tagged ethe 1/1/1 ethe 1/1/47 to 1/1/48
 router-interface ve 53
!
vlan 54 name IoT by port
 tagged ethe 1/1/1 ethe 1/1/47 to 1/1/48
 router-interface ve 54
!
vlan 55 name Management by port
 tagged ethe 1/1/1 ethe 1/1/13 to 1/1/40 ethe 1/2/1 to 1/2/8
 router-interface ve 55
!
vlan 60 name "Windows Server" by port
 tagged ethe 1/1/1 ethe 1/1/13 to 1/1/40
 router-interface ve 60
!
vlan 65 name "Linux Server" by port
 tagged ethe 1/1/1 ethe 1/1/13 to 1/1/40
 router-interface ve 65
!
vlan 70 name WORK by port
 tagged ethe 1/1/1 ethe 1/1/13 to 1/1/48 ethe 1/2/1 to 1/2/8
 router-interface ve 70
!
vlan 80 name "Dev Network" by port
 tagged ethe 1/1/1 ethe 1/1/13 to 1/1/40
 router-interface ve 80
!
vlan 90 name Storage by port
 tagged ethe 1/1/1 ethe 1/2/1 to 1/2/8
 router-interface ve 90
!
vlan 91 name Storage2 by port
 tagged ethe 1/1/1 ethe 1/2/1 to 1/2/8
 router-interface ve 91
!
vlan 92 name vMotion by port
 tagged ethe 1/1/1 ethe 1/2/1 to 1/2/8
 router-interface ve 92
!
!
!
!
!
!
!
!
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable aaa console
no fast port-span
hostname icx7250
ip dhcp-client disable
ip dns domain-list padrosys.net
ip dns server-address 192.168.50.5 192.168.50.6
ip route 0.0.0.0/0 192.168.40.1
ip route 0.0.0.0/0 192.168.50.1
!
no telnet server
username root password .....
!
!
snmp-server community ..... rw
snmp-server contact vpadro
snmp-server location Noneedtoknow
!
!
clock summer-time
clock timezone gmt GMT-06
!
!
ntp
 disable serve
 server 192.168.50.1
!
!
no web-management http
web-management https
web-management page-menu
web-management session-timeout 3000
web-management list-menu
!
!
!
!
router ospf
 area 0
!
!
!
!
!
!
!
interface ethernet 1/1/1
 port-name Uplink
 no flow-control both
!
interface ethernet 1/1/2
 no flow-control both
!
interface ethernet 1/1/3
 no flow-control both
!
interface ethernet 1/1/4
 no flow-control both
!
interface ethernet 1/1/5
 no flow-control both
!
interface ethernet 1/1/6
 no flow-control both
!
interface ethernet 1/1/7
 no flow-control both
!
interface ethernet 1/1/8
 no flow-control both
!
interface ethernet 1/1/9
 no flow-control both
!
interface ethernet 1/1/10
 no flow-control both
!
interface ethernet 1/1/11
 no flow-control both
!
interface ethernet 1/1/12
 no flow-control both
!
interface ethernet 1/1/13
 no flow-control both
!
interface ethernet 1/1/14
 no flow-control both
!
interface ethernet 1/1/15
 no flow-control both
!
interface ethernet 1/1/16
 no flow-control both
!
interface ethernet 1/1/17
 no flow-control both
!
interface ethernet 1/1/18
 no flow-control both
!
interface ethernet 1/1/19
 no flow-control both
!
interface ethernet 1/1/20
 no flow-control both
!
interface ethernet 1/1/21
 no flow-control both
!
interface ethernet 1/1/22
 no flow-control both
!
interface ethernet 1/1/23
 no flow-control both
!
interface ethernet 1/1/24
 no flow-control both
!
interface ethernet 1/1/25
 no flow-control both
!
interface ethernet 1/1/26
 no flow-control both
!
interface ethernet 1/1/27
 no flow-control both
!
interface ethernet 1/1/28
 no flow-control both
!
interface ethernet 1/1/29
 no flow-control both
!
interface ethernet 1/1/30
 no flow-control both
!
interface ethernet 1/1/31
 no flow-control both
!
interface ethernet 1/1/32
 no flow-control both
!
interface ethernet 1/1/33
 no flow-control both
!
interface ethernet 1/1/34
 no flow-control both
!
interface ethernet 1/1/35
 no flow-control both
!
interface ethernet 1/1/36
 no flow-control both
!
interface ethernet 1/1/37
 no flow-control both
!
interface ethernet 1/1/38
 no flow-control both
!
interface ethernet 1/1/39
 no flow-control both
!
interface ethernet 1/1/40
 no flow-control both
!
interface ethernet 1/1/41
 no flow-control both
!
interface ethernet 1/1/42
 no flow-control both
!
interface ethernet 1/1/43
 no flow-control both
!
interface ethernet 1/1/44
 no flow-control both
!
interface ethernet 1/1/45
 no flow-control both
!
interface ethernet 1/1/46
 no flow-control both
!
interface ethernet 1/1/47
 no flow-control both
!
interface ethernet 1/1/48
 no flow-control both
!
interface ethernet 1/2/1
 no flow-control both
!
interface ethernet 1/2/2
 no flow-control both
!
interface ethernet 1/2/3
 no flow-control both
!
interface ethernet 1/2/4
 no flow-control both
!
interface ethernet 1/2/5
 no flow-control both
!
interface ethernet 1/2/6
 no flow-control both
!
interface ethernet 1/2/7
 no flow-control both
!
interface ethernet 1/2/8
 no flow-control both
!
interface ve 1
 ip address 192.168.50.254 255.255.255.0
!
interface ve 40
 ip address 192.168.40.2 255.255.255.252
!
interface ve 51
 ip address 192.168.51.1 255.255.255.224
!
interface ve 52
 ip address 192.168.52.1 255.255.255.224
!
interface ve 53
 ip address 192.168.53.1 255.255.255.224
!
interface ve 54
 ip address 192.168.54.1 255.255.255.224
!
interface ve 55
 ip address 192.168.55.1 255.255.255.192
!
interface ve 60
 ip address 192.168.60.1 255.255.255.0
!
interface ve 65
 ip address 192.168.65.1 255.255.255.0
!
interface ve 70
 ip address 192.168.70.1 255.255.255.192
!
interface ve 80
 ip address 192.168.80.1 255.255.255.192
!
interface ve 90
 ip address 192.168.90.1 255.255.255.128
!
interface ve 91
 ip address 192.168.91.1 255.255.255.128
!
interface ve 92
 ip address 10.10.76.1 255.255.255.192
!
!
!
!
!
!
!
!
!
!
end

ICX7250
Code:
Total number of IP routes: 15
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          192.168.40.1    ve 40         1/1           S    8m37s
        0.0.0.0/0          192.168.50.1    ve 1          1/1           S    8m37s
2       10.10.76.0/26      DIRECT          ve 92         0/0           D    1d1h
3       192.168.40.0/30    DIRECT          ve 40         0/0           D    8m38s
4       192.168.50.0/24    DIRECT          ve 1          0/0           D    1d9h
5       192.168.51.0/27    DIRECT          ve 51         0/0           D    1d8h
6       192.168.52.0/27    DIRECT          ve 52         0/0           D    1d8h
7       192.168.53.0/27    DIRECT          ve 53         0/0           D    1d8h
8       192.168.54.0/27    DIRECT          ve 54         0/0           D    1d8h
9       192.168.55.0/26    DIRECT          ve 55         0/0           D    1d8h
10      192.168.60.0/24    DIRECT          ve 60         0/0           D    1d8h
11      192.168.65.0/24    DIRECT          ve 65         0/0           D    1d8h
12      192.168.70.0/26    DIRECT          ve 70         0/0           D    1d8h
13      192.168.80.0/26    DIRECT          ve 80         0/0           D    1d8h
14      192.168.90.0/25    DIRECT          ve 90         0/0           D    1d8h
15      192.168.91.0/25    DIRECT          ve 91         0/0           D    1d8h

Code:
icx7250#ping 192.168.40.1
Sending 1, 16-byte ICMP Echo to 192.168.40.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.40.1    : bytes=16 time<1ms TTL=64
Success rate is 100 percent (1/1), round-trip min/avg/max=0/0/0 ms.
icx7250#ping 192.168.50.50
Sending 1, 16-byte ICMP Echo to 192.168.50.50, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.
icx7250#
PfSense

pfping.jpg

pfinterfaces.jpg
pfnat.jpg
pfgateways.jpg
pfstaticroutes.jpg
pflanrules.jpg
pftransitrules.jpg

If you need more info I'd gladly provide it.
 

JJ27

New Member
Oct 24, 2018
8
1
3
All your stuff
Probably 99% of the forum will be better than me to answer this, but in your case, isn't it better to use the Cisco as a L2+ switch, the ICX as L3 and the FW in the transit VLAN ?

So you trunk all the VLANs from ICX to Cisco (with a LACP if too much traffic) and let the ICX handle the routing as it will do it line rate?

With that you can have the 10g separated from 1g and when not internal the ICX will go to the transit?
 

vpadro

New Member
Jul 16, 2015
14
2
3
45
Probably 99% of the forum will be better than me to answer this, but in your case, isn't it better to use the Cisco as a L2+ switch, the ICX as L3 and the FW in the transit VLAN ?

So you trunk all the VLANs from ICX to Cisco (with a LACP if too much traffic) and let the ICX handle the routing as it will do it line rate?

With that you can have the 10g separated from 1g and when not internal the ICX will go to the transit?
Yup, that was my third attempt, but haven't been able to do all the routing on the ICX since the first attempt, hence the wall post, LACP seems a additional step which I'm not sure if I will take it, previously had the SG300-52 with a SG300-28 doing exactly that topology and had more troubles than stability, probably to my old CAT5e cabling was to be blamed but with 1Gb line from ICX to the SG300 will be ok for now until I get another ICX for my core/rack networking, thank you so much!
 

richtj99

Member
Jul 8, 2017
56
0
6
48
I have three Brocade questions - I am hoping I can get some help on:

1. Lag - I have 6 strand, 3 pair fiber cable running between two switches (6450 & 7250) - Is a lag for redundancy or extra speed? If I setup 3 SFP+ ports on the two switches in a lag and one goes down, does my network go down?

2. NTP: I have a NTP server setup on my network & the Brocade 6450 & 7250 can ping the server, they dont seem to be getting updates?

3. Full factory reset? I have done the "factory set-default" - is there a way to confirm everything is gone? Is there something more I can do to make sure there are no "phone home" things left on the switches? Overwrite firmware, etc?

Thanks,
Rich
 

kapone

Well-Known Member
May 23, 2015
1,049
621
113
@vpadro (and others) - You guys do realize that you're now getting into serious networking configs, which are more analogous to SMB and small companies than just a homelab, right? :)

These kinds of configs with multiple switches, LAGs, VLANs, router config, DHCP/DNS config, FW config...are not easy or straight forward. I'm not sure what to say other than, I'd love to help you, but I can't. I'm way too overwhelmed with family and work at the moment.

Maybe @fohdeesha or others can jump in! :)
 

LodeRunner

Active Member
Apr 27, 2019
221
90
28
1. Lag - I have 6 strand, 3 pair fiber cable running between two switches (6450 & 7250) - Is a lag for redundancy or extra speed? If I setup 3 SFP+ ports on the two switches in a lag and one goes down, does my network go down?
Can be both redundancy and speed, though the throughput of a single stream will never exceed the speed of a single link in the LAG.
 

vpadro

New Member
Jul 16, 2015
14
2
3
45
So I'm trying to figure out what went wrong trying to replicate kapone's post but can't seem to find the culprit of it.

First of all, you'll have to forgive me if I don't make too much sense, since English is not my first language but always try to do my best to communicate with it.

Right now my home gear network consists of the following:

PC Engines APU2 - PfSense
Cisco SG300-52 L3 enabled
ICX7250-48 L3 10Gb license (Thanks @fohdeesha)
ICX6610-48 Fully licensed too (1 PSU rev3, 1 FAN) - Not in use, too loud after a few minutes (maybe will sell it since PSU and fan will cost me as much as another unit)

What I am trying to achive?

Have a native L3 network after a while since FW was handling the VLANs as a Router on a Stick approach, right now it can't handle inter-vlan 1Gb network traffic after doing so for a "some time", since I have the gear that can do all the L3 at its core.

My core networking/services/servers (more APU2s) are on a 12U startech rack is hangin on my apartment's entrance corner and my office is a few meters away cabled with 6 CAT6 ethernet drops, maybe more, but don't tell my wife.

My first approach was to use the ICX7250 as a Core switch for my place on the aforementioned startech rack, and the ICX6610 for my 24U rack with 6 SM servers all with 10Gb NICs and a 40Gbps NIC on my main ESX/NAS server, short long story, as I mentioned the switch is too loud to have it 24/7 on my apartment I didn't even setup up correctly on both ends. - Currently discarded until further notice or until christmas bonus. lol

Then I tried to use the SG300 as my Core L3 switch and the ICX7250 as my rack switch with inter-VLAN routing on my main LAN, this worked "well" can access the SVIs, setup the firewall rules, static routes both the switches and FW can see and communicate, everything was ok but then I realized L3 routing was performed at the Cisco so 10Gb traffic was limited to 1Gb as you might guessed. - Discarded for the time being, maybe will get back to this if could find the routing issue on the ICX.

So third attempt, since the APU has 3 ethernet ports (WAN, LAN, OPT1) tried the Kapone's post guideline, using the OPT1 which was unused to connect one of the cable drops to my office directly to the ICX, created a /30 transit VLAN, gateway, static route on FW, static route on SW, FW rules and what not, but can't communicate from my main home network to the VLANs associated on the ICX, I'm still using the SG300 on my LAN, nothing has done yet in there (No L3 switching, VLANs SVIs, nothing really, just a dumb SW ATM), was thinking to use it as an access L2 SW for the VLANs needed for the APU2s VMs and LXC containers using a second drop back from my rack to the startech rack.

So with all this, which approach will be the best to execute, and more important, am I missing something on my config, steps that might be overlooked?

Basically TDLR;

Need to setup a L3 network using Cisco SG300, ICX7250 and pfsense, but have failed doing so.


Here's the precious data if needed:

VLANs: 40 (Transit), 51-54, 60, 65, 70, 80, 90-92

PfSense
LAN IP: 192.168.50.1/24
Transit IP: 192.168.40.1/30

ICX7250
VLAN/VE 1 IP: 192.168.50.254/24
VLAN/VE 40 (Transit) IP: 192.168.40.2/30

ICX7250 sh run:

Code:
Current configuration:
!
ver 08.0.80eT213
!
stack unit 1
  module 1 icx7250-48-port-management-module
  module 2 icx7250-sfp-plus-8port-80g-module
!
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
loop-detection
!
vlan 40 name Transit by port
tagged ethe 1/1/1
router-interface ve 40
!
vlan 51 name IPMI by port
tagged ethe 1/1/1
untagged ethe 1/1/3 to 1/1/12
router-interface ve 51
!
vlan 52 name VoIP by port
tagged ethe 1/1/1 ethe 1/1/13 to 1/1/36
router-interface ve 52
!
vlan 53 name "Guest WiFi" by port
tagged ethe 1/1/1 ethe 1/1/47 to 1/1/48
router-interface ve 53
!
vlan 54 name IoT by port
tagged ethe 1/1/1 ethe 1/1/47 to 1/1/48
router-interface ve 54
!
vlan 55 name Management by port
tagged ethe 1/1/1 ethe 1/1/13 to 1/1/40 ethe 1/2/1 to 1/2/8
router-interface ve 55
!
vlan 60 name "Windows Server" by port
tagged ethe 1/1/1 ethe 1/1/13 to 1/1/40
router-interface ve 60
!
vlan 65 name "Linux Server" by port
tagged ethe 1/1/1 ethe 1/1/13 to 1/1/40
router-interface ve 65
!
vlan 70 name WORK by port
tagged ethe 1/1/1 ethe 1/1/13 to 1/1/48 ethe 1/2/1 to 1/2/8
router-interface ve 70
!
vlan 80 name "Dev Network" by port
tagged ethe 1/1/1 ethe 1/1/13 to 1/1/40
router-interface ve 80
!
vlan 90 name Storage by port
tagged ethe 1/1/1 ethe 1/2/1 to 1/2/8
router-interface ve 90
!
vlan 91 name Storage2 by port
tagged ethe 1/1/1 ethe 1/2/1 to 1/2/8
router-interface ve 91
!
vlan 92 name vMotion by port
tagged ethe 1/1/1 ethe 1/2/1 to 1/2/8
router-interface ve 92
!
!
!
!
!
!
!
!
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable aaa console
no fast port-span
hostname icx7250
ip dhcp-client disable
ip dns domain-list padrosys.net
ip dns server-address 192.168.50.5 192.168.50.6
ip route 0.0.0.0/0 192.168.40.1
ip route 0.0.0.0/0 192.168.50.1
!
no telnet server
username root password .....
!
!
snmp-server community ..... rw
snmp-server contact vpadro
snmp-server location Noneedtoknow
!
!
clock summer-time
clock timezone gmt GMT-06
!
!
ntp
disable serve
server 192.168.50.1
!
!
no web-management http
web-management https
web-management page-menu
web-management session-timeout 3000
web-management list-menu
!
!
!
!
router ospf
area 0
!
!
!
!
!
!
!
interface ethernet 1/1/1
port-name Uplink
no flow-control both
!
interface ethernet 1/1/2
no flow-control both
!
interface ethernet 1/1/3
no flow-control both
!
interface ethernet 1/1/4
no flow-control both
!
interface ethernet 1/1/5
no flow-control both
!
interface ethernet 1/1/6
no flow-control both
!
interface ethernet 1/1/7
no flow-control both
!
interface ethernet 1/1/8
no flow-control both
!
interface ethernet 1/1/9
no flow-control both
!
interface ethernet 1/1/10
no flow-control both
!
interface ethernet 1/1/11
no flow-control both
!
interface ethernet 1/1/12
no flow-control both
!
interface ethernet 1/1/13
no flow-control both
!
interface ethernet 1/1/14
no flow-control both
!
interface ethernet 1/1/15
no flow-control both
!
interface ethernet 1/1/16
no flow-control both
!
interface ethernet 1/1/17
no flow-control both
!
interface ethernet 1/1/18
no flow-control both
!
interface ethernet 1/1/19
no flow-control both
!
interface ethernet 1/1/20
no flow-control both
!
interface ethernet 1/1/21
no flow-control both
!
interface ethernet 1/1/22
no flow-control both
!
interface ethernet 1/1/23
no flow-control both
!
interface ethernet 1/1/24
no flow-control both
!
interface ethernet 1/1/25
no flow-control both
!
interface ethernet 1/1/26
no flow-control both
!
interface ethernet 1/1/27
no flow-control both
!
interface ethernet 1/1/28
no flow-control both
!
interface ethernet 1/1/29
no flow-control both
!
interface ethernet 1/1/30
no flow-control both
!
interface ethernet 1/1/31
no flow-control both
!
interface ethernet 1/1/32
no flow-control both
!
interface ethernet 1/1/33
no flow-control both
!
interface ethernet 1/1/34
no flow-control both
!
interface ethernet 1/1/35
no flow-control both
!
interface ethernet 1/1/36
no flow-control both
!
interface ethernet 1/1/37
no flow-control both
!
interface ethernet 1/1/38
no flow-control both
!
interface ethernet 1/1/39
no flow-control both
!
interface ethernet 1/1/40
no flow-control both
!
interface ethernet 1/1/41
no flow-control both
!
interface ethernet 1/1/42
no flow-control both
!
interface ethernet 1/1/43
no flow-control both
!
interface ethernet 1/1/44
no flow-control both
!
interface ethernet 1/1/45
no flow-control both
!
interface ethernet 1/1/46
no flow-control both
!
interface ethernet 1/1/47
no flow-control both
!
interface ethernet 1/1/48
no flow-control both
!
interface ethernet 1/2/1
no flow-control both
!
interface ethernet 1/2/2
no flow-control both
!
interface ethernet 1/2/3
no flow-control both
!
interface ethernet 1/2/4
no flow-control both
!
interface ethernet 1/2/5
no flow-control both
!
interface ethernet 1/2/6
no flow-control both
!
interface ethernet 1/2/7
no flow-control both
!
interface ethernet 1/2/8
no flow-control both
!
interface ve 1
ip address 192.168.50.254 255.255.255.0
!
interface ve 40
ip address 192.168.40.2 255.255.255.252
!
interface ve 51
ip address 192.168.51.1 255.255.255.224
!
interface ve 52
ip address 192.168.52.1 255.255.255.224
!
interface ve 53
ip address 192.168.53.1 255.255.255.224
!
interface ve 54
ip address 192.168.54.1 255.255.255.224
!
interface ve 55
ip address 192.168.55.1 255.255.255.192
!
interface ve 60
ip address 192.168.60.1 255.255.255.0
!
interface ve 65
ip address 192.168.65.1 255.255.255.0
!
interface ve 70
ip address 192.168.70.1 255.255.255.192
!
interface ve 80
ip address 192.168.80.1 255.255.255.192
!
interface ve 90
ip address 192.168.90.1 255.255.255.128
!
interface ve 91
ip address 192.168.91.1 255.255.255.128
!
interface ve 92
ip address 10.10.76.1 255.255.255.192
!
!
!
!
!
!
!
!
!
!
end

ICX7250
Code:
Total number of IP routes: 15
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          192.168.40.1    ve 40         1/1           S    8m37s
        0.0.0.0/0          192.168.50.1    ve 1          1/1           S    8m37s
2       10.10.76.0/26      DIRECT          ve 92         0/0           D    1d1h
3       192.168.40.0/30    DIRECT          ve 40         0/0           D    8m38s
4       192.168.50.0/24    DIRECT          ve 1          0/0           D    1d9h
5       192.168.51.0/27    DIRECT          ve 51         0/0           D    1d8h
6       192.168.52.0/27    DIRECT          ve 52         0/0           D    1d8h
7       192.168.53.0/27    DIRECT          ve 53         0/0           D    1d8h
8       192.168.54.0/27    DIRECT          ve 54         0/0           D    1d8h
9       192.168.55.0/26    DIRECT          ve 55         0/0           D    1d8h
10      192.168.60.0/24    DIRECT          ve 60         0/0           D    1d8h
11      192.168.65.0/24    DIRECT          ve 65         0/0           D    1d8h
12      192.168.70.0/26    DIRECT          ve 70         0/0           D    1d8h
13      192.168.80.0/26    DIRECT          ve 80         0/0           D    1d8h
14      192.168.90.0/25    DIRECT          ve 90         0/0           D    1d8h
15      192.168.91.0/25    DIRECT          ve 91         0/0           D    1d8h

Code:
icx7250#ping 192.168.40.1
Sending 1, 16-byte ICMP Echo to 192.168.40.1, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.40.1    : bytes=16 time<1ms TTL=64
Success rate is 100 percent (1/1), round-trip min/avg/max=0/0/0 ms.
icx7250#ping 192.168.50.50
Sending 1, 16-byte ICMP Echo to 192.168.50.50, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.
icx7250#
PfSense

View attachment 18764


If you need more info I'd gladly provide it.
So removing the ve 1 did the trick, but really would like to know why I don't need that SVI since every guide I've read including fohdeesha's suggest creating the ve 1 as a starting point when configuring the switch, maybe some documentation that I am overlooking or misreading could be helpful.

Thank you.
 

vpadro

New Member
Jul 16, 2015
14
2
3
45
@vpadro (and others) - You guys do realize that you're now getting into serious networking configs, which are more analogous to SMB and small companies than just a homelab, right? :)

These kinds of configs with multiple switches, LAGs, VLANs, router config, DHCP/DNS config, FW config...are not easy or straight forward. I'm not sure what to say other than, I'd love to help you, but I can't. I'm way too overwhelmed with family and work at the moment.
Thanks for taking the time, yes indeed, my enviroment is suppossed to emulate a few of what I've seen on the wild west, and it will be more complicated afterwards, hehe.

Best of luck.
 
  • Like
Reactions: EngineerNate
Quick (and possibly stupid) question on a different topic. My 6450 does not retain the clock setting when rebooted or when power is pulled. I'm tired of being in 1969 even though it may have been a good year. I have done write memory after setting the clock. I changed the battery and tested both the old and new, and they are both good.

Have tried @fohdeesha tutorial:

NTP
To have the switch keep its time synced via NTP (so its logs make more sense), use the following. If you live in an area that doesn't use Daylight Savings, skip the clock summer-time command. Use tab completion for the timezone command to see what's available. The IP's in the following example are google's NTP servers and work well for most cases:

clock summer-time
clock timezone gmt GMT-05
ntp
disable serve
server 216.239.35.0
server 216.239.35.4
exit
I have tried all this and even attempted through the GUI. I tried just setting the time with no ntp and no servers.

wtf? Am I missing something very obvious or is my switch actually broken?

Thanks for any help!
craigr
 
Also this was a quick and dirty way to spoof the 6450-24P into thinking it still has two fans to avoid the fan fail message.

f1.jpg

(I actually did wind up putting the sole fan in the normal position after experimenting not as shown)

f2.jpg

It would be better form to remove the board and solder onto the bottom, but this was so much easier. You can of course also splice into the fan wire if you don't want to solder. A three pin header is also an option so as not to accidentally cross, but I have these made up and handy.

Could this be a possible solution for 6610 and 6650 users to quiet down the fans by using less of them? Just throwing it out there as I have never used those switches.

I switched the fan to the popular Sunon MagLev KDE1204PKV3.MS.AR.GN and also wound up adding a Noctua speed reducer/silencer, and the 6450 still reports that the fans are OK. It's really quiet now and my stead state temp under very heavy load is 60.5 degrees. As always, @fohdeesha is absolutely correct, this switch just won't get too hot for a home lab as long as there is just some air going through. And, the fan can indeed start when the switch is booted and the fan is disconnected and reconnected at low voltage.

craigr
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,294
2,383
113
30
fohdeesha.com
Quick (and possibly stupid) question on a different topic. My 6450 does not retain the clock setting when rebooted or when power is pulled. I'm tired of being in 1969 even though it may have been a good year. I have done write memory after setting the clock. I changed the battery and tested both the old and new, and they are both good.

Have tried @fohdeesha tutorial:



I have tried all this and even attempted through the GUI. I tried just setting the time with no ntp and no servers.

wtf? Am I missing something very obvious or is my switch actually broken?

Thanks for any help!
craigr
if you post your config (output of "show run") I can have a look. also post the output of "show ntp ass" and "show ntp stat"
 

richtj99

Member
Jul 8, 2017
56
0
6
48
Can be both redundancy and speed, though the throughput of a single stream will never exceed the speed of a single link in the LAG.
Thank you - I have been playing with this a bit - if I have three 10gb ports, the speed wont exceed 10gb but I can have three separate transfers of up to 10 gb x 3 (theoretical?) at the same time?


if you post your config (output of "show run") I can have a look. also post the output of "show ntp ass" and "show ntp stat"
So this wasnt at me specifically but i am having an identical problem.

Code:
BrO6450-200#show ntp ass
   address         ref clock      st  when  poll reach  delay   offset  disp
 ~192.168.1.29  LOCL             1    30    64   377  1.880 21474836  0.997
* synced, # selected, + candidate, - outlayer, x falseticker, ~ configured
BrO6450-200#show ntp stat
 Clock is unsynchronized, no reference clock
 NTP server mode is disabled, NTP client mode is enabled
 NTP master mode is disabled, NTP master stratum is 8
 NTP is not in panic mode
Sh Run (without the junk)
Code:
ip dhcp-client disable
ip dns server-address 192.168.1.29
ip route 0.0.0.0/0 192.168.1.1
!
!
!
clock timezone us Eastern
!
!
ntp
 disable serve
 server 192.168.1.29
!
BrO6450-200#ping 192.168.1.29
Sending 1, 16-byte ICMP Echo to 192.168.1.29, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 192.168.1.29  : bytes=16 time=1ms TTL=128
Success rate is 100 percent (1/1), round-trip min/avg/max=1/1/1 ms.
Code:
BrO6450-200#sh clock
12:07:49.121 Eastern Fri Jan 30 1970
A manual reset in the web interface fixes it until it reboots

I think i am missing something basic but cant figure what it is.
 
Oh one other tip for fan mods with the Sunon MagLev KDE1204PKV3.MS.AR.GN. If you don't want to
if you post your config (output of "show run") I can have a look. also post the output of "show ntp ass" and "show ntp stat"
Thank you! The config is very simple as I have tried resetting it again:

SSH@switch#show run
Current configuration:
!
ver 08.0.30tT313
!
stack unit 1
module 1 icx6450-24p-poe-port-management-module
module 2 icx6450-sfp-plus-4port-40g-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
!
optical-monitor
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable telnet authentication
hostname switch
ip dhcp-client disable
!
username root password .....
!
!
clock summer-time
clock timezone us Central
!
!
ntp
disable serve
!
!
web-management refresh tftp 30
web-management refresh front-panel 30
web-management connection-receive-timeout 30
web-management session-timeout 3000
!
!
!
interface ethernet 1/1/1
port-name WAN
!
interface ethernet 1/1/2
port-name IPMI
!
interface ethernet 1/1/4
port-name Office
!
interface ethernet 1/2/1
port-name unRAID1
!
interface ethernet 1/2/3
port-name unRAID2
!
interface ve 1
ip address 192.168.1.2 255.255.255.0
!
!
!
!
!
!
!
!
!
end

I get no response from " show ntp ass ".

SSH@switch#show ntp stat
Clock is unsynchronized, no reference clock
NTP server mode is disabled, NTP client mode is enabled
NTP master mode is disabled, NTP master stratum is 8
NTP is not in panic mode

Like I said in the post above, I am pretty sure I disabled ntp yesterday and tried to just manually set the clock.


Regarding recoding my 10G RJ45 "optics" with i2c to official Brocade; the only reason I want to do this is so that I can get temperature information on the optic's chips to see if they remain normal. I know they are always hot, but I have been testing running iperf both directions at full 10G up and down simultaneously with -t 60000. The 6450 hovers around 98% utilization on both ports Tx and Rx and is moving near theoretical top speed.

After some time iperf will just stop (about 10-15 minutes) and also the system becomes erratic, loss of LAN and/or WAN. I am trying to see what may have been the culprit in this. I have been doing the testing with a single Intel X550-T2 (high quality Chinese knock off) with one port assigned to a host OS (slackware) and the other port passed through bare metal (each port is in it's own IOMMU group by default) to the guest VM Win10. So the X550-T2 is basically running in a two circles between host and guest. The Brocade shows around 98% utilization on both ports up and down. The optics get very hot and the 6450 is not hot. The X550-T2 seems to be at a bit hot but at a tolerable temperature.

Thanks again,
craigr
 

kapone

Well-Known Member
May 23, 2015
1,049
621
113
@CIR-Engineering - Looking at your config, you don't seem to have an NTP "server" (that the switch should sync with) defined for the switch??

Mine (it's a 6610 btw, but that should make no difference) looks like this (the 172.16.x.2 are my pair of firewalls that also serve as NTP servers):

Code:
clock summer-time                                                 
clock timezone us Eastern

ntp
 disable serve
 server 172.16.0.2
 server 172.16.1.2
And NTP stats work as expected.

Code:
show ntp stat
 Clock is synchronized, stratum 3, reference clock is 172.16.1.2
 precision is 2**-16
 reference time is 3831128794.1405142281 (13:26:34.1405142281 Eastern Thu May 27 2021)
 clock offset is -0.1163 msec, root delay is 35.2241 msec
 root dispersion is 42.8897 msec,  peer dispersion is 40.4451 msec
 system poll interval is 64,  last clock update was 334 sec ago
 NTP server mode is disabled, NTP client mode is enabled
 NTP master mode is disabled, NTP master stratum is 8
 NTP is not in panic mode
Code:
show ntp ass
   address         ref clock      st  when  poll reach  delay   offset  disp
 ~172.16.0.2      STEP            16     -  1024     0   0.00    0.000 15937.
*~172.16.1.2      50.192.156.119   2    33    64   377  0.445  -0.1163  3.667
* synced, # selected, + candidate, - outlayer, x falseticker, ~ configured
 
  • Like
Reactions: CIR-Engineering
@CIR-Engineering - Looking at your config, you don't seem to have an NTP "server" (that the switch should sync with) defined for the switch??

Mine (it's a 6610 btw, but that should make no difference) looks like this (the 172.16.x.2 are my pair of firewalls that also serve as NTP servers):

Code:
clock summer-time                                                
clock timezone us Eastern

ntp
disable serve
server 172.16.0.2
server 172.16.1.2
And NTP stats work as expected.

Code:
show ntp stat
Clock is synchronized, stratum 3, reference clock is 172.16.1.2
precision is 2**-16
reference time is 3831128794.1405142281 (13:26:34.1405142281 Eastern Thu May 27 2021)
clock offset is -0.1163 msec, root delay is 35.2241 msec
root dispersion is 42.8897 msec,  peer dispersion is 40.4451 msec
system poll interval is 64,  last clock update was 334 sec ago
NTP server mode is disabled, NTP client mode is enabled
NTP master mode is disabled, NTP master stratum is 8
NTP is not in panic mode
Code:
show ntp ass
   address         ref clock      st  when  poll reach  delay   offset  disp
~172.16.0.2      STEP            16     -  1024     0   0.00    0.000 15937.
*~172.16.1.2      50.192.156.119   2    33    64   377  0.445  -0.1163  3.667
* synced, # selected, + candidate, - outlayer, x falseticker, ~ configured
I think I had set it up before, but deleted it. So I just did this:

Code:
SSH@switch#show ntp stat
 Clock is unsynchronized, no reference clock
 NTP server mode is disabled, NTP client mode is enabled
 NTP master mode is disabled, NTP master stratum is 8
 NTP is not in panic mode

SSH@switch#config t
SSH@switch(config)#clock summer-time
SSH@switch(config)#clock timezone us central
SSH@switch(config)#ntp
SSH@switch(config-ntp)#disable serve
SSH@switch(config-ntp)# server 172.16.0.2
SSH@switch(config-ntp)# server 172.16.1.2
SSH@switch(config-ntp)#write mem

SSH@switch(config-ntp)#Flash Memory Write (8192 bytes per dot) .
Write startup-config done.
Copy Done.
SSH@switch(config-ntp)#show ntp stat
 Clock is unsynchronized, no reference clock
 NTP server mode is disabled, NTP client mode is enabled
 NTP master mode is disabled, NTP master stratum is 8
 NTP is not in panic mode
This does not seem right. What the heck am I doing wrong?!? Or is it broken. Seems odd that the 6450 is working fine with other commands and not ntp :mad:. I did follow @fohdeesha's reset and config procedure to the letter when I got the switch last week.

Please advise and thanks as always,
craigr
 

itronin

Well-Known Member
Nov 24, 2018
742
428
63
Denver, Colorado
I think I had set it up before, but deleted it. So I just did this:

Code:
SSH@switch#config t
SSH@switch(config)#clock summer-time
SSH@switch(config)#clock timezone us central
SSH@switch(config)#ntp
SSH@switch(config-ntp)#disable serve
SSH@switch(config-ntp)# server 172.16.0.2
SSH@switch(config-ntp)# server 172.16.1.2
SSH@switch(config-ntp)#write mem

interface ve 1
ip address 192.168.1.2 255.255.255.0
Per your preivous config I did not see a default route in your switch *and* your switch's IP address is in a different subnet ... Did you simply cut'n paste @kapone 's config? If so its not gonna work for ya.

First question, do you have a local NTP server on your network?
If Yes substitute its IP address for what kapone had.
If not you can try using one of the global NTP server addresses however your switch will need to be configured to get to the outside world.