I thought @fohdeesha doesn’t drink alcoholIn case you were wondering, he accepts donations for beer and maybe food
Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
liquor is the only reason I still reply to this threadI thought @fohdeesha doesn’t drink alcohol
I don't mind going to the CLI when I need to but junos just feels like unneeded pain. I'm by far not an expert but I've always managed to get a switch up and running pretty quickly, the EX3300? not so much.I'm sure you've figured out I've spent (and am still spending) a lot of time with the IOS CLI.
JunOS is...verbose. Which is amazing in a production environment where I need to create nested lists that feed into BGP filters etc, but at home, yeah you're probably gunna feel like you're wasting time if you don't already have a bunch of junOS experience. Wanna trade for an ICX6450? lmao
I very much like
Aside from that, if one doesn't use it regularly it's difficult to remember the higher level node names and once you do see the config scrolling up is necessary to see them--a pain in the ass. For that
commit confirmed exists in JunOS even though I typically don't use it. I normally use commit check. I've been bit before where issuing a change in conf t immediately borked my config forcing me to use the console or power cycle.Aside from that, if one doesn't use it regularly it's difficult to remember the higher level node names and once you do see the config scrolling up is necessary to see them--a pain in the ass. For that
show config|display set displays the full command in a non-structured format.I swear you just described my experience with my EX2300. lol. The initial configuration was fine but dang, creating Firewall Filters are like writing a novel. I can create Cisco ACLs in my sleep but these firewall filters have me stuck.
Liquor is the only reason half my co workers are still alive.....liquor is the only reason I still reply to this thread
Ok ive got a weird issue going on.
So i had issues getting breakout working, so ended up getting a new 6610. I was able to get the breakouts working on the new one.
I then setup stacking between the new and old. And am now not able to get breakout working on the new one. And it only works on the old one.
I used the directions above. So its like the breakout port fixed itself on the old and broke on the new. So it has to do with stacking somehow.
I have 2 diff breakout cables. one is FS, and one is Brocade from FS.
I have the exact same config/setup as you used here.
Any other thoughts on how to make breakout work on unit 1? It was working before stacking.
ID Type Role Mac Address Pri State Comment
1 S ICX6610-48P active 748e.f8fb.b8b6 128 local Ready
2 S ICX6610-48P standby cc4e.2413.2d44 0 remote Ready
active standby
+---+ +---+
-2/6| 1 |2/1--2/1| 2 |2/6-
| +---+ +---+ |
| |
|------------------------|
Standby u2 - protocols ready, can failover
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
priority 128
stack-port 1/2/1 1/2/6
stack unit 2
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack-port 2/2/1 2/2/6
stack enable
Unit 1 is the new one, unit 2 is the old. Breakout never worked on the old one pre stacking. It worked fine on the new one prestacking. Now it is reversed, and does not work on the new one, but works on the old.
Thoughts?
So i had issues getting breakout working, so ended up getting a new 6610. I was able to get the breakouts working on the new one.
I then setup stacking between the new and old. And am now not able to get breakout working on the new one. And it only works on the old one.
I used the directions above. So its like the breakout port fixed itself on the old and broke on the new. So it has to do with stacking somehow.
I have 2 diff breakout cables. one is FS, and one is Brocade from FS.
I have the exact same config/setup as you used here.
Any other thoughts on how to make breakout work on unit 1? It was working before stacking.
ID Type Role Mac Address Pri State Comment
1 S ICX6610-48P active 748e.f8fb.b8b6 128 local Ready
2 S ICX6610-48P standby cc4e.2413.2d44 0 remote Ready
active standby
+---+ +---+
-2/6| 1 |2/1--2/1| 2 |2/6-
| +---+ +---+ |
| |
|------------------------|
Standby u2 - protocols ready, can failover
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
priority 128
stack-port 1/2/1 1/2/6
stack unit 2
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack-port 2/2/1 2/2/6
stack enable
Unit 1 is the new one, unit 2 is the old. Breakout never worked on the old one pre stacking. It worked fine on the new one prestacking. Now it is reversed, and does not work on the new one, but works on the old.
Thoughts?
Having trouble using the breakout stack ports on a 6610 to a server. Stacking is not enabled. Using a decent quality breakout DAC cable.
I'm sure I'm missing something stupid.
I configured a static LAG to go to an ESXi server using 1/2/9 and 1/2/10, primary port 1/2/9. Deployed. Links were up, lights on the back of the server were lit. Then I added 1/2/9 as tagged to one of the vlans and the link went down on both ports. I have not been able to get them to come back up. I know the cable/nic/switchports are physically fine because I saw link before. The link dropped the instant that I did the "tagged ethe 1/2/9" in the vlan config. I saw it out of the corner of my eye because I was tailing the vmkernel.log on the server. I have since completely removed the ports from all vlans, the lag itself, added the two ports to my main vlan (happens to be 101) and rebooted the ESXi. There is no chance of a loop as the NICs are "unused adapters" on the esxi side. STP appears to be enabled on the port but not on any of my vlans.
I cannot for the life of me get these links to come back up. I have not reloaded the switch because it is passing other traffic right now but I can try that if necessary. It doesn't seem like this is the sort of thing that should require reboot of the switch. What stupid thing am I missing?
I'm sure I'm missing something stupid.
I configured a static LAG to go to an ESXi server using 1/2/9 and 1/2/10, primary port 1/2/9. Deployed. Links were up, lights on the back of the server were lit. Then I added 1/2/9 as tagged to one of the vlans and the link went down on both ports. I have not been able to get them to come back up. I know the cable/nic/switchports are physically fine because I saw link before. The link dropped the instant that I did the "tagged ethe 1/2/9" in the vlan config. I saw it out of the corner of my eye because I was tailing the vmkernel.log on the server. I have since completely removed the ports from all vlans, the lag itself, added the two ports to my main vlan (happens to be 101) and rebooted the ESXi. There is no chance of a loop as the NICs are "unused adapters" on the esxi side. STP appears to be enabled on the port but not on any of my vlans.
I cannot for the life of me get these links to come back up. I have not reloaded the switch because it is passing other traffic right now but I can try that if necessary. It doesn't seem like this is the sort of thing that should require reboot of the switch. What stupid thing am I missing?
Code:
10GigabitEthernet 1/2/9 is down, line protocol is down
Port down for 11 minute(s) 48 second(s)
Hardware is 10GigabitEthernet , address is 748e.f8e9.a6d0 (bia 748e.f8e9.a709)
Configured speed 10Gbit, actual unknown, configured duplex fdx, actual unknown
Configured mdi mode AUTO, actual unknown
Member of L2 VLAN ID 101, port is untagged, port state is BLOCKING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Openflow is Disabled, Openflow Hybrid mode is Disabled, Flow Control is config enabled, oper enabled, negotiation disabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
Port name is Trunk vSphere Umbra 10G
MTU 1500 bytes, encapsulation ethernet
300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
0 packets output, 0 bytes, 0 underruns
Transmitted 0 broadcasts, 0 multicasts, 0 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled
Last edited:
Protip:
If you upgrade/migrate from the "switch mode" firmware to the "router mode" firmware in @fohdeesha 's guides, you will break your management interface and have to console into the switch
If you upgrade/migrate from the "switch mode" firmware to the "router mode" firmware in @fohdeesha 's guides, you will break your management interface and have to console into the switch
Thank God for the console!Protip:
If you upgrade/migrate from the "switch mode" firmware to the "router mode" firmware in @fohdeesha 's guides, you will break your management interface and have to console into the switch
reason #12324 why it's easier to just start on the layer3 FW in the first place, even if you don't need l3 features at the momentProtip:
If you upgrade/migrate from the "switch mode" firmware to the "router mode" firmware in @fohdeesha 's guides, you will break your management interface and have to console into the switch
zero downsideFor the two people above having breakout link issues, once the other end is plugged in and the server is booted up etc, can you reboot all the switches? On some chassis I've seen some weird stuff where the breakout stack ports only want to link up if the stuff is all plugged in and ready on switch boot
Ill give it a try. Thanksreason #12324 why it's easier to just start on the layer3 FW in the first place, even if you don't need l3 features at the moment
For the two people above having breakout link issues, once the other end is plugged in and the server is booted up etc, can you reboot all the switches? On some chassis I've seen some weird stuff where the breakout stack ports only want to link up if the stuff is all plugged in and ready on switch boot
Regarding mine, the following comes out of the serial console on startup. A clue? I'm using 1/2/2, 1/2/3 in a static LAG, and same for 1/2/7, 1/2/8. No physical connections on 1/2/1, 1/2/4, 1/2/5, 1/2/6, 1/2/9, 1/2/10 - yet. Do all four lanes in the QSFP+ on these breakout stacking ports have to be physically connected to something?
Code:
Parsing Config Data ...
------------------------------------------------------------------
M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/2 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/2
------------------------------------------------------------------
M:9 L:0 - chow_qsfp_read, qsfp 3, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 3, port 1/2/7 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/7If this switch doesn't like servers at the end of the breakout QSFP+ ports going up and down without itself being reloaded too this may not meet my use case - I like to keep one ESXi up most of the time and only spin up the others if I need them - power use and all... Going to have to experiment and will report.
The conversation starting here sounds almost exactly like what I'm experiencing - including adding tagged vlans to them causing the links to go down and never come back up until the switch is reloaded. I think the conclusion was that the switch was bad, but could the fact that I'm seeing the same thing show this is a bug instead? I do have another 6610 to try if it's necessary but they came from the same place so who knows if trying it will show anything conclusive. I'm running the latest 08.0.30u firmware. I do have some different breakouts coming in the mail supposedly today too - another thing to try.
Last edited:
Hi all - two questions about the ICX7xxx series:
1. Can the SFP+ "uplink" ports be used just like any other port on the switch? Or are there limitations on what they can/cannot be used for. (I know only certain ports can be uplinks, but can uplinks be "normal" ports)?
2. Are you able to stack any model ICX7xxx with any other model ICX7xxx? For example a ICX7150-C12P at my desk and a "bigger beef" ICX7450 in a sever rack?
I found this, but it ONLY mentions the 7150:
docs.commscope.com
And this, but it doesn't mention the 7150:
So I'm concerned I can't mix the entry-level switches with the higher-end ones in a single stack.
Thank you!!
1. Can the SFP+ "uplink" ports be used just like any other port on the switch? Or are there limitations on what they can/cannot be used for. (I know only certain ports can be uplinks, but can uplinks be "normal" ports)?
2. Are you able to stack any model ICX7xxx with any other model ICX7xxx? For example a ICX7150-C12P at my desk and a "bigger beef" ICX7450 in a sever rack?
I found this, but it ONLY mentions the 7150:
Commscope Technical Content Portal
And this, but it doesn't mention the 7150:
So I'm concerned I can't mix the entry-level switches with the higher-end ones in a single stack.
Thank you!!
Does anyone know if there is a complete replacement for the icx6650? Needs to have all fiber ports at 10g. Sorry if this was covered in the thread already.
Uplink ports can be use as normal switchports.Hi all - two questions about the ICX7xxx series:
1. Can the SFP+ "uplink" ports be used just like any other port on the switch? Or are there limitations on what they can/cannot be used for. (I know only certain ports can be uplinks, but can uplinks be "normal" ports)?
2. Are you able to stack any model ICX7xxx with any other model ICX7xxx? For example a ICX7150-C12P at my desk and a "bigger beef" ICX7450 in a sever rack?
I found this, but it ONLY mentions the 7150:
Commscope Technical Content Portal
And this, but it doesn't mention the 7150:
So I'm concerned I can't mix the entry-level switches with the higher-end ones in a single stack.
Thank you!!
No cross-model stacking. Must be in same family. So 71xx, 72xx, 74xx, etc.
The only exception is if you are using them as 802.1br SPX extenders with a 76, 77, or 78 series as the CB, but that's apparently gone away in v9 of the firmware, possibly indicating EoL/discontinuation of 802.1br support.