Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[shpitz461]

Any idea why i can't set mtu on a port?
trying this Commscope Technical Content Portal

device(config)# interface ethernet 1/1/5
device(config-if-1/2/1)# ip mtu 1500
device(config-if-1/2/1)# write memory
device(config-if-1/2/1)# end
device# reload

but when I run the commands, I get an error about 'ip mtu':

SSH@icx6610(config)#interface ethernet 1/2/1
SSH@icx6610(config-if-e40000-1/2/1)#ip mtu 1500
Invalid input -> mtu 1500
Type ? for a list

Also, this is what Terry's videos show, the 'ip mtu' command, but it doesn't work on my 6610.

Is there a proper user manual for the commandline of the 6610?

Thanks

 

[jode]

(ICX6610) Rear QSFP+ Ports​

Describing the rear ports can be a little confusing in text, so below is an extremely professional graphic showing how they break down. 2 of them are 40gbE only ports - for your freenas, hypervisors, whatever. The other 2 are 4x 10gbE breakout (each) only:

To break out those 2x breakout ports, you have many options:

You can use a DAC breakout cable: Dell Networking Cable QSFP+ 40GBE - 4X SFP+ 10GBE 3M 27GG5 | eBay

Or you can use a 40gbE optic to give you an MPO fiber connector: Brocade XBR-000232 QSFP 57-1000267-01 4*16Gb SWL 100m for DCX8510 EMC | eBay

Note: the above optic is a 4x 16gbps fibrechannel optic, which makes them available for super cheap. They work perfectly fine in brocade ethernet switches, as they are not picky and simply clock the optic down to 10gbps per channel. I have been using quite a few for a while now

Then using an optic like above, use MPO fiber like this: 5m 50/125 Standard MTP Multimode Fiber Optic Patch Cable Key-up to Key-down | eBay

To go to an MPO breakout panel like this: Systimax SCS InstaPATCH 24-Port Fiber Patch Panel Module 642337970740 | eBay

A single one of those panels will break out 4x MPO connectors - with 4x 10gbe LC connectors per MPO, that's 16x 10gbe ports - for instance if you had a stack of 2x ICX6610's - that's 4x total QSFP+ breakout ports, to one panel like this.

I had the hardest time wrapping my head around the section above, probably because eBay listings have long since expired. Allow me to suggest the following links to add 8 additional 10gb LC ports in an additional 1U:

Brocade XBR-000232 (~$7)
QSFP MPO/MTP 8F to 4x LC Fiber Optic Breakout Cable, Multimode OM3 Aqua (~$35)
Fiber Optic Adapter Patch Panel with 6 LC UPC Duplex Connectors (~$10 x2)
1U 19" Blank Rack Mount Fiber Optic Optical Patch Panel (~$20)

There may be cheaper alternatives, there may be better alternatives. I was looking for links that may stay active on eBay for a little bit and descriptions that are easy to search for in case they aren't.

Just ordered my ICX6610-24P-PI (I like my ports on the back of the rack).

 

[jode]

Is there a proper user manual for the commandline of the 6610?

In @fohdeesha 's zip file there is a folder called "ICX6xxx manuals". Look for the file called "fastiron-08030p-commandref.pdf".

Look in document "fastiron-08030mb-l3guide.pdf" on page 46ff for "changing the MTU".

 

[bwahaha]

@jode

Here's one of the 24x mpo lc thingys from that link.

Fiber Optic Cassette - 24F Elite MPO MTP - LC | eBay

24F Elite MPO MTP - LC Cassette. But, we know nearly zero about fiber optic components. We have described them to the best of our limited understanding of what they are.

www.ebay.com

If it were me, I'd go with lc keystones and a keysone patch panel This gives a bit of expansion for the future.

8x lc keystones for about $25

Keystone Insert - Fiber Optic LC Type (9/125 50/125) OM2/3/4 | eBay

Suitable for OM4 and OM4 10G Multi-mode 50/125, Singlemode Duplex 9/125, and 10/40/100G multi-mode duplex installs. Connector Type: LC both front and back. Supports Single Mode (9/125) and Multimode (50/125) fiber cables.

www.ebay.com

$16 for 16 keystone slots in 1u

16-Port Keystone Jack Blank Patch Panel Plate Cat5e Cat6 RJ45 19" Rack Mount 1U | eBay

Rack Height: 1U (1.75"). The perfect patch panel when you need more than one type of port. Just load with your favorite keystone jacks. 16 Blank Keystone Style Ports. 16-Port Blank Patch Panel. Standard 19" Rack Mountable.

www.ebay.com

Realistically, using external links in a faq is stupid, but it's been the norm on forums since before forums and ebay. But what are you gonna do? (post ebay links!)

 

[shpitz461]

In @fohdeesha 's zip file there is a folder called "ICX6xxx manuals". Look for the file called "fastiron-08030p-commandref.pdf".

Look in document "fastiron-08030mb-l3guide.pdf" on page 46ff for "changing the MTU".

Ok, I'm there on page 46 (actually the commands are shown on page 48), I ran it exactly as shown, but i get in SSH the following error:

SSH@icx6610(config-if-e1000-1/1/1)#ip mtu 1500
Invalid input -> mtu 1500
Type ? for a list

Also, I've enabled jumbo frames and reloaded the switch.
Is there a way to default MTU to 1500 on all ports and then set 9000 on specific ones? e.g. on the 40G in the back that is connected to my NAS.

I can't seem to be able to run the commands in the manuals, some work and some don't.

Does the switch need to be in a specific mode?

When I ssh into it, I do
'enable'
'config terminal'
then run commands on ports.

Spoiler: ssh login and running-config

login as: root
Keyboard-interactive authentication prompts from server:
| Password:
End of keyboard-interactive prompts from server
SSH@icx6610>
SSH@icx6610>enable
No password has been assigned yet...
SSH@icx6610#config t
SSH@icx6610(config)#show running
running-config Current running-config
SSH@icx6610(config)#show running-config
Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
no legacy-inline-power
stack disable
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
!
mtu-exceed forward
optical-monitor
aaa authentication web-server default local
aaa authentication login default local
jumbo
hostname icx6610
ip dhcp-client disable
ip default-network 192.168.1.0/24
ip dns server-address 192.168.1.251
ip show-portname
ip route 0.0.0.0/0 192.168.1.254
!
no telnet server
username root password .....
username maor privilege 4 password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone us Eastern
!
!
ntp
disable serve
server 216.239.35.0
server 216.239.35.4
!
!
no web-management http
web-management https
!
!
!
!
!
!
!
interface ethernet 1/1/1
inline power power-limit 1000
!
interface ethernet 1/1/2
inline power priority 1
!
interface ethernet 1/1/3
inline power
!
interface ethernet 1/1/4
port-name GWN_7610
inline power
!
interface ethernet 1/1/5
inline power
!
interface ethernet 1/1/6
inline power
!
interface ethernet 1/1/7
port-name Office
inline power
!
interface ethernet 1/1/8
inline power
!
interface ethernet 1/2/1
port-name Rome_40G_eth1
!
interface ethernet 1/2/6
port-name Z790
!
interface ethernet 1/3/1
port-name Rome_10G_eth1
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
port-name Rome_10G_eth2
speed-duplex 10G-full
!
interface ethernet 1/3/5
speed-duplex 10G-full
!
interface ethernet 1/3/6
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
port-name Aiden MoCA
speed-duplex 10G-full
!
interface ve 1
ip address 192.168.1.2 255.255.255.0
!
!
!
!
!
!
!
!
!
end

I appreciate the help!

 

[shpitz461]

So I'm playing with GUI, I go to IP->Interface menu.

I can make changes to port v1, but no other ports, e.g.

Where does this v1 port come from and what does it mean?
Do i need to create a v-port for each physical port in the switch?
Completely confused...

 

[dmerillat]

Ok, I'm there on page 46 (actually the commands are shown on page 48), I ran it exactly as shown, but i get in SSH the following error:


Also, I've enabled jumbo frames and reloaded the switch.
Is there a way to default MTU to 1500 on all ports and then set 9000 on specific ones? e.g. on the 40G in the back that is connected to my NAS.

I can't seem to be able to run the commands in the manuals, some work and some don't.

Does the switch need to be in a specific mode?

When I ssh into it, I do
'enable'
'config terminal'
then run commands on ports.

In my limited playing with the 6610, MTU seems to be only configurable on the virtual interfaces.

SSH@coreswitch#show run int ve 2
interface ve 2
ip address 192.168.0.3 255.255.255.0
ip mtu 1500

Everything else gives an error. I think that's correct behavior for L2 switching rather than L3 routing?

Edit:

Where does this v1 port come from and what does it mean?
Do i need to create a v-port for each physical port in the switch?

Imagine you've got a physical router sitting above a switch in the rack. You patch the LAN port to the switch on a physical port, and now all other devices on the switch can access the router's gateway.

Now build the router into the switch, and you can connect it to a vlan internally by creating a virtual port rather than using a patch cable.

 

[shpitz461]

As you can tell from my questions, I'm a total noob// Best I can do at this point is setup basic networking an AP's.
So when it comes to MTU, i need to create a virtual interface on each physical port?
Right now I'm using an Ubiquity ERLite-3 as my router, in the near future I'm planning on replacing it with a OPNSense router running on a Lenovo ThinkCentre with 10GbE card.
If I need to set an IP address on a virtual interface on the 6610 like @dmerillat is showing above, how will VLAN DHCP work in OPNSense?
Is there a guide somewhere that explains the terms and shows how to set up a switch step-by-step?
Thanks!

 

[dmerillat]

As you can tell from my questions, I'm a total noob// Best I can do at this point is setup basic networking an AP's.
So when it comes to MTU, i need to create a virtual interface on each physical port?
Right now I'm using an Ubiquity ERLite-3 as my router, in the near future I'm planning on replacing it with a OPNSense router running on a Lenovo ThinkCentre with 10GbE card.
If I need to set an IP address on a virtual interface on the 6610 like @dmerillat is showing above, how will VLAN DHCP work in OPNSense?
Is there a guide somewhere that explains the terms and shows how to set up a switch step-by-step?
Thanks!

I don't think you need to do anything at all. It doesn't hurt anything to be jumbo-frame capable on a LAN that's not using it. Cheap, unmanaged gig-e switches usually pass jumbo frames without any configuration and they just work when everybody is set to the default 1500.

The only time anything should care is if you have the 6610 configured to route between a large mtu vlan and a smaller one, which is what the virtual interface MTU is for.

 

[Ralph_IT]

Hi, looking for some help here.

I purchased a 7250-48P and have been trying to get into the console with Putty. I made a DIY console cable by cutting up a mini usb cable and putting the other end on a RJ45 keystone as I had seen others do in this thread. I then used this with a new OIKWAN usb to RJ45 console cable off Amazon (Blue cisco cable).

I used the following pinout
Mini USB - Keystone
1 (VCC/red) - NC
2 (RX/white) - 6 (TX)
3 (TX/green) - 3 (RX)
4 (GND/black) - 4 (GND)

When I first tried opening Putty with the cable, I got output but when it got to the point to press b to go into the bootloader it didn't appear to be working. I tried to look online to see if anyone else had similar issues with Putty not sending input to the switch, but it seemed like the most likely cause was my cable. So I looked back at my cable and thought I'd try repunching the wires. I pulled the wires off the keystone and repunched it on another keystone trying to ensure I punched everything down well. After this, I tried again and now I am not even getting any output on Putty. I double checked my settings and the COM port didn't change, but nothing seems to work.

Did I somehow fry my console port? Is it still potentially just a cable issue? Any other ideas?

I had the same issue with a DIY cable, and had to re-punch the keystone more than 10 times to work properly.
This is the wiring I used, and it works fine (if you do it right, that is).

 

[fohdeesha]

it's almost 2024 don't use jumbo frames

 

[oudude07]

I had the same issue with a DIY cable, and had to re-punch the keystone more than 10 times to work properly.
This is the wiring I used, and it works fine (if you do it right, that is).

Thanks for the reply!

After trying to get the cable working for a few hours yesterday (punched it down a few more times and was getting frustrated) I was ready to give up on this DIY cable and was looking at other options on ebay today. After I read your comment I decided to try again today and after punching it down another time I hooked it up and everything worked! I guess the wires in the mini USB are just hard to get a good connection on a keystone. An easier solution may just be to cut up the cisco cable as well as I had seen some people do, but I wanted to keep that cable if I needed it for other hardware in the future. Anyways, getting everything set up with fohdeesha's guide now!

 

[linuxsrc]

Hi all, I have an ICX6650 and I am attempting to connect a Mellanox MSX6720. I am attempting to use ports 1/2/3 and ports 1/2/4 to connect to the Mellanox switch configured in ethernet mode for ports 1/35 and 1/36. Both systems when I run show media, both systems show the type of connected cable.
Below is the output from the Mellanox side

Spoiler: Mellanox Interface

Eth1/36:
Admin state : Enabled
Operational state : Up
Last change in operational status : 14:40:24 ago (77 oper change)
Boot delay time : 0 sec
Description : UpLink-ICX6650
Mac address : 24:8a:07:d1:05:4f
MTU : 9216 bytes (Maximum packet size 9238 bytes)
Fec : auto
Flow-control : receive off send off
Actual speed : 40 Gbps
Auto-negotiation : Enabled
Width reduction mode : Unknown
Switchport mode : trunk
MAC learning mode : Enabled
Last clearing of "show interface" counters: Never
60 seconds ingress rate : 0 bits/sec, 0 bytes/sec, 0 packets/sec
60 seconds egress rate : 0 bits/sec, 0 bytes/sec, 0 packets/sec

Rx:
0 packets
0 unicast packets
0 multicast packets
0 broadcast packets
0 bytes
0 discard packets
0 error packets
0 fcs errors
0 undersize packets
0 oversize packets
0 pause packets
0 unknown control opcode
0 symbol errors

Tx:
7 packets
0 unicast packets
7 multicast packets
0 broadcast packets
570 bytes
2789797 discard packets
0 error packets
0 hoq discard packets

Spoiler: Connected Cable Mellanox Side

Identifier :		QSFP+
Cable/ Module type :		Passive copper, unequalized
Ethernet speed and type :		56GigE
Vendor :		Mellanox

Cable length :		3 m
Part number :		670759-B25
Revision :		A1
Serial number :		6C2634028J

Spoiler: ICX Switch Interface

show interfaces ethernet 1/2/3
40GigabitEthernet1/2/3 is down, line protocol is down
Port down for 14 hours 47 minutes 37 seconds
Hardware is 40GigabitEthernet, address is 748e.f8fb.1c00 (bia 748e.f8fb.1c3b)
Interface type is 40Gig Copper
Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
Configured mdi mode AUTO, actual unknown
Member of 9 L2 VLANs, port is tagged, port state is BLOCKING
BPDU guard is Disabled, ROOT protect is Enabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
MTU 10240 bytes, encapsulation ethernet
300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
145 packets input, 19778 bytes, 0 no buffer
Received 0 broadcasts, 145 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
228 packets output, 30627 bytes, 0 underruns
Transmitted 0 broadcasts, 228 multicasts, 0 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

Egress queues:
Queue counters Queued packets Dropped Packets
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
7 228 0

Spoiler: ICX Interface Media

show media ethernet 1/2/3
Port 1/2/3: Type : 40GBASE-Passive Copper
Vendor Name: Mellanox Serial Num: 6C2634028J Revision: A1

I am not getting any traffic across, it seems as if the ICX switch just is not happy with the link. I have tried a Plain Jane non-Mellanox 40gb cable as well and it has the same issue. The Mellanox switch lights the port, but the ICX switch will not.

Any pointers on making this work would be great.
Thanks

 

[kevindd992002]

it's almost 2024 don't use jumbo frames

Why? I was looking into enabling jumbo frames as I thought they're good?

 

[fohdeesha]

in 2003

 

[bwahaha]

in 2003

I totally profess my ignorance on all things layer 2 and up, but how so? I'm not running them, the effort isn't worth it for my home lab, but it's often recommended, especially when discussions of poor 10g performance are brought up.

 

[fohdeesha]

I totally profess my ignorance on all things layer 2 and up, but how so? I'm not running them, the effort isn't worth it for my home lab, but it's often recommended, especially when discussions of poor 10g performance are brought up.

because in 2023 if you are having performance issues with 10gbE, you have much deeper issues with your setup and jumbo frames will be a bandaid. I can saturate a 10gbps link with default 1500 MTU on a $10 15 year old xeon, if you need jumbo frames to make 10gbe work on something newer you have bigger fish to fry. 40gbe and up is a different story depending on hardware and traffic type

 

[kevindd992002]

because in 2023 if you are having performance issues with 10gbE, you have much deeper issues with your setup and jumbo frames will be a bandaid. I can saturate a 10gbps link with default 1500 MTU on a $10 15 year old xeon, if you need jumbo frames to make 10gbe work on something newer you have bigger fish to fry. 40gbe and up is a different story depending on hardware and traffic type

I am in the process of revamping my home network to 10GbE. I installed a duplex foc from my server room to one of the bedrooms in the house. The bedroom has a windows 11 pc with a mellanox cx3 card. The server room has a windows 11 laptop with a qnap 10gbe sfp+ to thunderbolt adapter. To test, I used iperf3. I cannot saturate the link until I enabled jumbo frames. So is this a limitation from the qnap adapter or iperf3?

 

[fohdeesha]

Hi all, I have an ICX6650 and I am attempting to connect a Mellanox MSX6720. I am attempting to use ports 1/2/3 and ports 1/2/4 to connect to the Mellanox switch configured in ethernet mode for ports 1/35 and 1/36. Both systems when I run show media, both systems show the type of connected cable.
Below is the output from the Mellanox side

Spoiler: Mellanox Interface

Eth1/36:
Admin state : Enabled
Operational state : Up
Last change in operational status : 14:40:24 ago (77 oper change)
Boot delay time : 0 sec
Description : UpLink-ICX6650
Mac address : 24:8a:07:d1:05:4f
MTU : 9216 bytes (Maximum packet size 9238 bytes)
Fec : auto
Flow-control : receive off send off
Actual speed : 40 Gbps
Auto-negotiation : Enabled
Width reduction mode : Unknown
Switchport mode : trunk
MAC learning mode : Enabled
Last clearing of "show interface" counters: Never
60 seconds ingress rate : 0 bits/sec, 0 bytes/sec, 0 packets/sec
60 seconds egress rate : 0 bits/sec, 0 bytes/sec, 0 packets/sec

Rx:
0 packets
0 unicast packets
0 multicast packets
0 broadcast packets
0 bytes
0 discard packets
0 error packets
0 fcs errors
0 undersize packets
0 oversize packets
0 pause packets
0 unknown control opcode
0 symbol errors

Tx:
7 packets
0 unicast packets
7 multicast packets
0 broadcast packets
570 bytes
2789797 discard packets
0 error packets
0 hoq discard packets

Spoiler: Connected Cable Mellanox Side

Identifier :		QSFP+
Cable/ Module type :		Passive copper, unequalized
Ethernet speed and type :		56GigE
Vendor :		Mellanox

Cable length :		3 m
Part number :		670759-B25
Revision :		A1
Serial number :		6C2634028J

Spoiler: ICX Switch Interface

show interfaces ethernet 1/2/3
40GigabitEthernet1/2/3 is down, line protocol is down
Port down for 14 hours 47 minutes 37 seconds
Hardware is 40GigabitEthernet, address is 748e.f8fb.1c00 (bia 748e.f8fb.1c3b)
Interface type is 40Gig Copper
Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
Configured mdi mode AUTO, actual unknown
Member of 9 L2 VLANs, port is tagged, port state is BLOCKING
BPDU guard is Disabled, ROOT protect is Enabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
MTU 10240 bytes, encapsulation ethernet
300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
145 packets input, 19778 bytes, 0 no buffer
Received 0 broadcasts, 145 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
228 packets output, 30627 bytes, 0 underruns
Transmitted 0 broadcasts, 228 multicasts, 0 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

Egress queues:
Queue counters Queued packets Dropped Packets
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
7 228 0

Spoiler: ICX Interface Media

show media ethernet 1/2/3
Port 1/2/3: Type : 40GBASE-Passive Copper
Vendor Name: Mellanox Serial Num: 6C2634028J Revision: A1

I am not getting any traffic across, it seems as if the ICX switch just is not happy with the link. I have tried a Plain Jane non-Mellanox 40gb cable as well and it has the same issue. The Mellanox switch lights the port, but the ICX switch will not.

Any pointers on making this work would be great.
Thanks

what's the output of "show pod"

 

[Jason Antes]

Why? I was looking into enabling jumbo frames as I thought they're good?

The only place I use jumbo frames is with storage, generally FCIP doing replication traffic out of an FC network. Not something I've had to deal with much for years though since many storage arrays that I've dealt with more recently have ethernet ports for replication. Some of them strongly suggest using jumbo frames, though they'll settle for standard frames and complain about it.