Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[diode]

I think the issue with the cables were on 7k series devices that had USB-C form factor ports for console access that didn't use standard USB-C pinouts. I believe it would try and negotiate power over some of the pins and fry something.

As far as I know, any router serial cable with an RJ45 end for the switch and labeled as Cisco compatible will work for a 6k device. But I don't have a 6610 specifically, so I can't 100% confirm. I use this one with a 6450.

Ahh, thanks for the clarification on the USB-C issue. I guess I won’t have to worry about that with my 6610 since after looking at the manual I see that the serial console ports are rj45 and on the back with the 48 port switch.

For some reason I thought that the serial console cable plugged into a micro usb on the switch and then you would use the rj45 connector on the PC side. I think I just confused myself while trying to digest all of the info over the last few months.

 

[Midvalley]

That's my expectation (RJ45-Switch, USB-PC), the data sheet says the management port on the 6610 is RJ45. Is there a USB-C port on it? The clearest picture I found shows a 10101 RJ45 port on the back (serial connection) next to the managment port and the QSFP's.

 

[jei]

That's my expectation (RJ45-Switch, USB-PC), the data sheet says the management port on the 6610 is RJ45. Is there a USB-C port on it? The clearest picture I found shows a 10101 RJ45 port on the back (serial connection) next to the managment port and the QSFP's.

No USB-C on 6000 series.

 

[ano]

Has anyone replaced the power supply in a 7250-48P?

I have a 7250-48P with fried PoE. I'd like to remove the PoE board and the original power supply with PoE (first, to have a spare, second, to save energy).

Does anyone have closeup pics of the non-P power supply? Seems to be a Great Wall, but I cannot see voltage + current from the photos in the forum and the photos I have. It's probably 12V, but how many amps?
Also I'd have to measure the dimensions of the original PSU.

Meanwell EPP-150-12 might be a good replacement.

wouldnt happen to know the power usage exactly of that unit idle, and with some ports connected?

 

[56ip1234]

I working with a new to me 7150-48P that has Fastiron 09.0.10fT213 installed and I'm trying to get the web interface working to access some of the views. The configuration is setup with "web-management http" and "web-management https" enables. I'd like to disable http but it is enabled for testing. When running nmap from a computer on the same subnet the switch is not responding on port 80 or 443. Is there some sort of access list I need to update? I use the CLI for configuration but the web interface can be useful for viewing things so I would like to have access.

Any suggestions are welcome.

Thanks,
Ian

 

[oudude07]

Hi, looking for some help here.

I purchased a 7250-48P and have been trying to get into the console with Putty. I made a DIY console cable by cutting up a mini usb cable and putting the other end on a RJ45 keystone as I had seen others do in this thread. I then used this with a new OIKWAN usb to RJ45 console cable off Amazon (Blue cisco cable).

I used the following pinout
Mini USB - Keystone
1 (VCC/red) - NC
2 (RX/white) - 6 (TX)
3 (TX/green) - 3 (RX)
4 (GND/black) - 4 (GND)

When I first tried opening Putty with the cable, I got output but when it got to the point to press b to go into the bootloader it didn't appear to be working. I tried to look online to see if anyone else had similar issues with Putty not sending input to the switch, but it seemed like the most likely cause was my cable. So I looked back at my cable and thought I'd try repunching the wires. I pulled the wires off the keystone and repunched it on another keystone trying to ensure I punched everything down well. After this, I tried again and now I am not even getting any output on Putty. I double checked my settings and the COM port didn't change, but nothing seems to work.

Did I somehow fry my console port? Is it still potentially just a cable issue? Any other ideas?

 

[safado]

The 6650 however, I'm learning, takes on a whole new meaning for getting console! I've had to order several cables (from serial USB to USB mini) and now finally going the USB to DB9 Male --> DB9 female to USB mini. For anyone coming by here, the prior links to ebay from fohdeesha and others aren't working. I'll update with specifics if this actually works.

 

[shpitz461]

Any idea why i can't set mtu on a port?
trying this Commscope Technical Content Portal

device(config)# interface ethernet 1/1/5
device(config-if-1/2/1)# ip mtu 1500
device(config-if-1/2/1)# write memory
device(config-if-1/2/1)# end
device# reload

but when I run the commands, I get an error about 'ip mtu':

SSH@icx6610(config)#interface ethernet 1/2/1
SSH@icx6610(config-if-e40000-1/2/1)#ip mtu 1500
Invalid input -> mtu 1500
Type ? for a list

Also, this is what Terry's videos show, the 'ip mtu' command, but it doesn't work on my 6610.

Is there a proper user manual for the commandline of the 6610?

Thanks

 

[jode]

(ICX6610) Rear QSFP+ Ports​

Describing the rear ports can be a little confusing in text, so below is an extremely professional graphic showing how they break down. 2 of them are 40gbE only ports - for your freenas, hypervisors, whatever. The other 2 are 4x 10gbE breakout (each) only:

To break out those 2x breakout ports, you have many options:

You can use a DAC breakout cable: Dell Networking Cable QSFP+ 40GBE - 4X SFP+ 10GBE 3M 27GG5 | eBay

Or you can use a 40gbE optic to give you an MPO fiber connector: Brocade XBR-000232 QSFP 57-1000267-01 4*16Gb SWL 100m for DCX8510 EMC | eBay

Note: the above optic is a 4x 16gbps fibrechannel optic, which makes them available for super cheap. They work perfectly fine in brocade ethernet switches, as they are not picky and simply clock the optic down to 10gbps per channel. I have been using quite a few for a while now

Then using an optic like above, use MPO fiber like this: 5m 50/125 Standard MTP Multimode Fiber Optic Patch Cable Key-up to Key-down | eBay

To go to an MPO breakout panel like this: Systimax SCS InstaPATCH 24-Port Fiber Patch Panel Module 642337970740 | eBay

A single one of those panels will break out 4x MPO connectors - with 4x 10gbe LC connectors per MPO, that's 16x 10gbe ports - for instance if you had a stack of 2x ICX6610's - that's 4x total QSFP+ breakout ports, to one panel like this.

I had the hardest time wrapping my head around the section above, probably because eBay listings have long since expired. Allow me to suggest the following links to add 8 additional 10gb LC ports in an additional 1U:

Brocade XBR-000232 (~$7)
QSFP MPO/MTP 8F to 4x LC Fiber Optic Breakout Cable, Multimode OM3 Aqua (~$35)
Fiber Optic Adapter Patch Panel with 6 LC UPC Duplex Connectors (~$10 x2)
1U 19" Blank Rack Mount Fiber Optic Optical Patch Panel (~$20)

There may be cheaper alternatives, there may be better alternatives. I was looking for links that may stay active on eBay for a little bit and descriptions that are easy to search for in case they aren't.

Just ordered my ICX6610-24P-PI (I like my ports on the back of the rack).

 

[jode]

Is there a proper user manual for the commandline of the 6610?

In @fohdeesha 's zip file there is a folder called "ICX6xxx manuals". Look for the file called "fastiron-08030p-commandref.pdf".

Look in document "fastiron-08030mb-l3guide.pdf" on page 46ff for "changing the MTU".

 

[bwahaha]

@jode

Here's one of the 24x mpo lc thingys from that link.

Fiber Optic Cassette - 24F Elite MPO MTP - LC | eBay

24F Elite MPO MTP - LC Cassette. But, we know nearly zero about fiber optic components. We have described them to the best of our limited understanding of what they are.

www.ebay.com

If it were me, I'd go with lc keystones and a keysone patch panel This gives a bit of expansion for the future.

8x lc keystones for about $25

Keystone Jack - Fiber Optic LC Type (9/125 50/125) OM2/3/4 | eBay

Suitable for OM4 and OM4 10G Multi-mode 50/125, Singlemode Duplex 9/125, and 10/40/100G multi-mode duplex installs. Connector Type: LC both front and back. Supports Single Mode (9/125) and Multimode (50/125) fiber cables.

www.ebay.com

$16 for 16 keystone slots in 1u

16-Port Keystone Jack Blank Patch Panel Plate Cat5e Cat6 RJ45 19" Rack Mount 1U | eBay

Rack Height: 1U (1.75"). The perfect patch panel when you need more than one type of port. Just load with your favorite keystone jacks. 16 Blank Keystone Style Ports. 16-Port Blank Patch Panel. Standard 19" Rack Mountable.

www.ebay.com

Realistically, using external links in a faq is stupid, but it's been the norm on forums since before forums and ebay. But what are you gonna do? (post ebay links!)

 

[shpitz461]

In @fohdeesha 's zip file there is a folder called "ICX6xxx manuals". Look for the file called "fastiron-08030p-commandref.pdf".

Look in document "fastiron-08030mb-l3guide.pdf" on page 46ff for "changing the MTU".

Ok, I'm there on page 46 (actually the commands are shown on page 48), I ran it exactly as shown, but i get in SSH the following error:

SSH@icx6610(config-if-e1000-1/1/1)#ip mtu 1500
Invalid input -> mtu 1500
Type ? for a list

Also, I've enabled jumbo frames and reloaded the switch.
Is there a way to default MTU to 1500 on all ports and then set 9000 on specific ones? e.g. on the 40G in the back that is connected to my NAS.

I can't seem to be able to run the commands in the manuals, some work and some don't.

Does the switch need to be in a specific mode?

When I ssh into it, I do
'enable'
'config terminal'
then run commands on ports.

Spoiler: ssh login and running-config

login as: root
Keyboard-interactive authentication prompts from server:
| Password:
End of keyboard-interactive prompts from server
SSH@icx6610>
SSH@icx6610>enable
No password has been assigned yet...
SSH@icx6610#config t
SSH@icx6610(config)#show running
running-config Current running-config
SSH@icx6610(config)#show running-config
Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
no legacy-inline-power
stack disable
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
!
mtu-exceed forward
optical-monitor
aaa authentication web-server default local
aaa authentication login default local
jumbo
hostname icx6610
ip dhcp-client disable
ip default-network 192.168.1.0/24
ip dns server-address 192.168.1.251
ip show-portname
ip route 0.0.0.0/0 192.168.1.254
!
no telnet server
username root password .....
username maor privilege 4 password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone us Eastern
!
!
ntp
disable serve
server 216.239.35.0
server 216.239.35.4
!
!
no web-management http
web-management https
!
!
!
!
!
!
!
interface ethernet 1/1/1
inline power power-limit 1000
!
interface ethernet 1/1/2
inline power priority 1
!
interface ethernet 1/1/3
inline power
!
interface ethernet 1/1/4
port-name GWN_7610
inline power
!
interface ethernet 1/1/5
inline power
!
interface ethernet 1/1/6
inline power
!
interface ethernet 1/1/7
port-name Office
inline power
!
interface ethernet 1/1/8
inline power
!
interface ethernet 1/2/1
port-name Rome_40G_eth1
!
interface ethernet 1/2/6
port-name Z790
!
interface ethernet 1/3/1
port-name Rome_10G_eth1
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
port-name Rome_10G_eth2
speed-duplex 10G-full
!
interface ethernet 1/3/5
speed-duplex 10G-full
!
interface ethernet 1/3/6
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
port-name Aiden MoCA
speed-duplex 10G-full
!
interface ve 1
ip address 192.168.1.2 255.255.255.0
!
!
!
!
!
!
!
!
!
end

I appreciate the help!

 

[shpitz461]

So I'm playing with GUI, I go to IP->Interface menu.

I can make changes to port v1, but no other ports, e.g.

Where does this v1 port come from and what does it mean?
Do i need to create a v-port for each physical port in the switch?
Completely confused...

 

[dmerillat]

Ok, I'm there on page 46 (actually the commands are shown on page 48), I ran it exactly as shown, but i get in SSH the following error:


Also, I've enabled jumbo frames and reloaded the switch.
Is there a way to default MTU to 1500 on all ports and then set 9000 on specific ones? e.g. on the 40G in the back that is connected to my NAS.

I can't seem to be able to run the commands in the manuals, some work and some don't.

Does the switch need to be in a specific mode?

When I ssh into it, I do
'enable'
'config terminal'
then run commands on ports.

In my limited playing with the 6610, MTU seems to be only configurable on the virtual interfaces.

SSH@coreswitch#show run int ve 2
interface ve 2
ip address 192.168.0.3 255.255.255.0
ip mtu 1500

Everything else gives an error. I think that's correct behavior for L2 switching rather than L3 routing?

Edit:

Where does this v1 port come from and what does it mean?
Do i need to create a v-port for each physical port in the switch?

Imagine you've got a physical router sitting above a switch in the rack. You patch the LAN port to the switch on a physical port, and now all other devices on the switch can access the router's gateway.

Now build the router into the switch, and you can connect it to a vlan internally by creating a virtual port rather than using a patch cable.

 

[shpitz461]

As you can tell from my questions, I'm a total noob// Best I can do at this point is setup basic networking an AP's.
So when it comes to MTU, i need to create a virtual interface on each physical port?
Right now I'm using an Ubiquity ERLite-3 as my router, in the near future I'm planning on replacing it with a OPNSense router running on a Lenovo ThinkCentre with 10GbE card.
If I need to set an IP address on a virtual interface on the 6610 like @dmerillat is showing above, how will VLAN DHCP work in OPNSense?
Is there a guide somewhere that explains the terms and shows how to set up a switch step-by-step?
Thanks!

 

[dmerillat]

As you can tell from my questions, I'm a total noob// Best I can do at this point is setup basic networking an AP's.
So when it comes to MTU, i need to create a virtual interface on each physical port?
Right now I'm using an Ubiquity ERLite-3 as my router, in the near future I'm planning on replacing it with a OPNSense router running on a Lenovo ThinkCentre with 10GbE card.
If I need to set an IP address on a virtual interface on the 6610 like @dmerillat is showing above, how will VLAN DHCP work in OPNSense?
Is there a guide somewhere that explains the terms and shows how to set up a switch step-by-step?
Thanks!

I don't think you need to do anything at all. It doesn't hurt anything to be jumbo-frame capable on a LAN that's not using it. Cheap, unmanaged gig-e switches usually pass jumbo frames without any configuration and they just work when everybody is set to the default 1500.

The only time anything should care is if you have the 6610 configured to route between a large mtu vlan and a smaller one, which is what the virtual interface MTU is for.

 

[Ralph_IT]

Hi, looking for some help here.

I purchased a 7250-48P and have been trying to get into the console with Putty. I made a DIY console cable by cutting up a mini usb cable and putting the other end on a RJ45 keystone as I had seen others do in this thread. I then used this with a new OIKWAN usb to RJ45 console cable off Amazon (Blue cisco cable).

I used the following pinout
Mini USB - Keystone
1 (VCC/red) - NC
2 (RX/white) - 6 (TX)
3 (TX/green) - 3 (RX)
4 (GND/black) - 4 (GND)

When I first tried opening Putty with the cable, I got output but when it got to the point to press b to go into the bootloader it didn't appear to be working. I tried to look online to see if anyone else had similar issues with Putty not sending input to the switch, but it seemed like the most likely cause was my cable. So I looked back at my cable and thought I'd try repunching the wires. I pulled the wires off the keystone and repunched it on another keystone trying to ensure I punched everything down well. After this, I tried again and now I am not even getting any output on Putty. I double checked my settings and the COM port didn't change, but nothing seems to work.

Did I somehow fry my console port? Is it still potentially just a cable issue? Any other ideas?

I had the same issue with a DIY cable, and had to re-punch the keystone more than 10 times to work properly.
This is the wiring I used, and it works fine (if you do it right, that is).

 

[fohdeesha]

it's almost 2024 don't use jumbo frames

 

[oudude07]

I had the same issue with a DIY cable, and had to re-punch the keystone more than 10 times to work properly.
This is the wiring I used, and it works fine (if you do it right, that is).

Thanks for the reply!

After trying to get the cable working for a few hours yesterday (punched it down a few more times and was getting frustrated) I was ready to give up on this DIY cable and was looking at other options on ebay today. After I read your comment I decided to try again today and after punching it down another time I hooked it up and everything worked! I guess the wires in the mini USB are just hard to get a good connection on a keystone. An easier solution may just be to cut up the cisco cable as well as I had seen some people do, but I wanted to keep that cable if I needed it for other hardware in the future. Anyways, getting everything set up with fohdeesha's guide now!

 

[linuxsrc]

Hi all, I have an ICX6650 and I am attempting to connect a Mellanox MSX6720. I am attempting to use ports 1/2/3 and ports 1/2/4 to connect to the Mellanox switch configured in ethernet mode for ports 1/35 and 1/36. Both systems when I run show media, both systems show the type of connected cable.
Below is the output from the Mellanox side

Spoiler: Mellanox Interface

Eth1/36:
Admin state : Enabled
Operational state : Up
Last change in operational status : 14:40:24 ago (77 oper change)
Boot delay time : 0 sec
Description : UpLink-ICX6650
Mac address : 24:8a:07:d1:05:4f
MTU : 9216 bytes (Maximum packet size 9238 bytes)
Fec : auto
Flow-control : receive off send off
Actual speed : 40 Gbps
Auto-negotiation : Enabled
Width reduction mode : Unknown
Switchport mode : trunk
MAC learning mode : Enabled
Last clearing of "show interface" counters: Never
60 seconds ingress rate : 0 bits/sec, 0 bytes/sec, 0 packets/sec
60 seconds egress rate : 0 bits/sec, 0 bytes/sec, 0 packets/sec

Rx:
0 packets
0 unicast packets
0 multicast packets
0 broadcast packets
0 bytes
0 discard packets
0 error packets
0 fcs errors
0 undersize packets
0 oversize packets
0 pause packets
0 unknown control opcode
0 symbol errors

Tx:
7 packets
0 unicast packets
7 multicast packets
0 broadcast packets
570 bytes
2789797 discard packets
0 error packets
0 hoq discard packets

Spoiler: Connected Cable Mellanox Side

Identifier :		QSFP+
Cable/ Module type :		Passive copper, unequalized
Ethernet speed and type :		56GigE
Vendor :		Mellanox

Cable length :		3 m
Part number :		670759-B25
Revision :		A1
Serial number :		6C2634028J

Spoiler: ICX Switch Interface

show interfaces ethernet 1/2/3
40GigabitEthernet1/2/3 is down, line protocol is down
Port down for 14 hours 47 minutes 37 seconds
Hardware is 40GigabitEthernet, address is 748e.f8fb.1c00 (bia 748e.f8fb.1c3b)
Interface type is 40Gig Copper
Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
Configured mdi mode AUTO, actual unknown
Member of 9 L2 VLANs, port is tagged, port state is BLOCKING
BPDU guard is Disabled, ROOT protect is Enabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
MTU 10240 bytes, encapsulation ethernet
300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
145 packets input, 19778 bytes, 0 no buffer
Received 0 broadcasts, 145 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
228 packets output, 30627 bytes, 0 underruns
Transmitted 0 broadcasts, 228 multicasts, 0 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

Egress queues:
Queue counters Queued packets Dropped Packets
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
7 228 0

Spoiler: ICX Interface Media

show media ethernet 1/2/3
Port 1/2/3: Type : 40GBASE-Passive Copper
Vendor Name: Mellanox Serial Num: 6C2634028J Revision: A1

I am not getting any traffic across, it seems as if the ICX switch just is not happy with the link. I have tried a Plain Jane non-Mellanox 40gb cable as well and it has the same issue. The Mellanox switch lights the port, but the ICX switch will not.

Any pointers on making this work would be great.
Thanks