Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[EngineerNate]

Wanted to drop by and say thanks to Fohdeesha again, I updated a second one of these switches last night using his guide and no hiccups.

The first I had to step up the firmware a few times before 8090 would take, thing was waaaay old on the firmware side. No problems once it finally got up to date.

I'm using amphenol branded QSFP+ DAC cables and they're linking up just fine at 40g to my server's ConnectX-3 cards.

 

[kevinSTH]

Hello, really new to the server space. If I were to get a CX6450-24P and it is fully reset. Would it function as an unmanaged switch?

 

[danb35]

How do i identify these breakout ports in the console?

See the "Port Layout" section here:

FCX / ICX6610 - Fohdeesha Docs

fohdeesha.com

 

[kpfleming]

Hello, really new to the server space. If I were to get a CX6450-24P and it is fully reset. Would it function as an unmanaged switch?

It's possible, although if you try to use VLAN tags that may not work. If all of your traffic is untagged then yes, it would operate as a plain layer 2 switch.

 

[luks]

I don't quite understand how the switch management works when using the L3 firmware on ICX6450 and 6610. On the Fodeesha's guide, a static ip is set for ve 1 and that will become the IP where I can connect to the switch via SSH. But that wasn't explained very well in the guide and I got a bit confused.

I managed to set up some VLANs and routing between them. Now the switch management is also accessible through every VLAN as the switch works as a router for each of those VLANs. I would like the management access to be only accessible from a specified management VLAN. I couldn't find information from the manuals on how to do that.

Also why is the dedicated management port not used? I would prefer to have the SSH access to the switch only through the management ethernet port if that's possible.

 

[kpfleming]

You can use access-lists (ACLs) to restrict the ability to talk to the management IP address over SSH/HTTP/HTTPS/etc. if you wish. This is the same technique you'd use to restrict access across any other routed paths (between hosts, for example).

If you prefer to use the management port you certainly can, although you'll have to provide a way to connect to it If you are going to route traffic between your VLANs you'll still need addresses on the VEs, though, so you'll have to use access-lists if you want to block access to the management interfaces through those addresses. I don't think there is any way to tell the device to *not* listen on the VE addresses for management traffic.

 

[EngineerNate]

You can use access-lists (ACLs) to restrict the ability to talk to the management IP address over SSH/HTTP/HTTPS/etc. if you wish. This is the same technique you'd use to restrict access across any other routed paths (between hosts, for example).

If you prefer to use the management port you certainly can, although you'll have to provide a way to connect to it If you are going to route traffic between your VLANs you'll still need addresses on the VEs, though, so you'll have to use access-lists if you want to block access to the management interfaces through those addresses. I don't think there is any way to tell the device to *not* listen on the VE addresses for management traffic.

Correct me if I'm wrong here, but if you create a VE it should only be exposed on the vlans you put it in right? If you run the command:

no router-interface ve 1

Inside the vlans where you don't want that interface, that interface shouldn't be accessible in those vlans right?

 

[kpfleming]

Nope, that's not correct

'router-interface ve 1' creates a virtual ethernet interface in VLAN 1. That interface only exists in VLAN 1, it does not exist anywhere else. Think of it as if the management CPU was in a box outside the switch and you connected it to a port that you put into VLAN 1... it's functionally the same, just virtual.

Once that VE exists, it is a layer 3 interface, and when you assign addresses to it those addresses are reachable from any other layer 3 host in the network, unless access-lists or some other mechanism stops the traffic.

 

[EngineerNate]

I guess I misunderstood the original question, I thought he was saying that it was reachable from outside the VLAN. I understand now and the behavior you describe is what I was trying to convey.

 

[kevinSTH]

It's possible, although if you try to use VLAN tags that may not work. If all of your traffic is untagged then yes, it would operate as a plain layer 2 switch.

Thanks

 

[Drewy]

Just plugged a Mikrotik S+RJ10 into one of my 7250's and it's not wanting to play.
The 7250 is running 08.0.95.

No lights on the sfp+ cage, no traffic. The other end (QNAP nas) does show a 10Gb link.

SSH@core#show media ethe 2/2/5
Port 2/2/5: Type : 1GE M-SX(SFP)
Vendor: MikroTik Version: 2.16
Part# : S+RJ10 Serial#: F060030B02B9

SSH@core#show interface ethe 2/2/5
10GigabitEthernet2/2/5 is down, line protocol is down
Port down for 1 day(s) 19 hour(s) 4 minute(s) 23 second(s)
Hardware is 10GigabitEthernet, address is 609c.9f9d.352c (bia 78a6.e12a.7235)
Configured speed optic-based, actual unknown, configured duplex fdx, actual unknown
Configured mdi mode AUTO, actual unknown
Untagged member of L2 VLAN 1, port state is BLOCKING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
MACsec is Disabled
Openflow is Disabled, Openflow Hybrid mode is Disabled, Flow Control is config enabled, oper enabled, negotiation disabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
VLAN-Mapping is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
IPG XGMII 96 bits-time
MTU 1500 bytes, encapsulation ethernet
MMU Mode is Store-and-forward
300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
0 packets output, 0 bytes, 0 underruns
Transmitted 0 broadcasts, 0 multicasts, 0 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled
Protected: No
MAC Port Security: Disabled

trying to set link speed manually and I get:

SSH@core(config-if-e10000-2/2/7)#speed-duplex 10g-full
ERROR: 2/2/7: optics <-> speed mismatch. Replace with SFP+ to enable link.

that and the output from show media above kinda looks like the 7250 thinks it's a sfp module...

any ideas?

 

[Shaun Bosworth]

longshot, but have you tried rebooting the switch?

Hi Fohdeesha

Just coming back to this post, I had the switch in storage for a while, but have managed to get it up and running again, but still having an issue with the 40G links between the switch and my XL710-QDA2 card.

The switch has been setup as per the previous and i have rebooted the switch on multiple occasions.

Any help would be appreciated.

Thanks

 

[OKGolombRuler]

BLUF: Possible to stack via 10G copper SFPs for Stacking.... over 2.5gbps MOCA links?

I feel like I could just as well have titled this request "How to get your OSI card revoked" but the question stands. I have spent a fair bit of quality time juggling VLANs and switch configs across a handful of small 7150s which are connected by, respectively, 1 fiber link, one CAT5 link, and a few point-to-point, I believe 2.5gbps max speed, MOCA bridges. I'd like to collapse most or all of it into one stack. I think I can get the CAT5 online either with a pair of 10G copper SFPs, or pulling another piece of glass (which, while annoying, would be mostly through "open"/unfinished space). The MOCA bridges are a pickle of another brine, however.

When I tried using 1gbps copper SFPs, I was unable to get the switch to stack; it complained the link wasn't 10gbps. I was hoping someone here might have experience, or eldritch knowledge, about whether I could use a cheap 10gbps SFP that might report 10gbps despite only being connected at 2.5gbps, and rate limit those interfaces to prevent drops. Anybody been down this road? If so, which SFPs do you like for this kind of adventure?

 

[kpfleming]

When I tried using 1gbps copper SFPs, I was unable to get the switch to stack; it complained the link wasn't 10gbps. I was hoping someone here might have experience, or eldritch knowledge, about whether I could use a cheap 10gbps SFP that might report 10gbps despite only being connected at 2.5gbps, and rate limit those interfaces to prevent drops. Anybody been down this road? If so, which SFPs do you like for this kind of adventure?

You may be out of luck there. IIRC the switch stacking links don't even run in normal 'Ethernet' mode, they use something called HiGig and it isn't a protocol that you will be able to pass across bridges.

 

[Shaun Bosworth]

Hi Guys

Okay the mystery deepens, I can see that my Windows DHCP server is issuing IP addresses to the Intel 40G link cables, but the card within the OS does not acknowledge that the leases has been issued, as it only shows an AutoIP.


I have confirmed that the switch is linking at 40G to the card on both ports

Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack disable
!
global-stp
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
spanning-tree 802-1w
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable aaa console
fast port-span exclude ethe 1/2/1 ethe 1/2/6
hostname brocade-sw
ip dhcp-client disable
ip dns domain-list bozzy.info
ip dns server-address 192.168.10.15 192.168.10.16
ip route 0.0.0.0/0 192.168.10.1
!
no telnet server
username admin password .....
!
!
clock timezone gmt GMT+02
!
!
ntp
disable serve
server 192.168.10.1
!
!
web-management https
web-management refresh port-statistic 10
web-management refresh stp 10
web-management refresh tftp 10
web-management refresh rmon 10
web-management frame bottom
web-management page-size 20
!
!
router rip
!
!
!
!
!
!
!
interface ethernet 1/2/1
no spanning-tree
no flow-control
!
interface ethernet 1/2/6
no spanning-tree
no flow-control
!
interface ethernet 1/3/1
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
speed-duplex 10G-full
!
interface ethernet 1/3/5
speed-duplex 10G-full
!
interface ethernet 1/3/6
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
speed-duplex 10G-full
!
interface ve 1
ip address 192.168.10.7 255.255.255.0
!
!
!
!
!
!
!
!
!
end

This is the current switch config. I know a lot of you will say that I should have gotten a Mellanox card, but I prefer to use Intel NICs. Also, just a note that this XL710 is a Dell OEM model, obviously used in a Dell PowerEdge server, along with Dell 3M QSFP DAC cables, so i think there shouldnt be any compatibility issues.

Any thoughts or resolution would be appreciated

 

[kevindd992002]

I see that the ICX6610 has 4 different versions:



Questions:

1) What is the use case of an "intake air flow"? I though all rack switches have their rear fans exhaust airflow out?
2) What's the difference between Base and Premium software? Does it matter for home lab use?

 

[i386]

1) What is the use case of an "intake air flow"? I though all rack switches have their rear fans exhaust airflow out?

It's for managing cooling in a datacenter: all network ports are on the same side of the rack (servers AND switches) AND the air flows in the same direction

 

[kevindd992002]

Ok, so why would the fan be intake if all network ports are on the same side of the rack? I guess I still don't understand the use case. And what I want is the standard exhaust airflow switch if I have the switch ports in front and the server ports at the back, correct?

 

[i386]

Ok, so why would the fan be intake if all network ports are on the same side of the rack?

If you have a switch with front to back airflow and use it as a tor switch it would pull hot air into a server rack -> more heat -> higher risk for failure (not just the switch but all other stuff inside the rack)

And what I want is the standard exhaust airflow switch if I have the switch ports in front and the server ports at the back, correct?

Yes.
But if you have many devices connected the cable management will be a lot harder and (you would need longer cables going from front to back of the rack)

 

[klui]

In a lot of cases switches are mounted in the rear of a rack in a data center. It helps with minimizing cable length and if they are mounted that way, front-to-rear airflow will route server exhaust air through the switch. That's why there are rear-to-front airflow SKUs to go along with vented blank panels so air from the cold aisle can properly cool a switch mounted in that configuration. And while the term is "top" of rack, they may not be mounted there, instead in the middle for the same reason re: cable length.