Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

kpfleming

Active Member
Dec 28, 2021
147
61
28
Pelham NY USA
Nope, that's not correct :)

'router-interface ve 1' creates a virtual ethernet interface in VLAN 1. That interface only exists in VLAN 1, it does not exist anywhere else. Think of it as if the management CPU was in a box outside the switch and you connected it to a port that you put into VLAN 1... it's functionally the same, just virtual.

Once that VE exists, it is a layer 3 interface, and when you assign addresses to it those addresses are reachable from any other layer 3 host in the network, unless access-lists or some other mechanism stops the traffic.
 

EngineerNate

Member
Jun 3, 2017
67
16
8
33
I guess I misunderstood the original question, I thought he was saying that it was reachable from outside the VLAN. I understand now and the behavior you describe is what I was trying to convey.
 

Drewy

Active Member
Apr 23, 2016
206
54
28
52
Just plugged a Mikrotik S+RJ10 into one of my 7250's and it's not wanting to play.
The 7250 is running 08.0.95.

No lights on the sfp+ cage, no traffic. The other end (QNAP nas) does show a 10Gb link.

SSH@core#show media ethe 2/2/5
Port 2/2/5: Type : 1GE M-SX(SFP)
Vendor: MikroTik Version: 2.16
Part# : S+RJ10 Serial#: F060030B02B9

SSH@core#show interface ethe 2/2/5
10GigabitEthernet2/2/5 is down, line protocol is down
Port down for 1 day(s) 19 hour(s) 4 minute(s) 23 second(s)
Hardware is 10GigabitEthernet, address is 609c.9f9d.352c (bia 78a6.e12a.7235)
Configured speed optic-based, actual unknown, configured duplex fdx, actual unknown
Configured mdi mode AUTO, actual unknown
Untagged member of L2 VLAN 1, port state is BLOCKING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
MACsec is Disabled
Openflow is Disabled, Openflow Hybrid mode is Disabled, Flow Control is config enabled, oper enabled, negotiation disabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
VLAN-Mapping is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
IPG XGMII 96 bits-time
MTU 1500 bytes, encapsulation ethernet
MMU Mode is Store-and-forward
300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
0 packets output, 0 bytes, 0 underruns
Transmitted 0 broadcasts, 0 multicasts, 0 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled
Protected: No
MAC Port Security: Disabled

trying to set link speed manually and I get:

SSH@core(config-if-e10000-2/2/7)#speed-duplex 10g-full
ERROR: 2/2/7: optics <-> speed mismatch. Replace with SFP+ to enable link.

that and the output from show media above kinda looks like the 7250 thinks it's a sfp module...

any ideas?
 

Shaun Bosworth

New Member
Dec 23, 2017
3
0
1
43
longshot, but have you tried rebooting the switch?
Hi Fohdeesha

Just coming back to this post, I had the switch in storage for a while, but have managed to get it up and running again, but still having an issue with the 40G links between the switch and my XL710-QDA2 card.

The switch has been setup as per the previous and i have rebooted the switch on multiple occasions.

Any help would be appreciated.

Thanks
 

OKGolombRuler

New Member
Mar 13, 2020
16
6
3
BLUF: Possible to stack via 10G copper SFPs for Stacking.... over 2.5gbps MOCA links?

I feel like I could just as well have titled this request "How to get your OSI card revoked" but the question stands. I have spent a fair bit of quality time juggling VLANs and switch configs across a handful of small 7150s which are connected by, respectively, 1 fiber link, one CAT5 link, and a few point-to-point, I believe 2.5gbps max speed, MOCA bridges. I'd like to collapse most or all of it into one stack. I think I can get the CAT5 online either with a pair of 10G copper SFPs, or pulling another piece of glass (which, while annoying, would be mostly through "open"/unfinished space). The MOCA bridges are a pickle of another brine, however.

When I tried using 1gbps copper SFPs, I was unable to get the switch to stack; it complained the link wasn't 10gbps. I was hoping someone here might have experience, or eldritch knowledge, about whether I could use a cheap 10gbps SFP that might report 10gbps despite only being connected at 2.5gbps, and rate limit those interfaces to prevent drops. Anybody been down this road? If so, which SFPs do you like for this kind of adventure?
 

kpfleming

Active Member
Dec 28, 2021
147
61
28
Pelham NY USA
When I tried using 1gbps copper SFPs, I was unable to get the switch to stack; it complained the link wasn't 10gbps. I was hoping someone here might have experience, or eldritch knowledge, about whether I could use a cheap 10gbps SFP that might report 10gbps despite only being connected at 2.5gbps, and rate limit those interfaces to prevent drops. Anybody been down this road? If so, which SFPs do you like for this kind of adventure?
You may be out of luck there. IIRC the switch stacking links don't even run in normal 'Ethernet' mode, they use something called HiGig and it isn't a protocol that you will be able to pass across bridges.
 

Shaun Bosworth

New Member
Dec 23, 2017
3
0
1
43
Hi Guys

Okay the mystery deepens, I can see that my Windows DHCP server is issuing IP addresses to the Intel 40G link cables, but the card within the OS does not acknowledge that the leases has been issued, as it only shows an AutoIP.
1658595664433.png
1658595720807.png1658595777109.png
I have confirmed that the switch is linking at 40G to the card on both ports
1658595850081.png1658595870879.png
Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack disable
!
global-stp
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
spanning-tree 802-1w
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable aaa console
fast port-span exclude ethe 1/2/1 ethe 1/2/6
hostname brocade-sw
ip dhcp-client disable
ip dns domain-list bozzy.info
ip dns server-address 192.168.10.15 192.168.10.16
ip route 0.0.0.0/0 192.168.10.1
!
no telnet server
username admin password .....
!
!
clock timezone gmt GMT+02
!
!
ntp
disable serve
server 192.168.10.1
!
!
web-management https
web-management refresh port-statistic 10
web-management refresh stp 10
web-management refresh tftp 10
web-management refresh rmon 10
web-management frame bottom
web-management page-size 20
!
!
router rip
!
!
!
!
!
!
!
interface ethernet 1/2/1
no spanning-tree
no flow-control
!
interface ethernet 1/2/6
no spanning-tree
no flow-control
!
interface ethernet 1/3/1
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
speed-duplex 10G-full
!
interface ethernet 1/3/5
speed-duplex 10G-full
!
interface ethernet 1/3/6
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
speed-duplex 10G-full
!
interface ve 1
ip address 192.168.10.7 255.255.255.0
!
!
!
!
!
!
!
!
!
end

This is the current switch config. I know a lot of you will say that I should have gotten a Mellanox card, but I prefer to use Intel NICs. Also, just a note that this XL710 is a Dell OEM model, obviously used in a Dell PowerEdge server, along with Dell 3M QSFP DAC cables, so i think there shouldnt be any compatibility issues.

Any thoughts or resolution would be appreciated
 
Last edited:

kevindd992002

Member
Oct 4, 2021
47
0
6
I see that the ICX6610 has 4 different versions:

1658596647095.png

Questions:

1) What is the use case of an "intake air flow"? I though all rack switches have their rear fans exhaust airflow out?
2) What's the difference between Base and Premium software? Does it matter for home lab use?
 

kevindd992002

Member
Oct 4, 2021
47
0
6
Ok, so why would the fan be intake if all network ports are on the same side of the rack? I guess I still don't understand the use case. And what I want is the standard exhaust airflow switch if I have the switch ports in front and the server ports at the back, correct?
 

i386

Well-Known Member
Mar 18, 2016
3,121
997
113
33
Germany
Ok, so why would the fan be intake if all network ports are on the same side of the rack?
If you have a switch with front to back airflow and use it as a tor switch it would pull hot air into a server rack -> more heat -> higher risk for failure (not just the switch but all other stuff inside the rack)
And what I want is the standard exhaust airflow switch if I have the switch ports in front and the server ports at the back, correct?
Yes.
But if you have many devices connected the cable management will be a lot harder and (you would need longer cables going from front to back of the rack)
 

klui

Well-Known Member
Feb 3, 2019
552
250
63
In a lot of cases switches are mounted in the rear of a rack in a data center. It helps with minimizing cable length and if they are mounted that way, front-to-rear airflow will route server exhaust air through the switch. That's why there are rear-to-front airflow SKUs to go along with vented blank panels so air from the cold aisle can properly cool a switch mounted in that configuration. And while the term is "top" of rack, they may not be mounted there, instead in the middle for the same reason re: cable length.
 

kevindd992002

Member
Oct 4, 2021
47
0
6
Ok, I think I have a good idea of what you guys are saying. I have just a couple of devices in my home rack. A Synology NAS, NAS extension bay, and pfsense. I think I'm set with the ICX6610-48p-E (exhaust airflow) then so all my devices have front-to-back airflow.
 

juju

New Member
Sep 29, 2021
25
1
3
I have an IoT device on my network which is not using the dns server on the ICX7250 and always defaults to 0.0.0.0. ( so cant resolve dns names). All other devices on my network correctly use the dns server ip I have setup on my 7250. Other devices on the same vlan as this IoT device have no such issue with dns resolution. Is there a way to force a specific device to use the dns server of its vlan ?
 

Rttg

Member
May 21, 2020
30
19
8
I have an IoT device on my network which is not using the dns server on the ICX7250 and always defaults to 0.0.0.0. ( so cant resolve dns names).
Are you sure the IoT device isn’t trying to use a hardcoded DNS server?

Some will to ensure they can ‘phone home’ (and avoid ad blockers). If that’s the case, you may need to use a DNAT rule to rewrite DNS traffic to your preferred server(s).
 

juju

New Member
Sep 29, 2021
25
1
3
Are you sure the IoT device isn’t trying to use a hardcoded DNS server?
It is entirely possible it is - I suspect that is the issue. In this case, it has a default dns server of 0.0.0.0 ( if you query the device for dns server info). Not sure if its returning 0.0.0.0 because its not getting the dns info from the ICX switch or its hardcoded. Can you give some pointers how to do the DNAT rule ? Where ? On my pfsense switch or on the ICX7250?
 

Rttg

Member
May 21, 2020
30
19
8
Can you give some pointers how to do the DNAT rule ? Where ? On my pfsense switch or on the ICX7250?
The ICX can’t do NAT - it’d have to run on your firewall/internet gateway.

Can’t say how to do it on pfsense - I’m running VyOS so don’t know the syntax/config there.
 

seatrope

New Member
Oct 5, 2018
20
6
3
Maine
www.ychng.com
Hi all, thanks for the accumulated wealth of information. Have had a ICX6610 and a couple of ICX6450s for a while now but just started to use the L3 features.

I've gotten VLANs and inter-VLAN routing on the ICX6610 up and running, with pfSense only serving firewall duties. DHCP/DNS is via a piHole with dnsmasq serving all the subnets.

I have been searching and can't figure out how to do this. @fohdeesha any help would be appreciated - thanks again for your help with the switches a few years back!!
I either need to:
1) Stack the ICX6610 and one ICX6450 (i know only the front ports can be used as a "peri-trunk") but still preserve all the back 40Gbe ports for data use (I have a couple of servers connected via 40Gbe 1/2/1 and 1/2/6). It won't let me enable stacking without pulling the back ports into it too. I saw where you have used one 40Gbe and one breakout port for stacking btw two ICX6610 but this is somewhat different, i guess.
OR
2) if the above is impossible, what's the best way to pass all the VLANs defined on the 6610 to the 6450 and distribute them as untagged ports? I tried making the link between the 6610-6450 a dual-mode port on both sides with the same VLAN numbers - that did not work and I could not ping the 6610 interface IP after I did that.

Apologize in advance for the n00b questions but am slowly learning L3 networking.

Thanks so much!