Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Shaun Bosworth

New Member
Dec 23, 2017
3
0
1
45
Hey Guys

Long time lurker, first time poster.

Trying to get the 2 rear 40G QSFP stacking ports on my ICX6610-48P to play nice with my Intel XL710-QDA2 card in a Dell R720XD. I have two 3m QSFP DAC cables between the card and the ports on the switch, and the link is negotiated at 40G full duplex on Windows Server, but there is no traffic been allowed, not even DHCP calls for IP address, comes back as unidentified network after a few minutes trying. The switch also shows full 40G on the web interface, as well as the SSH console.

I have disabled the stack configuration and all ports are added to the default VLAN 1 with all ports as untagged. Running the latest firmware from Brocade. Running config below:

Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack disable
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable aaa console
hostname brocadesw-rack
ip dhcp-client disable
ip dns domain-list xxxxxxxxx
ip dns server-address 192.168.10.15 192.168.10.16
ip route 0.0.0.0/0 192.168.10.1
!
no telnet server
username root password .....
!
!
clock timezone gmt GMT+02
!
!
ntp
disable serve
server 192.168.10.1
!
!
web-management page-menu
!
!
!
!
!
!
!
interface ethernet 1/1/45
inline power priority 1 power-by-class 4
!
interface ethernet 1/1/46
inline power priority 1 power-by-class 4
!
interface ethernet 1/1/47
inline power priority 1 power-by-class 4
!
interface ethernet 1/1/48
inline power priority 1 power-by-class 4
!
interface ve 1
ip address 192.168.10.3 255.255.255.0
!
!
!
!
!
!
!
!
!
end


Any ideas, as I have checked all the options within the Intel config tool and cannot see what is going on. Any help would be appreciated
 

bcook

New Member
Sep 3, 2021
2
1
3
Why not keeping both? :p

7000-series should be better particularly on the software side, and pretty much mandatory if you want to be seriously in business with IPv6 (IIRC, no SLAAC on the 6450 and 6610... or some other things IPv6-related).
If only :) Don't think I that would fly with the wife, lol, plus my little lab isn't big enough right now to justify two switches like that. If the 6450 is already overkill for my current setup then the 7250 would be even more so....but when is any of what we do with homelabs of any size not overkill? :D

I may still see what fohdeesha has available in the ICX7250-48P or 24P flavor....
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Hey Guys

Long time lurker, first time poster.

Trying to get the 2 rear 40G QSFP stacking ports on my ICX6610-48P to play nice with my Intel XL710-QDA2 card in a Dell R720XD. I have two 3m QSFP DAC cables between the card and the ports on the switch, and the link is negotiated at 40G full duplex on Windows Server, but there is no traffic been allowed, not even DHCP calls for IP address, comes back as unidentified network after a few minutes trying. The switch also shows full 40G on the web interface, as well as the SSH console.

I have disabled the stack configuration and all ports are added to the default VLAN 1 with all ports as untagged. Running the latest firmware from Brocade. Running config below:

Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack disable
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable aaa console
hostname brocadesw-rack
ip dhcp-client disable
ip dns domain-list xxxxxxxxx
ip dns server-address 192.168.10.15 192.168.10.16
ip route 0.0.0.0/0 192.168.10.1
!
no telnet server
username root password .....
!
!
clock timezone gmt GMT+02
!
!
ntp
disable serve
server 192.168.10.1
!
!
web-management page-menu
!
!
!
!
!
!
!
interface ethernet 1/1/45
inline power priority 1 power-by-class 4
!
interface ethernet 1/1/46
inline power priority 1 power-by-class 4
!
interface ethernet 1/1/47
inline power priority 1 power-by-class 4
!
interface ethernet 1/1/48
inline power priority 1 power-by-class 4
!
interface ve 1
ip address 192.168.10.3 255.255.255.0
!
!
!
!
!
!
!
!
!
end


Any ideas, as I have checked all the options within the Intel config tool and cannot see what is going on. Any help would be appreciated
longshot, but have you tried rebooting the switch?
 

neonclash

New Member
Aug 10, 2015
13
4
3
Does anyone near Indianapolis want to buy any switches? I have so many I need to offload but I don't have the time or energy to deal with shipping. ICX6610s, ICX7250s, ICX6450s, Dell S4810P, Brocade ADX, brocade CES, probably more I'm forgetting
A relative of mine returns to the Indianapolis area this evening, so this is perfect timing. I just can't settle on what hardware would fit my use case the best, so recommendations would be welcome.

The eventual goal for my home setup is reliability and mandatory access control on edge devices to assigned VLANs using 802.1x. Most of the 10gbit (three servers and growing) stays in the rack, so I need a stack there with PoE for my pi hats and backup AP. I also have an office with a PoE AP, and need 10gbit to a workstation there. The Z-series line would be the best fit because of the 2.5Gbase-T that my APs can take advantage of, but they're impossible to find.

Right now I have a 7150-c12p in the office and a 7150-24P in the closet. It's 10gbit from end to end, and I have it working the way I want to with VLANs and even a guest PVLAN. I just need to scale out soon, preferably while adding the redundancy I am after.

The 7250s are still maintained and use less power, but that MACSEC on the 6610 is nice, and I've never played with OpenFlow before, not to mention they're cheaper. What do I lose out on that I might not expect, and what do you recommend?
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
A relative of mine returns to the Indianapolis area this evening, so this is perfect timing. I just can't settle on what hardware would fit my use case the best, so recommendations would be welcome.

The eventual goal for my home setup is reliability and mandatory access control on edge devices to assigned VLANs using 802.1x. Most of the 10gbit (three servers and growing) stays in the rack, so I need a stack there with PoE for my pi hats and backup AP. I also have an office with a PoE AP, and need 10gbit to a workstation there. The Z-series line would be the best fit because of the 2.5Gbase-T that my APs can take advantage of, but they're impossible to find.

Right now I have a 7150-c12p in the office and a 7150-24P in the closet. It's 10gbit from end to end, and I have it working the way I want to with VLANs and even a guest PVLAN. I just need to scale out soon, preferably while adding the redundancy I am after.

The 7250s are still maintained and use less power, but that MACSEC on the 6610 is nice, and I've never played with OpenFlow before, not to mention they're cheaper. What do I lose out on that I might not expect, and what do you recommend?
I don't think you'd lose out on anything going with the 7250 over the 6610 - MACSEC and openflow don't really have real applications in a setup like that - plus with the 7 series you get newer software/features. Just looked and this is what I have that I'm willing to part with (no PoE 7250s left sadly):

ICX7250-48 X2
ICX7250-24 X1

ICX6610-48P X3
ICX6610-48 X2

ICX6450-24 X2
ICX6450-48P X1

FCX648S X4
FCX648S-HPOE X1

Juniper EX4300 X1
Dell S4810P X1
CES-2024C X1
 

neonclash

New Member
Aug 10, 2015
13
4
3
I don't think you'd lose out on anything going with the 7250 over the 6610 - MACSEC and openflow don't really have real applications in a setup like that - plus with the 7 series you get newer software/features. Just looked and this is what I have that I'm willing to part with (no PoE 7250s left sadly):

ICX7250-48 X2
ICX7250-24 X1

ICX6610-48P X3
ICX6610-48 X2

ICX6450-24 X2
ICX6450-48P X1

FCX648S X4
FCX648S-HPOE X1

Juniper EX4300 X1
Dell S4810P X1
CES-2024C X1
Thanks for the response! I'm gonna poke some ebay listings to see if I can get a couple of 7250-xxPs first. In the meantime, what's your asking price for each 6610-48P?
 
  • Like
Reactions: fohdeesha

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,318
800
113
Is there any way to *lower* the temperature where the fans spin up on ICX6610?

I'm currently using them in a quite warm environment and the fans spin up every ~30-60s and then spin down again... spin up again... spin down...

I'd like to have them permanently running on level 2...
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Thanks for the response! I'm gonna poke some ebay listings to see if I can get a couple of 7250-xxPs first. In the meantime, what's your asking price for each 6610-48P?
cheapest I see on ebay for 48Ps with dual PSUs is 190, I could do 160? Obviously no shipping fee with the local pickup, and they've been graced by the brocade loser himself
 

Rootless

Thank you
Aug 9, 2021
17
7
3
I can't wait for the surprise. I love this thread. I now have two ICX-6610, one of the Dell Networking Cable QSFP+ 40GBE - 4X SFP+ 10GBE 3M 27GG5 cables. All of the licenses are good and both units are online. The Dell cable works fine between the two ICX6610, but I'm having problems getting it working with the " Cavium QLogic 1010/1020/1007/1741 10Gbps CNA (rev 01)" installed in my Dell R910.

The card is visible (lspci):
05:00.2 Ethernet controller: Cavium QLogic 1010/1020/1007/1741 10Gbps CNA (rev 01)
05:00.3 Ethernet controller: Cavium QLogic 1010/1020/1007/1741 10Gbps CNA (rev 01)

But the links always show as down both on the server:
6: ens5f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
link/ether 8c:7c:ff:20:c0:32 brd ff:ff:ff:ff:ff:ff
altname enp5s0f2
7: ens5f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
link/ether 8c:7c:ff:20:c0:33 brd ff:ff:ff:ff:ff:ff
altname enp5s0f3

and on the switch:
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/2/7 Down None None None None No 1 0 cc4e.2484.0568
1/2/8 Down None None None None No 1 0 cc4e.2484.0568
1/2/9 Down None None None None No 1 0 cc4e.2484.0568

(the one connected between the switches at the moment is 1/2/10:
SSH@racksw1>sh int br e 1/2/10
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/2/10 Up Forward Full 10G None No 1 0 cc4e.2484.0568
)

The Dell breakout DAC
SSH@racksw1>sh media e 1/2/7
Port 1/2/7:Type : 40GBASE-Passive Copper
Vendor Name: Amphenol Serial Num: CN027GG554Q4JWR Revision: C

All of the QSFP ports are removed from stacking and stacking is disabled:
SSH@racksw1>sh conf
!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.30tT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack disable
stack mac cc4e.2484.0568
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
hostname racksw1
ip dhcp-client disable
ip dns server-address 192.168.1.1
ip route 0.0.0.0/0 192.168.1.1
!
no telnet server
username root password .....
no snmp-server
!
!
clock timezone gmt GMT-07
!
!
ntp
disable serve
server 216.239.35.0
server 216.239.35.4
!
web-management https
hitless-failover enable
!
!
interface ethernet 1/3/1
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
speed-duplex 10G-full
!
interface ethernet 1/3/5
speed-duplex 10G-full
!
interface ethernet 1/3/6
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
speed-duplex 10G-full
!
interface ve 1
ip address 192.168.1.58 255.255.255.0
!
!
end

I picked up the Brocade 1020 NIC because I read in a forum post that they just work. I have since found another post that describes them as finicky. Not sure where to go from here. Do I need to pickup something like a MCX354A-QCBT Mellanox CX354A ConnectX-3 VPI QDR IB 10GbE Dual-Port QSFP NIC? If so, why do some of the MCX345A-QCBT list themselves as 10Gb/40Gb, some as 10Gb only, and some as 40Gb only?

Thank you for all of the incredible documentation and information sharing. It is what got me this far. Thank you.
 
  • Like
Reactions: Yunia and fohdeesha

neonclash

New Member
Aug 10, 2015
13
4
3
cheapest I see on ebay for 48Ps with dual PSUs is 190, I could do 160? Obviously no shipping fee with the local pickup, and they've been graced by the brocade loser himself
The US sellers for 24 and 48 port 7250s with POE were all asking stupidly high prices and didn't take any sane offers, but I did find a Canadian seller willing to send me two 7250-48Ps for $300CAD each, for a total of ~$550 USD shipped.
 
  • Like
Reactions: itronin

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
You found switches for 2.5, 5, 10, 25, 50 & 100 gbit ethernet that are as awesome as the icx in the op? :D
damn now the surprise is going to seem really lame in comparison :p it's a form of info a lot of people in this thread already have, hint hint. also shit we're on 300 everyone uhh take a vacation real quick while I finish this guide
 

Jason Antes

Active Member
Feb 28, 2020
224
76
28
Twin Cities
I don't think you'd lose out on anything going with the 7250 over the 6610 - MACSEC and openflow don't really have real applications in a setup like that - plus with the 7 series you get newer software/features. Just looked and this is what I have that I'm willing to part with (no PoE 7250s left sadly):

ICX7250-48 X2
ICX7250-24 X1

ICX6610-48P X3
ICX6610-48 X2

ICX6450-24 X2
ICX6450-48P X1

FCX648S X4
FCX648S-HPOE X1

Juniper EX4300 X1
Dell S4810P X1
CES-2024C X1
Nice. Too bad I live to far away to pick one up. Need another POE switch to put in my pole barn once I get it built (who knows when that will be...depends on lumber prices) that I can uplink to my 6610 via the 40GbE optic SFP+'s I have. I know I can with the VDX, just doesn't have PoE. I'm sure these will sell out quick! :)
 
  • Like
Reactions: fohdeesha