Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

fohdeesha · Jan 28, 2020

Wolfstar said:
Correction on this. The Stacking models take either the FCX-2XG or the FCX-2SFPP, which is a 2-port SFP+ 10G uplink model. However, there is precisely one that I can find on eBay

Indeed, I've never actually seen one for sale in the wild so I didn't even bother mentioning them - the one you mention on ebay right now is the first I've seen actually (granted I haven't been actively looking). that pricing is nuts! with how cheap XFPs are and the fact they work perfectly with SR/LR/etc SFP+ modules on the other end, no reason to spend extra otherwise honestly

Wolfstar · Jan 28, 2020

fohdeesha said:
Indeed, I've never actually seen one for sale in the wild so I didn't even bother mentioning them - the one you mention on ebay right now is the first I've seen actually (granted I haven't been actively looking). that pricing is nuts! with how cheap XFPs are and the fact they work perfectly with SR/LR/etc SFP+ modules on the other end, no reason to spend extra otherwise honestly

Oh, no doubt. And the reason you don't see them for sale anywhere is probably the particular organization I work for. I'm not actually exaggerating on the number of FCXs - if anything I'm low - but a lot of them are in stacks. Thing is, every stack has two of the modules in it, and we're sitting on a pile of them for spares and small remote sites that aren't ready to go to 10G. I'm hoping word comes down to replace the damned things with 7450s before that happens, but I'm not holding my breath; we're still running 3560v2s at some of these small sites.

camper8080 · Jan 29, 2020

vangoose said:
Yes.
I have 7250-24/P. The fans run avg 6500 rpm but it's very quiet.
Your is 48 port so fan runs a bit faster.

Thank you for the feedback. Thought I was doing something wrong or I missed something. I know this is an enterprise switch so I shouldn't expect it to be quiet.

danpage617 · Jan 29, 2020

Is diffie-hellman-group14-sha1 KEX supported on the ICX 6610 running latest firmware? Or only group1? I'm finding conflicting information from my searches.

jd.developer · Jan 29, 2020

fohdeesha said:
that message happens when the RPM reported by the PSU fan is not within the normal operating threshold, so it's either failing and spinning too slow, or it's RPM reporting/wire is busted altogether

Thank you. I ended up getting another PSU from the seller and it worked fine. Can the fan be repaired on a defective PSU?

fohdeesha · Jan 29, 2020

jd.developer said:
Thank you. I ended up getting another PSU from the seller and it worked fine. Can the fan be repaired on a defective PSU?

yeah just open up the PSU and remove the fan, get the part number off of it and buy another one, should be about $10. you'll have to splice the wires as it's not a standard pinout/connector if I remember right

iotapi322 · Jan 30, 2020

Ugh I hate to ask this question, but.... are there any special commands to enable the sfp+ ports on the front of the ICX 6610-48P? Just got a 6610 off of ebay and I went through the guide and updated the firmware and the Poe Firmware to the latest versions - setup the system just like the guide suggested.

However when I connect any of my dac cables I don't get any link light. They are connected to a Intel dual port sfp+ card. I'm using a cisco DAC cable but I've also tried this DAC cable from AWS ( 10Gtek 10G SFP+ DAC Cable - 10GBASE-CU Passive Direct Attach Copper Twinax SFP Cable for Cisco SFP-H10GB-CU3M, Ubiquiti, D-link, Supermicro, Netgear, Mikrotik, ZTE Devices, 3m, 1pack)

The cisco DAC cable seemed to work with other switches.... I am a newbie when it comes to 10G networking so this could clearly be a problem I've created.

running the show media command

Code:

Port 1/2/1:  Type : EMPTY
Port 1/2/2:  Type : EMPTY
Port 1/2/3:  Type : EMPTY
Port 1/2/4:  Type : EMPTY
Port 1/2/5:  Type : EMPTY
Port 1/2/6:  Type : EMPTY
Port 1/2/7:  Type : EMPTY
Port 1/2/8:  Type : EMPTY
Port 1/2/9:  Type : EMPTY
Port 1/2/10:  Type : EMPTY
Port 1/3/1:  Type : 10GE Twinax   3m (SFP +)
Port 1/3/2:  Type : EMPTY
Port 1/3/3:  Type : EMPTY
Port 1/3/4:  Type : EMPTY
Port 1/3/5:  Type : EMPTY
Port 1/3/6:  Type : EMPTY
Port 1/3/7:  Type : EMPTY
Port 1/3/8:  Type : 10GE Twinax   5m (SFP +)

seems to show something is plugged in.

Now running the command

Code:

show media validation

Code:

SSH@brocade01#show media validation

Port       Supported Vendor               Type
----------------------------------------------------------------------
1/3/1      Yes       OEM                   Type  : 10GE  Passive Twinax  3m (SFP +) (Not supported)
1/3/8      Yes       CISCO-TYCO            Type  : 10GE  Passive Twinax  5m (SFP +) (Not supported)

Is this telling me the DAC cables I have are not compatible?

kapone · Jan 30, 2020

fohdeesha said:
yeah just open up the PSU and remove the fan, get the part number off of it and buy another one, should be about $10. you'll have to splice the wires as it's not a standard pinout/connector if I remember right

You remember right. It's not.

fohdeesha · Jan 30, 2020

iotapi322 said:
Ugh I hate to ask this question, but.... are there any special commands to enable the sfp+ ports on the front of the ICX 6610-48P? Just got a 6610 off of ebay and I went through the guide and updated the firmware and the Poe Firmware to the latest versions - setup the system just like the guide suggested.

However when I connect any of my dac cables I don't get any link light. They are connected to a Intel dual port sfp+ card. I'm using a cisco DAC cable but I've also tried this DAC cable from AWS ( 10Gtek 10G SFP+ DAC Cable - 10GBASE-CU Passive Direct Attach Copper Twinax SFP Cable for Cisco SFP-H10GB-CU3M, Ubiquiti, D-link, Supermicro, Netgear, Mikrotik, ZTE Devices, 3m, 1pack)

The cisco DAC cable seemed to work with other switches.... I am a newbie when it comes to 10G networking so this could clearly be a problem I've created.

running the show media command

Code:

Port 1/2/1: Type : EMPTY Port 1/2/2: Type : EMPTY Port 1/2/3: Type : EMPTY Port 1/2/4: Type : EMPTY Port 1/2/5: Type : EMPTY Port 1/2/6: Type : EMPTY Port 1/2/7: Type : EMPTY Port 1/2/8: Type : EMPTY Port 1/2/9: Type : EMPTY Port 1/2/10: Type : EMPTY Port 1/3/1: Type : 10GE Twinax 3m (SFP +) Port 1/3/2: Type : EMPTY Port 1/3/3: Type : EMPTY Port 1/3/4: Type : EMPTY Port 1/3/5: Type : EMPTY Port 1/3/6: Type : EMPTY Port 1/3/7: Type : EMPTY Port 1/3/8: Type : 10GE Twinax 5m (SFP +)

seems to show something is plugged in.

Now running the command

Code:

show media validation

Code:

SSH@brocade01#show media validation Port Supported Vendor Type ---------------------------------------------------------------------- 1/3/1 Yes OEM Type : 10GE Passive Twinax 3m (SFP +) (Not supported) 1/3/8 Yes CISCO-TYCO Type : 10GE Passive Twinax 5m (SFP +) (Not supported)

Is this telling me the DAC cables I have are not compatible?

it's probably not licensed, 6th post in the thread https://forums.servethehome.com/ind...rful-10gbe-40gbe-switching.21107/#post-196476

iotapi322 · Jan 30, 2020

fohdeesha said:
it's probably not licensed, 6th post in the thread https://forums.servethehome.com/ind...rful-10gbe-40gbe-switching.21107/#post-196476

Yes, I think I just came up with that hypothesis.

Code:

SSH@brocade01#show license
License record empty

fohdeesha · Jan 30, 2020

PM me, your fate's now in the hands of someone who just had taco bell for breakfast

fohdeesha · Jan 30, 2020

camper8080 said:
Thank you for the feedback. Thought I was doing something wrong or I missed something. I know this is an enterprise switch so I shouldn't expect it to be quiet.

you can unplug all but one of the fans internally if you want it quieter (or replace the fans). If I recall correctly the 48 port PoE has THREE fans, intended to cool it if it's putting out hundreds of watts of PoE load, which you won't be doing

infoMatt · Jan 30, 2020

iotapi322 said:
Yes, I think I just came up with that hypothesis.

Code:

SSH@brocade01#show license License record empty

You can verify it by forcing the interface (on both ends!!) at 1000-fd, so it shouldn't require any licence...

EDIT: I've verified on the datasheed posted on the first page... yes, 1000-fd is free, 10000-fd requires an extra licence; for the purpose of testing the cables, you can go ahead, albeit at a "mere" 1G.

iotapi322 · Jan 30, 2020

ok so i am licensed now... and all the licenses now say Status as Active.
The ports on the switch are lit now as well...
however the show media validation still shows the "not supported" at the end of the line

Code:

SSH@brocade01>show media validation

Port       Supported Vendor               Type
----------------------------------------------------------------------
1/3/1      Yes       OEM                   Type  : 10GE  Passive Twinax  3m (SFP +) (Not supported)
1/3/8      Yes       CISCO-TYCO            Type  : 10GE  Passive Twinax  5m (SFP +) (Not supported)

the ubuntu computer (running 20.04 beta... i know iknow) is running a cisco dual VIC card

Code:

      resources: irq:17 memory:f0700000-f0707fff memory:f0708000-f0709fff
  *-network
       description: Ethernet interface
       product: VIC Ethernet NIC
       vendor: Cisco Systems Inc
       physical id: 0
       bus info: pci@0000:0b:00.0
       logical name: enp11s0
       version: a2
       serial: 00:22:bd:ea:9a:cd
       capacity: 10Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: msi msix pm pciexpress bus_master cap_list ethernet physical fibre 10000bt-fd
       configuration: autonegotiation=off broadcast=yes driver=enic driverversion=2.3.0.53 firmware=4.1(0.333) latency=0 link=no multicast=yes port=fibre
       resources: irq:18 memory:f0600000-f0607fff memory:f0608000-f0609fff

even when i try to set the interface up it comes back with no carrier

Code:

ubuntutest:~$ sudo ip link set dev enp10s0 up

ubuntutest:~$ ip address show dev enp10s0
3: enp10s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:22:bd:ea:9a:cc brd ff:ff:ff:ff:ff:ff

Once again, I'm a newbie at this 10G network stuff, so it's probably the interface between the chair and the keyboard causing the issue.

camper8080 · Jan 30, 2020

fohdeesha said:
you can unplug all but one of the fans internally if you want it quieter (or replace the fans). If I recall correctly the 48 port PoE has THREE fans, intended to cool it if it's putting out hundreds of watts of PoE load, which you won't be doing

I did remove two fans but the management thermal plane reached 80C. The fan threshold was still in speed 1 but I stopped and went back to 3 fans. Maybe it would've stabilized if I gave it a little more time. Only activity was console access to monitor temps. The ambient temp in the room was around 70F. I plan on keeping it in the basement which is cooler but summers are hot and humid and forced to run a dehumidifier so I'm a little worried. I also have a SuperMicro E300-9D-4CN8TP with 3 fans that are louder than I want.

If I replace the fans, is there a particular brand/model everyone is using?

Thank you!

fohdeesha · Jan 30, 2020

the "not supported" thing just means officially, so support has something to blame if you file an issue under a support contract. These things will link up up with literally anything, I've even stuck in fibrechannel optics and it just clocked them down and ran happily. I suspect you have a configuration issue somewhere. Can you post the output of "show run?"

edit: also the output of show int br e 1/3/1 and show int br e 1/3/8

iotapi322 · Jan 30, 2020

I tried moving to ports 1/3/1 and 1/3/2

show run

Code:

SSH@brocade01>show run
Current configuration:
!
ver 08.0.30tT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
hostname brocade01
ip dhcp-client disable
ip dns server-address 10.0.0.131
!
no telnet server
username snoby password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT-05
!
!
ntp
 disable serve
 server 10.0.0.1
!
!
!
!
!
!
!
!
!
interface ethernet 1/3/1
 speed-duplex 10G-full
!
interface ethernet 1/3/2
 speed-duplex 10G-full
!
interface ethernet 1/3/3
 speed-duplex 10G-full
!
interface ethernet 1/3/4
 speed-duplex 10G-full
!
interface ethernet 1/3/5
 speed-duplex 10G-full
!
interface ethernet 1/3/6
 speed-duplex 10G-full
!
interface ethernet 1/3/7
 speed-duplex 10G-full
!
interface ethernet 1/3/8
 speed-duplex 10G-full
!
interface ve 1
 ip address 10.0.0.3 255.255.255.0
!
!
!
!
!
!
!
!
!
end

commands outputs from show int br e 1/3/x

Code:

SSH@brocade01>show int br e 1/3/1

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/3/1      Up      Forward Full 10G   None  No  1    0   748e.f893.efb0
SSH@brocade01>show int br e 1/3/2

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/3/2      Up      Forward Full 10G   None  No  1    0   748e.f893.efb0

Maybe i need to change the cisco card ...

fohdeesha · Jan 30, 2020

OK yeah, the brocade is perfectly fine, your ubuntu install just doesn't like the card or isn't configured properly, no clue which. never seen or touched those cisco cards, I keep it simple with mellanox

jd.developer · Jan 30, 2020

kapone said:
You remember right. It's not.

Thank you both! I'll see what I can dig up in part numbers and see if I can replace.

csementuh · Jan 30, 2020

Anyone have a dead 6450 that still had working POE? My 48 port 6450P has a burnt POE board and I need a replacement.

Or I'll sell this 48P switch OR trade for a 24P with working POE. I replaced my main switch with a ICX7250-48P but need something with POE for my garage. Thanks!

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Kaini Industries

Active Member

New Member

New Member

New Member

Kaini Industries

Member

Well-Known Member

Kaini Industries

Member

Kaini Industries

Kaini Industries

Active Member

Member

New Member

Kaini Industries

Member

Kaini Industries

New Member

Member