Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Seba]

Hi guys,

thanks for this great and for the amazing guides fohdeesha.

I am looking for a stable switch for my new „old“ house.

I am not sure if the ICX 6610 or an ICX 7450 would the right choice.

With POE they both cost the same here in germany. The noise and the power conjunction from the 6610 are no problems for me and a commandline based config is also okay.

Do you have any suggestions for me guys?

 

[pinkypie]

Hello LR, trying to understand something. I put the static route in my laptop and can now ping devices on VLAN 2, VLAN 3 and those VLANs can ping VLAN 1. Also, VLAN 2 and VLAN 3 can ping between themselves, however, they could do this before the static route was added.

What I am trying to figure out is why the static route would go on the source device and not the device doing the routing, i.e. Nighthawk or the ICX 6450?

 

[DavidRa]

Hi guys,

I am looking for a stable switch for my new „old“ house.

I am not sure if the ICX 6610 or an ICX 7450 would the right choice.

Speaking as a 6450 owner, for me, it'd be the 7450. Newer tech, still updated I think where the 6610 is done. Likely lower power too.

 

[Serhan]

Hi guys,

thanks for this great and for the amazing guides fohdeesha.

I am looking for a stable switch for my new „old“ house.

I am not sure if the ICX 6610 or an ICX 7450 would the right choice.

With POE they both cost the same here in germany. The noise and the power conjunction from the 6610 are no problems for me and a commandline based config is also okay.

Do you have any suggestions for me guys?

I have them both. My 6610 was manufactured in 2012 and 7450 was manufactured in 2018. While I don't know how long these switches can operate, I would consider 7450 being a newer unit. Noise is the same on both units and they both have the a version of the psu.

 

[Serhan]

I am trying to stack two 7450's that are 70 feet apart. What is the cheapest way of doing this? Any suggestions?

 

[kpfleming]

I am trying to stack two 7450's that are 70 feet apart. What is the cheapest way of doing this? Any suggestions?

Can you run new cables between them, or do you have existing cables that you must use?

 

[Serhan]

Can you run new cables between them, or do you have existing cables that you must use?

I have 5 runs of OM4 LC/LC fiber between these two locations. Those were for 5 servers connected to one of the 7450s which was the previous set-up. I had. I wish there could be a way to repurpose those pre-terminated lc/lc cables to stack the 7450s

 

[kpfleming]

That should be fine, OM4 can handle 40GbE. QSFP+ transceivers like this could be used, and they can be purchased with Brocade coding.

 

[Jamy]

I bought an ICX6650. Has anyone put quieter fans in? What was used?

Thanks

 

[Ruchira]

I bought an ICX 7250-24P too I unplugged one fan out of two but It is significantly louder than my dell R630 at 20% fan load. I'm going to replace the fans and I'm torn between these 2 options

109P0412B3013 Sanyo Denki America Inc. | Fans, Thermal Management | DigiKey

Order today, ships today. 109P0412B3013 – Fan Tubeaxial 12VDC Square - 40mm L x 40mm H Ball 13.4 CFM (0.375m³/min) 3 Wire Leads from Sanyo Denki America Inc.. Pricing and Availability on millions of electronic components from Digi-Key Electronics.

www.digikey.com.au

MR4020X12B1-RSR Mechatronics Fan Group | Fans, Thermal Management | DigiKey

Order today, ships today. MR4020X12B1-RSR – Fan Tubeaxial 12VDC Square - 40mm L x 40mm H Ball 17.3 CFM (0.484m³/min) 3 Wire Leads from Mechatronics Fan Group. Pricing and Availability on millions of electronic components from Digi-Key Electronics.

www.digikey.com.au

Can anyone help to select the best fit between these?

Thanks

 

[LodeRunner]

I have a little Question to the License. I have bought an Brocade VDX6740-24.

I don´t know the exactly meaning of the output of "show license" and "show dpod".

Here is the output.

<snipped console output>

Does this mean, from 24 Ports are 16 for 10Gbit and also all 4 QSFTP Ports are licensed for 40 Gbit, yes?

Or does this mean 16 Ports @ 10 Gbit and 24 Ports @ 1Gbit too?

Why does a VDX6740-24 with normaly 24 Ports show 48 / 40 Ports? Can i license this 24 Port one to full 48 Ports? The Hardware is in the switch?

Thanks for your help I'am new at Brocade - sorry for the i think stupid question

As I understand the output you provided, the switch natively comes with 24 of the ports licensed for 10G operation and the remaining 24 require POD licensing. So 24 port base license, plus 16 port POD license means you have 40 ports that can be brought online in at 10G, plus the 4x40 G port license. It looks like as ports are brought up, the licenses are first assigned from the base pool; once that is consumed, it will start issuing licenses from the POD pool.

To get all 48 ports active at 10G you'd need to find an additional 8 port POD license.

There was a dedicated thread: Brocade VDX 6740 | ServeTheHome Forums Posters in that thread might have additional help/guidance for you in terms of firmware and licensing issues and how they might be resolved.

 

[LodeRunner]

Hello LR, trying to understand something. I put the static route in my laptop and can now ping devices on VLAN 2, VLAN 3 and those VLANs can ping VLAN 1. Also, VLAN 2 and VLAN 3 can ping between themselves, however, they could do this before the static route was added.

What I am trying to figure out is why the static route would go on the source device and not the device doing the routing, i.e. Nighthawk or the ICX 6450?

The static route on your laptop is to tell it how to get to the other subnets. Since those subnets are not connected via your Nighthawk, and your Nighthawk is your default gateway, you have to tell your client (source) how it can reach those networks. For devices in VLAN 2 and 3, they're using the ICX as the gateway. The ICX knows what networks are directly connected and can route them; then anything that does not match a direct connected network is sent to the ICX's default gateway, the Nighthawk.

In your specific case, this is required because the Nighthawk and the ICX are in the same subnet.

If your Nighthawk had the ability to do static routes, the more enterprise correct way of setting this up would be

• each VLAN has its own subnet (you have this)
• each VLAN has a VE with IP
• a dedicated VLAN between the ICX and firewall; this VLAN has its own subnet
• ICX uses firewall VLAN IP as its upstream gateway
• The firewall has reverse routes for each subnet using the ICX IP as its gateway

Theoretically, you could try setting the gateway on your laptop to use the ICX IP and the ICX would send any non-local traffic to the Nighthawk, but the Nighthawk can directly see your client, so return traffic would come direct instead of going back through the ICX leading to asymmetric routing which generally causes issues.

 

[Serhan]

That should be fine, OM4 can handle 40GbE. QSFP+ transceivers like this could be used, and they can be purchased with Brocade coding.

Thank you.

 

[Chow]

As I understand the output you provided, the switch natively comes with 24 of the ports licensed for 10G operation and the remaining 24 require POD licensing. So 24 port base license, plus 16 port POD license means you have 40 ports that can be brought online in at 10G, plus the 4x40 G port license. It looks like as ports are brought up, the licenses are first assigned from the base pool; once that is consumed, it will start issuing licenses from the POD pool.

To get all 48 ports active at 10G you'd need to find an additional 8 port POD license.

There was a dedicated thread: Brocade VDX 6740 | ServeTheHome Forums Posters in that thread might have additional help/guidance for you in terms of firmware and licensing issues and how they might be resolved.

Thank´s a lot LodeRunner ;-) I will write there ;-)

Ah ok that´s very cool. I thought, that i have only 16 Ports @ 10 Gbit and 4 Ports on 40 Gbit :-D

But you think a VDX6740-24 Model comes with normaly 24 x 10 Gbit Ports in Standard, yes? And the Licenses goes up on top, right?

 

[hibby50]

Hey Guys, I bought a 6610 based on this thread and I could use some help. I'm having an issue configuring extended acls. I have a port tagged with multiple vlans (10,20,40,50) going to an access point, and a machine with an untagged LAG on vlan 50. The problem is on my inbound rule (mgmt-in) for vlan 50 I have `permit tcp any any established` And when that rule is in the inbound acl, the other vlans can ssh into the device on the LAG. If I remove it they can't but then it also cant get out

I also tried enable acl-per-port-per-vlan despite not knowing what it means, but that did nothing

Edit: I dumbed it down and put one machine on an untagged vlan 20 port and the other on the LAG untagged vlan 50 port with only 1 acl on the vlan 50 ve: 'permit tcp any any established' and I was able to initiate an ssh session from the vlan 20 machine. Totally stumped.

SSH@ICX6610-48P-Router(config)#show lag

=== LAG "hibsrv-LAG" ID 2028 (dynamic Deployed) ===

LAG Configuration:
Ports: e 1/3/1 to 1/3/2
Port Count: 2
Primary Port: 1/3/1
Trunk Type: hash-based
LACP Key: 22028
LACP Timeout: long
Deployment: HW Trunk ID 2
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/3/1 Up Forward Full 10G 2028 No 50 0 748e.f8ff.fc4e
1/3/2 Up Forward Full 10G 2028 No 50 0 748e.f8ff.fc4e

Port [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/3/1 1 1 22028 Yes L Agg Syn Col Dis No No Ope
1/3/2 1 1 22028 Yes L Agg Syn Col Dis No No Ope


Partner Info and PDU Statistics
Port Partner Partner LACP LACP
System ID Key Rx Count Tx Count
1/3/1 65535-246e.9600.5080 15 11 9
1/3/2 65535-246e.9600.5080 15 8 8

SSH@ICX6610-48P-Router(config)#sh run int e 1/1/13

interface ethernet 1/1/13
inline power
!

SSH@ICX6610-48P-Router(config)#sh vlan br     

PORT-VLAN 10, Name guest, Priority level0, Spanning tree Off

Untagged Ports: None
Tagged Ports: (U1/M1) 13 14 15
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

PORT-VLAN 20, Name iot, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1) 39
Tagged Ports: (U1/M1) 13 14 15
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

!

PORT-VLAN 50, Name mgmt, Priority level0, Spanning tree Off

Untagged Ports: (U1/M1)  37  38
Untagged Ports: (U1/M2) 1 2 3 4 5 6 7 8 9 10
Untagged Ports: (U1/M3) 1 2 3 4 5 6 7 8
Tagged Ports: (U1/M1) 13 14
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

Extended IP access list  mgmt-in : 10 entry

ACL Remark:  ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
ACL Remark: ALLOW DNS
permit tcp 192.168.50.0 0.0.0.255 host 10.0.0.2 eq dns
permit udp 192.168.50.0 0.0.0.255 host 10.0.0.2 eq dns
ACL Remark: ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
ACL Remark: ALLOW mDNS
permit udp any host 224.0.0.251 eq 5353
ACL Remark: ALLOW HERE TO ANYWHERE
permit ip 192.168.50.0 0.0.0.255 any
ACL Remark: DENY INTER-VLAN TRAFFIC
deny ip 192.168.0.0 0.0.255.255 any
deny ip 10.0.0.0 0.255.255.255 any
deny ip 172.16.0.0 0.15.255.255 any

 

[hibby50]

I just solved it. I realized I had a fundamental misunderstanding of inbound and outbound acl. It is inbound and outbound to the PORT, not inbound outbound to the vlan/subnet

 

[LodeRunner]

Thank´s a lot LodeRunner ;-) I will write there ;-)

Ah ok that´s very cool. I thought, that i have only 16 Ports @ 10 Gbit and 4 Ports on 40 Gbit :-D

But you think a VDX6740-24 Model comes with normaly 24 x 10 Gbit Ports in Standard, yes? And the Licenses goes up on top, right?

I'm taking what the output you provided says at face value:

40 10G port assignments are provisioned for use in this switch:
        24 10G port assignments are provisioned by the base switch license
        16 10G port assignments are provisioned by the 10G Port Upgrade license
14 10G ports are assigned to installed licenses:
        14 10G ports are assigned to the base switch license
         0 10G ports are assigned to the 10G Port Upgrade license

 

[Chow]

Thanks for your answer LodeRunner ;-)

Hmm ok, i thought that the "base switch license" is only 1 Gbit, not 10 Gbit?!

 

[fohdeesha]

I have 5 runs of OM4 LC/LC fiber between these two locations. Those were for 5 servers connected to one of the 7450s which was the previous set-up. I had. I wish there could be a way to repurpose those pre-terminated lc/lc cables to stack the 7450s

the 7450's happily stack over 10gbe as well, if you have the 4x 10gbe port modules for your 7450s. if you only have 40gb qsfp modules, there's the much cheaper Kaiam LR4 lite modules that would allow you to use your existing lc fiber if it were singlemode, but you say it's multimode - there's multimode SR4 BiDi transceivers like @kpfleming linked, but I don't know if the ICX7450 QSFP+ ports can supply enough power for SR4 BiDi (haven't seen anyone try, although the module he links claims only 1.5W of draw)

 

[RoachedCoach]

I bought an ICX 7250-24P too I unplugged one fan out of two but It is significantly louder than my dell R630 at 20% fan load. I'm going to replace the fans and I'm torn between these 2 options

109P0412B3013 Sanyo Denki America Inc. | Fans, Thermal Management | DigiKey

Order today, ships today. 109P0412B3013 – Fan Tubeaxial 12VDC Square - 40mm L x 40mm H Ball 13.4 CFM (0.375m³/min) 3 Wire Leads from Sanyo Denki America Inc.. Pricing and Availability on millions of electronic components from Digi-Key Electronics.

www.digikey.com.au

MR4020X12B1-RSR Mechatronics Fan Group | Fans, Thermal Management | DigiKey

Order today, ships today. MR4020X12B1-RSR – Fan Tubeaxial 12VDC Square - 40mm L x 40mm H Ball 17.3 CFM (0.484m³/min) 3 Wire Leads from Mechatronics Fan Group. Pricing and Availability on millions of electronic components from Digi-Key Electronics.

www.digikey.com.au

Can anyone help to select the best fit between these?

Thanks

I replaced mine with Mechatronix, they've been perfect for me.

Best bet is sticking a fan on the ASIC as well, makes all the difference.