Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[mattaw]

Any recommendations for new fans for a ICX 6610 PSU? I have a failed fan in one I want to replace, but it seems to be a special delta fan, or at least the info on the label does not match those I see on Delta's site/eBay. I can find a bunch of recommendations to not downgrade the fans (especially @fohdeesha wise warnings - and you should trust anyone who has gone as far as hooking up a JTAG diagnostic/programming adapter to the switch in question), but not any good purchasing targets.

Thanks, M

 

[TZann]

First, thanks for the feedback about my questions on ACL to provide internet only access to some devices.
I decided to look at using VLANs to do this since this seems to be the best approach.
I am look to start by having some of my servers in their own vlan and trying to understand how inter-vlan routing works.

I have the following setup

Vlans

vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1

vlan 10 name servers by port
 untagged ethe 1/1/18
 router-interface ve 10

Virtual Interfaces

interface ve 1
 ip address 192.168.1.80 255.255.255.0
!
interface ve 10
 ip address 192.168.10.1 255.255.255.0
 ip helper-address 1 192.168.1.1

show ip route

Total number of IP routes: 3
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          192.168.1.1     ve 1          1/1           S    11d16h
2       192.168.1.0/24     DIRECT          ve 1          0/0           D    11d16h
3       192.168.10.0/24    DIRECT          ve 10         0/0           D    1d15h

I only have one wired client on 1/1/18 with ip 192.168.10.200 with static ip and gateway 192.168.10.1.

The issue I am having right now is that I cannot ping 192.168.10.200 from 192.168.1.143.
My understanding of vlan routing is that I should be able to reach that device on subnet 20 with the above configuration, without having to setup any additional static routes on the switch or on my firewall/router.

Am I missing something or is my understand of L3 routing wrong?

Thanks

 

[eduncan911]

Looks like someone is dumping a bunch of 7150-48ZP on ebay.

Ruckus ICX 7150-48ZP POE Switch p/n ICX7150-48ZP-E2X10G 2 x Power 2 x Fan + Ears | eBay

Minor scuffs consistent with ordinary use. Tested for key functions/R2 ready for resale.

www.ebay.com

Still out of my budget... So so close though.

For those that don't know, the "ZP" models have PoH 802.3bt for up to 90W equipment, per port, as well as 16x 1/2.5/10 RJ45 ports.

I'm sure with the largest PoE budget of all 7150 series, 1480 W w/2x 920W PSUs, and those PSUs being the largest PSUs in the 7150 series, it would be loud. Very loud. Specs say 52 dBA.

My 7250-48P is the loudest thing in my rack (yeah, haven't done the fan swap yet), and it's rated at 45.9 dB. It drove me crazy when playing with it in my office for a few days. Much louder than my stock SC846 chassis that I've IPMI'd the fans down to 0%. Louder than the 3D printers I haven't replaced the loud stepper drivers in.

 

[richtj99]

I appreciate the advice migrating from Cisco to Brocade. The video about cisco trunk vs everyone elses trunk was eye opening. My Cisco stuff is fairly old & I really like the Brocades so I am thinking I might dump the ciscos for the most part. My setup right now is a bit messy. I am not using Vlan1 (rather - not meaning to). But if I move my cisco off vlan 1 things are not working properly. I lost the cisco passwords and - well if i have to redo it, might as well make it uniform.

I want to keep using my router which has 5 vlans and have them go to my ubiquiti switch and AP's, and use them with the brocade 6450-48p, then the 2nd brocade 6450-48p.

So if i am using 5 vlans:

75 - PBX
85 - Cameras
95 - Data
105- IOT
115 - Kids

device# enable
device(config)# vlan 75 PBX 
device(config-vlan-75)# tagged ethernet 1/1/1  
device(config-vlan-75)# exit
device(config)# vlan 85 Cameras
device(config-vlan-85)# tagged ethernet 1/1/1  
device(config-vlan-85)# exit
device(config)# vlan 95 Data 
device(config-vlan-95)# tagged ethernet 1/1/1  
device(config-vlan-95)# exit 
device(config)# vlan 105 IOT 
device(config-vlan-105)# tagged ethernet 1/1/1  
device(config-vlan-105)# exit 
device(config)# vlan 115 Kids 
device(config-vlan-115)# tagged ethernet 1/1/1  
device(config-vlan-115)# exit

This gives me 5 vlans all available from port 1/1/1
Tagged means that it can see all 5 vlan traffic tags?

If I do the same for 1/1/2 (tag all 5 vlans as above) - this gives me two ports - port 1 - feed from Unifi switch, port 2 feed to the next Brocade switch.

If the above is correct - ports 1 & 2 are similar to a cisco trunk port?

Can I do:

device(config)# vlan 115
device(config-vlan-115)# untagged ethernet 1/1/20 to 1/1/25  
device(config-vlan-115)# exit

Do I need to do a int e 1/1/20 to 1/1/25 to do the untagged?

Will that make it so ports 20-25 are only able to see Vlan 115? Can they change the vlan tag on their device to access another vlan?

Thanks,
Rich

 

[LodeRunner]

Tagging a single VLAN to a specific port means that the device on that port will also need to be tagging it's traffic, or it won't pass. For edge devices that only need a single connection, UNtagged is appropriate. I only tag edge ports when dealing with VOIP phones with a computer hooked up on the pass-thru port, so the phone gets tagged to one VLAN and passes the untagged traffic to the workstation (so technically any edge port serving a phone + computer is a trunk).

I believe that if the ports are untagged to a VLAN, a device that is tagging its packets will also pass no traffic, because the untagged port is not a trunk.

Edit: Any port carrying multiple VLANs is a trunk. You don't have to change it to trunk mode like you would on a Cisco. If you need a default VLAN on it, depending on the ICX version, you either need to do dual-mode <vlan> or you simply go to which VLAN should be the default for untagged traffic and do 'untag e x/y/z'

 

[mattaw]

Any recommendations for new fans for a ICX 6610 PSU? I have a failed fan in one I want to replace, but it seems to be a special delta fan, or at least
...

I am just tired - the PSU swap did not work - it was the power cord that had fallen out. Lesson: Always check the LED on the back - if the supply is faulty it will not be steady green, it will flash a color enabling diagnosis.

 

[ArmedAviator]

@TZann
I suspect the devices on your 192.168.1.0/24 subnet are unaware of the routes on your switch. There are two ways to rectify this:

• Use the switch's ve as the gateway on each device (192.168.1.80)
• Add a static route on the gateway/router currently being used by those devices
  • Example: 192.168.10.0/24 via 192.168.1.80

Note: It is advisable to prevent using VLAN ID 1 for anything, or having routed traffic on any VLAN ID configured as the Default VLAN. This is more or less for security and troubleshooting reasons.

@richtj99
Your VLAN + tagged proposals are correct.
Yes, setting a port as untagged strictly limits all traffic through that port on that specific VLAN. Tagged packets are ignored since that port can be considered "unaware" of any VLAN IDs from a client device point of view.

If you want tagged AND untagged traffic (e.g. a tagged VoIP VLAN but default to a VLAN for PCs), you will tag all of the VLANs desired and then go to the interface and set dual-mode <VLAN ID>

Here's an example using all three options:

vlan 10 name voip
tagged eth 1/1/1 to 1/1/48 eth 1/2/2
untagged eth 1/2/1

vlan 20 name pcs
tagged eth 1/1/1 to 1/1/24 eth 1/2/2

vlan 30 name whatever
tagged eth 1/1/25 to 1/1/48 eth 1/2/2

interface 1/1/1 to 1/1/24
dual-mode 20
voice-vlan 10

interface 1/1/25 to 1/1/48
dual-mode 30
voice-vlan 10

interface 1/2/1
port-name PBX

interface 1/2/2
port-name Link-to-switch2

Don't forget a management VLAN. Use the management VLAN (whatever ID on whatever ports you choose/need) to manage your network devices. For example, my switches use VLAN ID 5 as management. Only network switches, the Unifi Controller, Unifi APs and wireless bridges are on that VLAN on my network - the absolute necessary devices to operate my LAN. I have one port on each switch left as untagged on that VLAN as a simple "management" port (seperate from the factory "Management" port on these switches which I have configured differently).

Example:

vlan 5 name Management
tagged eth 1/2/1 to 1/2/4
untagged eth 1/1/48
router-interface ve 5

interface ve 5
port-name Management
ip address 10.0.5.1/24

interface eth 1/1/48
port-name Management

 

[snowsnoot]

EDIT: Found out that one fan wasnt running, it was a loose 12V connector. For some reason the switch reported all 3 not running but either way all are reporting as OK status now that I fixed the loose connector.

Hey everyone, following the advice of some on this thread and on YouTube I have replace the stock fans with 3x Sunon MB40201VX-000U. The are much quieter and everything worked great for about 12 hours, at which point my switch decided that all 3 of them had 'failed' at the exact same time. They still work fine and are moving some air, but seems like not much. When I power cycle the switch they run full speed OK then are throttled down once the boot has completed.

The sensor A temp went from ~42C to ~62C since the fans were replaced, which is a bit high for my liking. I am running about 16.5W of PoE (few cameras and APs).

Does anyone know the reason why these fans report as failed? I did fix the pinout to match the stock fan (Pin1 = GND, Pin2 = 12V, Pin3 = Sensor) and I find it interesting that they all "failed" at the exact same time (to the second) as per the switch log.

Lastly, do we know if there is a way to manually set the fan speed? I wouldnt mind running these full speed all the time, since they are lower CFM than the stock and my temps are a bit high. They are way quieter at full speed than the stock fans were at speed 1.

#show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 failed
Fan 2 failed
Fan 3 failed

Fan controlled temperature: 62.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 54.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 62.5 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C
Boot Prom MAC : xxxx.xxxx.xxxx
Management MAC: xxxx.xxxx.xxxx

 

[mintchipmadness]

Narrowed my options down to a few different Delta fans that I've had good luck with in the past. There's a lot of choices that appear to meet the specs for the Brocade, I went with the Delta FFB0412SHN-BF00 - they're slightly newer in design and quieter than its peers, coming in around 5 dB lower. They also run a max of 13000 RPM vs the Nidec's 18000 RPM (though I can't seem to manage to push it that high, ever).

It appears only the FFB0412SHN-F00 is available on digikey. They are about 17 dollars each which is fairly pricey. Where did you find the BF00 ones for sale in low quantities? Thank you for your help.

 

[eduncan911]

Trying to follow @fohdeesha 's guide here on setting up the 7150. Specifically 7150-C12P. These switches have newer versions of firmware 08.0.90BT211 (aka 8090?) and boot code 10.1.15T225. Fohdeesha's instructions are for 8080e.

Am I supposed to downgrade to 8080e? I started the process, but these must have the newer firmware that embeds the boot firmware with the primary image (that i read in this thread) - as I get the error: Invalid input -> bootrom when trying to copy the bootrom image.

When skipping past the firmware, and trying to license them, it says 45-day evaluation period. Searching this thread, I found a post by fohdeesha that said the guide in the first post handles this. For the 7150, there's just the datasheet though.

I am assuming I am supposed to downgrade to 8080e before registering the license?

Or, remain on 8090 and say Yes to the 45-day evaluation license, and it will remain unlocked?

Am I supposed to downgrade? And in doing so, ignore the bootrom option and just flash the .bin file only?

 

[nerdalertdk]

Looks like someone is dumping a bunch of 7150-48ZP on ebay.

Ruckus ICX 7150-48ZP POE Switch p/n ICX7150-48ZP-E2X10G 2 x Power 2 x Fan + Ears | eBay

Minor scuffs consistent with ordinary use. Tested for key functions/R2 ready for resale.

www.ebay.com

Still out of my budget... So so close though.

For those that don't know, the "ZP" models have PoH 802.3bt for up to 90W equipment, per port, as well as 16x 1/2.5/10 RJ45 ports. There's also a couple 1/2.5/5/10 ports as well (adding the "5Gbps" there).

I'm sure with the largest PoE budget of all 7150 series, 1480 W w/2x 920W PSUs, and those PSUs being the largest PSUs in the 7150 series, it would be loud. Very loud. Specs say 52 dBA.

My 7250-48P is the loudest thing in my rack (yeah, haven't done the fan swap yet), and it's rated at 45.9 dB. It drove me crazy when playing with it in my office for a few days. Much louder than my stock SC846 chassis that I've IPMI'd the fans down to 0%. Louder than the 3D printers I haven't replaced the loud stepper drivers in.

Hi its only 16x2.5gbps ports, i know since i have one

 

[eduncan911]

Hi its only 16x2.5gbps ports, i know since i have one

Humm. Are you saying only 2.5? Because the datasheet says in multiple places:

• 16x 100/1000 Mbps/2.5 Gbps RJ-45 PoH, 802.3bt ready (and 32× 10/100/1000 Mbps RJ-45 PoE+ ports).
• 100/1000 Mbps/2.5 Gbps RJ45 downlinks (full duplex only).....16

Oh, you were saying 2.5 - no 5Gbps. yeah, I now see in the specs that 1/2.5/5/10 ports are only on the 7150-C10ZP. Yeah, I'll fix that.

 

[nthu9280]

Can we get a floating web spectre that just says "RTFT, RTFM"?

Sorry, seems the "Cheerios" didn't work this am.

RTFM & Reading the first few posts require too much effort!

 

[TZann]

I suspect the devices on your 192.168.1.0/24 subnet are unaware of the routes on your switch. There are two ways to rectify this:

• Use the switch's ve as the gateway on each device (192.168.1.80)
• Add a static route on the gateway/router currently being used by those devices
  • Example: 192.168.10.0/24 via 192.168.1.80

Note: It is advisable to prevent using VLAN ID 1 for anything, or having routed traffic on any VLAN ID configured as the Default VLAN. This is more or less for security and troubleshooting reasons.

Thanks for your feedback. Some follow up questions if that's ok.

You are correct, the devices do not have ve 1 ip (192.168.1.80) as the gateway. I (wrongly) assumed that the switch would handle the routing based on the destination address. Based on what you wrote it's more clear now, and the way I understand it is that for this to happen a device needs to at a minimum be using as gateway one of the switch ve ips (which of course has it's vlan configured with a router-interface).

Ideally if all devices were in their own vlan (as per what you wrote avoid using vlan 1) they would anyway need to use their own vlan's gateway and this would solve the issue.

About the second option to add static route to the gateway/router. I tried this and it works. But with this setup isn't the gateway doing inter vlan routing instead of the switch? Doing a trace route shows the first hop as my internet gateway.

I guess static routes are needed anyway for all subnets that require at least internet access (so the gateway can route traffic to them).

Thanks

 

[Dade49]

RTFM & Reading the first few posts require too much effort!

This thread convinced me to buy an ICX6610-24P-PE recently. I followed all instructions and got the firmware updated to 8030u. Thanks to everyone for all the help.

I wish I had the license for the 8x SFP+ 10Gb ports on the front. I did PM @fohdeesha a couple of days ago and I'm hoping he's still able to help. This is the perfect switch to replace two other switches I have in my home. I decided to order a couple of Mellanox CX314A ConnectX-3 Pro cards for my NAS and Server. Going from 1GbE to 40GbE is exciting.

 

[ArmedAviator]

Thanks for your feedback. Some follow up questions if that's ok.

You are correct, the devices do not have ve 1 ip (192.168.1.80) as the gateway. I (wrongly) assumed that the switch would handle the routing based on the destination address. Based on what you wrote it's more clear now, and the way I understand it is that for this to happen a device needs to at a minimum be using as gateway one of the switch ve ips (which of course has it's vlan configured with a router-interface).

Ideally if all devices were in their own vlan (as per what you wrote avoid using vlan 1) they would anyway need to use their own vlan's gateway and this would solve the issue.

About the second option to add static route to the gateway/router. I tried this and it works. But with this setup isn't the gateway doing inter vlan routing instead of the switch? Doing a trace route shows the first hop as my internet gateway.

I guess static routes are needed anyway for all subnets that require at least internet access (so the gateway can route traffic to them).

Thanks

@TZann

The switch only handles routing that has traffic sent to it that needs to be routed (not on same L2 network). This only happens when the appropriate switch VE is used as a gateway (somewhere along the line) as you discovered.

Adding the static router to your internet router may or may not be ideal for your setup. That's up to you to determine your needs. If you're interested in maximizing the use of your Brocade and having all LAN routing on it, set up whatever VLANs you need (perhaps as few as one).

Here's an example:

vlan 10 name servers
untagged eth 1/1/18
router-interface ve 10

vlan 20 name clients
untagged eth int eth 1/1/1 to 1/1/17
router-interface ve 20

ip route 0.0.0.0/0 192.168.1.1
ip dns server-address 192.168.1.1

interface ve 10
ip address 192.168.10.1/24

interface ve 20
ip address 192.168.20.1/24

interface eth 1/1/18
port-name server1

interface eth 1/1/24
port-name internet
route-only
ip address 192.168.1.80/24

Regarding DHCP and the ip helper-address - you will need a DHCP server that works properly with multiple subnets. Your internet router may or may not support that (probably not).

 

[klui]

I wish I had the license for the 8x SFP+ 10Gb ports on the front. I did PM @fohdeesha a couple of days ago and I'm hoping he's still able to help.

@fohdeesha probably gets at least 50 messages/day and I'm sure he'll appreciate all inquirers give him leeway to respond as he finds the time. This thread is pretty unwieldy but packed with a lot of great information; make good use of the Search function w/ This Thread option selected.

The 6610 is quite loud and you can't really replace the fans to quiet them down, especially if you have Rev.A or I[ntake] fans/PSUs which have orange filled arrows. Get one with Rev.B/C and E[xhaust] fans which have green filled arrows. Search for RPS16-E and RPS16-I for comparison on eBay. Others have installed additional fans by modifying the lid/insides.

 

[Dade49]

@fohdeesha probably gets at least 50 messages/day and I'm sure he'll appreciate all inquirers give him leeway to respond as he finds the time. This thread is pretty unwieldy but packed with a lot of great information; make good use of the Search function w/ This Thread option selected.

The 6610 is quite loud and you can't really replace the fans to quiet them down, especially if you have Rev.A or I[ntake] fans/PSUs which have orange filled arrows. Get one with Rev.B/C and E[xhaust] fans which have green filled arrows. Search for RPS16-E and RPS16-I for comparison on eBay. Others have installed additional fans by modifying the lid/insides.

Absolutely, no offense intended. I'm sure there are many requests for help with this and I will be patient. In the mean time, I decided to order a QSFP breakout DAC and test that at 10Gb x4. I am also on the hunt for a backup PSU that doesn't cost as much as another switch. I read the info on the Rev B/C PSU's and will be mindful. Thanks for the help.

I've read about 12% of this thread, but clicked all the links in the first few pages and read the last 12 pages or so. It's a long thread for sure, but I'll get there.

 

[fohdeesha]

Trying to follow @fohdeesha 's guide here on setting up the 7150. Specifically 7150-C12P. These switches have newer versions of firmware 08.0.90BT211 (aka 8090?) and boot code 10.1.15T225. Fohdeesha's instructions are for 8080e.

Am I supposed to downgrade to 8080e? I started the process, but these must have the newer firmware that embeds the boot firmware with the primary image (that i read in this thread) - as I get the error: Invalid input -> bootrom when trying to copy the bootrom image.

When skipping past the firmware, and trying to license them, it says 45-day evaluation period. Searching this thread, I found a post by fohdeesha that said the guide in the first post handles this. For the 7150, there's just the datasheet though.

I am assuming I am supposed to downgrade to 8080e before registering the license?

Or, remain on 8090 and say Yes to the 45-day evaluation license, and it will remain unlocked?

Am I supposed to downgrade? And in doing so, ignore the bootrom option and just flash the .bin file only?

I need to update the fw on the guide but I haven't had the chance yet - don't downgrade - the codetrain I recommend is 8092 - flash whatever the latest in that is (8092e I'm pretty sure). The licenses are still honor only on all the versions, the 45 day thing is just....in spirit. Nothing happens in 45 days

 

[vswitchzero]

Not sure if anyone has done this before, but just wanted to mention that it is possible to run Noctua 40mm fans in the ICX 7250. By default, fan voltage is way too low for them. Even if they continue spinning after startup, they'll be marked as failed (and would not flow enough air to keep the switch cool anyway).

I got around this by pulling +12V off the RPSU board so that the fans would always spin at maximum RPM. The fans continue to report the tach signal to the board via the usual fan headers. At full speed, the switch has no problem with the RPM and considers the fans good. The noise level is SO much better even with the two fans at full speed. Switch temps are about 10'C higher in my case (from low 70s to low 80s now). This breaks fan speed control, obviously, but the switch has no idea.

I soldered two leads to one of the +12V pins on the rear of the RPSU board:

It's a bit ugly, but its on there very well. Here is a closeup of the fan wiring:

The fans I used were 3-pin 40x25mm NF-A4x20FLX models. I may see if I can get some additional cooling into the switch. Maybe a 3rd fan, but this would require a rear cutout. I may also see if a 10mm fan will have enough clearance to be mounted on top of the heatsink.