Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[logixworx]

anyone know if Brocade 10G-SFPP-TWX-0101 cables are cisco compatible? can I use them with brocade icx 6450 as well as intel x520-da cards?

 

[dbvader]

And if I statically assign the phone a different IP address, then all of a sudden now it can reach the internet and locally-hosted services again

It seems assigning a different rather than just a static address is what's fixing the issue.

When a client (e.g. 10.10.10.1) roams from an AP to a different AP, and those APs are attached to different ports on the Brocade switch (e.g. ports 1/1/4 and 1/1/5), you should see log messages akin to;

<timestamp>:next hop router 10.10.10.1 moved from port 1/1/4 to port 1/1/5

which is something to look for.

I'd run a traceroute from the client to the internet, and from the expected hop after where traffic is being blocked back to the client, or at least from further upstream if you can't run a traceroute on the hop immediately after where traffic is being blocked.

 

[dbvader]

@ProxmoxProphet Ah, I see @jode already suggested checking logs. Also agree there's a good chance it's due to the other switch or even the opnsense node.

Also, where is the DHCP server running and how intelligent are the APs? Are they just doing L2 or also L3?

 

[Koop]

Hey, what the dog doin?

 

[nlj]

Hi everybody. Someone over on Reddit asked me to document my fan mod for my ICX 7150-24; I hope this is helpful to someone. See my previous post in this thread for my research on various different makes/models of fans. This is a soft mod, meaning it's easily reversible and you can put the stock fan back without any issues.

Items needed:

• Delta FFB0412VHN-F00 fan [Digikey]
• A 3-pin Molex fan header [Digikey]
  • You can cut off the existing fan's connector, but then this becomes a hard mod that you can't easily revert
• [optional] if soldering: solder and flux
• electrical tape or [optional] heat shrink tubing (especially if soldering)

Equipment needed:

• Wire cutter and/or wire stripper
• [optional] if soldering: soldering iron and related supplies
• [optional] if using heat shrink tubing: heat source like hairdryer or heat gun

Process:

The motherboard fan pinout on the ICX-7150 does not match the standard ATX pinout, so pay careful attention to the pinout to make sure you connect the right wires. The motherboard pins are: 1) fan speed, 2) 12VDC+, 3) 12VDC–.

• Cut the fan header so you have a single, appropriately-sized "pigtail" that has a Molex connector on one end.
• Strip the pigtail wires, and if not already stripped, the fan wires. You may consider trimming the wire lengths so there's less excess wiring in the case.
• If using heat shrink tubing, thread onto the wires BEFORE soldering them.
• Attach the fan to the header pigtail using the following pinout chart, either by electrical tape or soldering.
  • If soldering, cover with electrical tape or heat shrink tubing.
• Remove the old fan and replace in the same way with the new fan.

Pinout (for ATX-style Molex connectors):

Pin	Molex ATX fan wire color	Fan wire color	Purpose
1	Black	Blue	Speed/Tach
2	Red	Red	12VDC+
3	Yellow	Black	12VDC–

Finally, I've attached a photo of a completed project for reference. I hope this helps all you fan modders out there! Cheers!

 

[anomaly]

Back to this behemoth of a thread: I am helping a friend with a 6450 and doing the intra-vlan routing for him. I'm beyond rusty since I touched ACLs and forgot how to assign them/limit them to specific VEs. I have cleared all the ACLs, added VEs for the VLANs that are meant to be routed by the switch, and yet, trying to reach any host between any two VLANs (with the correct VEs) isn't working.

I'm testing with a static route using the IP of the VE in the VLAN of the test host (ex. 10.7.6.3 in VLAN 3), for the address of the target host (ex. 10.7.7.3 in VLAN 2). THe switch has the VEs at .2 for both VLANs (ex. 10.7.6.2). And yes, it is the routing-enabled firmware variant

What am I missing?

Nevermind, had to adjust the gateway for the clients in VLAN 3.

This is still something I have not sorted out:

SSH@switch-1(config-vif-10)#ip access-group "test_traffic" out                                 

ERROR: ACL test_traffic has filters with "log" action. Output Access Lists cannot be configured.

I was able to set an ACL with log directives before. Running ICX64R08030t.

 

[anomaly]

Pinout (for ATX-style Molex connectors):

Pin	Molex ATX fan wire color	Fan wire color	Purpose
1	Black	Blue	Speed/Tach
2	Red	Red	12VDC+
3	Yellow	Black	12VDC–

Finally, I've attached a photo of a completed project for reference. I hope this helps all you fan modders out there! Cheers!

View attachment 38358

You can get a cheap crimp tool and connectors from Aliexpress. Once you get the gist of it it's quicker than bothering yourself with solder joints and cutting wires. Search for "crimping JST kit", I believe the pitch in these is 2.0mm. @fohdeesha can probably confirm.

 

[DangerMc]

I'm having some issues connecting 3 ICX7450s together using the QSFP port on the back. I have one with two qsfp ports and then from there a MTP cable out to each of the two other switches. (I'm using brocade transceivers from FS along with MTP cables from them too) However, they don't seem to be showing as active/up. Im trying to configure these in a uplink mode and not stack, so I have gone in and disabled the stacking feature but that didn't change anything.
Is there anything else I should have done or should this just work as an uplink port "out of the box?", i.e. is there some likely hood that the cable is broken?
Thanks

 

[Fluxor]

Having an issue with upgrade on a 7150-C12P, Seems stuck in a boot loop. I get this message during boot and then it reboots

CPLD is upgraded Successfully !!!Applying factory defaults..
sil_ramfs_creat: Failed to create file /var/log/pdc/pdc_errorlog.txt

Could not create ramfs file!
pdc_errorlog
Could not open ramfs file!

Got my answer from a thread titled "Brocade ICX 7150-C12P stuck in boot loop after upgrade"

If this happens to you, just hard power cycle the unit and it will continue

 

[anomaly]

Setting up a 7250-48p for a friend, the fans are stuck on speed 2:

Fan 1 ok, speed (auto): 1<->[[2]]

Fan 2 ok, speed (auto): 1<->[[2]]

Fan 3 ok, speed (auto): 1<->[[2]]

I tried 'chassis fanless' but it is not supported on the 7250. Is there a reason the fans are not ramping down? (case is open)

 

[anomaly]

Putting the cover on:

The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 60.7 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 28.5 deg-C

Fan speed switching temperature thresholds:
        Rule 1/2 (MGMT THERMAL PLANE):
                Speed 1: NM<-----> 95       deg-C
                Speed 2:        85<----->105 deg-C (shutdown)
        Rule 2/2 (AIR OUTLET NEAR PSU):
                Speed 1: NM<-----> 41       deg-C
                Speed 2:        34<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back 
Fan 2 Air Flow Direction:  Front to Back                          
Fan 3 Air Flow Direction:  Front to Back 
Slot 1 Current Temperature: 61.2 deg-C (Sensor 1), 28.5 deg-C (Sensor 2)

So, without the cover:

Slot 1 Current Temperature: 63.5 deg-C (Sensor 1), 56.0 deg-C (Sensor 2)

How can we configure the rules?? Are these somewhere we can reach with commands or inside the baremetal Linux OS? Obviously the threshold of 63.5 is too low.

 

[RuckusVol]

I'm sorry for not following up on this sooner. We have a newborn and there's been some time-consuming challenges and I had to set this aside for a few weeks.

There are multiple ways to do this, but since your eventual goal is to have the SFP provide routes to the Internet (which it will do over one or more VLANs), you don't want to use the layer 3 (router-only) method.

Which version of the ICX software are you running? Is it the 'S' (layer 2) or 'R' (layer 3) flavor? If it's 'R', are you currently using the ICX as a router, or are all of your ports and VLANs configured for layer 2 connectivity only?

You say the 'recommended' address for the SFP is 192.168.1.200, but what's critical to know is how the device will get its address. It will either be hardcoded in the device, or it will use BOOTP or DHCP to obtain an address (in which case there is no real need to use the 'recommended' address).

With the answers to all of those questions, it is possible to figure out a working configuration.

I'm not sure whether I'm on the S or R version of software. show run ver 08.0.95mT213.

I've been able to access the SFP module on 192.168.1.1. I've configured it and get the O5 status indicating it's getting what it wants from the ISP. I have not been able to see any traffic on the back side of the SFP that's trying to communicate with my router so I can get a DHCP lease.

I'm not intentionally using the switch as a router. I'm not very good at networking and I'm trying really hard to learn. I believe there are a few IP routes in there, from the guide here and maybe one I added to get to the SFP. I've spent 12+ hours on this this weekend and I'm still stuck haha. But currently, I'm trying to use port 1/2/8 for the GPON SFP and 1/1/22 for the ethernet to router WAN. I created a VLAN 1202 (which is the VLAN I get from the ISP) and I've tried creating it with both tagged and untagged ports for the members, with and without any other vlans assigned to the ports. When I do a tcp dump I don't see any traffic from the GPON.

I have the GPON set to auto use the ISP VLAN. I also tried the forward anything option from the guides I've seen.

which SFP ONT? you'll most likely not want to do any IP addressing or routing on the icx itself. Just plug the SFP in, then on a computer on the same network (plugged into an ICX port that's on the same VLAN as the sfp+ port), and assign that computer the .200 address. the ICX shouldn't have an IP in that SFP+ vlan. Which leads me to - if you haven't, the sfp should be in its own isolated vlan, as it will be providing your internet feed. then you can connect that vlan to your routers WAN port, by either putting another icx port in that vlan, or trunking the vlan over something to your router. This is what I'm doing with my WAS-110 ONT in my icx6610

I'm using a DFP-34X-2C2 and following the PON Madness guide. I currently have VLAN 1202 setup with all ports tagged and the only traffic on that vlan I see is from the router requesting a DHCP lease. When I do a tcpdump on the whole interface I still don't see other traffic from the GPON SFP. I also tried untagged ports.

What's the proper setup? Just those two ports on VLAN 1202 and no other VLAN? Tagged or Untagged? Should a third port used to access the SFP GPON be tagged or untagged to be able to get to it? Any other ideas why I'm not getting traffic through to the ethernet side that goes to the router WAN? I really appreciate the input so far. You guys take care.

 

[R3Z3N]

snip....Fan speed 2 on complaint....snip

There is no software/firmware logic to change on these ICX models beside the model that supports fanless mode. Either add more cooling via some hackey method, run ac to where this is located, or buy a more energy efficient router/switch.

 

[kpfleming]

I'm not sure whether I'm on the S or R version of software. show run ver 08.0.95mT213.

You'll need to show the full output of 'show version', that alone is not enough.

 

[compufritz]

Not understanding what my problem is here, but I assume it's something I configured incorrectly. I have an ICX6450-48P that I reset and updated following Fodeesha's guide. I set up a few VLANs and a transit network to go to my pfSense. Right now this is all offline as I finish configuring before changing the whole network over, but I am setting up 2 Windows Server VMs acting as DCs handling DHCP and DNS, pfSense acting as the firewall on the transit, and the 6450 as the L3 switch.

I have my desktop connected to one of the SFP+ ports using a Brocade optic and a fiber cable. I can connect just fine to the switch using putty and modify configuration, however I am unable to ping the other IPs of the ve's I set up (i.e. can't ping 10.10.5.1 from PC which is 10.10.10.50). The IP address on the desktop is configured manually (10.10.10.50, 255.255.255.0, 10.10.10.1 as the gateway). If I connect my laptop into port 1/1/2 it will pull an IP address from DHCP and I am able to ping 10.10.5.1 and 10.10.10.1, but not 10.10.10.50 (desktop). Switch config

SSH@icx6450>show run
Current configuration:
!
ver 08.0.30uT313
!
stack unit 1
  module 1 icx6450-48p-poe-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 5 name MGMT by port
tagged ethe 1/1/11 to 1/1/12 ethe 1/1/46 ethe 1/1/48
untagged ethe 1/1/2 to 1/1/4
router-interface ve 5
!
vlan 10 name Trusted by port
tagged ethe 1/1/11 to 1/1/12 ethe 1/1/46 ethe 1/1/48
untagged ethe 1/1/5 to 1/1/10 ethe 1/1/13 to 1/1/36 ethe 1/2/1 to 1/2/3
router-interface ve 10
!
vlan 20 name Lights by port
tagged ethe 1/1/11 to 1/1/12 ethe 1/1/46 ethe 1/1/48
untagged ethe 1/1/38 ethe 1/1/40
router-interface ve 20
!
vlan 30 name NVR by port
untagged ethe 1/1/37 ethe 1/1/39 ethe 1/1/41 ethe 1/1/43 ethe 1/1/45 ethe 1/1/4                                       7
router-interface ve 30
!
vlan 40 name IOT by port
tagged ethe 1/1/11 to 1/1/12 ethe 1/1/46 ethe 1/1/48
untagged ethe 1/1/42
router-interface ve 40
!
vlan 50 name Guest by port
tagged ethe 1/1/11 to 1/1/12 ethe 1/1/46 ethe 1/1/48
untagged ethe 1/1/44
router-interface ve 50
!
vlan 172 name Transit by port
untagged ethe 1/1/1 ethe 1/2/4
router-interface ve 172
!
!
!
!
!
optical-monitor
aaa authentication web-server default local
aaa authentication login default local
hostname icx6450
ip dhcp-client disable
ip route 0.0.0.0/0 172.16.1.1
!
no telnet server
username admin password .....
!
!
web-management https
!
!
!
interface ethernet 1/1/11
dual-mode  5
!
interface ethernet 1/1/12
dual-mode  5
!
interface ethernet 1/1/37
inline power
!
interface ethernet 1/1/38
inline power
!
interface ethernet 1/1/39
inline power
!
interface ethernet 1/1/40
inline power
!
interface ethernet 1/1/41
inline power
!
interface ethernet 1/1/42
inline power
!
interface ethernet 1/1/43
inline power
!
interface ethernet 1/1/44
inline power
!
interface ethernet 1/1/45
inline power
!
interface ethernet 1/1/46
dual-mode  5
inline power
!
interface ethernet 1/1/47
inline power
!
interface ethernet 1/1/48
dual-mode  5
inline power
!
interface ethernet 1/2/1
speed-duplex 1000-full-master
!
interface ve 5
ip address 10.10.5.1 255.255.255.0
!
interface ve 10
ip address 10.10.10.1 255.255.255.0
!
interface ve 20
ip address 10.10.20.1 255.255.255.0
!
interface ve 30
ip address 10.10.30.1 255.255.255.0
!
interface ve 40
ip address 10.10.40.1 255.255.255.0
!
interface ve 50
ip address 10.10.50.1 255.255.255.0
!
interface ve 172
ip address 172.16.1.2 255.255.255.252
!
!
!
!
!
!
!
!
!
end

I noticed that at the very beginning of the config it doesn't say 'stack disable' like I've seen in some other configs posted here, so I ran a 'stack unconfigure clean' and got the following:

Leads me to believe that it isn't part of a stack. The 10G ports are licensed following the guide. It seems like it's not routing from module 2 to module 1 and vice versa and I can't seem to figure it out. Any help would be appreciated. Thanks

 

[N1h1l1ty]

Been a long time since I posted here... (6 years) but huge thanks @fohdeesha, buying an ICX6610 may or may not have sent me down a deep spiral that ended up with me as a sysadmin for a career.

Said career actually led to me recently buying my first home and naturally, despite the place being pretty tiny, I wanted 10 gig... and POE... and all the bells and whistles. Fast forward a few months and I'm the relatively happy new owner of an ICX7250-24P. I'd figure I'd share my experience trying to quiet this thing down and also the kind of silly oopsie I made that I've now figured out and may as well mention in case someone else runs into the same thing.

Now, context here is that I'm making do with the top of my laundry closet doubling for server/network closet duties. Its not exactly the largest space and even though I've got a door and a sliding bookshelf in front of it, noise is definitely a concern, so when I got the 7250 in I fully expected to have to, ahem, do a little work - as expected, the Foxconn fans were just too loud even at speed 1. Naturally I perused this thread, found the Delta EFB0412VHD-F00 + Sunon MF60101V3-1000U-A99 on asic recommendation, went ahead and ordered. Installation was fairly straightforward, I did not bother with new connectors and just cut the connectors off the Foxconns and soldered in the Deltas and Sunon:

http://imgur.com/a/VNOsWq3

Closed er up, looked at temps, all was swell... and this is where I make a critical error. What I did not realize (and I hope this will be informative for anyone that does this and then sees temps climb up and down and up and down between speeds 1 and 2) is that the ICX7250 relies on diagonal airflow, with air being pulled in not from the front but from the left (Power supply) side of the chassis and traversing across the board / PSU on its way out when pulled by the fans. I made the mistake of sandwiching this poor thing beside my UPS, which itself is not actually running hot at all... but completely eliminating any actual potential for the switch to draw in fresh (relatively for a tiny closet) air:

http://imgur.com/0y1Xpvl

This didn't actually lead to the switch overheating - in Speed 2 the Deltas are actually fully capable of keeping PSU temps below 50 C and the ASIC is completely fine the entire time, stabilizing at 68 when running at Speed 1 and 60 at Speed 2, even with the airflow restriction. However, once I moved the switch 2 inches away from the UPS (and also elevated it for good measure to stop the bottom of the switch radiating heat inwards) temps have completely stabilized at 67-68 on the ASIC and 45 on the PSU air outlet. The best part? Its dead silent. I mean, the M720q above it makes more noise, and that thing is running at a cool 25C.

Overall, lessons have been learned... I actually ordered some more potent Mechatronix fans while I was trying to puzzle out this (silly) temp issue, but I think I'm going to experiment with adding one as a third fan just to completely eliminate any chance of the PSU running too hot.

 

[ManoftheSea]

I have my desktop connected to one of the SFP+ ports using a Brocade optic and a fiber cable.

SSH@icx6450>show run
...
vlan 10 name Trusted by port
tagged ethe 1/1/11 to 1/1/12 ethe 1/1/46 ethe 1/1/48
untagged ethe 1/1/5 to 1/1/10 ethe 1/1/13 to 1/1/36 ethe 1/2/1 to 1/2/3
router-interface ve 10
!
vlan 172 name Transit by port
untagged ethe 1/1/1 ethe 1/2/4
router-interface ve 172
!
end

Any help would be appreciated. Thanks

You don't happen to have the desktop plugged into 1/2/4, do you?

 

[compufritz]

You don't happen to have the desktop plugged into 1/2/4, do you?

No, it's definitely plugged into 1/2/1. I actually tested plugging my laptop in 1/2/1 using a RJ45 SFP adapter and it is able to ping the desktop at 10.10.10.50, but is also doesn't get an IP from DHCP servers on the DCs and is unable to ping 10.10.5.1 or either of the DC VMs(10.10.5.2 & 10.10.5.3)

 

[ManoftheSea]

It seems like it's not routing from module 2 to module 1

It looks like you can plug the laptop into 1/1/5, manually assign 10.10.10.49/24, and ping the desktop at 10.10.10.50. This will demonstrate that modules 1 and 2 can pass traffic to each other.

You might also try running tcpdump at various points to determine where traffic is breaking down.

I'd also recommend the "show ip route" and "show ip cache" commands.

 

[compufritz]

It looks like you can plug the laptop into 1/1/5, manually assign 10.10.10.49/24, and ping the desktop at 10.10.10.50. This will demonstrate that modules 1 and 2 can pass traffic to each other.

You might also try running tcpdump at various points to determine where traffic is breaking down.

I'd also recommend the "show ip route" and "show ip cache" commands.

Duh moment...eliminate variable to isolate the problem. Didn't think of that last night.

I'll try that later tonight. I had run "show ip route" and it looked normal. I'll run both of those commands later and post the output. Thanks