Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[CTurtle]

Surely, I´ve found mentions but so far no real confirmation. Normally I order from fs.com and they have Brocade options with DOM but I wanted to check if there were any confirmations.

I guess I will just try and order one

I was obviously joking, but from memory and a cursory search, I remember there being issues with a DAC cable not being detected, and that small snippet of this huge thread does post a lot of details about adapter eeprom formats and how to manipulate them, I didn't try to find the start of that topic, but here's one post somewhere in the meat of it: https://forums.servethehome.com/ind...erful-10gbe-40gbe-switching.21107/post-198322

I just tried to enable optical monitoring on my two QLogic branded 10G SFP+ adapters at ports 1/3/1 and 1/3/3 and:

SSH@puft-prince(config-mif-1/3/1-1/3/8)#optical-monitor
port 1/3/1 SFP can't support Optical Monitoring feature.
Port 1/3/2 is EMPTY: Can't Enable Optical Monitor
port 1/3/3 SFP can't support Optical Monitoring feature.
Port 1/3/4 is EMPTY: Can't Enable Optical Monitor
Port 1/3/5 is EMPTY: Can't Enable Optical Monitor
Port 1/3/6 is EMPTY: Can't Enable Optical Monitor
Port 1/3/7 is EMPTY: Can't Enable Optical Monitor
Port 1/3/8 is EMPTY: Can't Enable Optical Monitor

Does not seem to work. I also have an FS.com branded QSFP 40G but that shows up as

Port 1/2/1:  Type : 40GBASE-Passive Copper

It works fine, but trying to turn on optical monitor there:

Port 1/2/1 is EMPTY: Can't Enable Optical Monitor

I bought all these adapters used on ebay, so I have no idea for the FS.com what flavour they were bought as, which I assume boils down to what the eeprom says they are.

 

[tazplex]

Hi @tazplex that is strange especially since you say all that’s changed is the switch. Have you upgraded firmware on the APs recently or are they running the same since before you changed the switch?

Two things I can think of to help would be to put the old switch back in to confirm if you still get the problem. If you do then likely something with the switch like you say and we can continue troubleshooting here.

Other idea would be to keep the switch in place and disconnect some APs and test individually over the course of a few days to see which isn’t working correctly. Or put one of them (at a time) on a separate SSID connect to that specific AP for the day to test to see if you get the slowdowns? This will help you at least narrow down if one AP is misbehaving.

Thanks. Same firmware as before the changeover. I do plan on putting the old switch back in the rotation this weekend to narrow down if its an AP issue.

 

[TeleFragger]

1) make sure it's a poe model. There are some without.
2) sfp module? mine doesn't care about brand, so yes.
3) Follow the guides for the 6610, all licensing will be applied. You want the licensing for 10gbe.

Follow the guide, licensing is applied for "free"; you pay nothing. Good seller, imo. I got mine from them.

I won't comment on licensing for the fcx, since I don't know.

https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/page-143#post-248899

Bit more info in the link to the 648s.

This is the NON POE version..

Brocade IronPort FCX648S


thx.. status update...
i just bought this...
' Brocade ES4625M-FLF-08 2x 10GB XFP Switch Module
Brocade ES4625M-FLF-08 2x 10GB XFP Switch Module | eBay


To buy???

* 7 meter OM4 LC to SC Fiber Optic Patch Cable Multimode Duplex Aqua 50/125 LC SC 1M-15M

OM4 LC to SC Fiber Optic Patch Cable Multimode Duplex Aqua 50/125 LC SC 1M-15M | eBay

OM4 patch cords are commonly used in passive applications (GPON, FTTX and LANs), | OM4 Bandwidth Overfilled Launch At 850nm / 1500 Mhz/Km | Overfilled Launch At 1300nm / 500 Mhz/Km | Laser Launch At 850nm / 2000 Mhz/Km | Attenuation At 850nm / 3.5db/Km | Attenuation At 1300nm / 1.5db/Km.

www.ebay.com

Hope this is what I need next...
* Juniper SRX-XFP-10GE-SR 10GBase-SR 10G XFP optical transceiver
FOR Juniper SRX-XFP-10GE-SR 10GBase-SR 10G XFP optical transceiver w/60 days WRT | eBay

* Intel X520-DA2
DELL Intel X520-DA2 X520 10Gb SFP+ Network Adapter 0942V6 0XYT17 E10G42BTDAPG1P5

* Intel E10GSFPSR Ethernet SFP SR Optics Transceiver Module
Intel E10GSFPSR Ethernet SFP SR Optics Transceiver Module 101556666464 | eBay


If my to buy list looks good, I will go for it today... need to get all the parts in, then hit the website to walk through licensing, etc.. updating to latest firmware and see if i can pull out any straggler hairs i have left on top of my head!!!!1

 

[Gerhen]

Hi All,
I don't know how to troubleshoot this so hoping one of you might have a suggestion.

I have an IOT device that doesn't like being connected directly to my Brocade ICX 7250 (in router mode). Any port I try, the ethernet port light simply doesn't light up. It's not the cable either, I've tried swapping (see photo A)

Here's where it gets strange... If I connect an unmanaged switch to the Brocade and then plug the IOT device into the unmanaged switch -> it works and the IOT device will self-configure and get connected/become visible on the network (see photo B).

Any suggestions as to why the IOT device would only work through an intermediary switch and not when directly connected to the Brocade? I have two other IOT hubs, both from different manufacturers, that worked when directly connected to the Brocade so am stuck on how to troubleshoot this further so that I can directly connect.

Attaching a couple photos to show.

 

[sic0048]

Are you using the Brocade as the DHCP server?

If so, can you assign a static IP address to the misbehaving IOT device and see if it will connect directly to the switch then? I know the Brocade DHCP server does not function as an "authoritative" DHCP server. This is known to cause problems with some devices (usually IOT devices) which prevents them from being assigned a valid ip address. I don't know why the dumb switch would change this, but perhaps it changes the way the DHCP negotiation occurs just enough that the device gets assigned a valid DHCP address when connected this way and it doesn't get a valid address when connected directly to the switch. On the other hand, I could also be completely wrong in my diagnosis! It wouldn't be the first time....

It's easy enough to test out this theory however.

 

[Gerhen]

Are you using the Brocade as the DHCP server?
If so, can you assign a static IP address to the misbehaving IOT device and see if it will connect directly to the switch then? ...

No, DHCP is run upstream on the firewall. I do have static assignments working for other components, so I'll give a try here and report back.

 

[NablaSquaredG]

Any thoughts or suggestions are most appreciated. Thanks.

You mention that the switch doesn't exceed 1% usage.
Have you checked the traffic / packet per second count on individual ports?


FYI, there's a new firmware 10.0.10c, which has "FI-282109" closed (No access to Internet from WbC client, Condition: When WiFi client is moved from one AP to another AP)

 

[dbvader]

First of all, thanks to @fohdeesha and other contributors for their time and effort. Currently running a 6450 24P with POE access points and now have good wifi coverage around the house. Also dove into L3 switching which allowed me to keep the access points as dumb as possible while having the 6450 handle routing and acls. Yes, the DHCP server on the 6450 didn't work with my roku devices but the 6450 has a LAG with another (TP-Link) switch on the network which runs a DHCP server which works with the roku devices. Since the 6450 apparently has linux underlying it I'm curious whether its DHCP server is in linux or within the switch (fastiron?) OS. Not an immediate concern as DHCP is currently covered by the TP-Link switch.

Any suggestions as to why the IOT device would only work through an intermediary switch and not when directly connected to the Brocade?

Curious what

show interfaces ethernet <switchport for IOT device>

says when the IOT device is connected vs when the unmanaged switch is connected? Notably, the 'Configured speed' and 'Configured mdi mode'. Can't imagine speed or mdi would be set to anything other than auto but it's worth checking. Also curious to know what ethernet capabilities (speed, mdi/mdix) the IOT device has. Finally, is the unmanaged switch gigabit or otherwise?

 

[Andydude]

Just an update on my noisy fan replacement on my 6450. Basically even with the latest firmware etc when ever I turned it on the single fan would run at 100% and never go down no matter what. I tried 3 different fans and they all behaved the same way. I tried plugging all 3 in even though there is only one hole in the cover and they still never slowed down. Show Chassis would always say the fan was ok, on profile 1 and nothing was too hot but it would still run at 100%.

After trawling through this thread I found Fohdeesha commenting that on some switches the controller seems to be burnt out and it just runs at a full 12v all the time. Guess I was just unlucky with my £75 ebay purchase....

I bought a Delta EFB0412VHD fan from eBay but they sent me a Delta EFB0412MD instead. This was actually a blessing as it was very quiet but isn't very powerful and shows up as failed. I left that in and then bought a proper Delta EFB0412VHD and used an Evercool EC-DF004 Fan speed reduction cable (12v- 7.5v) to quieten it down and it's much, much better and is quiet and actually keeping it cool while also reporting that the fan is OK and not failed. Definitely recommend the reduction cable for anyone else with the same issue.

 

[Gerhen]

No, DHCP is run upstream on the firewall. I do have static assignments working for other components, so I'll give a try here and report back.

Update: the suggestion to assign a static IP seems to have fixed it i did not expect that.

I ran a show interface ethernet hoping to see if anything is strange for this device vs any of the other devices which worked without special treatment but nothing stood out. The other hubs negotiated 100Mbit connections with some running MDI and others MDIX but this was all auto-configured by the Brocade. Seems like the static assignment was the only thing preventing it from connecting.

 

[sic0048]

Update: the suggestion to assign a static IP seems to have fixed it i did not expect that.

I ran a show interface ethernet hoping to see if anything is strange for this device vs any of the other devices which worked without special treatment but nothing stood out. The other hubs negotiated 100Mbit connections with some running MDI and others MDIX but this was all auto-configured by the Brocade. Seems like the static assignment was the only thing preventing it from connecting.

Well even a broken clock is correct twice a day! While my initial diagnosis was incorrect (because you aren't using the Brocade as the DHCP server), I am glad my suggestion helped you find a solution in spite of this!

PS - I would double check to make sure that VLAN has access to the DHCP device. Perhaps the dumb switch was removing the VLAN tag and so the device was able to access the DHCP server because of that. Now that you have assigned a static address, it is bypassing that problem.

 

[Gerhen]

First of all, thanks to @fohdeesha and other contributors for their time and effort. Currently running a 6450 24P with POE access points and now have good wifi coverage around the house. Also dove into L3 switching which allowed me to keep the access points as dumb as possible while having the 6450 handle routing and acls. Yes, the DHCP server on the 6450 didn't work with my roku devices but the 6450 has a LAG with another (TP-Link) switch on the network which runs a DHCP server which works with the roku devices. Since the 6450 apparently has linux underlying it I'm curious whether its DHCP server is in linux or within the switch (fastiron?) OS. Not an immediate concern as DHCP is currently covered by the TP-Link switch.


Curious what

show interfaces ethernet <switchport for IOT device>

says when the IOT device is connected vs when the unmanaged switch is connected? Notably, the 'Configured speed' and 'Configured mdi mode'. Can't imagine speed or mdi would be set to anything other than auto but it's worth checking. Also curious to know what ethernet capabilities (speed, mdi/mdix) the IOT device has. Finally, is the unmanaged switch gigabit or otherwise?

Hi @dbvader I didn't try the show interface ethernet command when the hub was connected but did try after trying the static ip assignment (which fixed it). With the IOT hub now connected this is what came up on show interfaces ... I have other IOT hubs that also connected at 100Mbit with an MDI setting of either MDI or MDIX ... didn't have to do anything specific on the Brocade -> it auto-configured all of that so not sure what the problem is other than to say a static IP assignment resolved it (I would have never thought to try that thank you @sic0048 for the idea).

Configured speed auto, actual 100Mbit, configured duplex fdx, actual fdx
Configured mdi mode AUTO, actual MDIX

 

[ViciousXUSMC]

I went with RoachedCoach's mod and that worked for me, swap out three fans (got them from digikey) and add another on top of one of the heatsinks: https://forums.servethehome.com/ind...be-40gbe-switching.21107/page-159#post-256375

So the fans are here going to try the same mod, it seems all 4 fans come with... no connectors.

Id rather notcut and solder the old ones to the new fans and ruin them, so what connectors do I need? I am pretty sure its JST, but there are so many different ones, does anyone know exactly the specific one for the fans?

Also I was running my new Zyxel WBE660S AP's using the 10gb RJ45 SFP and after a couple of hours of working fine suddenly the AP started messing up.

I think it was honestly the SFP got too hot! Because when I touched it with my hand it was like OMFG hot, so I wonder if I will have the same issue with this switch after this specific fan mod.

Just to be safeI did already order a multi gig switch that has RJ45 and SFP ports so I do not have to use the RJ45 SFPs anymore.

Update: Just got the switch housing off, where does the thin larger fan go? Over the CPU heatsink? I thought it was supposed to sit above the SFP cage to keep it cool (as thats where I am having heat issues) but no space for it there.

 

[tazplex]

You mention that the switch doesn't exceed 1% usage.
Have you checked the traffic / packet per second count on individual ports?


FYI, there's a new firmware 10.0.10c, which has "FI-282109" closed (No access to Internet from WbC client, Condition: When WiFi client is moved from one AP to another AP)

Thanks for this. Is it something I can check via CLI? I'm just referring to the chart on the dashboard. I've disabled all the APs except for the R850 to see if I can narrow down the issue as well. Updated FW to 10.0.10c. Was getting dropped packets every 3-4 pings of the R850s IP, just sent a few hundred through all received.

 

[TonyArrr]

So the fans are here going to try the same mod, it seems all 4 fans come with... no connectors.

Id rather notcut and solder the old ones to the new fans and ruin them, so what connectors do I need? I am pretty sure it’s JST, but there are so many different ones, does anyone know exactly the specific one for the fans?

I used these for my 7250-24p fans: https://www.jaycar.com.au/3-pin-0-1...os=2&queryId=731e76cc052c83b4c1642bf7d7c14fae

Update: Just got the switch housing off, where does the thin larger fan go? Over the CPU heatsink? I thought it was supposed to sit above the SFP cage to keep it cool (as thats where I am having heat issues) but no space for it there.

The 60mm fan is for the ASIC, not the SFP cage: https://forums.servethehome.com/ind...erful-10gbe-40gbe-switching.21107/post-331607

I considered installing a usb fan at the bottom of my rack blowing air up past the SFP+ slots for mine, but ended up going all optical instead. Maybe a consideration for your situation? Wouldn’t need to be super powerful, just enough to move the air consistently

 

[ViciousXUSMC]

I used these for my 7250-24p fans: https://www.jaycar.com.au/3-pin-0-1...os=2&queryId=731e76cc052c83b4c1642bf7d7c14fae



The 60mm fan is for the ASIC, not the SFP cage: https://forums.servethehome.com/ind...erful-10gbe-40gbe-switching.21107/post-331607

I considered installing a usb fan at the bottom of my rack blowing air up past the SFP+ slots for mine, but ended up going all optical instead. Maybe a consideration for your situation? Wouldn’t need to be super powerful, just enough to move the air consistently

For some reason I thought this was a solution to make the switch quiet and keep the SFP's really cool, because when I tested the RJ45 SFP the other day they worked fine, even at multi gig speeds and my desktop has been hooked to one for weeks no issues, but soon as I added my access point using one it got super super hot and the AP messed up and I cant say if it was a fluke with the AP or not but I think the SFP got hot enough to cause issues.

My brain took this heat problem and just assumed that was the problem you were addressing, as everything else is running cool enough without an extra fan.

Adding an external fan is a solution, and I have a great fan for it already (the ones I use ontop of my AVR in the theater room) but I wanted something clean and intergrated so that way my rack is self contained and my dog or whatever cant mess it up.

Looks like I may just have to go with plan B, I got this switch and can just use it in conjunction with my ICX 6450 and just have the 7250 for (another) backup.

https://www.bhphotovideo.com/c/product/1790774-REG/qnap_qsw_m3216r_8s8t_us_1_2_rackmount_layer2_16_port_managed.html

 

[NablaSquaredG]

Thanks for this. Is it something I can check via CLI? I'm just referring to the chart on the dashboard.

show int <interface> should show all the info

 

[Andydude]

So the fans are here going to try the same mod, it seems all 4 fans come with... no connectors.

Id rather notcut and solder the old ones to the new fans and ruin them, so what connectors do I need? I am pretty sure its JST, but there are so many different ones, does anyone know exactly the specific one for the fans?

Also I was running my new Zyxel WBE660S AP's using the 10gb RJ45 SFP and after a couple of hours of working fine suddenly the AP started messing up.

I think it was honestly the SFP got too hot! Because when I touched it with my hand it was like OMFG hot, so I wonder if I will have the same issue with this switch after this specific fan mod.

Just to be safeI did already order a multi gig switch that has RJ45 and SFP ports so I do not have to use the RJ45 SFPs anymore.

Update: Just got the switch housing off, where does the thin larger fan go? Over the CPU heatsink? I thought it was supposed to sit above the SFP cage to keep it cool (as thats where I am having heat issues) but no space for it there.

I bought 5 Pcs x Shakmods 3 Pin Female Black Fan Power Connector + 15 Free Pins from UK eBay. You then just strip the wires and use long nose pliers or a crimping tool to put the ends on and make sure you slide them into the connector in the right order.

If you look at RoachedCoach mod links he has some pics, the thin larger fan goes over the CPU heatsink to keep that cool to cope with the reduced airflow from the quieter fans.

Not sure about keeping the RJ45 SPF's cool, my friend had the same issue with them getting too hot and shutting down and did the same as you and bought a separate switch.

 

[ViciousXUSMC]

I bought 5 Pcs x Shakmods 3 Pin Female Black Fan Power Connector + 15 Free Pins from UK eBay. You then just strip the wires and use long nose pliers or a crimping tool to put the ends on and make sure you slide them into the connector in the right order.

If you look at RoachedCoach mod links he has some pics, the thin larger fan goes over the CPU heatsink to keep that cool to cope with the reduced airflow from the quieter fans.

Not sure about keeping the RJ45 SPF's cool, my friend had the same issue with them getting too hot and shutting down and did the same as you and bought a separate switch.

Ok going to give these a try.

https://www.amazon.com/gp/product/B097MKLWCY/ref=ppx_yo_dt_b_asin_title_o01_s01?ie=UTF8&psc=1

I also maybe would/could/should test the RJ45 SFP's a bit more as like I said my desktop has been on one for weeks no issues, but the AP overhreated.

Could be a difference in brand and the chipset used, or simply and more likely because the AP is constatly sending traffic and my computer is not doing too much.

But I have 2 other brands I could test, but if I am going to return them for a refund I hate to open them.

 

[kpfleming]

I also maybe would/could/should test the RJ45 SFP's a bit more as like I said my desktop has been on one for weeks no issues, but the AP overhreated.

What's the difference in copper cable length between the desktop and the AP? More cable == more power consumed by the SFP+ module.