Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[CIR-Engineering]

Snagged an R750 last night for $70 on eBay! Seller later wrote he mispriced but would ship anyway. I feel kinda bad for him, I think they left off a zero. I might write back and let him cancel? I also need four and don’t know if I’ll want to pay for more 750’s.

 

[fohdeesha]

Snagged an R750 last night for $70 on eBay! Seller later wrote he mispriced but would ship anyway. I feel kinda bad for him, I think they left off a zero. I might write back and let him cancel? I also need four and don’t know if I’ll want to pay for more 750’s.

Damn, I thought the $200 employee pricing I got a couple years ago was good. They are incredible APs

 

[CIR-Engineering]

Damn, I thought the $200 employee pricing I got a couple years ago was good. They are incredible APs

Yeah, it has Unleashed already on it too... And it's new, not used. I'm wondering if just one could manage our new house instead of four 650's or something. I did have them run dual Ethernet right to the panty in the smack middle of the house and between the basement and second floor.

EDIT: They already shipped it!

Rebooted the switch and still no PoE errors. It's been running perfectly for twelve hours (still fingers crossed).

So hopefully (fingers crossed) I got a fully usable ICX7450-32ZP that may or may not have some problems that aren't relevant at $237, and stumbled on an R750 for $77. If this holds I'm going to Vegas!

I once bought two Lexicon LX7's off eBay that were sold as-is not working (audio dropping in and out on multiple channels) with grainy photos that didn't look great. Paid $1200 shipped for the pair. Got them here and repaired them in less than three hours, AND they cleaned up to 9.5/10. That was my best score on eBay ever and they are still in my system today. But if this 32ZP holds this will be a close second when combined with the R750.

Sometimes you win and sometimes you lose. This has been a good week (keeping fingers crossed!).

 

[CircuitSlinger]

I was one of the other winners on those ICX7450-32ZPs. Currently running the Diags just to make sure I don't run into any of the issues described by CIR-Engineering. I also ordered some extra SFP+ modules to replace the QSFP modules on the rear, hoping to replace my ICX6610-48P.

 

[CIR-Engineering]

I was one of the other winners on those ICX7450-32ZPs. Currently running the Diags just to make sure I don't run into any of the issues described by CIR-Engineering. I also ordered some extra SFP+ modules to replace the QSFP modules on the rear, hoping to replace my ICX6610-48P.

PLEASE! report back with your findings either way! I’m highly curious if mine is in fact “normal” or divergent.

I will probably also get one or two SFP+ modules at some point. I have zero need for 40 GbE but sometimes could benefit from a couple more 10 GbE ports.

How much did you pay for yours? Some went for $405, lucky dogs. I did focus on cleaner looking units with rack ears and two power supplies though. I bid on every unit, but after I scored I never bid higher than $390. I was hoping one would slip through cheaper.

 

[86turbodsl]

I had something very similar to this (search under my name in this thread) - 3 x ESXi hosts in a cluster - trying to use dual 10Gb ethernet ports (on a variety of different cards to the same switch - could not consistently get both connections to run - tried across various versions of ESXi (6.7, 7,8) and still the same - each time a restarted a host i was a fair chance of losing one of the 10GB connections and the only fix was a switch reset. Happened across multip 6610 as well.

In the end i gave up and use the 40GB ports for two of the hosts.

IN the other one i had to place two single port 10GB cards- dual port just would not do it

Craig

Yeah I played around with it a good long while and think it's down to the mellanox card. The 10g ports on the other server are copper rj45 and work fine lacp. I have an Intel card on order to get rid of the mellanox. Not very impressed with that card or Nvidia support. Nothing I did was able to get that card to work with the provided tools. HP branded connectx3pro. I gave up.

 

[CircuitSlinger]

PLEASE! report back with your findings either way! I’m highly curious if mine is in fact “normal” or divergent.

I will probably also get one or two SFP+ modules at some point. I have zero need for 40 GbE but sometimes could benefit from a couple more 10 GbE ports.

How much did you pay for yours? Some went for $405, lucky dogs. I did focus on cleaner looking units with rack ears and two power supplies though. I bid on every unit, but after I scored I never bid higher than $390. I was hoping one would slip through cheaper.

I paid $488 after shipping/tax. So far so good, self tests all passed. Little dissapointed in the power draw, I'm seeing around 120w from the wall when just sitting idle and messing around in the console, was hoping it was do a little better than the 6610. I had spare rack ears from a 7250 that should work, and I didn't care about dual PSU for my lab.

Edit
I forgot to put my Kil-A-Watt into Watts instead of Volts. Rechecking idle power draw numbers.
Boot: 68w
Idle: 105w

Still not the improvement I was hoping for.

Edit 2
ICX7250 rack ears work perfectly.

 

[CIR-Engineering]

I paid $488 after shipping/tax. So far so good, self tests all passed. Little dissapointed in the power draw, I'm seeing around 120w from the wall when just sitting idle and messing around in the console, was hoping it was do a little better than the 6610. I had spare rack ears from a 7250 that should work, and I didn't care about dual PSU for my lab.

Edit
I forgot to put my Kil-A-Watt into Watts instead of Volts. Rechecking idle power draw numbers.
Boot: 68w
Idle: 105w

Still not the improvement I was hoping for.

Edit 2
ICX7250 rack ears work perfectly.

Thanks. Which self tests did you run?

EDIT: Did you run dm diag and dm alt-diag?

 

[CircuitSlinger]

Thanks. Which self tests did you run?

dm diag and dm alt-diag

As mentioned by donedeal19 earlier in this thread, everything passed. So I updated, licensed, and through my APs onto it without issue. Just did the cutover from my 6610 and so far everything looks good.

 

[CIR-Engineering]

dm diag and dm alt-diag

As mentioned by donedeal19 earlier in this thread, everything passed. So I updated, licensed, and through my APs onto it without issue. Just did the cutover from my 6610 and so far everything looks good.

EDIT: I guess it actually doesn't matter what OS I have on there. I'll try running the diagnostics again now that I am on v9, but like fohdeesha advised, I'm not going to sweat it too much either way especially since Fastcrap gave me the nice partial refund. However, I will continue searching for another ICX7450-32ZP. I can't believe I have searched for one for like four years and then when 13 sell in three days I get a defective one. Oh well, it is running fine despite whatever is wrong with it.

I just hate these lines during boot. I don't like unexpected things in hardware

SOC unit 0 attached to PCI device BCM56548_A0
0:soc_do_init: soc_do_init: total cpu and arm cosq 0030 unexpected
Initializing the Port pair for the P2 board
u=0 p=50 Using SDK PHY driver for BCM84756/7/9
u=0 p=51 Using SDK PHY driver for BCM84756/7/9
Setting the CMIC registers for the link scan
SOC unit 1 attached to PCI device BCM56548_A0
1:soc_do_init: soc_do_init: total cpu and arm cosq 0030 unexpected
u=1 p=42 Using SDK PHY driver for BCM84756/7/9
u=1 p=43 Using SDK PHY driver for BCM84756/7/9
Setting the CMIC registers for the link scan

I wonder if it's the RAM. That is not too difficult to replace. I'm not going to try reballing either of those Broadcom chips!

 

[CIR-Engineering]

Is there a diagnostic command for just testing memory and just testing RAM one at a time so I can see where my error is.

 

[GvySmily]

Hi everyone,

I picked up an ICX6430-C12 and followed the very helpful Fohdeesha Docs guide to get set up. However, I'm now at the point where I need to get my VLANs working, which is where the guides stop.

I'm trying to trunk several VLANs from the ICX6430 to a Catalyst 3650, but when I physically make the connection, the entire network goes down.

I'm just trying to mirror/replace a cheap TPLink TL-SG105E managed switch - nothing too crazy. That said, I'm a CCNA student and this is for a home network - I'm not a professional.

I'm guessing this is a problem with STP or VLAN config or some other sort of looping issue, but I've reached the point where I want to ask for help instead of just spinning my wheels.

It's a little harder to find info for these switches than for Cisco stuff. Does anyone have any recommended reading? I'll continue looking through this thread in the meantime.

I've attached logs, configs, diagrams, etc in case anyone can to take a look at my issue specifically.

Thanks!
(Sorry if this is the wrong place to ask - please let me know!)

 

[TonyArrr]

It's a little harder to find info for these switches than for Cisco stuff. Does anyone have any recommended reading?

Just checking you’ve looked through fohdeesha’s download. I took me longer than I am willing to admit to notice all the manuals that were included with the firmware files, though I use 7xxx units, so dunno if there are guides for 6xxx that cover what you might need?

 

[EmnaX]

Having some internet speed issues that I think I have narrowed down to the Brocade ICX-6450-48P switch and maybe something called Virtual Output Queuing (VOQ)??

Essentially, I am getting slow speeds when accessing the internet from any of the 1 G Ethernet ports on the Brocade switch. I am running an internet speed test (speedtest.net) and only getting about 600 Mbps on the 1 G ports. The internet connection is 1500 Mbps download. If I run a speed test on a 10 G port from the Brocade switch, I get 1500 Mbps. So, I would expect to get somewhere around 940 Mbps on any of the 1 G Ethernet ports. I am not an IT guy and I've been reading about pause frames and virtual output queuing and trying to get this figured out but I think I need some help

More Details:
My ISP provides 1500 Mbps asymmetric connection to my cable modem with a 2.5 G Ethernet port. I am using a media converter that is setup for Ethernet at 1/2.5/5/10G and has a SFP+ port on the other side. I am using a 10 G DAC to connect that media convert to my router. Another 10 G DAC connects my router (pfSense) to the Brocade switch. From the switch, I have a couple servers using 10 G DACs and a bunch of other devices using the 1 G ports. When I run a speed test (speedtest.net) on any device connected to the Brocade's 1 G ports, I only get around 600 Mbps download. The servers that are connected to the Brocade switch on the 10 G ports get around 1500 Mbps. If I connect my laptop directly to my router, I am able to get about 940 Mbps (which is what I would expect since my laptop has a 1 G Ethernet port). I also have another 24 port UniFi switch that only has 1 G ports connected to my router and the devices over there are able to get about 940 Mbps as well.

So everything seems to work fine (cable modem, media converter, and router) except for this strange behavior across the Brocade switch. If I use a different media converter that only has a 1 G SFP connection and only negotiates a 1 G Ethernet link, then everything works as expected (about 940 Mbps on all devices). The only downside to this is that the servers on the 10 G links also only get about 940 Mbps instead of 1500 Mbps.

Is there something I can enable so the Brocade switch can mitigate the issue?

 

[CIR-Engineering]

Having some internet speed issues that I think I have narrowed down to the Brocade ICX-6450-48P switch and maybe something called Virtual Output Queuing (VOQ)??

Essentially, I am getting slow speeds when accessing the internet from any of the 1 G Ethernet ports on the Brocade switch. I am running an internet speed test (speedtest.net) and only getting about 600 Mbps on the 1 G ports. The internet connection is 1500 Mbps download. If I run a speed test on a 10 G port from the Brocade switch, I get 1500 Mbps. So, I would expect to get somewhere around 940 Mbps on any of the 1 G Ethernet ports. I am not an IT guy and I've been reading about pause frames and virtual output queuing and trying to get this figured out but I think I need some help

More Details:
My ISP provides 1500 Mbps asymmetric connection to my cable modem with a 2.5 G Ethernet port. I am using a media converter that is setup for Ethernet at 1/2.5/5/10G and has a SFP+ port on the other side. I am using a 10 G DAC to connect that media convert to my router. Another 10 G DAC connects my router (pfSense) to the Brocade switch. From the switch, I have a couple servers using 10 G DACs and a bunch of other devices using the 1 G ports. When I run a speed test (speedtest.net) on any device connected to the Brocade's 1 G ports, I only get around 600 Mbps download. The servers that are connected to the Brocade switch on the 10 G ports get around 1500 Mbps. If I connect my laptop directly to my router, I am able to get about 940 Mbps (which is what I would expect since my laptop has a 1 G Ethernet port). I also have another 24 port UniFi switch that only has 1 G ports connected to my router and the devices over there are able to get about 940 Mbps as well.

So everything seems to work fine (cable modem, media converter, and router) except for this strange behavior across the Brocade switch. If I use a different media converter that only has a 1 G SFP connection and only negotiates a 1 G Ethernet link, then everything works as expected (about 940 Mbps on all devices). The only downside to this is that the servers on the 10 G links also only get about 940 Mbps instead of 1500 Mbps.

Is there something I can enable so the Brocade switch can mitigate the issue?

This is odd to me because if you change media converters to the 1GbE everything works as expected right? Like you get nearly 1GbE on you 6450 ports?

I’d say the media converter is the issue but on the 10GbE ports you get expected full bandwidth with either converter?

Have you tried just an SFP+ to RJ45? I’ve done well with these in my 6450 in the past and currently have one in my PFS router for my modem:

https://www.amazon.com/ipolex-10GBase-T-Transceiver-SFP-10G-T-S-Supermicro/dp/B01M5LIUK5/

 

[BossHoss]

i've had my icx7250 working as just a basic switch for a few weeks since getting help here but when i came home this morning i could no longer get internet to anything i would plug into it, can not get access via ssh, or web gui. was able to get in using serial but no idea what im looking for or what happened. any im guessing i dont have to go through the whole update thing from scratch but no idea where to start to figure out what happened or how to begin to fix. any ideas?

 

[seany]

i've had my icx7250 working as just a basic switch for a few weeks since getting help here but when i came home this morning i could no longer get internet to anything i would plug into it, can not get access via ssh, or web gui. was able to get in using serial but no idea what im looking for or what happened. any im guessing i dont have to go through the whole update thing from scratch but no idea where to start to figure out what happened or how to begin to fix. any ideas?

If you do a `show run` you should see what config is loaded. Wild ass guess, it rebooted for some reason (power blip?) and you had for got to save things after you first configured things.

 

[BossHoss]

switchblade(config)#show interfaces ve 1
ve1 is up, line protocol is up
Hardware is Virtual Ethernet, address is 609c.9f4f.dd08 (bia 609c.9f4f.dd08)
No port name
Internet address is 10.0.0.252/24, IP MTU 1500 bytes, encapsulation ethernet
switchblade(config)#show run
Current configuration:
!
ver 08.0.95mT213
!
stack unit 1
module 1 icx7250-48p-poe-port-management-module
module 2 icx7250-sfp-plus-8port-80g-module
stack-port 1/2/1
stack-port 1/2/3
!
!
global-stp
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
spanning-tree
!
!
!
!
!
!
!
!
!
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
hostname switchblade
ip dhcp-client disable
ip dns server-address 10.0.0.1
ip route 0.0.0.0/0 10.0.0.1
!
no telnet server
username super password .....
!
!
!
!
--More--, next page: Space, next line: Return key, quit: Control-c

im not sure if maybe i should have put googles 75.75.75.75 instead of the 10.0.0.1 (my isp gateway) and thats the lack of internet or not also not sure why but i cant access the web gui still through http or https. cant seem to ssh in either. serial seems to be the only way i can access the switch.

 

[itronin]

im not sure if maybe i should have put googles 75.75.75.75 instead of the 10.0.0.1 (my isp gateway) and thats the lack of internet or not also not sure why but i cant access the web gui still through http or https. cant seem to ssh in either. serial seems to be the only way i can access the switch.

what you posted is missing the ve1 definition (ip address etc) however there may be more to your config (see the bottom "-More-, next page"...

you need to post your whole config. not just the first page. suggestion: use the "insert" and wrap it in a code or spoiler block. alternatively attach your config as a file.

If that is your entire config then @seany is correct that you forgot to "write mem" somewhere along the way...

 

[GvySmily]

Just checking you’ve looked through fohdeesha’s download. I took me longer than I am willing to admit to notice all the manuals that were included with the firmware files, though I use 7xxx units, so dunno if there are guides for 6xxx that cover what you might need?

Thanks for the tip! Between reading through docs in the zip (I learned a lot.. gave me a bunch more to-dos), and making a few tweaks from this presentation:

http://www.netadmin.us/docs/Brocade-Cisco-spanningtree.pdf

I was able to get things working!

Specifically, I made these changes on the Brocade side:

• Changed default vlan id: default-vlan-id 2
• Created vlan 1 and tagged ethe 1/2/1 (my trunk port)
• Set spanning-tree 802-1w for all vlans
• Added spanning-tree 802-1w admin-pt2pt-mac to my trunk port

Lastly, perhaps the only change that actually mattered:

• Instead of freaking out and unplugging the ethernet cable when the network went down, I waited. Everything came back up after ~30 seconds.

So... I'm not sure if the config changes were necessary. I can revert and test later later - my girlfriend is trying to run Final Fantasy XIV dungeons and I feel bad for interrupting her over and over.

Still, seems weird for the network to go down for 30 seconds on connection. But I can figure that out at my leisure.

ps - interesting music video in the zip too.