Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[BoGs]

If the DAC works its not the switches and its either your optics or your fiber cables. 40g port will require qsfp breakout (aoc or dac or MTP optic).

If you plug the optics sfp to sfp between the switches is that the only link? or are there other links like RJ45 because that might be loop block protection blocking the port.

if you show the ethernet device should show if its up or blocked or errored. I have never had a problem with any brocade switches (I own 6610 7250 7150 lots)

SSH@sw>show interfaces ethernet 1/2/1
10GigabitEthernet1/2/1 is up, line protocol is up
  Port up for 65 day(s) 8 hour(s) 42 minute(s) 47 second(s)
  Hardware is 10GigabitEthernet, address is xxxxxx
  Configured speed optic-based, actual 10Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual MDI
  Tagged member of 6 L2 VLANs, untagged in VLAN 1, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  MACsec is Disabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  VLAN-Mapping is disabled
  Member of active trunk ports 1/2/1,1/2/2,lg1, Lag Interface is lg1
  Member of configured trunk ports 1/2/1,1/2/2,lg1, Lag Interface is lg1
  No port name
  IPG XGMII 96 bits-time
  MTU 1500 bytes, encapsulation ethernet
  MMU Mode is Store-and-forward
  300 second input rate: 3335744 bits/sec, 320 packets/sec, 0.02% utilization
  300 second output rate: 114640 bits/sec, 160 packets/sec, 0.00% utilization
  2235483312 packets input, 2831962133325 bytes, 0 no buffer
  Received 468270 broadcasts, 27515482 multicasts, 2207499560 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  865566395 packets output, 130093800022 bytes, 0 underruns
  Transmitted 884661 broadcasts, 1808744 multicasts, 862872990 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled
  Protected: No
  MAC Port Security: Disabled

UC Egress queues:
Queue counters    Queued packets    Dropped Packets
         0           859445449                   0
         1                   0                   0
         2                   0                   0
         3                   0                   0
         4             2116533                   0
         5                  89                   0
         6                   0                   0
         7             1665509                   0


MC Egress queues:
Queue counters    Queued packets    Dropped Packets
         0              789089                   0
         1              837401                   0
         2              671440                   0
         3               40899                   0

 

[Kahooli]

It's MM OM3. Blue. I used the same cable to get links to the servers using SFP optics. Recently switched to DAC on that one that used fiber using the breakout on the 40G port on the back of the 6610. It was working fine before. I have several different kinds of optics including actual brocades. I don't think it's those. I'll go get PN's for optics.

Edit: Optics i have are Gigamon SFP-532, labeled 10G 850nm MM, SR
and Brocade 57-0000075-01

I believe both are MM, short range. the links in the house are under 50' and the link to the outbuilding is about 300'. There are 2 fiber cables to the building. Both are armored, LC-LC, one was cheap ebay and the other was EXPENSIVE custom ordered to try to fix whatever this link issue is.

I had 2 lagged 5' links to one server in the rack, and still have 1 50' link to my workstation using a mellanox cx-3 with same optics. everything other than switch to switch worked/works fine.

The switches have never had a good sfp link using fiber, only DAC. Short range or long. I repurposed one of the server cables to try this 5' away from the 6610.

Dumb question: are the front ports configured properly?

 

[86turbodsl]

I followed fohdeesha's guide. Is there something i need to configure for the front ports for switch to switch? They work fine connecting to devices like nics.

 

[86turbodsl]

I have a SFP DAC coming in tomorrow and i'll try front port to front port. If that works, i'll dig into the optics. I don't know what would be the issue though. The sfp adapters work for nics to computers.

To answer your question, yes, the front port sfp to front port sfp was the only connection between. No loops.

If the DAC works its not the switches and its either your optics or your fiber cables. 40g port will require qsfp breakout (aoc or dac or MTP optic).

If you plug the optics sfp to sfp between the switches is that the only link? or are there other links like RJ45 because that might be loop block protection blocking the port.

if you show the ethernet device should show if its up or blocked or errored. I have never had a problem with any brocade switches (I own 6610 7250 7150 lots)

SSH@sw>show interfaces ethernet 1/2/1
10GigabitEthernet1/2/1 is up, line protocol is up
  Port up for 65 day(s) 8 hour(s) 42 minute(s) 47 second(s)
  Hardware is 10GigabitEthernet, address is xxxxxx
  Configured speed optic-based, actual 10Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual MDI
  Tagged member of 6 L2 VLANs, untagged in VLAN 1, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  MACsec is Disabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  VLAN-Mapping is disabled
  Member of active trunk ports 1/2/1,1/2/2,lg1, Lag Interface is lg1
  Member of configured trunk ports 1/2/1,1/2/2,lg1, Lag Interface is lg1
  No port name
  IPG XGMII 96 bits-time
  MTU 1500 bytes, encapsulation ethernet
  MMU Mode is Store-and-forward
  300 second input rate: 3335744 bits/sec, 320 packets/sec, 0.02% utilization
  300 second output rate: 114640 bits/sec, 160 packets/sec, 0.00% utilization
  2235483312 packets input, 2831962133325 bytes, 0 no buffer
  Received 468270 broadcasts, 27515482 multicasts, 2207499560 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  865566395 packets output, 130093800022 bytes, 0 underruns
  Transmitted 884661 broadcasts, 1808744 multicasts, 862872990 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled
  Protected: No
  MAC Port Security: Disabled

UC Egress queues:
Queue counters    Queued packets    Dropped Packets
         0           859445449                   0
         1                   0                   0
         2                   0                   0
         3                   0                   0
         4             2116533                   0
         5                  89                   0
         6                   0                   0
         7             1665509                   0


MC Egress queues:
Queue counters    Queued packets    Dropped Packets
         0              789089                   0
         1              837401                   0
         2              671440                   0
         3               40899                   0

 

[notgeneric]

Hey all!

I'm in a bit of a pickle - I recently purchased an ICX7250-24P to upgrade my trusty ICX6450-24P, however, after receiving the new switch, my smart ass decided that it was a good idea to do a firmware upgrade on the switch (S to R image too) without any form of serial access (I didn't have the Mini USB to DB9 cable at time time).

The upgrade seemingly went smoothly at the time and the system reloaded itself - but it never came back online. The fans stay at full speed and more importantly there is no serial output (I have since purchased the correct cable and verified that it is wired correctly).

My question - is there some sort of flash recovery/bootloader on these switches (similar to ABOOT on Arista) or do I need to get the flash programmer out? I suspect that in some way of form, I must have overwritten the bootloader on this thing.

Thanks!

 

[i386]

In the first post is a link to an extensive documentation, if I remeber it correctly there is also a section about dealing with failed updates/recovery procedures.

 

[86turbodsl]

If the DAC works its not the switches and its either your optics or your fiber cables. 40g port will require qsfp breakout (aoc or dac or MTP optic).

If you plug the optics sfp to sfp between the switches is that the only link? or are there other links like RJ45 because that might be loop block protection blocking the port.

if you show the ethernet device should show if its up or blocked or errored. I have never had a problem with any brocade switches (I own 6610 7250 7150 lots)

The DAC works from front port to front port. I'm going to go swap in brocade optics and a known working cable. Report back in a minute.

Edit: Brocade optics, same ports, 4' cable, works fine. I am perplexed.

I guess i'll haul the 6450 back out there and try again.

Edit2: ARGH. It's working, same ports, same brocade optics, the 300' armored cable. Same ports as last time with no working.

Synopsis: Tech can be a b*tchy mistress.

Postscript: All this after i got frustrated and ordered a 6610-24P and a couple of bidi optics to just stack everything. I'm probably going to be more happy with that anyway. It's only money right??

 

[BoGs]

> Postscript: All this after i got frustrated and ordered a 6610-24P and a couple of bidi optics to just stack everything. I'm probably going to be more happy with that anyway. It's only money right??

6610s eat a lot of power and are loud as long as that is ok you will love it. I am debating if I am going to replace my 7250 with 7150-48zp just too loud.

 

[86turbodsl]

The 6610 in the basement is livable, i don't spend much time down there. The other one is in a noisy workshop environment and i could care less. The power is something i will live with probably. Having both switches in a single management environment is very appealing. The backup for my system will be out there also, so having 40Gb connection is also appealing.

 

[scottshotgg]

Has anyone ever used a RADIUS server to do MAC auth on their Brocade switch? I have an ICX7250-48P and am trying to get MAC auth working, however for some reason I seem to have what I can only call "client VLAN flip-flop" where the switch continuously calls my RADIUS server and the client flip-flops back and forth from the VLAN the RADIUS server sent back to the default VLAN set. Its very odd and I can provide some more context if anyone has any ideas that would be great

 

[Cheburashka]

Hello everyone, thank you for this fantastic thread.

I've been exploring getting 10GB connectivity between my QNAP and desktop. I purchased the ICX7150-C12 and I have Mellanox ConnectX3 cards in my desktop and QNAP. I am unsure where in the Brocade/Ruckus is the ability to set a larger MTU than 1500. I have browsed through the web gui but I haven't been able to find a spot for it.

I've tried to find this answer in many posts here but haven't found much information on it.


Edit: I was checking the CLI mode and I entered interface 1/3/1 (10gb). I checked MTU and Jumbo and was able to enable Jumbo but it looks like it is switch wide?

Doing a show run, I see jumbo listed but it is not assigned to a port or vlan.

 

[kpfleming]

Edit: I was checking the CLI mode and I entered interface 1/3/1 (10gb). I checked MTU and Jumbo and was able to enable Jumbo but it looks like it is switch wide?

Yes, that is correct. It's a switch-level configuration setting. It has to be, because otherwise a port with 'jumbo' enabled would be unable to practically communicate with a port that had 'jumbo' disabled. In fact that's still an issue even with the switch-level setting, because if any node on the LAN isn't configured for the same MTU as the other nodes, there will be communication problems.

In other words: either all of the nodes must have the increased MTU, or none of them do. Making effective use of jumbo frames can be tricky.

 

[fohdeesha]

Hello everyone, thank you for this fantastic thread.

I've been exploring getting 10GB connectivity between my QNAP and desktop. I purchased the ICX7150-C12 and I have Mellanox ConnectX3 cards in my desktop and QNAP. I am unsure where in the Brocade/Ruckus is the ability to set a larger MTU than 1500. I have browsed through the web gui but I haven't been able to find a spot for it.

I've tried to find this answer in many posts here but haven't found much information on it.


Edit: I was checking the CLI mode and I entered interface 1/3/1 (10gb). I checked MTU and Jumbo and was able to enable Jumbo but it looks like it is switch wide?

Doing a show run, I see jumbo listed but it is not assigned to a port or vlan.

Why do you want to run jumbo frames in 2024? They're almost entirely unnecessary for 10gb, have you done any throughput testing etc showing that you need them for some reason?

 

[BBergle]

Hello! I just got a 7250-24P from ebay and after I connect to the console I only get these characters. ����
I have hit b and enter multiple times but nothing shows up but those question mark characters. I know my cable is good since I have another switch I just tested it on and it works fine. I reset the switch to factory settings and nothing has changed. Is the switch fried? Thank you

 

[fohdeesha]

Hello! I just got a 7250-24P from ebay and after I connect to the console I only get these characters. ����
I have hit b and enter multiple times but nothing shows up but those question mark characters. I know my cable is good since I have another switch I just tested it on and it works fine. I reset the switch to factory settings and nothing has changed. Is the switch fried? Thank you

are you using 9600 - 8 - n - 1 ? you could try a different baudrate on the offchance the last user changed it n the bootloader, 19200, 38400, 115200

 

[BBergle]

are you using 9600 - 8 - n - 1 ? you could try a different baudrate on the offchance the last user changed it n the bootloader, 19200, 38400, 115200

Yes I have tried 9600 and 11520

 

[fohdeesha]

Yes I have tried 9600 and 11520

did you try 115200 ?

 

[BBergle]

did you try 115200 ?

Sorry, yes that's what I meant

 

[BackupProphet]

Sometimes the issue is bad serial cable. Some of the Arista and Celestica switches I bought came with their own serial cables, and they work much better than old one I had.

 

[klui]

Then you have the situation with Mellanox. ESPCommunity