Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
1. Setup switch according to instructions from first post in this thread
2. Get a client computer with a 1gb onboard NIC and a free PCIe (x4-x8) slot.
3. Add a NIC capable of 40gb (e.g. Mellanox CX3+).
4. Connect CAT5e (or better) cable to each of the 1gb ports, verify that physical connection is established by looking at leds both on switch and NIC. Bonus points: validate established connection in switch management and on computer OS. Assign IP address to computer NIC and ping switch management address.
5. Use a 40gb to 10gb adapter and a SFP+ DAC connecting NIC to 10gb ports. Verify physical connection by looking at leds both on switch and NIC. Same bonus points.
6. (on 6610) use QSFP+ to SFP+ breakout cable to test each of the 4x10gb stacking ports on the back. Connect each of the SFP+ ends to NIC (via adapter from last step) and verify physical connection.
7. Use a 40gb QSFP+ cable to connect 40gb port to 40gb NIC (remove adapter from last step). Verify connection.
Bonus step: connect CAT5e cable to switch management port and your onboard NIC, verify connectivity.
@NablaSquaredG and @fohdeesha Sorry to bother (and sorry for my ignorance), but you seem to know a lot about the Mellanox SX6036. I recently purchased one, enabled VPI through licensing changes, and have changed my 1/1 and 1/3 ports to ETH in an attempt to get my switch connected to my Brocade ICX 6610 (48p) switch through one of the "stack only" ports on which I have disabled the default stacking (stack trunk to no stack trunk) configuration.That's not correct.
I have connected the ICX6610 breakout ports to Mellanox SX6036 QSFP Port in Split Mode with a Single QSFP cable and they connected at 4x10G.
A QSFP cables is essentially just 4 SFP cables bundled into one
For the life of me, I cannot get the link to go up. I've also run the fae cable-stamping-unlock fdr10 and 40g_lr4, thinking it might be the cables.
For reference, I bought these cables (which I suspect are the problem):
10Gtek 40G QSFP+ DAC Cable - 40GBASE-CR4 Passive Direct Attach Copper Twinax QSFP Cable for Cisco QSFP-H40G-CU1M, Meraki MA-CBL-40G-1M, Supermicro, Mikrotik, Open Switch Devices, 1-Meter(3.3ft)
as well as:
40G QSFP+ AOC Cable - 40GBASE Ethernet Active Optical Cable, QDR, MMF for Cisco QSFP-H40G-AOC1M, Supermicro CBL-QSFP+AOC-1M and Open Switch, 1-Meter
Can either of you help me? I'm so sorry for my ignorance, again, but I've scoured this forum and everyone talks about how Mellanox hardware plays nice with 3rd party cables and I'm legitimately demoralized at this point. Oh, also, the port I'm using on the Brocade is 1/2/6 and I am confident that IT works, because when I used the cable from Brocade ICX 6610 48p # 1 1/2/6 to Brocade ICX 6610 48p #2 1/2/1 it worked fine (I got 40G up on the second Brocade). This suggests it might not be the cables (but still might be... as they are "for Cisco" - though I did get some "for Mellanox" and those also don't seem to work - but, actually, my cable is Orange for that one when it appears Mellanox is supposed to be teal/green, so maybe all I got was Cisco?).
As you can tell, I'm very new to all of this stuff. So I apologize if there's something super obvious that I'm missing.
I'm so sorry. I think the issue was that I didn't have both PSUs plugged in. Since I've done that, it seems to be working now.
Ok, i'm at a loss here. I have a 6610 as my core switch in the basement. I am trying to run a line out to an outbuilding using 10G port.
My remote switch is a 6450-24P. I brought into the basement for testing.
I am not able to get a link on any of the front 10G ports using actual fiber. I've used a number of different sfp+ adapters to a known good fiber connection. 5' cable, in the same room. Nothing i do gets a link. I've tried from front port on 6610 to front ports on 6450. No links. I grabbed a QSFP to Direct copper splitter and threw one of those in rear 6610 port and front 6450 port and got a link immediately.
Is there something i'm doing wrong here, or something i need to clear in the 6610 or 6450 to get them to talk over fiber? I'm not able to use a DAC to the outbuilding obviously. I am not trying to stack right now. Just port to port 10G. It would be wonderful if i could get this to work direct on the brocades so i can vlan my traffic properly. If i can't get this to work, i'll be adding dumb fiber converters on both ends, and thinking that's just a kludge.
My remote switch is a 6450-24P. I brought into the basement for testing.
I am not able to get a link on any of the front 10G ports using actual fiber. I've used a number of different sfp+ adapters to a known good fiber connection. 5' cable, in the same room. Nothing i do gets a link. I've tried from front port on 6610 to front ports on 6450. No links. I grabbed a QSFP to Direct copper splitter and threw one of those in rear 6610 port and front 6450 port and got a link immediately.
Is there something i'm doing wrong here, or something i need to clear in the 6610 or 6450 to get them to talk over fiber? I'm not able to use a DAC to the outbuilding obviously. I am not trying to stack right now. Just port to port 10G. It would be wonderful if i could get this to work direct on the brocades so i can vlan my traffic properly. If i can't get this to work, i'll be adding dumb fiber converters on both ends, and thinking that's just a kludge.
Alternate ideas i've had is get another 6610 and use one of the OM3 fibers to the outbuilding for a 40G stack connection, assuming i can find a QSFP+ to LC adapter for both ends. I read that 40G connection can go 100M, which is about what the distance to the outbuilding is.
What kind of fiber cable do you have? and what kind of SFP+ optics do you have? Want to throw up some pics?
Brocade pretty much accepts any optics so I doubt its that, there are single mode fiber optics and multi mode optics. Single mode fiber and multi mode fiber goes with each of the optics. You cannot have SMF and multi mode optics.
Single Mode Fiber (SMF) is _usually_ yellow unless you got a burial one which will be black then.
Multi Mode Fiber (MMF) will _usually_ be blue or pink depending on the generation.
I use SMF everywhere and DAC cables where needed. You could get a 10gtek 1M dac off amazon to see if the ports work. I assume you applied the licenses based on the guide.
It's MM OM3. Blue. I used the same cable to get links to the servers using SFP optics. Recently switched to DAC on that one that used fiber using the breakout on the 40G port on the back of the 6610. It was working fine before. I have several different kinds of optics including actual brocades. I don't think it's those. I'll go get PN's for optics.
Edit: Optics i have are Gigamon SFP-532, labeled 10G 850nm MM, SR
and Brocade 57-0000075-01
I believe both are MM, short range. the links in the house are under 50' and the link to the outbuilding is about 300'. There are 2 fiber cables to the building. Both are armored, LC-LC, one was cheap ebay and the other was EXPENSIVE custom ordered to try to fix whatever this link issue is.
I had 2 lagged 5' links to one server in the rack, and still have 1 50' link to my workstation using a mellanox cx-3 with same optics. everything other than switch to switch worked/works fine.
The switches have never had a good sfp link using fiber, only DAC. Short range or long. I repurposed one of the server cables to try this 5' away from the 6610.
Edit: Optics i have are Gigamon SFP-532, labeled 10G 850nm MM, SR
and Brocade 57-0000075-01
I believe both are MM, short range. the links in the house are under 50' and the link to the outbuilding is about 300'. There are 2 fiber cables to the building. Both are armored, LC-LC, one was cheap ebay and the other was EXPENSIVE custom ordered to try to fix whatever this link issue is.
I had 2 lagged 5' links to one server in the rack, and still have 1 50' link to my workstation using a mellanox cx-3 with same optics. everything other than switch to switch worked/works fine.
The switches have never had a good sfp link using fiber, only DAC. Short range or long. I repurposed one of the server cables to try this 5' away from the 6610.
Last edited:
If the DAC works its not the switches and its either your optics or your fiber cables. 40g port will require qsfp breakout (aoc or dac or MTP optic).
If you plug the optics sfp to sfp between the switches is that the only link? or are there other links like RJ45 because that might be loop block protection blocking the port.
if you show the ethernet device should show if its up or blocked or errored. I have never had a problem with any brocade switches (I own 6610 7250 7150 lots)
If you plug the optics sfp to sfp between the switches is that the only link? or are there other links like RJ45 because that might be loop block protection blocking the port.
if you show the ethernet device should show if its up or blocked or errored. I have never had a problem with any brocade switches (I own 6610 7250 7150
lots)
Code:
SSH@sw>show interfaces ethernet 1/2/1
10GigabitEthernet1/2/1 is up, line protocol is up
Port up for 65 day(s) 8 hour(s) 42 minute(s) 47 second(s)
Hardware is 10GigabitEthernet, address is xxxxxx
Configured speed optic-based, actual 10Gbit, configured duplex fdx, actual fdx
Configured mdi mode AUTO, actual MDI
Tagged member of 6 L2 VLANs, untagged in VLAN 1, port state is FORWARDING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
MACsec is Disabled
Openflow is Disabled, Openflow Hybrid mode is Disabled, Flow Control is config enabled, oper enabled, negotiation disabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
VLAN-Mapping is disabled
Member of active trunk ports 1/2/1,1/2/2,lg1, Lag Interface is lg1
Member of configured trunk ports 1/2/1,1/2/2,lg1, Lag Interface is lg1
No port name
IPG XGMII 96 bits-time
MTU 1500 bytes, encapsulation ethernet
MMU Mode is Store-and-forward
300 second input rate: 3335744 bits/sec, 320 packets/sec, 0.02% utilization
300 second output rate: 114640 bits/sec, 160 packets/sec, 0.00% utilization
2235483312 packets input, 2831962133325 bytes, 0 no buffer
Received 468270 broadcasts, 27515482 multicasts, 2207499560 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
865566395 packets output, 130093800022 bytes, 0 underruns
Transmitted 884661 broadcasts, 1808744 multicasts, 862872990 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled
Protected: No
MAC Port Security: Disabled
UC Egress queues:
Queue counters Queued packets Dropped Packets
0 859445449 0
1 0 0
2 0 0
3 0 0
4 2116533 0
5 89 0
6 0 0
7 1665509 0
MC Egress queues:
Queue counters Queued packets Dropped Packets
0 789089 0
1 837401 0
2 671440 0
3 40899 0I followed fohdeesha's guide. Is there something i need to configure for the front ports for switch to switch? They work fine connecting to devices like nics.
I have a SFP DAC coming in tomorrow and i'll try front port to front port. If that works, i'll dig into the optics. I don't know what would be the issue though. The sfp adapters work for nics to computers.
To answer your question, yes, the front port sfp to front port sfp was the only connection between. No loops.
To answer your question, yes, the front port sfp to front port sfp was the only connection between. No loops.
If the DAC works its not the switches and its either your optics or your fiber cables. 40g port will require qsfp breakout (aoc or dac or MTP optic).
If you plug the optics sfp to sfp between the switches is that the only link? or are there other links like RJ45 because that might be loop block protection blocking the port.
if you show the ethernet device should show if its up or blocked or errored. I have never had a problem with any brocade switches (I own 6610 7250 7150
Code:SSH@sw>show interfaces ethernet 1/2/1 10GigabitEthernet1/2/1 is up, line protocol is up Port up for 65 day(s) 8 hour(s) 42 minute(s) 47 second(s) Hardware is 10GigabitEthernet, address is xxxxxx Configured speed optic-based, actual 10Gbit, configured duplex fdx, actual fdx Configured mdi mode AUTO, actual MDI Tagged member of 6 L2 VLANs, untagged in VLAN 1, port state is FORWARDING BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled Link Error Dampening is Disabled STP configured to ON, priority is level0, mac-learning is enabled MACsec is Disabled Openflow is Disabled, Openflow Hybrid mode is Disabled, Flow Control is config enabled, oper enabled, negotiation disabled Mirror disabled, Monitor disabled Mac-notification is disabled VLAN-Mapping is disabled Member of active trunk ports 1/2/1,1/2/2,lg1, Lag Interface is lg1 Member of configured trunk ports 1/2/1,1/2/2,lg1, Lag Interface is lg1 No port name IPG XGMII 96 bits-time MTU 1500 bytes, encapsulation ethernet MMU Mode is Store-and-forward 300 second input rate: 3335744 bits/sec, 320 packets/sec, 0.02% utilization 300 second output rate: 114640 bits/sec, 160 packets/sec, 0.00% utilization 2235483312 packets input, 2831962133325 bytes, 0 no buffer Received 468270 broadcasts, 27515482 multicasts, 2207499560 unicasts 0 input errors, 0 CRC, 0 frame, 0 ignored 0 runts, 0 giants 865566395 packets output, 130093800022 bytes, 0 underruns Transmitted 884661 broadcasts, 1808744 multicasts, 862872990 unicasts 0 output errors, 0 collisions Relay Agent Information option: Disabled Protected: No MAC Port Security: Disabled UC Egress queues: Queue counters Queued packets Dropped Packets 0 859445449 0 1 0 0 2 0 0 3 0 0 4 2116533 0 5 89 0 6 0 0 7 1665509 0 MC Egress queues: Queue counters Queued packets Dropped Packets 0 789089 0 1 837401 0 2 671440 0 3 40899 0
Last edited:
Hey all!
I'm in a bit of a pickle - I recently purchased an ICX7250-24P to upgrade my trusty ICX6450-24P, however, after receiving the new switch, my smart ass decided that it was a good idea to do a firmware upgrade on the switch (S to R image too) without any form of serial access (I didn't have the Mini USB to DB9 cable at time time).
The upgrade seemingly went smoothly at the time and the system reloaded itself - but it never came back online. The fans stay at full speed and more importantly there is no serial output (I have since purchased the correct cable and verified that it is wired correctly).
My question - is there some sort of flash recovery/bootloader on these switches (similar to ABOOT on Arista) or do I need to get the flash programmer out? I suspect that in some way of form, I must have overwritten the bootloader on this thing.
Thanks!
I'm in a bit of a pickle - I recently purchased an ICX7250-24P to upgrade my trusty ICX6450-24P, however, after receiving the new switch, my smart ass decided that it was a good idea to do a firmware upgrade on the switch (S to R image too) without any form of serial access (I didn't have the Mini USB to DB9 cable at time time).
The upgrade seemingly went smoothly at the time and the system reloaded itself - but it never came back online. The fans stay at full speed and more importantly there is no serial output (I have since purchased the correct cable and verified that it is wired correctly).
My question - is there some sort of flash recovery/bootloader on these switches (similar to ABOOT on Arista) or do I need to get the flash programmer out? I suspect that in some way of form, I must have overwritten the bootloader on this thing.
Thanks!
The DAC works from front port to front port. I'm going to go swap in brocade optics and a known working cable. Report back in a minute.If the DAC works its not the switches and its either your optics or your fiber cables. 40g port will require qsfp breakout (aoc or dac or MTP optic).
If you plug the optics sfp to sfp between the switches is that the only link? or are there other links like RJ45 because that might be loop block protection blocking the port.
if you show the ethernet device should show if its up or blocked or errored. I have never had a problem with any brocade switches (I own 6610 7250 7150
Edit: Brocade optics, same ports, 4' cable, works fine. I am perplexed.
I guess i'll haul the 6450 back out there and try again.
Edit2: ARGH. It's working, same ports, same brocade optics, the 300' armored cable. Same ports as last time with no working.
Synopsis: Tech can be a b*tchy mistress.
Postscript: All this after i got frustrated and ordered a 6610-24P and a couple of bidi optics to just stack everything. I'm probably going to be more happy with that anyway. It's only money right??
Last edited: