Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[jasonwc]

---edited, thought at first these were standard MPO LR4 modules-----

edit: just saw in another ebay listing that shows the connector side, and it's just regular old LC, so they appear to be BiDi indeed, just bought four - that is an absolute steal. will test on the juniper, dell, brocade, mellanox 40g stuff i have lying around and report back. being able to run 40gbe between stack members over regular old cheap and plentiful single pair duplex fiber would be great

Yup, if these work, they are a crazy value. Thanks for testing! The cheapest OM3 MTP/MPO cable I could find at 20M was $100. Monoprice has duplex LC OS2 SMF for $5 at the same length!

On a slightly unrelated topic, there's an Ebay seller offering lots of 4 genuine Brocade 10G-SR optics for $19.88 with free shipping, so less than $5 each. This is about as cheap as any of the 10G-SR optics I've seen on Ebay, and you then get digital optical monitoring on the switch. I just picked up a lot. Here's the link for those interested:

Lot of 4 Brocade 10G-SFPP-SR 57-0000075-01 10GB 10GBASE-SR SFP module | eBay

Original Brocade 57-1000075-01 modules.

www.ebay.com

 

[jasonwc]

Are you sure it was allowing successful logins with no keys loaded, when you had just ""ip ssh password-authentication no" enabled? According to the manual this should excplicitly disable any kind of password based login, it even warns that this in combination with key-authentication no will make the ssh server useless. I wonder if the password login you saw was one of the enable passwords or something? if you can confirm it lets full succesful logins with the current recommendations in the guide, I'll update the guide with the extra line

Yup, I’m sure. It allowed password login with user root and successfully logged in. I could then get a configuration shell by running enable and then config t. I’m happy to share my config if you think I’ve done something wrong.

 

[pinkypie]

Is the brocade able to be set up via a Mac or through Windows on VMware? I have tried both with no success. I am using RJ45 from console port to an ethernet-->USB C adapter. Unable to find any connected devices via the ls -ltr /dev/*usb* command. Also, the COM port does not show up in Windows 10 in VMware. I checked show "Hidden Devices", still no luck.

I figure the failure is probably in the adapter. Any helpful suggestions would be appreciated?

 

[itronin]

my palm commited the post before I was ready -editing.

Is the brocade able to be set up via a Mac

Yes.

or through Windows on VMware? I have tried both with no success.

Yes.

I am using RJ45 from console port to an ethernet-->USB C adapter.

If that is really what you are using then it will never - ever - ever work.
The RJ45 coming off your 6450 is a serial port - pinned to cisco RJ45 console. It is NOT ethernet. Please take a look at the guide from the pinned first post.

what you will need will depend greatly on what you already have. I have an old school usb 2.0 to serial adapter and I use a usb-c to usb 3.0 adapter, cable that up and then connect a cisco rj45 cable (one end) to DB9 and connect the db9 to my old school usb serial adapter.

Unable to find any connected devices via the ls -ltr /dev/*usb* command. Also, the COM port does not show up in Windows 10 in VMware. I checked show "Hidden Devices", still no luck.

I figure the failure is probably in the adapter. Any helpful suggestions would be appreciated?

To access this through vmware you'll need to look up how to make com ports available from the host to the guest.

 

[pinkypie]

Good deal. To minimize any incompatibility issues, I ordered a Cable Matters RJ45 to USB-C.

Damn, that is a lot of connectors you are using. Three correct? USB-C to USB --> USB to DB9 --> DB9 to RJ45?

 

[itronin]

Good deal. To minimize any incompatibility issues, I ordered a Cable Matters RJ45 to USB-C.

good

Damn, that is a lot of connectors you are using. Three correct? USB-C to USB --> USB to DB9 --> DB9 to RJ45?

yeah. well I still have my first cisco console kit, rj45 to rj45 with modular DB9 and DB25 and modular DB25 modem pinout. that kit is about 25 years old. I think the rj45 to db9 is only 10 years old.

Not every serial console cable is cisco rj45 pinned and some consoles are still DB9, some rj45 but pinned differently, and still on occasion I have to hook up a modem for a while so I have remote/oob access.

you use what works.

 

[yobigd20]

NEED HELP!

I need help trying to figure out why I can't seem to pass more than 11.9Gb/sec across my switches.

Pic of my setup below.

Basic is that I have a pfsense router with a 40Gb mellanox card in it, connected to an ICX6650.

I then have the ICX6650 connected to two ICX6610's using 40G QSFP+ DAC Cable cables.
And then I have 2 ESXI v7 servers , each is using 10Gb connections to the switches for most vms and management. But each ESXI server also has a 40Gb Mellanox nic that is direct passthrough to an ubuntu 21.04 vm. Those ubuntu VMs don't have any other nic configured - just each having 1 dedicated 40Gb card.

When I test network throughput between these ubuntu VMs, the max speed I get is around 11.9Gbps total. I have tried multiple things. Even tried multiple iperf clients/servers just in case they ran out of cpu (iperf 3 is single threaded, and maxes out a single core). But using multiple parrallel iperf3 to get around that, and same result anyway.

As far as I can tell, everything is set up right. Everything on the switches shows 40Gb for the relevant ports. licenses are configured fine. I'm not seeing any bottlenecks anywhere that I can tell. I ran a second test, on one of the ubuntu vm's I removed out the 40Gb from its configuration and took 2 of the 10Gb nics on the ESXI 2, connected to 10Gb ports on the ICX6650 and set them up in dynamic lag lacp with active hash based load balancing. Ran the same test again with multiple iperf instances and got the *exact* same result, max of 11.9Gb/sec total transfer across the switches. I would have expected that to be upwards of 18-19Gbps.

Actually the first test I did was not even pass through. I had SR-IOV enabled but just created a 40Gb port group/vmswitch and had VMXNET3 direct path i/o and taht test yielded the same 11.9Gbps throughput. Then I did the LAG LACP test. Finalyl I assumed it was something in the VM layer and had reconfigured the nics to be pass through (not using SR-IOV) so its not even going through any VM layers at this point and STILL hit the same 11.9Gb/sec throughput. It's driving me nuts!

What am I overlooking that is preventing higher throughput? 40Gb nic to 40Gb switches to 40Gb nic I should be getting something like 30-35Gbps but I am not even close to that. I'm not cpu or memory bottlenecked, nothing I can see in the network path should be bottlenecking it. Should I not be expecting higher throughput across these switches??

 

[fohdeesha]

Connect those two servers together directly with a qsfp dac and try again, you're probably CPU/interrupt bound somewhere, could be marginal cabling/optics as well, check port statistics on every port in the path looking for discards/crc errors etc

 

[yobigd20]

Connect those two servers together directly with a qsfp dac and try again, you're probably CPU/interrupt bound somewhere, could be marginal cabling/optics as well, check port statistics on every port in the path looking for discards/crc errors etc

ok so directly attached them and am getting 12.6Gbps. Swapped cable out for a different one and didn't make any difference. I'm wondering if these cards need different firmware or modes or something. Forgive me it's the first time I've dealt with any 40Gb nics. the specific cards that I bought off ebay were Mellanox MCX4131A-GCAT_C05 ConnectX-4 LX 50GbE PCIe Network Card Newest Firmware | eBay .

lshw -C net output:


What do you think? its not a cpu issue. when I do 2 VMs on the same ESXI server, and have SR-IOV configured and have them in hte same port group on the same vswitch and using VMXNET3 , I do get over 35Gbps, but in that scenario isn't it getting short circuited by the VMXNET3 driver and vSwitch and not actually hitting the physical nic and switch, right? I would think that rules out a few bottlenecks though.

 

[yobigd20]

Connect those two servers together directly with a qsfp dac and try again, you're probably CPU/interrupt bound somewhere, could be marginal cabling/optics as well, check port statistics on every port in the path looking for discards/crc errors etc

any chance it's these cables? I tried 3 different ones. Are these junk? Do I need official Mellanox cables like MCP1700-B003E or Mc2210128-003 ?

 

[juju]

On my 7250, I see the following ip options for multicast:

1. ip multicast
2. ip multicast-routing
3. ip multicast-nonstop-routing

what the difference?

Also, do I need to enable both multicast routing and snooping to get things like sonos and apple bonjour to work?

1. ip multicast version 3
2. router pim

 

[jasonwc]

I'm still waiting for the $10 40G-LR4 Lite (uses duplex LC SMF) transceivers to arrive, but I just noticed that the ICX6XXX series administrate guide lists the Brocade optics that will work with the ICX6610 (p. 229). It specifically lists the 57-1000263-01 which is a 40G-LR4 transceiver with a 10km reach using duplex SMF fiber. So, it's a pretty good bet the generic modules will work as well. While the datasheet indicates that the SFP+ ports are limited to DAC, 10G-SR, and 10G-LR, the administrative manual also lists Brocade 10G-ER (40km) and 10G-ZR (80km), but it says the ZR will only run on the 1/3/8 port. I doubt anyone would need 40km or 80km optics, but it's nice to know the switch will accept almost anything. I assume BiDi optics will work fine as well.

As for the SSH issue, with only

ip ssh password-authentication no

I see the following using putty:

login as: jason
Keyboard-interactive authentication prompts from server:
| Password:
End of keyboard-interactive prompts from server
SSH@ICX6610-48p>

After adding

ip ssh interactive-authentication no

I get the expected result:

No supported authentication methods available (server sent: publickey)

 

[Vesalius]

On my 7250, I see the following ip options for multicast:

1. ip multicast
2. ip multicast-routing
3. ip multicast-nonstop-routing

what the difference?

Also, do I need to enable both multicast routing and snooping to get things like sonos and apple bonjour to work?

1. ip multicast version 3
2. router pim

Read up in the manual (first link below). You can also search for those commands as well (use the second link below) and compare the difference. I had to turn igmp snooping and such off to get Homekit to work over my icx7150.

RUCKUS FastIron IP Multicast Configuration Guide, 08.0.95

RUCKUS FastIron Command Reference Guide, 08.0.95

 

[Vesalius]

FYI, for anyone still on the 9000 series firmware, 9010 was released yesterday.

RUCKUS ICX FastIron 09.0.10 (GA) Software Release (.zip)

FastIron 09.0.10

 

[juju]

Read up in the manual (first link below). You can also search for those commands as well (use the second link below) and compare the difference. I had to turn igmp snooping and such off to get Homekit to work over my icx7150.

RUCKUS FastIron IP Multicast Configuration Guide, 08.0.95

RUCKUS FastIron Command Reference Guide, 08.0.95

@Vesalius Thx for those links. I have read through them a few times but still not exactly sure what do. Did you turn igmp snooping off or on? Isn't it off by default? I did the following but still don't have connectivity for my sonos: I am sure I have it completely wrong.

    # from main conf t:
    ip multicast version 3
    # then for each vlan
    vlan 110 ( home devices with sonos controller app)
    multicast version 3
    vlan 130 ( iot vlan with sonos devices)
    multicast version 3

Can you share how you have yours implemented?

 

[Vesalius]

@Vesalius Thx for those links. I have read through them a few times but still not exactly sure what do. Did you turn igmp snooping off or on? Isn't it off by default? I did the following but still dont have connectivity for my sonos: I am sure I have it competely wrong.

    # from main conf t:
    ip multicast version 3
    # then for each vlan
    vlan 110 ( home devices with sonos controller app)
    multicast version 3
    vlan 130 ( iot vlan wiht sonos devices)
    multicast version 3

Can you share how you have yours implemented?

I don't have sonos so can't help there. Do not remember what the default is now, but I do know when/if igmp snooping was on for my network/vlan homekit devices went unresponsive. I had to make sure igmp snooping was off. This was even when my homekit hubs and devices lived on the same vlan.

If you are trying to cross vlans for sonos control and or any multicast control then you will need something to do that for you. It gets complicated quickly, but what are you using to do the routing for your network? Things like Avahi among others come into play.

 

[jasonwc]

Will a 10G-SR or 10G-LR transceiver enable its laser if there is no link on an ICX6610? The Brocade docs indicate that there is a command (port-down-disable-laser) for the ICX7250 and ICX7450, but the docs state this isn't supported on the ICX6XXX switches (Commscope Technical Content Portal). I see digital optical monitoring output for the active ports but not for the inactive ports, which would suggest the laser is not on. The show media and show media validation commands do show the inactive transceivers. I removed the transceivers and they aren't warm, so there's no indication the laser is on. Just wondering if I should remove them. I plan to do two runs to each room for future needs and was planning to just keep it connected, as I would do with a copper port.

Port       Supported Vendor               Type
----------------------------------------------------------------------
1/3/1      Yes       BROCADE               Type  : 10GE SR 300m (SFP +)
1/3/2      Yes       BROCADE               Type  : 10GE SR 300m (SFP +)
1/3/3      Yes       BROCADE               Type  : 10GE SR 300m (SFP +)
1/3/4      Yes       BROCADE               Type  : 10GE SR 300m (SFP +)
1/3/5      Yes       BROCADE               Type  : 10GE LR 10km (SFP +)

SSH@ICX6610-48p(config)#show optic 1/3/1
 Port  Temperature   Tx Power     Rx Power       Tx Bias Current
+----+-----------+--------------+--------------+---------------+
1/3/1   32.0195 C  -002.1666 dBm -002.1197 dBm    5.934 mA
        Normal      Normal        Normal         Normal

SSH@ICX6610-48p(config)#show optic 1/3/2
 Port  Temperature   Tx Power     Rx Power       Tx Bias Current
+----+-----------+--------------+--------------+---------------+
1/3/2   32.7656 C  -002.5305 dBm -002.7140 dBm    5.796 mA
        Normal      Normal        Normal         Normal

SSH@ICX6610-48p(config)#show optic 1/3/3
 Port  Temperature   Tx Power     Rx Power       Tx Bias Current
+----+-----------+--------------+--------------+---------------+
1/3/3   34.4648 C  -002.2380 dBm -002.7794 dBm    5.492 mA
        Normal      Normal        Normal         Normal

SSH@ICX6610-48p(config)#show optic 1/3/4
SSH@ICX6610-48p(config)#show optic 1/3/5

 

[jasonwc]

@Vesalius Thx for those links. I have read through them a few times but still not exactly sure what do. Did you turn igmp snooping off or on? Isn't it off by default? I did the following but still don't have connectivity for my sonos: I am sure I have it completely wrong.

    # from main conf t:
    ip multicast version 3
    # then for each vlan
    vlan 110 ( home devices with sonos controller app)
    multicast version 3
    vlan 130 ( iot vlan with sonos devices)
    multicast version 3

Can you share how you have yours implemented?

I believe that you'll need Avahi to allow network discovery across VLANs for Homebridge. I set that up when I was trying to get Homebridge to work in my segmented network. See Homekit and Avahi.

 

[juju]

@jasonwc avahi on the 7250 ? or somewhere else? I have avahi setup on pfsense, but the layer 3 traffic will not hit pfsense for intervlan traffic, no? So pim or multicast routing setup on the 7250 has no impact on mdns traffic between vlans ?

 

[fohdeesha]

I'm still waiting for the $10 40G-LR4 Lite (uses duplex LC SMF) transceivers to arrive, but I just noticed that the ICX6XXX series administrate guide lists the Brocade optics that will work with the ICX6610 (p. 229). It specifically lists the 57-1000263-01 which is a 40G-LR4 transceiver with a 10km reach using duplex SMF fiber. So, it's a pretty good bet the generic modules will work as well. While the datasheet indicates that the SFP+ ports are limited to DAC, 10G-SR, and 10G-LR, the administrative manual also lists Brocade 10G-ER (40km) and 10G-ZR (80km), but it says the ZR will only run on the 1/3/8 port. I doubt anyone would need 40km or 80km optics, but it's nice to know the switch will accept almost anything. I assume BiDi optics will work fine as well.

As for the SSH issue, with only

ip ssh password-authentication no

I see the following using putty:

login as: jason
Keyboard-interactive authentication prompts from server:
| Password:
End of keyboard-interactive prompts from server
SSH@ICX6610-48p>

After adding

ip ssh interactive-authentication no

I get the expected result:

No supported authentication methods available (server sent: publickey)

thanks for verifying, guide updated: disable interactive auth for SSH configs · Fohdeesha/lab-docu@8bf0361