Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

jasonwc

Member
Dec 31, 2018
39
14
8
I followed the advanced guide for the ICX6000 series to enable RSA public key authentication for SSH while disabling password authentication. While RSA key authentication worked, the switch continued to allow password authentication. However, I noticed that when logging in via Putty, the server listed "keyboard-interactive" as the authentication method when logging in via a password. So, I tried "ip ssh interactive-authentication no" (page 92 of Security Configuration manual). This worked. If I don't select a public key for authentication, Putty now complains that there's no acceptable authentication method, and it only lists publickey. Further testing indicates that you need BOTH "ip ssh interactive-authentiation no" and ip ssh password-authentication no" to disable password authentication.

TLDR - To disable password authentication, use "ip ssh interactive-authentiation no" AND "ip ssh password-authentication no"
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,668
113
31
fohdeesha.com
Has anyone tried the XQX2502 KAIAM QSFP+40G-LR4 Lite transceivers to connect an ICX6610 with a Mellanox MCX354A-FCBT? An STH user indicated that Cisco-coded generic 40G-LR optics worked with his Mellanox card (https://forums.servethehome.com/ind...-back-using-fs-qsfp-bd-40g.22302/#post-210165). In addition, the ConnectX-3 Pro firmware release notes (2.42.5000) list Cisco 40G-BiDi optics (QSFP-40G-SR-BD) as supported. FS lists power consumption as 3.5W for their generic version of this optic.

The Kaiam card appears to just be a low power version (2.3W) that is limited to 2km versus the standard 10km. An Ebay seller with good reviews is offering these transceivers for $10 each. Thus, I was wondering if anyone gave it a shot. There are also cheap LR Lite optics from AOI.

I'm looking to do a 20M run between the switch an my server. A 20M 8 fiber MPO Type B cable is $100 from a seller in China. FS charges $166 for a MTP®-12 (Female) to MTP®-12 (Female) OM4 Multimode Elite Trunk Cable, 12 Fibers, Type B, Plenum (OFNP), Magenta. They don't offer OM3 or 8-fiber trunk cables. A 20M SMF duplex cable would cost around $15.
---edited, thought at first these were standard MPO LR4 modules-----

edit: just saw in another ebay listing that shows the connector side, and it's just regular old LC, so they appear to be BiDi indeed, just bought four - that is an absolute steal. will test on the juniper, dell, brocade, mellanox 40g stuff i have lying around and report back. being able to run 40gbe between stack members over regular old cheap and plentiful single pair duplex fiber would be great
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,668
113
31
fohdeesha.com
I followed the advanced guide for the ICX6000 series to enable RSA public key authentication for SSH while disabling password authentication. While RSA key authentication worked, the switch continued to allow password authentication. However, I noticed that when logging in via Putty, the server listed "keyboard-interactive" as the authentication method when logging in via a password. So, I tried "ip ssh interactive-authentication no" (page 92 of Security Configuration manual). This worked. If I don't select a public key for authentication, Putty now complains that there's no acceptable authentication method, and it only lists publickey. Further testing indicates that you need BOTH "ip ssh interactive-authentiation no" and ip ssh password-authentication no" to disable password authentication.

TLDR - To disable password authentication, use "ip ssh interactive-authentiation no" AND "ip ssh password-authentication no"
Are you sure it was allowing successful logins with no keys loaded, when you had just ""ip ssh password-authentication no" enabled? According to the manual this should excplicitly disable any kind of password based login, it even warns that this in combination with key-authentication no will make the ssh server useless. I wonder if the password login you saw was one of the enable passwords or something? if you can confirm it lets full succesful logins with the current recommendations in the guide, I'll update the guide with the extra line
 

LodeRunner

Active Member
Apr 27, 2019
383
158
43
---edited, thought at first these were standard MPO LR4 modules-----

edit: just saw in another ebay listing that shows the connector side, and it's just regular old LC, so they appear to be BiDi indeed, just bought four - that is an absolute steal. will test on the juniper, dell, brocade, mellanox 40g stuff i have lying around and report back. being able to run 40gbe between stack members over regular old cheap and plentiful single pair duplex fiber would be great
Since they're 2km rated, would one need an attenuator for sub 100m runs?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,668
113
31
fohdeesha.com
Since they're 2km rated, would one need an attenuator for sub 100m runs?
nah, regular LR power is fine over short singlemode runs and is what makes up the majority of runs inside DCs etc. to top it off these are "LR lite" which isn't really a standard, but LR running at lower power, so it's limited to "only" 2km versus ~10 (they're still backwards compatible with regular LR transceivers)
 

jasonwc

Member
Dec 31, 2018
39
14
8
---edited, thought at first these were standard MPO LR4 modules-----

edit: just saw in another ebay listing that shows the connector side, and it's just regular old LC, so they appear to be BiDi indeed, just bought four - that is an absolute steal. will test on the juniper, dell, brocade, mellanox 40g stuff i have lying around and report back. being able to run 40gbe between stack members over regular old cheap and plentiful single pair duplex fiber would be great
Yup, if these work, they are a crazy value. Thanks for testing! The cheapest OM3 MTP/MPO cable I could find at 20M was $100. Monoprice has duplex LC OS2 SMF for $5 at the same length!

On a slightly unrelated topic, there's an Ebay seller offering lots of 4 genuine Brocade 10G-SR optics for $19.88 with free shipping, so less than $5 each. This is about as cheap as any of the 10G-SR optics I've seen on Ebay, and you then get digital optical monitoring on the switch. I just picked up a lot. Here's the link for those interested:

 
Last edited:

jasonwc

Member
Dec 31, 2018
39
14
8
Are you sure it was allowing successful logins with no keys loaded, when you had just ""ip ssh password-authentication no" enabled? According to the manual this should excplicitly disable any kind of password based login, it even warns that this in combination with key-authentication no will make the ssh server useless. I wonder if the password login you saw was one of the enable passwords or something? if you can confirm it lets full succesful logins with the current recommendations in the guide, I'll update the guide with the extra line
Yup, I’m sure. It allowed password login with user root and successfully logged in. I could then get a configuration shell by running enable and then config t. I’m happy to share my config if you think I’ve done something wrong.
 

pinkypie

New Member
Dec 2, 2021
19
3
3
Is the brocade able to be set up via a Mac or through Windows on VMware? I have tried both with no success. I am using RJ45 from console port to an ethernet-->USB C adapter. Unable to find any connected devices via the ls -ltr /dev/*usb* command. Also, the COM port does not show up in Windows 10 in VMware. I checked show "Hidden Devices", still no luck.

I figure the failure is probably in the adapter. Any helpful suggestions would be appreciated?
 

itronin

Well-Known Member
Nov 24, 2018
891
555
93
Denver, Colorado
my palm commited the post before I was ready -editing.

Is the brocade able to be set up via a Mac
Yes.

or through Windows on VMware? I have tried both with no success.
Yes.

I am using RJ45 from console port to an ethernet-->USB C adapter.
If that is really what you are using then it will never - ever - ever work.
The RJ45 coming off your 6450 is a serial port - pinned to cisco RJ45 console. It is NOT ethernet. Please take a look at the guide from the pinned first post.

what you will need will depend greatly on what you already have. I have an old school usb 2.0 to serial adapter and I use a usb-c to usb 3.0 adapter, cable that up and then connect a cisco rj45 cable (one end) to DB9 and connect the db9 to my old school usb serial adapter.

Unable to find any connected devices via the ls -ltr /dev/*usb* command. Also, the COM port does not show up in Windows 10 in VMware. I checked show "Hidden Devices", still no luck.

I figure the failure is probably in the adapter. Any helpful suggestions would be appreciated?
To access this through vmware you'll need to look up how to make com ports available from the host to the guest.
 
Last edited:
  • Like
Reactions: pinkypie

itronin

Well-Known Member
Nov 24, 2018
891
555
93
Denver, Colorado
Good deal. To minimize any incompatibility issues, I ordered a Cable Matters RJ45 to USB-C.
good

Damn, that is a lot of connectors you are using. Three correct? USB-C to USB --> USB to DB9 --> DB9 to RJ45?
yeah. well I still have my first cisco console kit, rj45 to rj45 with modular DB9 and DB25 and modular DB25 modem pinout. that kit is about 25 years old. I think the rj45 to db9 is only 10 years old.

Not every serial console cable is cisco rj45 pinned and some consoles are still DB9, some rj45 but pinned differently, and still on occasion I have to hook up a modem for a while so I have remote/oob access.

you use what works.
 
  • Like
Reactions: pinkypie

yobigd20

Member
Jul 8, 2016
45
30
18
NEED HELP!

I need help trying to figure out why I can't seem to pass more than 11.9Gb/sec across my switches.

Pic of my setup below.

Basic is that I have a pfsense router with a 40Gb mellanox card in it, connected to an ICX6650.

I then have the ICX6650 connected to two ICX6610's using 40G QSFP+ DAC Cable cables.
And then I have 2 ESXI v7 servers , each is using 10Gb connections to the switches for most vms and management. But each ESXI server also has a 40Gb Mellanox nic that is direct passthrough to an ubuntu 21.04 vm. Those ubuntu VMs don't have any other nic configured - just each having 1 dedicated 40Gb card.

When I test network throughput between these ubuntu VMs, the max speed I get is around 11.9Gbps total. I have tried multiple things. Even tried multiple iperf clients/servers just in case they ran out of cpu (iperf 3 is single threaded, and maxes out a single core). But using multiple parrallel iperf3 to get around that, and same result anyway.

As far as I can tell, everything is set up right. Everything on the switches shows 40Gb for the relevant ports. licenses are configured fine. I'm not seeing any bottlenecks anywhere that I can tell. I ran a second test, on one of the ubuntu vm's I removed out the 40Gb from its configuration and took 2 of the 10Gb nics on the ESXI 2, connected to 10Gb ports on the ICX6650 and set them up in dynamic lag lacp with active hash based load balancing. Ran the same test again with multiple iperf instances and got the *exact* same result, max of 11.9Gb/sec total transfer across the switches. I would have expected that to be upwards of 18-19Gbps.

Actually the first test I did was not even pass through. I had SR-IOV enabled but just created a 40Gb port group/vmswitch and had VMXNET3 direct path i/o and taht test yielded the same 11.9Gbps throughput. Then I did the LAG LACP test. Finalyl I assumed it was something in the VM layer and had reconfigured the nics to be pass through (not using SR-IOV) so its not even going through any VM layers at this point and STILL hit the same 11.9Gb/sec throughput. It's driving me nuts!

What am I overlooking that is preventing higher throughput? 40Gb nic to 40Gb switches to 40Gb nic I should be getting something like 30-35Gbps but I am not even close to that. I'm not cpu or memory bottlenecked, nothing I can see in the network path should be bottlenecking it. Should I not be expecting higher throughput across these switches??

40gb_help.png
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,668
113
31
fohdeesha.com
Connect those two servers together directly with a qsfp dac and try again, you're probably CPU/interrupt bound somewhere, could be marginal cabling/optics as well, check port statistics on every port in the path looking for discards/crc errors etc
 

yobigd20

Member
Jul 8, 2016
45
30
18
Connect those two servers together directly with a qsfp dac and try again, you're probably CPU/interrupt bound somewhere, could be marginal cabling/optics as well, check port statistics on every port in the path looking for discards/crc errors etc
ok so directly attached them and am getting 12.6Gbps. Swapped cable out for a different one and didn't make any difference. I'm wondering if these cards need different firmware or modes or something. Forgive me it's the first time I've dealt with any 40Gb nics. the specific cards that I bought off ebay were Mellanox MCX4131A-GCAT_C05 ConnectX-4 LX 50GbE PCIe Network Card Newest Firmware | eBay .

lshw -C net output:
firmware.JPG

What do you think? its not a cpu issue. when I do 2 VMs on the same ESXI server, and have SR-IOV configured and have them in hte same port group on the same vswitch and using VMXNET3 , I do get over 35Gbps, but in that scenario isn't it getting short circuited by the VMXNET3 driver and vSwitch and not actually hitting the physical nic and switch, right? I would think that rules out a few bottlenecks though.
 
Last edited:

yobigd20

Member
Jul 8, 2016
45
30
18
Connect those two servers together directly with a qsfp dac and try again, you're probably CPU/interrupt bound somewhere, could be marginal cabling/optics as well, check port statistics on every port in the path looking for discards/crc errors etc
any chance it's these cables? I tried 3 different ones. Are these junk? Do I need official Mellanox cables like MCP1700-B003E or Mc2210128-003 ?

cables1.png
 

juju

New Member
Sep 29, 2021
22
1
3
On my 7250, I see the following ip options for multicast:
  1. ip multicast
  2. ip multicast-routing
  3. ip multicast-nonstop-routing
what the difference?

Also, do I need to enable both multicast routing and snooping to get things like sonos and apple bonjour to work?
  1. ip multicast version 3
  2. router pim
 
Last edited:

jasonwc

Member
Dec 31, 2018
39
14
8
I'm still waiting for the $10 40G-LR4 Lite (uses duplex LC SMF) transceivers to arrive, but I just noticed that the ICX6XXX series administrate guide lists the Brocade optics that will work with the ICX6610 (p. 229). It specifically lists the 57-1000263-01 which is a 40G-LR4 transceiver with a 10km reach using duplex SMF fiber. So, it's a pretty good bet the generic modules will work as well. While the datasheet indicates that the SFP+ ports are limited to DAC, 10G-SR, and 10G-LR, the administrative manual also lists Brocade 10G-ER (40km) and 10G-ZR (80km), but it says the ZR will only run on the 1/3/8 port. I doubt anyone would need 40km or 80km optics, but it's nice to know the switch will accept almost anything. I assume BiDi optics will work fine as well.

As for the SSH issue, with only
Code:
ip ssh password-authentication no
I see the following using putty:

Code:
login as: jason
Keyboard-interactive authentication prompts from server:
| Password:
End of keyboard-interactive prompts from server
SSH@ICX6610-48p>
After adding
Code:
ip ssh interactive-authentication no
I get the expected result:

Code:
No supported authentication methods available (server sent: publickey)
 
Last edited:
  • Like
Reactions: fohdeesha

Vesalius

Active Member
Nov 25, 2019
199
149
43
On my 7250, I see the following ip options for multicast:
  1. ip multicast
  2. ip multicast-routing
  3. ip multicast-nonstop-routing
what the difference?

Also, do I need to enable both multicast routing and snooping to get things like sonos and apple bonjour to work?
  1. ip multicast version 3
  2. router pim
Read up in the manual (first link below). You can also search for those commands as well (use the second link below) and compare the difference. I had to turn igmp snooping and such off to get Homekit to work over my icx7150.

RUCKUS FastIron IP Multicast Configuration Guide, 08.0.95

RUCKUS FastIron Command Reference Guide, 08.0.95
 

juju

New Member
Sep 29, 2021
22
1
3
Read up in the manual (first link below). You can also search for those commands as well (use the second link below) and compare the difference. I had to turn igmp snooping and such off to get Homekit to work over my icx7150.

RUCKUS FastIron IP Multicast Configuration Guide, 08.0.95

RUCKUS FastIron Command Reference Guide, 08.0.95
@Vesalius Thx for those links. I have read through them a few times but still not exactly sure what do. Did you turn igmp snooping off or on? Isn't it off by default? I did the following but still don't have connectivity for my sonos: I am sure I have it completely wrong.

Code:
    # from main conf t:
    ip multicast version 3
    # then for each vlan
    vlan 110 ( home devices with sonos controller app)
    multicast version 3
    vlan 130 ( iot vlan with sonos devices)
    multicast version 3
Can you share how you have yours implemented?
 
Last edited: