Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Scarlet]

I guess I could either block inbound on ve 2 or block outbound on ve 3. However, it probably makes more sense to block incoming on ve 2.

Do you need the management VLAN to be routed at all? If not you could just remove the ve and ip address from vlan 2 and the switch would not route anything to it.

If you have only one physical link from your pfsense to the switch you could always define vlans on the pfsense box and use a tagged port to access both vlans from pfsense.

 

[narapon]

Has anyone managed to 3d-print the rack ears for the ICX7150-C12P? Would be great to have the blueprints for those.

 

[pinkypie]

Do you need the management VLAN to be routed at all? If not you could just remove the ve and ip address from vlan 2 and the switch would not route anything to it.

Hi Scarlett, yes I need the Management VLAN to be routed because I need to be able to configure the cameras. It is management not only for switch but for the cameras as well. I want to keep routing off the pfSense box.

 

[etnoy]

Just wanted to jump in and say thanks, got my 6450-48p for €250 including shipping here in Europe. Works great, and the guide was very helpful for the initial setup.

Is there a command to shut down the switch before pulling power? Is that even needed? Didn't see that mentioned.

 

[infoMatt]

Is there a command to shut down the switch before pulling power?

Nope, just pull the power. All the software runs in a ramdisk/read only flash, so it's expected to be hard-powered off, no worries.

 

[LodeRunner]

If you made config changes you want to keep, then a “write mem” before pulling power is required. Otherwise, no. Pulling power is the standard way to revert a bad config change as almost all config changes are immediately applied.

 

[dswartz]

If you made config changes you want to keep, then a “write mem” before pulling power is required. Otherwise, no. Pulling power is the standard way to revert a bad config change as almost all config changes are immediately applied.

Back in the day when I had to manage cisco routers, calling the NOC and telling the night shift guy "yeah, i borked XXX, can you power cycle it? thanks bro!"

 

[jasonwc]

Has anyone tried the XQX2502 KAIAM QSFP+40G-LR4 Lite transceivers to connect an ICX6610 with a Mellanox MCX354A-FCBT? An STH user indicated that Cisco-coded generic 40G-LR optics worked with his Mellanox card (https://forums.servethehome.com/ind...-back-using-fs-qsfp-bd-40g.22302/#post-210165). In addition, the ConnectX-3 Pro firmware release notes (2.42.5000) list Cisco 40G-BiDi optics (QSFP-40G-SR-BD) as supported. FS lists power consumption as 3.5W for their generic version of this optic.

The Kaiam card appears to just be a low power version (2.3W) that is limited to 2km versus the standard 10km. An Ebay seller with good reviews is offering these transceivers for $10 each. Thus, I was wondering if anyone gave it a shot. There are also cheap LR Lite optics from AOI.

I'm looking to do a 20M run between the switch an my server. A 20M 8 fiber MPO Type B cable is $100 from a seller in China. FS charges $166 for a MTP®-12 (Female) to MTP®-12 (Female) OM4 Multimode Elite Trunk Cable, 12 Fibers, Type B, Plenum (OFNP), Magenta. They don't offer OM3 or 8-fiber trunk cables. A 20M SMF duplex cable would cost around $15.

 

[jasonwc]

I followed the advanced guide for the ICX6000 series to enable RSA public key authentication for SSH while disabling password authentication. While RSA key authentication worked, the switch continued to allow password authentication. However, I noticed that when logging in via Putty, the server listed "keyboard-interactive" as the authentication method when logging in via a password. So, I tried "ip ssh interactive-authentication no" (page 92 of Security Configuration manual). This worked. If I don't select a public key for authentication, Putty now complains that there's no acceptable authentication method, and it only lists publickey. Further testing indicates that you need BOTH "ip ssh interactive-authentiation no" and ip ssh password-authentication no" to disable password authentication.

TLDR - To disable password authentication, use "ip ssh interactive-authentiation no" AND "ip ssh password-authentication no"

 

[fohdeesha]

Has anyone tried the XQX2502 KAIAM QSFP+40G-LR4 Lite transceivers to connect an ICX6610 with a Mellanox MCX354A-FCBT? An STH user indicated that Cisco-coded generic 40G-LR optics worked with his Mellanox card (https://forums.servethehome.com/ind...-back-using-fs-qsfp-bd-40g.22302/#post-210165). In addition, the ConnectX-3 Pro firmware release notes (2.42.5000) list Cisco 40G-BiDi optics (QSFP-40G-SR-BD) as supported. FS lists power consumption as 3.5W for their generic version of this optic.

The Kaiam card appears to just be a low power version (2.3W) that is limited to 2km versus the standard 10km. An Ebay seller with good reviews is offering these transceivers for $10 each. Thus, I was wondering if anyone gave it a shot. There are also cheap LR Lite optics from AOI.

I'm looking to do a 20M run between the switch an my server. A 20M 8 fiber MPO Type B cable is $100 from a seller in China. FS charges $166 for a MTP®-12 (Female) to MTP®-12 (Female) OM4 Multimode Elite Trunk Cable, 12 Fibers, Type B, Plenum (OFNP), Magenta. They don't offer OM3 or 8-fiber trunk cables. A 20M SMF duplex cable would cost around $15.

---edited, thought at first these were standard MPO LR4 modules-----

edit: just saw in another ebay listing that shows the connector side, and it's just regular old LC, so they appear to be BiDi indeed, just bought four - that is an absolute steal. will test on the juniper, dell, brocade, mellanox 40g stuff i have lying around and report back. being able to run 40gbe between stack members over regular old cheap and plentiful single pair duplex fiber would be great

 

[fohdeesha]

I followed the advanced guide for the ICX6000 series to enable RSA public key authentication for SSH while disabling password authentication. While RSA key authentication worked, the switch continued to allow password authentication. However, I noticed that when logging in via Putty, the server listed "keyboard-interactive" as the authentication method when logging in via a password. So, I tried "ip ssh interactive-authentication no" (page 92 of Security Configuration manual). This worked. If I don't select a public key for authentication, Putty now complains that there's no acceptable authentication method, and it only lists publickey. Further testing indicates that you need BOTH "ip ssh interactive-authentiation no" and ip ssh password-authentication no" to disable password authentication.

TLDR - To disable password authentication, use "ip ssh interactive-authentiation no" AND "ip ssh password-authentication no"

Are you sure it was allowing successful logins with no keys loaded, when you had just ""ip ssh password-authentication no" enabled? According to the manual this should excplicitly disable any kind of password based login, it even warns that this in combination with key-authentication no will make the ssh server useless. I wonder if the password login you saw was one of the enable passwords or something? if you can confirm it lets full succesful logins with the current recommendations in the guide, I'll update the guide with the extra line

 

[LodeRunner]

---edited, thought at first these were standard MPO LR4 modules-----

edit: just saw in another ebay listing that shows the connector side, and it's just regular old LC, so they appear to be BiDi indeed, just bought four - that is an absolute steal. will test on the juniper, dell, brocade, mellanox 40g stuff i have lying around and report back. being able to run 40gbe between stack members over regular old cheap and plentiful single pair duplex fiber would be great

Since they're 2km rated, would one need an attenuator for sub 100m runs?

 

[fohdeesha]

Since they're 2km rated, would one need an attenuator for sub 100m runs?

nah, regular LR power is fine over short singlemode runs and is what makes up the majority of runs inside DCs etc. to top it off these are "LR lite" which isn't really a standard, but LR running at lower power, so it's limited to "only" 2km versus ~10 (they're still backwards compatible with regular LR transceivers)

 

[jasonwc]

---edited, thought at first these were standard MPO LR4 modules-----

edit: just saw in another ebay listing that shows the connector side, and it's just regular old LC, so they appear to be BiDi indeed, just bought four - that is an absolute steal. will test on the juniper, dell, brocade, mellanox 40g stuff i have lying around and report back. being able to run 40gbe between stack members over regular old cheap and plentiful single pair duplex fiber would be great

Yup, if these work, they are a crazy value. Thanks for testing! The cheapest OM3 MTP/MPO cable I could find at 20M was $100. Monoprice has duplex LC OS2 SMF for $5 at the same length!

On a slightly unrelated topic, there's an Ebay seller offering lots of 4 genuine Brocade 10G-SR optics for $19.88 with free shipping, so less than $5 each. This is about as cheap as any of the 10G-SR optics I've seen on Ebay, and you then get digital optical monitoring on the switch. I just picked up a lot. Here's the link for those interested:

Lot of 4 Brocade 10G-SFPP-SR 57-0000075-01 10GB 10GBASE-SR SFP module | eBay

Original Brocade 57-1000075-01 modules.

www.ebay.com

 

[jasonwc]

Are you sure it was allowing successful logins with no keys loaded, when you had just ""ip ssh password-authentication no" enabled? According to the manual this should excplicitly disable any kind of password based login, it even warns that this in combination with key-authentication no will make the ssh server useless. I wonder if the password login you saw was one of the enable passwords or something? if you can confirm it lets full succesful logins with the current recommendations in the guide, I'll update the guide with the extra line

Yup, I’m sure. It allowed password login with user root and successfully logged in. I could then get a configuration shell by running enable and then config t. I’m happy to share my config if you think I’ve done something wrong.

 

[pinkypie]

Is the brocade able to be set up via a Mac or through Windows on VMware? I have tried both with no success. I am using RJ45 from console port to an ethernet-->USB C adapter. Unable to find any connected devices via the ls -ltr /dev/*usb* command. Also, the COM port does not show up in Windows 10 in VMware. I checked show "Hidden Devices", still no luck.

I figure the failure is probably in the adapter. Any helpful suggestions would be appreciated?

 

[itronin]

my palm commited the post before I was ready -editing.

Is the brocade able to be set up via a Mac

Yes.

or through Windows on VMware? I have tried both with no success.

Yes.

I am using RJ45 from console port to an ethernet-->USB C adapter.

If that is really what you are using then it will never - ever - ever work.
The RJ45 coming off your 6450 is a serial port - pinned to cisco RJ45 console. It is NOT ethernet. Please take a look at the guide from the pinned first post.

what you will need will depend greatly on what you already have. I have an old school usb 2.0 to serial adapter and I use a usb-c to usb 3.0 adapter, cable that up and then connect a cisco rj45 cable (one end) to DB9 and connect the db9 to my old school usb serial adapter.

Unable to find any connected devices via the ls -ltr /dev/*usb* command. Also, the COM port does not show up in Windows 10 in VMware. I checked show "Hidden Devices", still no luck.

I figure the failure is probably in the adapter. Any helpful suggestions would be appreciated?

To access this through vmware you'll need to look up how to make com ports available from the host to the guest.

 

[pinkypie]

Good deal. To minimize any incompatibility issues, I ordered a Cable Matters RJ45 to USB-C.

Damn, that is a lot of connectors you are using. Three correct? USB-C to USB --> USB to DB9 --> DB9 to RJ45?

 

[itronin]

Good deal. To minimize any incompatibility issues, I ordered a Cable Matters RJ45 to USB-C.

good

Damn, that is a lot of connectors you are using. Three correct? USB-C to USB --> USB to DB9 --> DB9 to RJ45?

yeah. well I still have my first cisco console kit, rj45 to rj45 with modular DB9 and DB25 and modular DB25 modem pinout. that kit is about 25 years old. I think the rj45 to db9 is only 10 years old.

Not every serial console cable is cisco rj45 pinned and some consoles are still DB9, some rj45 but pinned differently, and still on occasion I have to hook up a modem for a while so I have remote/oob access.

you use what works.

 

[yobigd20]

NEED HELP!

I need help trying to figure out why I can't seem to pass more than 11.9Gb/sec across my switches.

Pic of my setup below.

Basic is that I have a pfsense router with a 40Gb mellanox card in it, connected to an ICX6650.

I then have the ICX6650 connected to two ICX6610's using 40G QSFP+ DAC Cable cables.
And then I have 2 ESXI v7 servers , each is using 10Gb connections to the switches for most vms and management. But each ESXI server also has a 40Gb Mellanox nic that is direct passthrough to an ubuntu 21.04 vm. Those ubuntu VMs don't have any other nic configured - just each having 1 dedicated 40Gb card.

When I test network throughput between these ubuntu VMs, the max speed I get is around 11.9Gbps total. I have tried multiple things. Even tried multiple iperf clients/servers just in case they ran out of cpu (iperf 3 is single threaded, and maxes out a single core). But using multiple parrallel iperf3 to get around that, and same result anyway.

As far as I can tell, everything is set up right. Everything on the switches shows 40Gb for the relevant ports. licenses are configured fine. I'm not seeing any bottlenecks anywhere that I can tell. I ran a second test, on one of the ubuntu vm's I removed out the 40Gb from its configuration and took 2 of the 10Gb nics on the ESXI 2, connected to 10Gb ports on the ICX6650 and set them up in dynamic lag lacp with active hash based load balancing. Ran the same test again with multiple iperf instances and got the *exact* same result, max of 11.9Gb/sec total transfer across the switches. I would have expected that to be upwards of 18-19Gbps.

Actually the first test I did was not even pass through. I had SR-IOV enabled but just created a 40Gb port group/vmswitch and had VMXNET3 direct path i/o and taht test yielded the same 11.9Gbps throughput. Then I did the LAG LACP test. Finalyl I assumed it was something in the VM layer and had reconfigured the nics to be pass through (not using SR-IOV) so its not even going through any VM layers at this point and STILL hit the same 11.9Gb/sec throughput. It's driving me nuts!

What am I overlooking that is preventing higher throughput? 40Gb nic to 40Gb switches to 40Gb nic I should be getting something like 30-35Gbps but I am not even close to that. I'm not cpu or memory bottlenecked, nothing I can see in the network path should be bottlenecking it. Should I not be expecting higher throughput across these switches??