Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[kpfleming]

10Gtek optics coded as Brocade also work (I'm using their 10G SM BiDi optics).

 

[liv3010m]

Hi guys!
To those who own a 6610 or had access to one, do you know if low profile fans (those that are lesser than 15mm in depth) will fit in its 1U chassis without having to implement those franken mods where the coolers are running from outside of the 1U space? Maybe not on the 48P / PoE variants, but on the simpler ones?

Thanks!

 

[RobstarUSA]

@RobstarUSA - Did you ever get around to trying to setup mDNS/Bonjour ? I am interested in the outcome of this as well. It's quite surprising there isn't a good guide here on how to set this up properly on brocade switches. (working reliably, I mean ). Just like @nickf1227 , I can't seem to get it to work. I am however, much less knowledgeable in the space than he his. I have an ICX7250 which is working fine except this issue of mDNS/Bonjour traffic on my network. I won't mind switching to another brand just to get it to work .

From the Cisco DNA service for Bonjour documentation :



Is there an equivalent for Brocade/Rukus switches?

I have some backend work to do before I do this & I don't know when I'm going to get the time.

My test is going to be something like Lan -> Printers for printer discovery, however I need to move the default gateway off of my OPNSense box (opnsense doesn't support pim) to the brocade (and make an SVI) & put in a transit interface between the brocade & upstream OPNSense box. For routing to work I have to enable a routing protocol for v4/v6 between opnsense and the brocade (probably will do OSPF). This whole shebang is being held-up while I wait for opnsense to fix bugs in their unbound aliases which are not importing properlty for me, and the he.net tunnels to auto start. I've got 2 tickets open on opnsense forums/github and I don't want to complicate my opnsense box prior to those getting fixed.

So it may be a while....figure a few months before I get to try this.

 

[clcorbin]

either of these work fine

<snip>

Brocade 10G-SFPP-T Compatible 10GBASE-T SFP+ Copper RJ-45 30m Transceiver Module

I'm at a lose. The front 1/3/x SFP+ ports work perfectly with fiber transceivers (Brocade and others) at 10Gb. I've tried three different 10Gb copper transceivers, including the one above and they never power up. Always says network cable unplugged.

Could I have something configured wrong that would disable copper transceivers? Or is it the Intel X540 on the other end of the cable? If I plug the X540 into one of the standard front ports (1/1/x), everything works perfectly fine. Just at 1Gb/s...

Push comes to shove, I could probably run another fiber run, but I know the copper cables work perfectly fine at 10GB from my old Aruba S3500s.

Of course, if I DID run another fiber cable, then I could use one of my 40Gb cards instead of the X540...

Either way, any ideas are appreciated!

 

[fohdeesha]

I'm at a lose. The front 1/3/x SFP+ ports work perfectly with fiber transceivers (Brocade and others) at 10Gb. I've tried three different 10Gb copper transceivers, including the one above and they never power up. Always says network cable unplugged.

Could I have something configured wrong that would disable copper transceivers? Or is it the Intel X540 on the other end of the cable? If I plug the X540 into one of the standard front ports (1/1/x), everything works perfectly fine. Just at 1Gb/s...

Push comes to shove, I could probably run another fiber run, but I know the copper cables work perfectly fine at 10GB from my old Aruba S3500s.

Of course, if I DID run another fiber cable, then I could use one of my 40Gb cards instead of the X540...

Either way, any ideas are appreciated!

Edit: just saw your added edit that plugging the link into a regular 1gbe port works, are you sure you're still using the 10gbe module on the Intel side when doing that, or switching to a different sfp module? Those mikrotik copper modules would never run at 1gbe for me, at least not the rev 1s. Sounds like your Intel card is only running the module at 1gbe if that's the case, which sounds like some of their vendor locking ****ery

 

[clcorbin]

The x540 doesn't use SFP+ modules. The transceivers are built in. I've used it at 10Gb with my old Aruba S3500 with no issues. And using the exact same module from the Aruba in the ICX 6610 1/3/x slots (tried multiple slots), it won't connect.

I realized I hadn't tested the cable from end to end yet, so I was hoping that would be a fail, but the cable tester came back with a pass and verified I am using the correct cable at both ends.

So do they make any QSFP transceivers for copper 10Gb? I have three of the dual port Mellanox cards gathering dust I could use if I could find a transceiver for my computers end.

 

[clcorbin]

I've been having a problem with one port (work laptop 1Gb LAN port) since I switched to the ICX 6610 stack. One 6610 is in the hobby room in the computer stack, the other is out in the garage (one switch would have been MORE THAN ENOUGH ports, but about half my network drops led to the hobby room (stuff I've added) and half to the garage (original drops)) linked by dual 40GB fiber on x/2/1 and x/2/6 stacking ports.

One port from the garage switch on vlan 50 comes back to my hobby room (again, existing cabling in the right place) and goes to a basic, unmanaged Netgear GS108 switch. My work laptop and my personal laptop are connected to it.

The personal laptop (Windows 10 Pro on my AD domain) has worked flawlessly. But the work laptop is being a bit of a pain. It will disconnect the LAN from the network at (what appears to be) random times and then reconnect like nothing happened a few HOURS later. I've seen it happen two or three times in a day. There IS a physical connection during this time, but it gets the default ip address and has no connectivity to anything. But it DOES know there is a cable connected to SOMETHING out there.

Every single time it has happened, I've checked my personal laptop and it is connected just fine. I've checked (and then swapped) the cable with no change. Once, I saw a message on the network adapter show it had failed authentication. Which is a bit strange, as I don't have 802.1x setup on my network.

Using the same network (other than the switches), I never saw this problem in the 1 1/2 years I've had this laptop at my house on my network.

Also, the WiFi (from an Aruba IAP-305 connected to the server rack switch) has remained flawless as well. Same computer, but it IS brought into a different VLAN while connected to WiFi.

Any suggestions on what to look at would be greatly appreciated!

Clint

 

[fohdeesha]

The x540 doesn't use SFP+ modules. The transceivers are built in. I've used it at 10Gb with my old Aruba S3500 with no issues. And using the exact same module from the Aruba in the ICX 6610 1/3/x slots (tried multiple slots), it won't connect.

I realized I hadn't tested the cable from end to end yet, so I was hoping that would be a fail, but the cable tester came back with a pass and verified I am using the correct cable at both ends.

So do they make any QSFP transceivers for copper 10Gb? I have three of the dual port Mellanox cards gathering dust I could use if I could find a transceiver for my computers end.

Hmm, have you followed the setup and licensing guide for the 6610 to actually unlock the 10gbe ports? And followed the part of the guide that includes setting int 1/3/1 - 1/3/8 to speed 10g? I'd test a regular 10gb fiber link to something to rule something weird out first

Edit: just saw your first post you already tested the front ports at 10gbe with fiber already, I need to not browse while half asleep. Maybe see if you can find a short cable to test a link with, and/or reboot the 6610 once the copper module has been installed

 

[fohdeesha]

Last thought: has that x540 ever successfully linked up with 10gbe copper modules? (Eg is that what it was connected to on the Aruba previously?) I would probably test that if not, kind of sounds like the x540 might not like linking with these modules, could be the length of the run or something

 

[_Adrian_]

It's literally at the top of the first post.

Brocade Overview - Fohdeesha Docs

fohdeesha.com

Sorry, noticed it last night when i was at home. was browsing on my cell and didn't notice the link.
Followed the setup down to a T and worked 90% of it.

Problem in the end was that i could not access the GUI with any credentials including the default ones
had to resort to using the no password option.

Any insight on this ?

Thanks for the info dump BTW

 

[clcorbin]

Hmm, have you followed the setup and licensing guide for the 6610 to actually unlock the 10gbe ports? And followed the part of the guide that includes setting int 1/3/1 - 1/3/8 to speed 10g? I'd test a regular 10gb fiber link to something to rule something weird out first

Edit: just saw your first post you already tested the front ports at 10gbe with fiber already, I need to not browse while half asleep. Maybe see if you can find a short cable to test a link with, and/or reboot the 6610 once the copper module has been installed

You are the man! As I expected, something obvious and trivial. I had enabled 10g on the front ports. Of unit 2 (rack). But never unit 1!

A quick "speed-duplex 10g-full" and unplugging and replugging the cable had everything up and running perfectly.

Thank you Kind Sir!

 

[NateR]

Hello all,

A question regarding LAG on an ICX6450:

I've enabled LAG on a couple of ports on the switch, and on both NICs within Proxmox (box has 2x 1gbe connections, remote device I'm testing with has 1x 10gbe connection). Outbound (from 2x 1gbe to 1x 10gbe device) iperf with 2 threads results in 1.8GB/s, whereas inbound (1x 10gbe to the 2x 1gbe device) maxes at about 1.2GB/s. The more threads, the higher the throughput, but once I get it up to 6 or more threads, it hits the 1.8GB/s point (5 threads or less are under 1.8GB/s).

I understand that with LAG, the different connections could be assigned to one NIC or the other, but with 2 threads it's definitely using both ports (1.2GB/s vs. the single-thread speed of 0.9GB/s), but still not maxing out both NICs. Is it routing more than half of the packets through the port that is set as the "primary?" That's what it looks like may be happening. If so, is there a way to more-equally weight the ports, or configure it to be more of a round-robin back-and-forth type of thing?

Thank you

 

[NateR]

I'm relatively happy with my ICX 6450-24P - it's certainly the loudest thing in the room, but FAR quieter than when it arrived. Mods:
-2x Gryphon GDA4028-12BB (ended up being about the same loudness as the stock fans, but the tone is less annoying).
-330μf electrolytic capacitor on one of the fans to smooth the annoying PWM buzzing.
-Custom-designed 3D printed TPU fan isolators (looks like crap because I've not yet tuned TPU, but it works):

 

[kpfleming]

Hello all,

A question regarding LAG on an ICX6450:

I've enabled LAG on a couple of ports on the switch, and on both NICs within Proxmox (box has 2x 1gbe connections, remote device I'm testing with has 1x 10gbe connection). Outbound (from 2x 1gbe to 1x 10gbe device) iperf with 2 threads results in 1.8GB/s, whereas inbound (1x 10gbe to the 2x 1gbe device) maxes at about 1.2GB/s. The more threads, the higher the throughput, but once I get it up to 6 or more threads, it hits the 1.8GB/s point (5 threads or less are under 1.8GB/s).

I understand that with LAG, the different connections could be assigned to one NIC or the other, but with 2 threads it's definitely using both ports (1.2GB/s vs. the single-thread speed of 0.9GB/s), but still not maxing out both NICs. Is it routing more than half of the packets through the port that is set as the "primary?" That's what it looks like may be happening. If so, is there a way to more-equally weight the ports, or configure it to be more of a round-robin back-and-forth type of thing?

Thank you

802.3ad/LACP (the underlying protocol used for LAGs) has a limited set of path-assignment algorithms; feel free to look them up on Wikipedia or you favorite other resource

Also keep in mind that each node decides how it will send traffic; the switch may be using a different algorithm than the 'remote device', there is no guarantee that you will see the same behavior in both directions unless you've explicitly configured them to use the same algorithms. It is not unusual to see traffic in one direction (of a bidrectional link) choose one path while traffic in the other direction chooses a different path.

 

[geeky_rand]

following the Setup/Configure/Licensing Guide V2 at the top of this post, I've noticed the brocade-12-20-2021.zip does not include a Switch firmware for the ICX6610, is that on purpose?

 

[dwalme]

Look again. I just updated one yesterday.

 

[geeky_rand]

No, only bin files in there are FCXR08030u.bin and grz10100.bin. Thus the reason I'm asking

~/tftp/brocade-12-20-2021$ find . -name '*.bin'
./TFTP Content/ICX7xxx/mnz10118.bin
./TFTP Content/ICX7xxx/SPR08095fufi.bin
./TFTP Content/ICX7xxx/spz10118.bin
./TFTP Content/ICX7xxx/SPR08090mc.bin
./TFTP Content/ICX6610-FCX/grz10100.bin
./TFTP Content/ICX6610-FCX/FCXR08030u.bin
./TFTP Content/ICX6650/ICXR08030u.bin
./TFTP Content/ICX6650/fxz10101.bin
./TFTP Content/ICX64xx/ICX64S08030u.bin
./TFTP Content/ICX64xx/ICX64R08030u.bin
./TFTP Content/ICX64xx/kxz10105.bin

 

[NablaSquaredG]

./TFTP Content/ICX6610-FCX/FCXR08030u.bin

From the guide "Setting up Brocade FCX & ICX6610":

copy tftp flash 192.168.1.8 ICX6610-FCX/FCXR08030u.bin primary

 

[geeky_rand]

From the guide "Setting up Brocade FCX & ICX6610":

copy tftp flash 192.168.1.8 ICX6610-FCX/FCXR08030u.bin primary

yeah, I get that. Thats the ROUTER firmware, for L3, I'm just curious if the SWITCH firmware is available. thats all

 

[NablaSquaredG]

I don't think there even is a L2 firmware for the ICX6610... And even if there was, there would be absolutely no point in using it, because the ICX6610 does routing at wirespeed.

If you don't need L3 features, don't enable / use them. Pretty simple. Don't make your life unnecessary hard