Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[LodeRunner]

Official noob man here. I am having trouble with the transfer timing out when I attempt to install the new bootloader on my 6610 from the tftp server. I went through the whole thread and found one other dude who had the same issue but it was because his firewall was enabled. I already tried doing it with my firewall disabled (as that fixed it for him) and, son of a biscuit, it still didn't work. I have tried multiple ethernet cables for the connection from the management port to the network and from my laptop to the network but to no avail. I'll try another cable each tomorrow and see but I'm doubting that is the fix at this point.

I saw someone else who had a kind of similar issue and it was because they had some sort of janky digit-ery where their mac address should have been, so I checked mine and it seemed (to my untrained eye) to be formatted correctly for a mac address. I'm really hoping this is just some stupid thing I've overlooked but I just can't figure out what that might be at this point.

What TFTP server are you using? Do you have a second computer where you can setup a TFTP client and test that server? If you do a netstat to show UDP ports, is port 69 bound to a physical interface or loopback? On Windows this would be "netstat -an -p udp | findstr 69" for most Linux "netstat -aun | grep 69" should work.

 

[ossicones]

I have four 6610 switches - I am installing two in one rack and two in another rack that is roughly 8-10m away. The question is, should I stack them, or trunk them together?
In the stacking guide, for the 6610 specifically, it states the max length DAC cable is 5m, with no specification for an optical option. I'm guessing this means I cannot stack them using fiber, meaning I would have to trunk them together. Can anyone speak to stacking over distances?

If I trunk, can I use fiber out the QSFP+ ports to get two 40GB links between switches?

 

[berezovskyi]

Hello folks. I have ICX6450-48P (firmware ICX64R08030u) and it seems that periodically crashes. I can't find any pattern for this. Previously it crashed sometimes and I added a big noctua fan to cool it down (`show chassis` showed pretty low temp) and crashes went away for more than a month.
Yesterday it came back. Today I had a chance to get the console cable plugged in and I saw a message that the crash was triggered by the hardware watchdog.
The device configuration wasn't changed for a long time ago. No new devices were connected to the network.

P.S. I found a command `#dd watchdog disable`. Will try it while the console cable is still connected.

 

[LodeRunner]

I have four 6610 switches - I am installing two in one rack and two in another rack that is roughly 8-10m away. The question is, should I stack them, or trunk them together?
In the stacking guide, for the 6610 specifically, it states the max length DAC cable is 5m, with no specification for an optical option. I'm guessing this means I cannot stack them using fiber, meaning I would have to trunk them together. Can anyone speak to stacking over distances?

If I trunk, can I use fiber out the QSFP+ ports to get two 40GB links between switches?

From the ICX6610 manual

ICX 6610 devices have two 40-Gbps QSFP stacking ports and two 4 x 10-Gbps QSFP stacking ports
on the rear panel. These ports can perform data transmission directly through copper links of up to 5
meters. ICX 6610 devices also support 40G-QSFP-SR4 at distances up to 100 meters

Also from the manual

Use 1 meter passive copper QSFP stacking cables or SFP+ fiber-optic cables to connect ICX 6610
devices in a traditional stack. 40G-QSFP-SR4 is also supported.

I'm thinking that when the 6610 came out, DACs might not have exceeded 5 meters. I don't know what spec covers DACs or if it initially had a limit of 5m that was then extended to 10m. So the -SR4 optic is called out, but with no length limit. If you have DACs or optics and 10m cabling, it's worth testing.

 

[lilkeeney]

I am trying to use my 6450-48P as a POE injector. I have a cellular modem in the attic that is POE powered plugged into one port of my switch and then I have another port plugged into the WAN2 port of my router. These two ports are on a seperate VLAN that nothing else is on. This works fine most of the time. However, if one gets disconnected it doens't always reconnected without a reboot. I have a feeling that some settings may be interfering with this. Could Flow Control or STP be an issue here? Any thoughts on how to dumb down the switch and act just like a POE injector? And yes, I could just use a POE injector, but this helps with the wiring and allows me to reboot the cellular modem if needed remotely.

Here are the settings and diagnostics for the cellular modem.




And here are the configurations and stats for the WAN2 port on my router.

 

[Kevlar]

What TFTP server are you using? Do you have a second computer where you can setup a TFTP client and test that server? If you do a netstat to show UDP ports, is port 69 bound to a physical interface or loopback? On Windows this would be "netstat -an -p udp | findstr 69" for most Linux "netstat -aun | grep 69" should work.

I'm just using the one straight from the zip that Fohdeesha made and pointing it to the correct folder replacing the ip address in the instructions with my tftp server ip address. Tested it on a second windows pc and had the same issue where it opens up the file transfer window 4 times and then times out on each of them after not transferring anything.

When I enter that netstat command I get:
UDP 0.0.0.0:69 *:*

....never mind, I was in the middle of getting ready to send this reply and I found that I either needed to select something other than the local (127) server interface in the tftpd64 app or I needed to use a different ip address than I was previously using for the switch because it just worked. Thanks for your quick response!

 

[Cncjerry]

hello all, my 6610 arrived today and thanks to the great write-up it is up, licensed and operational.

Thanks again for the great work.

Jerry

 

[S!rius]

Hello together

Been following this thread for a while and now got myself a 7250-48P. Now before I do my network design I have a trivial question: Can I use the 8-uplink ports as "normal ports" to connect end devices. Must confess that I have not read all 315 pages. Maybe someone of you has asked this or a similar question. The search did not bring any results.

 

[Freebsd1976]

Hello together

Been following this thread for a while and now got myself a 7250-48P. Now before I do my network design I have a trivial question: Can I use the 8-uplink ports as "normal ports" to connect end devices. Must confess that I have not read all 315 pages. Maybe someone of you has asked this or a similar question. The search did not bring any results.

yes

 

[klui]

Uplink is the same as a "normal" port.

 

[Jason Antes]

I'm just using the one straight from the zip that Fohdeesha made and pointing it to the correct folder replacing the ip address in the instructions with my tftp server ip address. Tested it on a second windows pc and had the same issue where it opens up the file transfer window 4 times and then times out on each of them after not transferring anything.

When I enter that netstat command I get:
UDP 0.0.0.0:69 *:*

....never mind, I was in the middle of getting ready to send this reply and I found that I either needed to select something other than the local (127) server interface in the tftpd64 app or I needed to use a different ip address than I was previously using for the switch because it just worked. Thanks for your quick response!

I had it do something like that to me too, pretty sure it was because I had the TFTP set to the wrong folder as a source. It was either down 1 level too far or up 1 level too far.

 

[mbze430]

I can license them, just be aware the copper version (the vdx6740t) draws significant power, it's first gen 10gbase-T PHY, so like a few watts per port. I think they idle around 180w but I don't have any numbers written down from when I had one. also not quiet

Do you have another recommendation, but cheap? I need 10GBase-T ports with 40Gbe or higher to uplink to a DX010 I was looking at a VDX6740T but that power draw is a bit insane, I looked it up in the data sheet from Extreme Network

 

[klui]

Your best bet would probably be Mikrotik if you want 10GbaseT. Anything else you're looking at 48 ports idling at 200W with nothing connected. You can't disable/reconfigure them to save power like you could with the old Nortel 5510/5520. I hope I'm wrong here but I could not find any "energy saving" command in the CLI reference. Shutting down the ports made no difference in power consumption.

As an additional data point the current generation Arista 48-port 10GbaseT switches idle at 150W. The good thing is new or old, copper 10G switches' latency are the same per their datasheets, at least across Arista's entire portfolio and these VDX-Ts.

We all want cheap, good, fast. Here they are cheap because of high power use impacting your OpEx.

 

[Cncjerry]

pointer needed:

You bond two ports on a server and say they had addresses 192.168.1.16 and 192.168.1.116 before the bond. Then you make a bond0 for instance and static it to 192.168.1.16, does it present to the application as 192.168.1.16 or potentially both .116 and .16 depending on load?

I was playing around with bonding two 10Gb ports on win10 and ubuntu. Everything came up after some screwing around. An owncloud server was connected to my new 6610 through a bonded pair of the front 10Gb ports. That was the only connection to that server. The bond0 (bonded name) on the owncloud server address was set to 192.168.1.16, the usual address. I could ssh to it, share files using smb, etc but many, not all, of my clients couldn't connect to the owncloud server on .16. The .16 address pinged, etc. I took the server bond out and all is back up. To make things more interesting for me, the devices that could connect were on the same subnet but coming in wireless to an AP, all with their same addresses so it wasn't a DHCP plan sort of issue. All hitting the server had their usual addresses. All had to get to the server through the bond0 two 10Gb port link though. The devices that didn't work were all using 10Gb adapters, connected to the front of the 6610. The clients that worked came into the 6610 through a 1Gb link from the wireless access point. This doesn't make any sense.

Thanks, by the way, I love the 6610 performance. It allowed me to take out a switch needed to get to the outside world, merged it all into one box. Love it and this site.

Jerry

 

[Blue)(Fusion]

pointer needed:

You bond two ports on a server and say they had addresses 192.168.1.16 and 192.168.1.116 before the bond. Then you make a bond0 for instance and static it to 192.168.1.16, does it present to the application as 192.168.1.16 or potentially both .116 and .16 depending on load?

I was playing around with bonding two 10Gb ports on win10 and ubuntu. Everything came up after some screwing around. An owncloud server was connected to my new 6610 through a bonded pair of the front 10Gb ports. That was the only connection to that server. The bond0 (bonded name) on the owncloud server address was set to 192.168.1.16, the usual address. I could ssh to it, share files using smb, etc but many, not all, of my clients couldn't connect to the owncloud server on .16. The .16 address pinged, etc. I took the server bond out and all is back up. To make things more interesting for me, the devices that could connect were on the same subnet but coming in wireless to an AP, all with their same addresses so it wasn't a DHCP plan sort of issue. All hitting the server had their usual addresses. All had to get to the server through the bond0 two 10Gb port link though. The devices that didn't work were all using 10Gb adapters, connected to the front of the 6610. The clients that worked came into the 6610 through a 1Gb link from the wireless access point. This doesn't make any sense.

Thanks, by the way, I love the 6610 performance. It allowed me to take out a switch needed to get to the outside world, merged it all into one box. Love it and this site.

Jerry

Firstly, do not assign IPs whatsoever to the interfaces you intend to bond. Only apply an IP to the bond (bond0).

Secondly, it may be a listening interface issue with owncloud. I am not familiar with owncloud's configuration, but ensure it will listen in all interfaces or by an IP address (192.168.1.16). You can confirm what addresses it is listening on by using netstat -an | grep :<port> where <port> is the port used by the owncloud server.

 

[Cncjerry]

Firstly, do not assign IPs whatsoever to the interfaces you intend to bond. Only apply an IP to the bond (bond0).

Secondly, it may be a listening interface issue with owncloud. I am not familiar with owncloud's configuration, but ensure it will listen in all interfaces or by an IP address (192.168.1.16). You can confirm what addresses it is listening on by using netstat -an | grep :<port> where <port> is the port used by the owncloud server.

Thanks, I don't understand the address issue on the interfaces because the netplan didn't have them specified. I'll go back and look because "netplan apply" could be dragging a file in from someplace.

I did check owncloud and you have to specify the domain or IP address that would be the server's target. So I had the .16 address specified in owncloud and not the .116 address (the two addresses of the interfaces prior to bonding). Trying to access owncloud on any interface other than that in the domain configuration will post a message whereas the problem I was having was a hang.

This was the netplan I used when I bonded the interfaces. The indenting isn't working but you'll see no addresses. But before I bring up the bond0 or take it down, the interfaces have addresses. I don't know where it is getting them.


network:
version: 2
renderer: networkd
ethernets:
ens3f0:
dhcp4: no
ens3f1:
dhcp4: no
bonds:
bond0:
interfaces: [ens3f0, ens3f1]
addresses: [192.168.1.16/24]
gateway4: 192.168.1.1
nameservers:
addresses: [8.8.8.8,8.8.4.4]
parameters:
mode: 802.3ad
lacp-rate: fast
primary: ens3f0
mii-monitor-interval: 100


I'm going to bring the bond back up and see if I can figure it out. This command, netstat -an | grep :<port> , will help me.

Jerry

edit: one thing I just realized was that netplan apply adds to or changes configs but it doesn't look like it deletes other interfaces. so when I applied that plan above, the interfaces were still configured and therefore the addresses were still applied (to the interfaces.)

 

[Cncjerry]

maybe a simpler question:

I use a VPN for work and have to connect using Cisco Anyconnect from as many as 3 computers at a time. Problem is that while anyconnect is up, I can't get to my private email, printers, smb shares, etc. It blocks all of them.

Is there a way using the facilities of the ICX 6610 that I can bring-up an Anyconnect VPN from the switch? Maybe a dumb question but I am a sales person and I amaze myself that I got this far. I remember a while back Anyconnect allowed you to specify the TLD that was routed out that VPN bit that's been gone for a while.

Thanks

 

[klui]

You're experiencing that problem because your company has disallowed split-tunneling for its VPN connections. I've not used Cisco's AnyConnect for a long time and it was possible to override that in the past probably due to no formal setting.

 

[tinfoil3d]

Guys, I've recently bought 7150-24 and the screws it came with only permit to backside mount the ears(therefore, only mid-of-rack mount) which isn't gonna work for me. Does anyone know the exact screw type used there at the front? It's really crazy that they are designed different! 4 screw holes on the back of switch and on the front have completely different size!! I don't have these anywhere and need help finding these. Carrying around the switch at the hardware shop isn't gonna look or feel cool.

 

[RedX1]

Guys, I've recently bought 7150-24 and the screws it came with only permit to backside mount the ears(therefore, only mid-of-rack mount) which isn't gonna work for me. Does anyone know the exact screw type used there at the front? It's really crazy that they are designed different! 4 screw holes on the back of switch and on the front have completely different size!! I don't have these anywhere and need help finding these. Carrying around the switch at the hardware shop isn't gonna look or feel cool.

Hi

I have several Brocade switiches, including the 7150-24P


The Larger screws are 8/32 UNC and the Smaller screws are 6/32 UNC.

Check out - MS24693 MACHINE SCREWS (AN507) from some suppliers.

You will need 1/4" long countersunk, if you are using the standard brackets.


I hope that helps.


Take Care


RedX1