Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[itronin]

I'm a hobbyist, so I don't have extra stuff, and won't have extra stuff.

Tracert to 8.8.8.8 is nothing but requests timed out.

and that is useful. whether you realize it or not. sometimes network issues are simple, if you see X then Y is the cause and Z is the solution. sometimes there's no silver bullet to a network issue, its multiple indicators that clue you in.

I guess Monday I will have to get ahold of FS and try to get another cable. See if they know what would be the best bet for a DAC (if it needs to be programmed. For Intel on one end and Brocade on the other?)

fwiw, I got a bad batch of fs.com 5m DACS recently. every single one of them was a turd, every @#$@##$ one was bad. not saying that fs.com is bad. not saying their products are bad. it happens. Wanna know what I ended up using just to get things to work till I figured it out a permanent solution? Used 3m Cisco DACS from the bay that cost like 5USD each. I buy 'em test 'em throw in a box, pull 'em out when needed. They don't work I cut the end off and bin them. why do I have this box? cause cables don't work sometimes. sometimes they work and then go bad. same thing with spare nics. I get it - hobbyist so no you won't have a box of dacs you can dig through.

No idea what to look for regarding looking at the interface statistics to see if there are errors on the connected switch port.

use the show inter command from earlier

look at this section in the output:

300 second input rate: 224 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 1976 bits/sec, 2 packets/sec, 0.00% utilization

Especially this part:

  7685 packets input, 826324 bytes, 0 no buffer
  Received 272 broadcasts, 7404 multicasts, 9 unicasts
  33 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 30 giants

And this part:

  502846 packets output, 56757049 bytes, 0 underruns
  Transmitted 181923 broadcasts, 319738 multicasts, 1185 unicasts
  0 output errors, 0 collisions

you will likely need to generate traffic to/from your target, a continuous ping with a 1400 byte payload would be good. let it run even if it is not always making it. look at the stats while this is happening. Note the numbers before you start and after finish. check the stats while the test is running? do any errors go up?

If you have a return window on your switch closing in - you could return it. get another from a different seller. If you have the same problems, could be a bad switch from the new seller, could be your cable(s), could be the nic.

IMO having a spare DAC is NOT a bad thing, nor is having a spare NIC NOT a bad thing but that's for me. Do what's right for you.

 

[Craig Curtin]

and that is useful. whether you realize it or not. sometimes network issues are simple, if you see X then Y is the cause and Z is the solution. sometimes there's no silver bullet to a network issue, its multiple indicators that clue you in.



fwiw, I got a bad batch of fs.com 5m DACS recently. every single one of them was a turd, every @#$@##$ one was bad. not saying that fs.com is bad. not saying their products are bad. it happens. Wanna know what I ended up using just to get things to work till I figured it out a permanent solution? Used 3m Cisco DACS from the bay that cost like 5USD each. I buy 'em test 'em throw in a box, pull 'em out when needed. They don't work I cut the end off and bin them. why do I have this box? cause cables don't work sometimes. sometimes they work and then go bad. same thing with spare nics. I get it - hobbyist so no you won't have a box of dacs you can dig through.



use the show inter command from earlier

look at this section in the output:

300 second input rate: 224 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 1976 bits/sec, 2 packets/sec, 0.00% utilization

Especially this part:

  7685 packets input, 826324 bytes, 0 no buffer
  Received 272 broadcasts, 7404 multicasts, 9 unicasts
  33 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 30 giants

And this part:

  502846 packets output, 56757049 bytes, 0 underruns
  Transmitted 181923 broadcasts, 319738 multicasts, 1185 unicasts
  0 output errors, 0 collisions

you will likely need to generate traffic to/from your target, a continuous ping with a 1400 byte payload would be good. let it run even if it is not always making it. look at the stats while this is happening. Note the numbers before you start and after finish. check the stats while the test is running? do any errors go up?

If you have a return window on your switch closing in - you could return it. get another from a different seller. If you have the same problems, could be a bad switch from the new seller, could be your cable(s), could be the nic.

IMO having a spare DAC is NOT a bad thing, nor is having a spare NIC NOT a bad thing but that's for me. Do what's right for you.

Great set of steps

I would just add to the OP - if this is a hobby project and you do not have necessary knowledge behind you - and do not intend to purchase spares to enable tracking down issues - then you are probably better off walking away now.

Have a look at my posts from a couple of days ago - i am still undergoing extensive troubleshooting trying to work out what is wrong in my environment and i am a network security guy by trade so have been working with lots of different manufacturers switches for years.

Craig

 

[ServerSemi]

ICX 7150-C10ZP

Are Ruckus and Brocade the same company? Like HP and Aruba?

 

[bitbckt]

Are Ruckus and Brocade the same company? Like HP and Aruba?

Commscope ate Arris, who ate Brocade, who ate Ruckus. And somebody swallowed the fly, but I don’t know why.

 

[i386]

Are Ruckus and Brocade the same company? Like HP and Aruba?

Brocade was acquired by broadcom. In order to complete/allow the acquisition brocade had to sell parts of the company.
The wireless and icx networking stuff was bought by ruckus.

 

[NablaSquaredG]

There's a nice table on wikipedia:

Brocade Communications Systems - Wikipedia

en.wikipedia.org

 

[ramicio]

Great set of steps

I would just add to the OP - if this is a hobby project and you do not have necessary knowledge behind you - and do not intend to purchase spares to enable tracking down issues - then you are probably better off walking away now.

Have a look at my posts from a couple of days ago - i am still undergoing extensive troubleshooting trying to work out what is wrong in my environment and i am a network security guy by trade so have been working with lots of different manufacturers switches for years.

Craig

Respectfully, the point of it being a hobby is that it's on a budget and it's not mission-critical. I don't need to have spare on-hand, because if it goes down, so what? It's not used for profit, so nothing is lost of it goes down. I'm not going to just walk away. I can buy another cable. If that doesn't work, then a NIC. If it ends up being the switch, then I will have 2 cables and 2 NICS, which is what I wanted in the end anyway.

 

[ramicio]

and that is useful. whether you realize it or not. sometimes network issues are simple, if you see X then Y is the cause and Z is the solution. sometimes there's no silver bullet to a network issue, its multiple indicators that clue you in.



fwiw, I got a bad batch of fs.com 5m DACS recently. every single one of them was a turd, every @#$@##$ one was bad. not saying that fs.com is bad. not saying their products are bad. it happens. Wanna know what I ended up using just to get things to work till I figured it out a permanent solution? Used 3m Cisco DACS from the bay that cost like 5USD each. I buy 'em test 'em throw in a box, pull 'em out when needed. They don't work I cut the end off and bin them. why do I have this box? cause cables don't work sometimes. sometimes they work and then go bad. same thing with spare nics. I get it - hobbyist so no you won't have a box of dacs you can dig through.



use the show inter command from earlier

look at this section in the output:

300 second input rate: 224 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 1976 bits/sec, 2 packets/sec, 0.00% utilization

Especially this part:

  7685 packets input, 826324 bytes, 0 no buffer
  Received 272 broadcasts, 7404 multicasts, 9 unicasts
  33 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 30 giants

And this part:

  502846 packets output, 56757049 bytes, 0 underruns
  Transmitted 181923 broadcasts, 319738 multicasts, 1185 unicasts
  0 output errors, 0 collisions

you will likely need to generate traffic to/from your target, a continuous ping with a 1400 byte payload would be good. let it run even if it is not always making it. look at the stats while this is happening. Note the numbers before you start and after finish. check the stats while the test is running? do any errors go up?

If you have a return window on your switch closing in - you could return it. get another from a different seller. If you have the same problems, could be a bad switch from the new seller, could be your cable(s), could be the nic.

IMO having a spare DAC is NOT a bad thing, nor is having a spare NIC NOT a bad thing but that's for me. Do what's right for you.

I did a ping from my one server (good) to the other (with the XL710) of 1400 bytes. 0% packet loss. Also zero additional input errors on the switch. I did the same, but in reverse, and the same thing happened. No errors, no packet loss.

I'm almost tempted to just buy 2 transceivers at this point. One for "Brocade" and one for "Intel." And just a fiber patch cable.

 

[itronin]

I did a ping from my one server (good) to the other (with the XL710) of 1400 bytes. 0% packet loss. Also zero additional input errors on the switch. I did the same, but in reverse, and the same thing happened. No errors, no packet loss.

I'm almost tempted to just buy 2 transceivers at this point. One for "Brocade" and one for "Intel." And just a fiber patch cable.

See that makes no sense now... you can't get DHCP to pass to the server, yet you have NO errors and NO packet loss with static IP's on the local lan.

I apologize in advance for what may seem like dumb questions but I feel they have to be asked:

static ip testing:
When testing with static IP's are you setting the test system's default gateway to your ASUS router?

dhcp testing:
While you have not stated you are using reservations based on your past comments with DHCP and your servers ip's assigned by dhcp it sounds like you have reservations set up.
If you have dhcp reserverations then are you changing the MAC address of the reservation to match the MAC address of the port you are using on the server?
I believe there are two ports on that card therefore 2 different mac addresses, which will also be different from your 10Gbe card's mac addresses?

 

[ramicio]

See that makes no sense now... you can't get DHCP to pass to the server, yet you have NO errors and NO packet loss with static IP's on the local lan.

I apologize in advance for what may seem like dumb questions but I feel they have to be asked:

static ip testing:
When testing with static IP's are you setting the test system's default gateway to your ASUS router?

dhcp testing:
While you have not stated you are using reservations based on your past comments with DHCP and your servers ip's assigned by dhcp it sounds like you have reservations set up.
If you have dhcp reserverations then are you changing the MAC address of the reservation to match the MAC address of the port you are using on the server?
I believe there are two ports on that card therefore 2 different mac addresses, which will also be different from your 10Gbe card's mac addresses?

Yes, I would set the gateway to 192.168.1.1.

This card's MAC address has been long gone from my DHCP reservations list. Since before I even made this thread. Once I can get things to work (may be with a different card, if that's the issue), it will be added back. It just grabs a DHCP address from the random range pool. Not really concerned about giving it a reserved address at this point.

This is the XL710-QDA1, the single-port card.

 

[ramicio]

See that makes no sense now... you can't get DHCP to pass to the server, yet you have NO errors and NO packet loss with static IP's on the local lan.

I apologize in advance for what may seem like dumb questions but I feel they have to be asked:

static ip testing:
When testing with static IP's are you setting the test system's default gateway to your ASUS router?

dhcp testing:
While you have not stated you are using reservations based on your past comments with DHCP and your servers ip's assigned by dhcp it sounds like you have reservations set up.
If you have dhcp reserverations then are you changing the MAC address of the reservation to match the MAC address of the port you are using on the server?
I believe there are two ports on that card therefore 2 different mac addresses, which will also be different from your 10Gbe card's mac addresses?

Oh, yes, when I set it to a static IP, that's when I get all of the errors. If I try DHCP, it can't do anything at all. It just hangs there endlessly looking for a response [to get an address].

 

[ramicio]

I just ordered 2 new cables. Customized for Brocade on one end and Intel on the other. Should arrive tomorrow. We'll see what happens.

 

[thebwack]

I've got a 7250-24 here on 08.0.30hT211 and I need to update it. I've successfully gone through the guide with two 6610's so I'm familiar but still new to FastIron.

I can't get the MiniUSB Serial Port to work at all, I know its not a USB connection. I've read all the posts and built a cable using the pinout and a usb/console adapter. I'll probably order a Brocade cable although I'm concerned the USB port might have been DOA but we'll see.

In the meantime I have access via Telnet and was able to create a super user and get GUI access so I can for the most part manage this switch but all the features I need are in v8.0.80 onward.

So my question: Is it possible to upgrade this unit from Telnet or GUI? I don't think I can get into the boot loader so I can't load the newer bootloader . Is there anyway to go directly from 8.0.30 to one of the newer packages via GUI or Telnet that would be safe?

Thanks!

 

[Rttg]

While it’s likely best practice to upload the new bootloader from the boot prompt regardless, have you checked to see whether you’re already on the newest bootloader?

A quick sh ver should do the trick. If you’re already on the latest boot code, then it’s fairly trivial to update the software image via SSH/telnet

 

[thebwack]

here is the output

telnet@ICX7250-24 Switch#sh ver                                                 
  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserv
ed.                                                                             
    UNIT 1: compiled on May 19 2016 at 01:22:26 labeled as SPS08030h           
      (28713156 bytes) from Primary SPS08030h.bin                               
        SW: Version 08.0.30hT211                                               
      Compressed Boot-Monitor Image size = 786944, Version:10.1.06T215 (spz10106
)                                                                               
  HW: Stackable ICX7250-24                                                     
==========================================================================     
UNIT 1: SL 1: ICX7250-24 24-port Management Module                             
      Serial  #:DUH3824N00C                                                     
      License: BASE_SOFT_PACKAGE   (LID: fwjINHJpFFe)                           
      P-ASIC  0: type B344, rev 01  Chip BCM56344_A0                           
==========================================================================     
UNIT 1: SL 2: ICX7250-SFP-Plus 8-port 80G Module                               
==========================================================================     
 1000 MHz ARM processor ARMv7 88 MHz bus

 

[thebwack]

wondering if I could load SPR08080f.bin or something newer from the telnet prompt to then allow me to flash the 8095 ufi

if not I'll wait while I order the correct serial cable and see if that works. seems like making a DIY console cable for the MiniUSB is hit or miss so Im probably doing something wrong there.

 

[js00]

Not a direct question for this series but how do you use public IPs with these switches do you need to configure the IP blocks or are they just plug and play like for private IP ranges.

 

[kpfleming]

Not a direct question for this series but how do you use public IPs with these switches do you need to configure the IP blocks or are they just plug and play like for private IP ranges.

If you are using the L2 firmware (in 'switch' mode), there is no need to manage IP blocks at all.
If you are using the L3 firmware (in 'router' mode), you set it up like you would any IPv4 or IPv6 router.

 

[js00]

If you are using the L2 firmware (in 'switch' mode), there is no need to manage IP blocks at all.
If you are using the L3 firmware (in 'router' mode), you set it up like you would any IPv4 or IPv6 router.

I don't currently have access I presume by default these ICX (after factory reset) are in L2 firmware mode?

 

[kpfleming]

No, it depends on the firmware that is installed; the device will either boot 'S' (switch) or 'R' (router) firmware depending on what has been loaded into its flash.