Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[ramicio]

This card acts the same in the other server. Try to set a static address, mentions the old one (192.168.1.3 on this machine) and some DHCP nonsense while killing it with ifdown. Won't DHCP, and RX errors out the wazoo with a static IP.

And yeah, both ports on the switch work. No idea why they do now, but I feel like a liar!

 

[Cobra0101]

hi, can now access the web portal, and things connected to the switch now work(thanks )

can't get my 2nd router which is connected to the switch icx6450 (which is connection to the 1st router ) to connect to the internet though i still can do dns look ip but can't access any websites including the admin page of router one


Current configuration:
!
ver 08.0.30qT313
!
stack unit 1
module 1 icx6450-24-port-management-module
module 2 icx6450-sfp-plus-4port-40g-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
!
aaa authentication snmp-server default local
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
hostname icx6450
ip dhcp-client disable
ip dns domain-list my-lan.ovh
ip dns server-address 10.0.0.1
ip route 0.0.0.0/0 10.0.0.1
!
no telnet server
username root password .....
!
ntp
server 129.250.35.251
server 85.199.214.99
server 185.103.117.60
server 90.155.73.34
server 109.74.206.120
server 103.214.44.30
server 162.159.200.1
server 193.47.147.20
!
router rip
neighbor 1 permit 10.0.0.2
learn-default
redistribute connected metric 1
redistribute static metric 1
redistribute bgp metric 1
redistribute ospf metric 1
!
interface ve 1
ip address 10.0.0.4 255.255.254.0
!
end




also any advice on setting up the vlans/ipsubnets please - my setup is below

Router1 <> 6450-switch <> router2

Router1 provides internet and main wifi – all running on vlan 1

Router3 has a VPN running on it, and multiple vlans 1, red, orange and green

6450-switch is running routing mode and has my nas and 6 interfaces spread over 2 severs all running on vlan 1. In addition I want to add the following vlans and/or subnets of /23 size.

For the Dell R710: Green(10.1.0.1/23), Yellow(10.2.0.1/23), Red(10.3.0.1/23)

For the Dell R810: Green(10.4.0.1/23), Yellow(10.5.0.1/23), Red(10.6.0.1/23)

Green has access to the internet and all device on vlan1, Orange has access to internet via VPN and can connect to any device in Orange), Red has no access to internet and can only talk to devices in red.

From reading the manuals think the right approach would be to create 3 vlans and 6 ip subnets (2 on each vlan) with an ip and port assigned to each ip subnet.

 

[itronin]

@ramicio

dumb question. setting it back to DHCP or with the server assigned to .17 turned off - you can't ping .17 right? just checking. Same thing happen if you pick some other address for a static. one you've tested for no replies before assigning it?

you got some other funk going on in your network.

If it were me I'd at least finish testing the breakout ports (assuming you have a DAC or AOC 40gbe to 10gbe breakout... the breakouts are probably fine but would be ncie to know all the ports are good before your return window closes...

dont' feel like a liar. Having someone else help you - often is nothing more than a different set of eyes giving you a (slightly even) different process which forces you out of behaviors which may have contributed to you false negative testing of the ports. its why we help each other.

besides you re-learned how to pull a config and look at port stats on the switch. sounds like a winner afternoon of network geekdom to me.

 

[ramicio]

@ramicio

dumb question. setting it back to DHCP or with the server assigned to .17 turned off - you can't ping .17 right? just checking. Same thing happen if you pick some other address for a static. one you've tested for no replies before assigning it?

you got some other funk going on in your network.

If it were me I'd at least finish testing the breakout ports (assuming you have a DAC or AOC 40gbe to 10gbe breakout... the breakouts are probably fine but would be ncie to know all the ports are good before your return window closes...

dont' feel like a liar. Having someone else help you - often is nothing more than a different set of eyes giving you a (slightly even) different process which forces you out of behaviors which may have contributed to you false negative testing of the ports. its why we help each other.

besides you re-learned how to pull a config and look at port stats on the switch. sounds like a winner afternoon of network geekdom to me.

If it's set to use DHCP it just hangs on the system looking for a DHCP lease forever and ever. On static it can send and receive pings, but it has no real connectivity. Pings only work within the LAN. It can't get any DNS info for some reason.

I really don't think it has anything to do with anything on my network. Why is it these 2 ports with this specific NIC, and I have zero problem with my 10 gig stuff?

I don't have any kind of breakout cable and I don't plan on buying one for any reason.

I need to know if it's the switch, the DAC, or the NIC. How can I determine this? I'm honestly about to just junk it all, because I'm willing to bet it's all a problem with the XL710 and it's probably never going to work, and I refuse to buy a Mellanox or any other brand.

 

[ramicio]

I ran a live Ubuntu Desktop thing. Couldn't do anything meaningful with that, but it did pull an address with DHCP and had some activity. But it was very flaky. Installed Windows, because the live Ubuntu thing was basically unusable. Got an IP address but can't do anything with the connection but ping. What information would one need from me at this point? Router is some older ASUS thing with DD-WRT installed. That's plugged into port 1 (RJ45) on this switch. No other devices on my network have a problem, it's only this 40gig nonsense.

 

[itronin]

If it's set to use DHCP it just hangs on the system looking for a DHCP lease forever and ever. On static it can send and receive pings, but it has no real connectivity. Pings only work within the LAN. It can't get any DNS info for some reason.

I really don't think it has anything to do with anything on my network. Why is it these 2 ports with this specific NIC, and I have zero problem with my 10 gig stuff?

I don't have any kind of breakout cable and I don't plan on buying one for any reason.

I need to know if it's the switch, the DAC, or the NIC. How can I determine this? I'm honestly about to just junk it all, because I'm willing to bet it's all a problem with the XL710 and it's probably never going to work, and I refuse to buy a Mellanox or any other brand.

the fact that you can ping the local lan with a static IP leads me to believe that dhcp packets aren't making it. ping with default is a very small packet. Have you looked at the interface statistics to see if there are errors (incrementing) on the connected switch port?

Have you tried traceroute to a known ip - say 8.8.8.8 - again very small packets, would not involve dns if you disable ip to name resolution (traceroute option) it will even go relatively quickly. If it goes slowly then I'm inclined to believe packet corruption is occurring.

typically when I see a problem like this I suspect the cable first, then the nic, then the switch port. Just my experience is all that leads me to things in that order. not saying its right or wrong to look at it other ways. If you have another DAC, AOC, set of optics with patch, you may want to try that. If that doesn't work and you have another nic then try that. You may have to build a matrix of tests and results to ultimately weed out what is wrong. could be the switch too, could be the port on the switch, the nic, the DAC all of the above in combination. Key thing here is change 1 thing at a time and try again. record the result.

regarding your last post - it is very clear you are quite frustrated - and I say this with all due respect (and not knowing what other efforts and time have been spent, nor your skill, knowledge level or experience history)

This may not be the best solution set for you. Only you can decide if the return on your investment - time and money - is worth the return that you will get from making this work. for some people its the end result, for some folks its the end result as well as what they learned getting there. I don't know the answer for you. only you do.

 

[ramicio]

the fact that you can ping the local lan with a static IP leads me to believe that dhcp packets aren't making it. ping with default is a very small packet. Have you looked at the interface statistics to see if there are errors (incrementing) on the connected switch port?

Have you tried traceroute to a known ip - say 8.8.8.8 - again very small packets, would not involve dns if you disable ip to name resolution (traceroute option) it will even go relatively quickly. If it goes slowly then I'm inclined to believe packet corruption is occurring.

typically when I see a problem like this I suspect the cable first, then the nic, then the switch port. Just my experience is all that leads me to things in that order. not saying its right or wrong to look at it other ways. If you have another DAC, AOC, set of optics with patch, you may want to try that. If that doesn't work and you have another nic then try that. You may have to build a matrix of tests and results to ultimately weed out what is wrong. could be the switch too, could be the port on the switch, the nic, the DAC all of the above in combination. Key thing here is change 1 thing at a time and try again. record the result.

regarding your last post - it is very clear you are quite frustrated - and I say this with all due respect (and not knowing what other efforts and time have been spent, nor your skill, knowledge level or experience history)

This may not be the best solution set for you. Only you can decide if the return on your investment - time and money - is worth the return that you will get from making this work. for some people its the end result, for some folks its the end result as well as what they learned getting there. I don't know the answer for you. only you do.

I'm a hobbyist, so I don't have extra stuff, and won't have extra stuff.

Tracert to 8.8.8.8 is nothing but requests timed out.

I guess Monday I will have to get ahold of FS and try to get another cable. See if they know what would be the best bet for a DAC (if it needs to be programmed. For Intel on one end and Brocade on the other?)

No idea what to look for regarding looking at the interface statistics to see if there are errors on the connected switch port.

 

[itronin]

I'm a hobbyist, so I don't have extra stuff, and won't have extra stuff.

Tracert to 8.8.8.8 is nothing but requests timed out.

and that is useful. whether you realize it or not. sometimes network issues are simple, if you see X then Y is the cause and Z is the solution. sometimes there's no silver bullet to a network issue, its multiple indicators that clue you in.

I guess Monday I will have to get ahold of FS and try to get another cable. See if they know what would be the best bet for a DAC (if it needs to be programmed. For Intel on one end and Brocade on the other?)

fwiw, I got a bad batch of fs.com 5m DACS recently. every single one of them was a turd, every @#$@##$ one was bad. not saying that fs.com is bad. not saying their products are bad. it happens. Wanna know what I ended up using just to get things to work till I figured it out a permanent solution? Used 3m Cisco DACS from the bay that cost like 5USD each. I buy 'em test 'em throw in a box, pull 'em out when needed. They don't work I cut the end off and bin them. why do I have this box? cause cables don't work sometimes. sometimes they work and then go bad. same thing with spare nics. I get it - hobbyist so no you won't have a box of dacs you can dig through.

No idea what to look for regarding looking at the interface statistics to see if there are errors on the connected switch port.

use the show inter command from earlier

look at this section in the output:

300 second input rate: 224 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 1976 bits/sec, 2 packets/sec, 0.00% utilization

Especially this part:

  7685 packets input, 826324 bytes, 0 no buffer
  Received 272 broadcasts, 7404 multicasts, 9 unicasts
  33 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 30 giants

And this part:

  502846 packets output, 56757049 bytes, 0 underruns
  Transmitted 181923 broadcasts, 319738 multicasts, 1185 unicasts
  0 output errors, 0 collisions

you will likely need to generate traffic to/from your target, a continuous ping with a 1400 byte payload would be good. let it run even if it is not always making it. look at the stats while this is happening. Note the numbers before you start and after finish. check the stats while the test is running? do any errors go up?

If you have a return window on your switch closing in - you could return it. get another from a different seller. If you have the same problems, could be a bad switch from the new seller, could be your cable(s), could be the nic.

IMO having a spare DAC is NOT a bad thing, nor is having a spare NIC NOT a bad thing but that's for me. Do what's right for you.

 

[Craig Curtin]

and that is useful. whether you realize it or not. sometimes network issues are simple, if you see X then Y is the cause and Z is the solution. sometimes there's no silver bullet to a network issue, its multiple indicators that clue you in.



fwiw, I got a bad batch of fs.com 5m DACS recently. every single one of them was a turd, every @#$@##$ one was bad. not saying that fs.com is bad. not saying their products are bad. it happens. Wanna know what I ended up using just to get things to work till I figured it out a permanent solution? Used 3m Cisco DACS from the bay that cost like 5USD each. I buy 'em test 'em throw in a box, pull 'em out when needed. They don't work I cut the end off and bin them. why do I have this box? cause cables don't work sometimes. sometimes they work and then go bad. same thing with spare nics. I get it - hobbyist so no you won't have a box of dacs you can dig through.



use the show inter command from earlier

look at this section in the output:

300 second input rate: 224 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 1976 bits/sec, 2 packets/sec, 0.00% utilization

Especially this part:

  7685 packets input, 826324 bytes, 0 no buffer
  Received 272 broadcasts, 7404 multicasts, 9 unicasts
  33 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 30 giants

And this part:

  502846 packets output, 56757049 bytes, 0 underruns
  Transmitted 181923 broadcasts, 319738 multicasts, 1185 unicasts
  0 output errors, 0 collisions

you will likely need to generate traffic to/from your target, a continuous ping with a 1400 byte payload would be good. let it run even if it is not always making it. look at the stats while this is happening. Note the numbers before you start and after finish. check the stats while the test is running? do any errors go up?

If you have a return window on your switch closing in - you could return it. get another from a different seller. If you have the same problems, could be a bad switch from the new seller, could be your cable(s), could be the nic.

IMO having a spare DAC is NOT a bad thing, nor is having a spare NIC NOT a bad thing but that's for me. Do what's right for you.

Great set of steps

I would just add to the OP - if this is a hobby project and you do not have necessary knowledge behind you - and do not intend to purchase spares to enable tracking down issues - then you are probably better off walking away now.

Have a look at my posts from a couple of days ago - i am still undergoing extensive troubleshooting trying to work out what is wrong in my environment and i am a network security guy by trade so have been working with lots of different manufacturers switches for years.

Craig

 

[ServerSemi]

ICX 7150-C10ZP

Are Ruckus and Brocade the same company? Like HP and Aruba?

 

[bitbckt]

Are Ruckus and Brocade the same company? Like HP and Aruba?

Commscope ate Arris, who ate Brocade, who ate Ruckus. And somebody swallowed the fly, but I don’t know why.

 

[i386]

Are Ruckus and Brocade the same company? Like HP and Aruba?

Brocade was acquired by broadcom. In order to complete/allow the acquisition brocade had to sell parts of the company.
The wireless and icx networking stuff was bought by ruckus.

 

[NablaSquaredG]

There's a nice table on wikipedia:

Brocade Communications Systems - Wikipedia

en.wikipedia.org

 

[ramicio]

Great set of steps

I would just add to the OP - if this is a hobby project and you do not have necessary knowledge behind you - and do not intend to purchase spares to enable tracking down issues - then you are probably better off walking away now.

Have a look at my posts from a couple of days ago - i am still undergoing extensive troubleshooting trying to work out what is wrong in my environment and i am a network security guy by trade so have been working with lots of different manufacturers switches for years.

Craig

Respectfully, the point of it being a hobby is that it's on a budget and it's not mission-critical. I don't need to have spare on-hand, because if it goes down, so what? It's not used for profit, so nothing is lost of it goes down. I'm not going to just walk away. I can buy another cable. If that doesn't work, then a NIC. If it ends up being the switch, then I will have 2 cables and 2 NICS, which is what I wanted in the end anyway.

 

[ramicio]

and that is useful. whether you realize it or not. sometimes network issues are simple, if you see X then Y is the cause and Z is the solution. sometimes there's no silver bullet to a network issue, its multiple indicators that clue you in.



fwiw, I got a bad batch of fs.com 5m DACS recently. every single one of them was a turd, every @#$@##$ one was bad. not saying that fs.com is bad. not saying their products are bad. it happens. Wanna know what I ended up using just to get things to work till I figured it out a permanent solution? Used 3m Cisco DACS from the bay that cost like 5USD each. I buy 'em test 'em throw in a box, pull 'em out when needed. They don't work I cut the end off and bin them. why do I have this box? cause cables don't work sometimes. sometimes they work and then go bad. same thing with spare nics. I get it - hobbyist so no you won't have a box of dacs you can dig through.



use the show inter command from earlier

look at this section in the output:

300 second input rate: 224 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 1976 bits/sec, 2 packets/sec, 0.00% utilization

Especially this part:

  7685 packets input, 826324 bytes, 0 no buffer
  Received 272 broadcasts, 7404 multicasts, 9 unicasts
  33 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 30 giants

And this part:

  502846 packets output, 56757049 bytes, 0 underruns
  Transmitted 181923 broadcasts, 319738 multicasts, 1185 unicasts
  0 output errors, 0 collisions

you will likely need to generate traffic to/from your target, a continuous ping with a 1400 byte payload would be good. let it run even if it is not always making it. look at the stats while this is happening. Note the numbers before you start and after finish. check the stats while the test is running? do any errors go up?

If you have a return window on your switch closing in - you could return it. get another from a different seller. If you have the same problems, could be a bad switch from the new seller, could be your cable(s), could be the nic.

IMO having a spare DAC is NOT a bad thing, nor is having a spare NIC NOT a bad thing but that's for me. Do what's right for you.

I did a ping from my one server (good) to the other (with the XL710) of 1400 bytes. 0% packet loss. Also zero additional input errors on the switch. I did the same, but in reverse, and the same thing happened. No errors, no packet loss.

I'm almost tempted to just buy 2 transceivers at this point. One for "Brocade" and one for "Intel." And just a fiber patch cable.

 

[itronin]

I did a ping from my one server (good) to the other (with the XL710) of 1400 bytes. 0% packet loss. Also zero additional input errors on the switch. I did the same, but in reverse, and the same thing happened. No errors, no packet loss.

I'm almost tempted to just buy 2 transceivers at this point. One for "Brocade" and one for "Intel." And just a fiber patch cable.

See that makes no sense now... you can't get DHCP to pass to the server, yet you have NO errors and NO packet loss with static IP's on the local lan.

I apologize in advance for what may seem like dumb questions but I feel they have to be asked:

static ip testing:
When testing with static IP's are you setting the test system's default gateway to your ASUS router?

dhcp testing:
While you have not stated you are using reservations based on your past comments with DHCP and your servers ip's assigned by dhcp it sounds like you have reservations set up.
If you have dhcp reserverations then are you changing the MAC address of the reservation to match the MAC address of the port you are using on the server?
I believe there are two ports on that card therefore 2 different mac addresses, which will also be different from your 10Gbe card's mac addresses?

 

[ramicio]

See that makes no sense now... you can't get DHCP to pass to the server, yet you have NO errors and NO packet loss with static IP's on the local lan.

I apologize in advance for what may seem like dumb questions but I feel they have to be asked:

static ip testing:
When testing with static IP's are you setting the test system's default gateway to your ASUS router?

dhcp testing:
While you have not stated you are using reservations based on your past comments with DHCP and your servers ip's assigned by dhcp it sounds like you have reservations set up.
If you have dhcp reserverations then are you changing the MAC address of the reservation to match the MAC address of the port you are using on the server?
I believe there are two ports on that card therefore 2 different mac addresses, which will also be different from your 10Gbe card's mac addresses?

Yes, I would set the gateway to 192.168.1.1.

This card's MAC address has been long gone from my DHCP reservations list. Since before I even made this thread. Once I can get things to work (may be with a different card, if that's the issue), it will be added back. It just grabs a DHCP address from the random range pool. Not really concerned about giving it a reserved address at this point.

This is the XL710-QDA1, the single-port card.

 

[ramicio]

See that makes no sense now... you can't get DHCP to pass to the server, yet you have NO errors and NO packet loss with static IP's on the local lan.

I apologize in advance for what may seem like dumb questions but I feel they have to be asked:

static ip testing:
When testing with static IP's are you setting the test system's default gateway to your ASUS router?

dhcp testing:
While you have not stated you are using reservations based on your past comments with DHCP and your servers ip's assigned by dhcp it sounds like you have reservations set up.
If you have dhcp reserverations then are you changing the MAC address of the reservation to match the MAC address of the port you are using on the server?
I believe there are two ports on that card therefore 2 different mac addresses, which will also be different from your 10Gbe card's mac addresses?

Oh, yes, when I set it to a static IP, that's when I get all of the errors. If I try DHCP, it can't do anything at all. It just hangs there endlessly looking for a response [to get an address].

 

[ramicio]

I just ordered 2 new cables. Customized for Brocade on one end and Intel on the other. Should arrive tomorrow. We'll see what happens.

 

[thebwack]

I've got a 7250-24 here on 08.0.30hT211 and I need to update it. I've successfully gone through the guide with two 6610's so I'm familiar but still new to FastIron.

I can't get the MiniUSB Serial Port to work at all, I know its not a USB connection. I've read all the posts and built a cable using the pinout and a usb/console adapter. I'll probably order a Brocade cable although I'm concerned the USB port might have been DOA but we'll see.

In the meantime I have access via Telnet and was able to create a super user and get GUI access so I can for the most part manage this switch but all the features I need are in v8.0.80 onward.

So my question: Is it possible to upgrade this unit from Telnet or GUI? I don't think I can get into the boot loader so I can't load the newer bootloader . Is there anyway to go directly from 8.0.30 to one of the newer packages via GUI or Telnet that would be safe?

Thanks!