Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[kh78]

A seperate question:
For a management VRF, is it better practice to put an IP address on a loopback and forward that to the VRF, rather than a VE?

I ask because I have seen posts elsewhere, with people implying it's somehow better, but I fail to see how in any practical sense. Different if you are using it for keeping a routing protocol up or something I guess?

 

[ramicio]

...


STOP! Take a deep breath.

I reviewed your previous posts. I'd like to get some clarity first. I'm going to give you some suggestions in a bit so bear with me.

Your interface output looks "fine" even the small number of giants (hoping they're just 1522 and not something else - which could be coming form something else on your network). I'm going with the test on the 1/2/6 was simply brief as the uptime on it is short. - please confirm

based on the show int I'd say both 40gbe ports are fine.

for each interface test did you have good link / status "blinky lights" on your NIC?
You previously reported you had no electrical activity on the bottom port - is that still the case?



STOP again! Please don't wipe your 14 year maintained system. If it has been maintained that long I'd either consider it "prod" or of significant sentimental value. Put it back on your network the way it was and leave it alone until you've finished testing the switch.

You are experimenting and troubleshooting - use something else for this process.

Beg, borrow, steal another system. Throw an OS on it (clean install) and let's troubleshoot the rest of the ports on your switch. yeah?

seriously. Don't EFF around with the system you've been using. Use something else to finish testing out the switch.

Next, I saw a comment that made me think you want to rely on DHCP for testing. Bad idea right now.
80% of most network issues are physical layer.
If you rely on DHCP you may question whether DHCP is behaving or you have something else going on. Remove DHCP from the equation. Use static IP's for the remainder of the testing and on a CLEAN system.
that way you can focus on do I have a bad port, do i have a bad cable.
simplify your troubleshooting by simplifying the number of variables in any given test.

When I tell it to use a static IP is when I can actually form a connection with it, but it's then doing all of those RX errors. And using a static IP is also when it says that thing with the old IP address and DHCP when I go to ifdown the connection. I am going to force it to use the old "eth" names and do some more poking. Later I will stick the card in another server I just got and installed 22.04 onto. Honestly, the 14 year old install has been a pain when I've gone and upgraded it. It's pretty janky at this point, which is probably why I'm having so many issues. I will report back some stuff later. Thank you.

 

[itronin]

When I tell it to use a static IP is when I can actually form a connection with it, but it's then doing all of those RX errors. And using a static IP is also when it says that thing with the old IP address and DHCP when I go to ifdown the connection. I am going to force it to use the old "eth" names and do some more poking.

this. Makes me think you may have a conflict on your network with your static or some other thing unrelated to the switch (L2).

Later I will stick the card in another server I just got and installed 22.04 onto. Honestly, the 14 year old install has been a pain when I've gone and upgraded it. It's pretty janky at this point, which is probably why I'm having so many issues. I will report back some stuff later. Thank you.

yep.

please do report back as to whether the blinkies were working correctly on your NIC when connected to BOTH 40Gbe interfaces on the switch.

 

[ramicio]

this. Makes me think you may have a conflict on your network with your static or some other thing unrelated to the switch (L2).



yep.

please do report back as to whether the blinkies were working correctly on your NIC when connected to BOTH 40Gbe interfaces on the switch.

Changed the naming to eth*. No change. I don't understand why, when I use a static IP address (under /etc/network/interfaces) and I go to ifdown it, it mentions releasing the old IP address of 192.168.1.2. The static IP I'm assigning it is 192.168.1.17. When I do an ifconfig, I get something like the following:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.17  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::3efd:feff:feb2:cd30  prefixlen 64  scopeid 0x20<link>
        ether 3c:fd:fe:b2:cd:30  txqueuelen 1000  (Ethernet)
        RX packets 3392  bytes 504447 (504.4 KB)
        RX errors 3819  dropped 0  overruns 0  frame 3819
        TX packets 5184  bytes 637121 (637.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Going to throw the card in my other server now.

 

[ramicio]

This card acts the same in the other server. Try to set a static address, mentions the old one (192.168.1.3 on this machine) and some DHCP nonsense while killing it with ifdown. Won't DHCP, and RX errors out the wazoo with a static IP.

And yeah, both ports on the switch work. No idea why they do now, but I feel like a liar!

 

[Cobra0101]

hi, can now access the web portal, and things connected to the switch now work(thanks )

can't get my 2nd router which is connected to the switch icx6450 (which is connection to the 1st router ) to connect to the internet though i still can do dns look ip but can't access any websites including the admin page of router one


Current configuration:
!
ver 08.0.30qT313
!
stack unit 1
module 1 icx6450-24-port-management-module
module 2 icx6450-sfp-plus-4port-40g-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
!
aaa authentication snmp-server default local
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
hostname icx6450
ip dhcp-client disable
ip dns domain-list my-lan.ovh
ip dns server-address 10.0.0.1
ip route 0.0.0.0/0 10.0.0.1
!
no telnet server
username root password .....
!
ntp
server 129.250.35.251
server 85.199.214.99
server 185.103.117.60
server 90.155.73.34
server 109.74.206.120
server 103.214.44.30
server 162.159.200.1
server 193.47.147.20
!
router rip
neighbor 1 permit 10.0.0.2
learn-default
redistribute connected metric 1
redistribute static metric 1
redistribute bgp metric 1
redistribute ospf metric 1
!
interface ve 1
ip address 10.0.0.4 255.255.254.0
!
end




also any advice on setting up the vlans/ipsubnets please - my setup is below

Router1 <> 6450-switch <> router2

Router1 provides internet and main wifi – all running on vlan 1

Router3 has a VPN running on it, and multiple vlans 1, red, orange and green

6450-switch is running routing mode and has my nas and 6 interfaces spread over 2 severs all running on vlan 1. In addition I want to add the following vlans and/or subnets of /23 size.

For the Dell R710: Green(10.1.0.1/23), Yellow(10.2.0.1/23), Red(10.3.0.1/23)

For the Dell R810: Green(10.4.0.1/23), Yellow(10.5.0.1/23), Red(10.6.0.1/23)

Green has access to the internet and all device on vlan1, Orange has access to internet via VPN and can connect to any device in Orange), Red has no access to internet and can only talk to devices in red.

From reading the manuals think the right approach would be to create 3 vlans and 6 ip subnets (2 on each vlan) with an ip and port assigned to each ip subnet.

 

[itronin]

@ramicio

dumb question. setting it back to DHCP or with the server assigned to .17 turned off - you can't ping .17 right? just checking. Same thing happen if you pick some other address for a static. one you've tested for no replies before assigning it?

you got some other funk going on in your network.

If it were me I'd at least finish testing the breakout ports (assuming you have a DAC or AOC 40gbe to 10gbe breakout... the breakouts are probably fine but would be ncie to know all the ports are good before your return window closes...

dont' feel like a liar. Having someone else help you - often is nothing more than a different set of eyes giving you a (slightly even) different process which forces you out of behaviors which may have contributed to you false negative testing of the ports. its why we help each other.

besides you re-learned how to pull a config and look at port stats on the switch. sounds like a winner afternoon of network geekdom to me.

 

[ramicio]

@ramicio

dumb question. setting it back to DHCP or with the server assigned to .17 turned off - you can't ping .17 right? just checking. Same thing happen if you pick some other address for a static. one you've tested for no replies before assigning it?

you got some other funk going on in your network.

If it were me I'd at least finish testing the breakout ports (assuming you have a DAC or AOC 40gbe to 10gbe breakout... the breakouts are probably fine but would be ncie to know all the ports are good before your return window closes...

dont' feel like a liar. Having someone else help you - often is nothing more than a different set of eyes giving you a (slightly even) different process which forces you out of behaviors which may have contributed to you false negative testing of the ports. its why we help each other.

besides you re-learned how to pull a config and look at port stats on the switch. sounds like a winner afternoon of network geekdom to me.

If it's set to use DHCP it just hangs on the system looking for a DHCP lease forever and ever. On static it can send and receive pings, but it has no real connectivity. Pings only work within the LAN. It can't get any DNS info for some reason.

I really don't think it has anything to do with anything on my network. Why is it these 2 ports with this specific NIC, and I have zero problem with my 10 gig stuff?

I don't have any kind of breakout cable and I don't plan on buying one for any reason.

I need to know if it's the switch, the DAC, or the NIC. How can I determine this? I'm honestly about to just junk it all, because I'm willing to bet it's all a problem with the XL710 and it's probably never going to work, and I refuse to buy a Mellanox or any other brand.

 

[ramicio]

I ran a live Ubuntu Desktop thing. Couldn't do anything meaningful with that, but it did pull an address with DHCP and had some activity. But it was very flaky. Installed Windows, because the live Ubuntu thing was basically unusable. Got an IP address but can't do anything with the connection but ping. What information would one need from me at this point? Router is some older ASUS thing with DD-WRT installed. That's plugged into port 1 (RJ45) on this switch. No other devices on my network have a problem, it's only this 40gig nonsense.

 

[itronin]

If it's set to use DHCP it just hangs on the system looking for a DHCP lease forever and ever. On static it can send and receive pings, but it has no real connectivity. Pings only work within the LAN. It can't get any DNS info for some reason.

I really don't think it has anything to do with anything on my network. Why is it these 2 ports with this specific NIC, and I have zero problem with my 10 gig stuff?

I don't have any kind of breakout cable and I don't plan on buying one for any reason.

I need to know if it's the switch, the DAC, or the NIC. How can I determine this? I'm honestly about to just junk it all, because I'm willing to bet it's all a problem with the XL710 and it's probably never going to work, and I refuse to buy a Mellanox or any other brand.

the fact that you can ping the local lan with a static IP leads me to believe that dhcp packets aren't making it. ping with default is a very small packet. Have you looked at the interface statistics to see if there are errors (incrementing) on the connected switch port?

Have you tried traceroute to a known ip - say 8.8.8.8 - again very small packets, would not involve dns if you disable ip to name resolution (traceroute option) it will even go relatively quickly. If it goes slowly then I'm inclined to believe packet corruption is occurring.

typically when I see a problem like this I suspect the cable first, then the nic, then the switch port. Just my experience is all that leads me to things in that order. not saying its right or wrong to look at it other ways. If you have another DAC, AOC, set of optics with patch, you may want to try that. If that doesn't work and you have another nic then try that. You may have to build a matrix of tests and results to ultimately weed out what is wrong. could be the switch too, could be the port on the switch, the nic, the DAC all of the above in combination. Key thing here is change 1 thing at a time and try again. record the result.

regarding your last post - it is very clear you are quite frustrated - and I say this with all due respect (and not knowing what other efforts and time have been spent, nor your skill, knowledge level or experience history)

This may not be the best solution set for you. Only you can decide if the return on your investment - time and money - is worth the return that you will get from making this work. for some people its the end result, for some folks its the end result as well as what they learned getting there. I don't know the answer for you. only you do.

 

[ramicio]

the fact that you can ping the local lan with a static IP leads me to believe that dhcp packets aren't making it. ping with default is a very small packet. Have you looked at the interface statistics to see if there are errors (incrementing) on the connected switch port?

Have you tried traceroute to a known ip - say 8.8.8.8 - again very small packets, would not involve dns if you disable ip to name resolution (traceroute option) it will even go relatively quickly. If it goes slowly then I'm inclined to believe packet corruption is occurring.

typically when I see a problem like this I suspect the cable first, then the nic, then the switch port. Just my experience is all that leads me to things in that order. not saying its right or wrong to look at it other ways. If you have another DAC, AOC, set of optics with patch, you may want to try that. If that doesn't work and you have another nic then try that. You may have to build a matrix of tests and results to ultimately weed out what is wrong. could be the switch too, could be the port on the switch, the nic, the DAC all of the above in combination. Key thing here is change 1 thing at a time and try again. record the result.

regarding your last post - it is very clear you are quite frustrated - and I say this with all due respect (and not knowing what other efforts and time have been spent, nor your skill, knowledge level or experience history)

This may not be the best solution set for you. Only you can decide if the return on your investment - time and money - is worth the return that you will get from making this work. for some people its the end result, for some folks its the end result as well as what they learned getting there. I don't know the answer for you. only you do.

I'm a hobbyist, so I don't have extra stuff, and won't have extra stuff.

Tracert to 8.8.8.8 is nothing but requests timed out.

I guess Monday I will have to get ahold of FS and try to get another cable. See if they know what would be the best bet for a DAC (if it needs to be programmed. For Intel on one end and Brocade on the other?)

No idea what to look for regarding looking at the interface statistics to see if there are errors on the connected switch port.

 

[itronin]

I'm a hobbyist, so I don't have extra stuff, and won't have extra stuff.

Tracert to 8.8.8.8 is nothing but requests timed out.

and that is useful. whether you realize it or not. sometimes network issues are simple, if you see X then Y is the cause and Z is the solution. sometimes there's no silver bullet to a network issue, its multiple indicators that clue you in.

I guess Monday I will have to get ahold of FS and try to get another cable. See if they know what would be the best bet for a DAC (if it needs to be programmed. For Intel on one end and Brocade on the other?)

fwiw, I got a bad batch of fs.com 5m DACS recently. every single one of them was a turd, every @#$@##$ one was bad. not saying that fs.com is bad. not saying their products are bad. it happens. Wanna know what I ended up using just to get things to work till I figured it out a permanent solution? Used 3m Cisco DACS from the bay that cost like 5USD each. I buy 'em test 'em throw in a box, pull 'em out when needed. They don't work I cut the end off and bin them. why do I have this box? cause cables don't work sometimes. sometimes they work and then go bad. same thing with spare nics. I get it - hobbyist so no you won't have a box of dacs you can dig through.

No idea what to look for regarding looking at the interface statistics to see if there are errors on the connected switch port.

use the show inter command from earlier

look at this section in the output:

300 second input rate: 224 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 1976 bits/sec, 2 packets/sec, 0.00% utilization

Especially this part:

  7685 packets input, 826324 bytes, 0 no buffer
  Received 272 broadcasts, 7404 multicasts, 9 unicasts
  33 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 30 giants

And this part:

  502846 packets output, 56757049 bytes, 0 underruns
  Transmitted 181923 broadcasts, 319738 multicasts, 1185 unicasts
  0 output errors, 0 collisions

you will likely need to generate traffic to/from your target, a continuous ping with a 1400 byte payload would be good. let it run even if it is not always making it. look at the stats while this is happening. Note the numbers before you start and after finish. check the stats while the test is running? do any errors go up?

If you have a return window on your switch closing in - you could return it. get another from a different seller. If you have the same problems, could be a bad switch from the new seller, could be your cable(s), could be the nic.

IMO having a spare DAC is NOT a bad thing, nor is having a spare NIC NOT a bad thing but that's for me. Do what's right for you.

 

[Craig Curtin]

and that is useful. whether you realize it or not. sometimes network issues are simple, if you see X then Y is the cause and Z is the solution. sometimes there's no silver bullet to a network issue, its multiple indicators that clue you in.



fwiw, I got a bad batch of fs.com 5m DACS recently. every single one of them was a turd, every @#$@##$ one was bad. not saying that fs.com is bad. not saying their products are bad. it happens. Wanna know what I ended up using just to get things to work till I figured it out a permanent solution? Used 3m Cisco DACS from the bay that cost like 5USD each. I buy 'em test 'em throw in a box, pull 'em out when needed. They don't work I cut the end off and bin them. why do I have this box? cause cables don't work sometimes. sometimes they work and then go bad. same thing with spare nics. I get it - hobbyist so no you won't have a box of dacs you can dig through.



use the show inter command from earlier

look at this section in the output:

300 second input rate: 224 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 1976 bits/sec, 2 packets/sec, 0.00% utilization

Especially this part:

  7685 packets input, 826324 bytes, 0 no buffer
  Received 272 broadcasts, 7404 multicasts, 9 unicasts
  33 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 30 giants

And this part:

  502846 packets output, 56757049 bytes, 0 underruns
  Transmitted 181923 broadcasts, 319738 multicasts, 1185 unicasts
  0 output errors, 0 collisions

you will likely need to generate traffic to/from your target, a continuous ping with a 1400 byte payload would be good. let it run even if it is not always making it. look at the stats while this is happening. Note the numbers before you start and after finish. check the stats while the test is running? do any errors go up?

If you have a return window on your switch closing in - you could return it. get another from a different seller. If you have the same problems, could be a bad switch from the new seller, could be your cable(s), could be the nic.

IMO having a spare DAC is NOT a bad thing, nor is having a spare NIC NOT a bad thing but that's for me. Do what's right for you.

Great set of steps

I would just add to the OP - if this is a hobby project and you do not have necessary knowledge behind you - and do not intend to purchase spares to enable tracking down issues - then you are probably better off walking away now.

Have a look at my posts from a couple of days ago - i am still undergoing extensive troubleshooting trying to work out what is wrong in my environment and i am a network security guy by trade so have been working with lots of different manufacturers switches for years.

Craig

 

[ServerSemi]

ICX 7150-C10ZP

Are Ruckus and Brocade the same company? Like HP and Aruba?

 

[bitbckt]

Are Ruckus and Brocade the same company? Like HP and Aruba?

Commscope ate Arris, who ate Brocade, who ate Ruckus. And somebody swallowed the fly, but I don’t know why.

 

[i386]

Are Ruckus and Brocade the same company? Like HP and Aruba?

Brocade was acquired by broadcom. In order to complete/allow the acquisition brocade had to sell parts of the company.
The wireless and icx networking stuff was bought by ruckus.

 

[NablaSquaredG]

There's a nice table on wikipedia:

Brocade Communications Systems - Wikipedia

en.wikipedia.org

 

[ramicio]

Great set of steps

I would just add to the OP - if this is a hobby project and you do not have necessary knowledge behind you - and do not intend to purchase spares to enable tracking down issues - then you are probably better off walking away now.

Have a look at my posts from a couple of days ago - i am still undergoing extensive troubleshooting trying to work out what is wrong in my environment and i am a network security guy by trade so have been working with lots of different manufacturers switches for years.

Craig

Respectfully, the point of it being a hobby is that it's on a budget and it's not mission-critical. I don't need to have spare on-hand, because if it goes down, so what? It's not used for profit, so nothing is lost of it goes down. I'm not going to just walk away. I can buy another cable. If that doesn't work, then a NIC. If it ends up being the switch, then I will have 2 cables and 2 NICS, which is what I wanted in the end anyway.

 

[ramicio]

and that is useful. whether you realize it or not. sometimes network issues are simple, if you see X then Y is the cause and Z is the solution. sometimes there's no silver bullet to a network issue, its multiple indicators that clue you in.



fwiw, I got a bad batch of fs.com 5m DACS recently. every single one of them was a turd, every @#$@##$ one was bad. not saying that fs.com is bad. not saying their products are bad. it happens. Wanna know what I ended up using just to get things to work till I figured it out a permanent solution? Used 3m Cisco DACS from the bay that cost like 5USD each. I buy 'em test 'em throw in a box, pull 'em out when needed. They don't work I cut the end off and bin them. why do I have this box? cause cables don't work sometimes. sometimes they work and then go bad. same thing with spare nics. I get it - hobbyist so no you won't have a box of dacs you can dig through.



use the show inter command from earlier

look at this section in the output:

300 second input rate: 224 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 1976 bits/sec, 2 packets/sec, 0.00% utilization

Especially this part:

  7685 packets input, 826324 bytes, 0 no buffer
  Received 272 broadcasts, 7404 multicasts, 9 unicasts
  33 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 30 giants

And this part:

  502846 packets output, 56757049 bytes, 0 underruns
  Transmitted 181923 broadcasts, 319738 multicasts, 1185 unicasts
  0 output errors, 0 collisions

you will likely need to generate traffic to/from your target, a continuous ping with a 1400 byte payload would be good. let it run even if it is not always making it. look at the stats while this is happening. Note the numbers before you start and after finish. check the stats while the test is running? do any errors go up?

If you have a return window on your switch closing in - you could return it. get another from a different seller. If you have the same problems, could be a bad switch from the new seller, could be your cable(s), could be the nic.

IMO having a spare DAC is NOT a bad thing, nor is having a spare NIC NOT a bad thing but that's for me. Do what's right for you.

I did a ping from my one server (good) to the other (with the XL710) of 1400 bytes. 0% packet loss. Also zero additional input errors on the switch. I did the same, but in reverse, and the same thing happened. No errors, no packet loss.

I'm almost tempted to just buy 2 transceivers at this point. One for "Brocade" and one for "Intel." And just a fiber patch cable.

 

[itronin]

I did a ping from my one server (good) to the other (with the XL710) of 1400 bytes. 0% packet loss. Also zero additional input errors on the switch. I did the same, but in reverse, and the same thing happened. No errors, no packet loss.

I'm almost tempted to just buy 2 transceivers at this point. One for "Brocade" and one for "Intel." And just a fiber patch cable.

See that makes no sense now... you can't get DHCP to pass to the server, yet you have NO errors and NO packet loss with static IP's on the local lan.

I apologize in advance for what may seem like dumb questions but I feel they have to be asked:

static ip testing:
When testing with static IP's are you setting the test system's default gateway to your ASUS router?

dhcp testing:
While you have not stated you are using reservations based on your past comments with DHCP and your servers ip's assigned by dhcp it sounds like you have reservations set up.
If you have dhcp reserverations then are you changing the MAC address of the reservation to match the MAC address of the port you are using on the server?
I believe there are two ports on that card therefore 2 different mac addresses, which will also be different from your 10Gbe card's mac addresses?