Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Craig Curtin]

Kudos for the assistance Craig. Everything you outlined above has worked like a charm. I was able to add 1/2/1 to the "new" VLANs and didn't need to re-add to dual-mode nor did I lose connection via SSH during the process. Setting up new downstream ports to my WAPs, switches and hypervisor was even easier.

Once I'm feeling brave enough I may dabble into L3 routing, but the UDM SE is working well enough right now as router-on-a-stick so no urgent need. Nice to know that the L3 is available if I grow into it in the future.

No worries - i too am new on the ICX bandwagon so glad i could help.

Remember if you move from the UDM pro for routing/filtering you will get more speed - BUT will have to learn how to write ACLs etc for the switch to perform the necessary filtering between VLANs etc - so thats another whole can of worms !

Once i get my ESXi hosts stable on the ICXs - that will be my next rabbit hole to go down !

Craig

 

[Craig Curtin]

I had exactly the same problem as you.
Try to add the line to each physical interface:

no spanning-tree
e.g.
interface ethernet 1/2/1
port-name xxxx
no spanning-tree

Let me know if that helps.

The explanation is somewhere within this thread. I think it was about spanning tree being active per port even though it is globally disabled.
At least it helped in my case.

OK been through it and did the following

conf t
int e 1/2/1 to 1/2/10
no spanning-tree

wr mem

No change at the ESXi host

Rebooted the Host - no difference

logged back into the switch and did a disable and enable on the ports

No change at the ESXI host

Pretty much at my wits end now

Got one of the 3 hosts running on one switch and another on a different switch - one using Arista breakout cables and the other using FC.com breakouts

Tried with Dualport Intel 520DA-2 card (Intel 82599 and ixgben drivers in ESXi)
Tried with 2 x Single port Intel 520 card
Tried with Intel Dual port 540T cards and Twisted pair transceivers in the 1/3/x slots

I am waiting for some dual port Mellanox CX-3 adapters to arrive (CX312a)

Not sure what else i can do

Anyone ?

Craig

 

[itronin]

Yeah i might move it back to street power for a little while and see what happens - hopefully it is not that finicky with power requirements !!

...

Craig

did you end up getting your switches to stop spontaneously rebooting? on street power? half n half?

btw, I did not go back and find the last copy of your config - so would you clarify: Are you LAGGing the ports to your ESXI hosts?

FWIW, I have not had issues with the SM dual 10gbe sfp+ which is a 520 card, nor chelsio 520's, nor CX312A's nor the 40Gbe version (using 40 or 10gbe with an step down adapter).

 

[Craig Curtin]

did you end up getting your switches to stop spontaneously rebooting? on street power? half n half?

btw, I did not go back and find the last copy of your config - so would you clarify: Are you LAGGing the ports to your ESXI hosts?

FWIW, I have not had issues with the SM dual 10gbe sfp+ which is a 520 card, nor chelsio 520's, nor CX312A's nor the 40Gbe version (using 40 or 10gbe with an step down adapter).

Hey thanks for the follow up

Nope i have removed the 6610-POE for the moment as that was a bridge too far with all the other issues i am having with my hosts. I have ordered the replacement memory module that Fodeesha recommended and will swap that in when it arrives and then set the 6610-poe back up in my testing system.

No i do not have any lagging to the hosts - i am trying to get two seperate 10G links to the hosts - one of which will go to a dedicated vSwitch for NFS and vMotion traffic and the other for VM communications - the onboard 1GB card is then used for Management.

It has got me baffled - definitely something weird happening - its like the slightest change of config sets something in the switch to tell it not to enable the port again.

I am trying to methodically step through and test everything but it is doing my head in !

I could blame the non certified Optiplex PCs - but it was also happening with (non certified) HP units also.

I have just retired an R710 here from a customer with ESXI 5.5. on it - my plan tomorrow is to fire it up and try it to see if i can make any more progress

Craig

 

[AndroidCat]

OK been through it and did the following

conf t
int e 1/2/1 to 1/2/10
no spanning-tree

wr mem

No change at the ESXi host

Rebooted the Host - no difference

logged back into the switch and did a disable and enable on the ports

No change at the ESXI host

Pretty much at my wits end now

Got one of the 3 hosts running on one switch and another on a different switch - one using Arista breakout cables and the other using FC.com breakouts

Tried with Dualport Intel 520DA-2 card (Intel 82599 and ixgben drivers in ESXi)
Tried with 2 x Single port Intel 520 card
Tried with Intel Dual port 540T cards and Twisted pair transceivers in the 1/3/x slots

I am waiting for some dual port Mellanox CX-3 adapters to arrive (CX312a)

Not sure what else i can do

Anyone ?

Craig

Sorry to hear that. It definitely helped in my case, even though the logic behind this had been unclear to me. Furthermore I also tried to find any log relevant to that port blocking and wasn't able to spot anything.
If that helps any, I've been using Mellanox X3 (flashed to ethernet) with both 10G and 40G links towards 6610. I don't have any LAG towards ESXi, it's all active+spare configuration (40G active + 10G spare).
Also ESXi 6.7 and 7.0 behaved exactly the same just reporting physical port down.

I'd ask @fohdeesha for any hints if you can 100% rule out HW/optics/DACs.

 

[Craig Curtin]

Sorry to hear that. It definitely helped in my case, even though the logic behind this had been unclear to me. Furthermore I also tried to find any log relevant to that port blocking and wasn't able to spot anything.
If that helps any, I've been using Mellanox X3 (flashed to ethernet) with both 10G and 40G links towards 6610. I don't have any LAG towards ESXi, it's all active+spare configuration (40G active + 10G spare).
Also ESXi 6.7 and 7.0 behaved exactly the same just reporting physical port down.

I'd ask @fohdeesha for any hints if you can 100% rule out HW/optics/DACs.

Thanks for the follow up.

Yep i am pretty sure that i have ruled out almost everything

Started on switch #1 (6610) with FS.COM QSFP to SFP+ breakouts - thats where the problems started - and i thought first it was the FS cables (had 3 of them) so tried each one and same problems - so that started me down the path that it was a cable issue. Then purchased some working Arista cables (same QSFP to SFP+ breakouts) and got the same sorts of intermittent issues.

So decided it must be a switch problem (even though it felt very much like a Spanning tree issue)

So purchased a 6610POE and cabled it to the 6610 with a Dell SFP+ DAC 3 metre cable (one that i had on hand that i have had for about 6 years)

Thats when i started having the problems with rebooting of the POE switch - so gave up on that and swapped in a spare 6610 - so i now have the first 6610 which has 2 x 1GB copper links in a Trunk/Channel to my Cisco 4948 (that i am trying to retire) - which is where nearly all my devices are connected.

On this 6610 i have one FS.COM breakout for QSFP to SFP+ and have two of the ports on there connected to a single Dell host on a dual port card and that is working and is solid - as long as i do not make any VLAN changes or other config changes on the ports.

This 6610 also has a fibre SFP+ module going to another Linux host that has not missed a beat at any point.

The 2nd 6610 is now mounted in my rack (and will become the permanent one) and is attached through a single copper DAC cable to one of the 1/3/x ports on the first 6610.

This is the one i am doing all the testing on at the moment and can not nail down.

So i am pretty confident it is not

a cable problem
a switch problem (as in faulty switch)
a card problem (although all of them i have tested have been Intel 82599 based)
a transceiver problem

Last night after a switch restart with the no-spanning tree lines on each of the 1/2/x ports and a host restart i now have two Intel cards in the one host talking to that switch - i will do more VLAN changes and updates tonight and see if it breaks again.

The other thing that sort of points to some form of spanning tree issue is that the port/cable remains blocked - rather than a card issue on the host i.e. if i start plugged into (say) 1/2/2 and it has a problem and drops out - then i can not connect that to anything else and get it to come back up - but i can take the cable for say 1/2/3 and connect that to the same port on the same host and it comes back up at both the ESXi level and the switch Int Brief level.

There must be a table somewhere on the switch of ports that are blocked for whatever reason - but it is not being reported in the logs (or to the sysylog server i have setup) - nor anywhere else i can find



Craig

 

[Craig Curtin]

Thanks for the follow up.

Yep i am pretty sure that i have ruled out almost everything

Started on switch #1 (6610) with FS.COM QSFP to SFP+ breakouts - thats where the problems started - and i thought first it was the FS cables (had 3 of them) so tried each one and same problems - so that started me down the path that it was a cable issue. Then purchased some working Arista cables (same QSFP to SFP+ breakouts) and got the same sorts of intermittent issues.

So decided it must be a switch problem (even though it felt very much like a Spanning tree issue)

So purchased a 6610POE and cabled it to the 6610 with a Dell SFP+ DAC 3 metre cable (one that i had on hand that i have had for about 6 years)

Thats when i started having the problems with rebooting of the POE switch - so gave up on that and swapped in a spare 6610 - so i now have the first 6610 which has 2 x 1GB copper links in a Trunk/Channel to my Cisco 4948 (that i am trying to retire) - which is where nearly all my devices are connected.

On this 6610 i have one FS.COM breakout for QSFP to SFP+ and have two of the ports on there connected to a single Dell host on a dual port card and that is working and is solid - as long as i do not make any VLAN changes or other config changes on the ports.

This 6610 also has a fibre SFP+ module going to another Linux host that has not missed a beat at any point.

The 2nd 6610 is now mounted in my rack (and will become the permanent one) and is attached through a single copper DAC cable to one of the 1/3/x ports on the first 6610.

This is the one i am doing all the testing on at the moment and can not nail down.

So i am pretty confident it is not

a cable problem
a switch problem (as in faulty switch)
a card problem (although all of them i have tested have been Intel 82599 based)
a transceiver problem

Last night after a switch restart with the no-spanning tree lines on each of the 1/2/x ports and a host restart i now have two Intel cards in the one host talking to that switch - i will do more VLAN changes and updates tonight and see if it breaks again.

The other thing that sort of points to some form of spanning tree issue is that the port/cable remains blocked - rather than a card issue on the host i.e. if i start plugged into (say) 1/2/2 and it has a problem and drops out - then i can not connect that to anything else and get it to come back up - but i can take the cable for say 1/2/3 and connect that to the same port on the same host and it comes back up at both the ESXi level and the switch Int Brief level.

There must be a table somewhere on the switch of ports that are blocked for whatever reason - but it is not being reported in the logs (or to the sysylog server i have setup) - nor anywhere else i can find



Craig

Still not making a lot of progress

I took a port that was fine 1/2/10 cabled it into a host that had previously been connected on another 6610 - port came up immediately on both the Host and Switch and looked fine

Then went and addedd it to a single VLAN as tagged and the port immediately dropped at the ESXI host and the switch shows it in blocking mode





Any ideas from the brains trust on this ? @Fodeesha - i have a kidney i can donate if you can work this out or even give me a hint

Craig

 

[itronin]

@Craig Curtin

if it walks like a duck quacks like a duck its a platypus. okay not a great analogy but maybe culturally appropriate... ? <-- I jest I jest (and guess)
Still smells like STP in some way almost like the switch is hearing itself from ESX? CDP? something really weird there for sure.

Dumb question:

Do you have ANOTHER PC / Server that you can bare metal a vlan capable OS on and try the same thing to see what happens?
if it works to me that would say something NOT right in ESX - would not explain how it worked on another mfg switch but ya never know...

Personally I think this is a better route if you can swing it than taking a scalpel to your back. but IDK.

 

[vangoose]

Still not making a lot of progress

I took a port that was fine 1/2/10 cabled it into a host that had previously been connected on another 6610 - port came up immediately on both the Host and Switch and looked fine

Then went and addedd it to a single VLAN as tagged and the port immediately dropped at the ESXI host and the switch shows it in blocking mode

View attachment 25426

View attachment 25427

Any ideas from the brains trust on this ? @Fodeesha - i have a kidney i can donate if you can work this out or even give me a hint

Craig

How many vlan have been configured and how stp is configured per vlan? And how your new switch is connected to the old switch?
I also think it's a stp issue.

Something like this?

6610----------old switch

  \                  /

    \             /

        host

 

[Craig Curtin]

@Craig Curtin

if it walks like a duck quacks like a duck its a platypus. okay not a great analogy but maybe culturally appropriate... ? <-- I jest I jest (and guess)
Still smells like STP in some way almost like the switch is hearing itself from ESX? CDP? something really weird there for sure.

Dumb question:

Do you have ANOTHER PC / Server that you can bare metal a vlan capable OS on and try the same thing to see what happens?
if it works to me that would say something NOT right in ESX - would not explain how it worked on another mfg switch but ya never know...

Personally I think this is a better route if you can swing it than taking a scalpel to your back. but IDK.

Yep i have another PC here that i could put Linux onto and start playing around with to see where that gets us - good suggestion - will fire up a machine and report back

Craig

 

[itronin]

Yep i have another PC here that i could put Linux onto and start playing around with to see where that gets us - good suggestion - will fire up a machine and report back

Craig

awesome!

though if you do decide to self remove a kidney please do take video and share!

 

[Craig Curtin]

How many vlan have been configured and how stp is configured per vlan? And how your new switch is connected to the old switch?
I also think it's a stp issue.

Something like this?

6610----------old switch

  \                  /

    \             /

        host

I have 6 VLANs configured
STP is off for all the VLANs

I have the old 6610 connected to the new one with a single SFP+ cable

This problem was happening before i introduced the 2nd 6610 - this problem was what made me introduce the 2nd switch so i could get an idea of what was happening and perform reboots etc without affecting my Linux NFS host that is connected to the 1/3/8 port on the original switch

So the sequence is something like this

1) Purchase and runup original 6610 - with a view to replace current Cisco 4948-10G
2) Deploy 6610 into same cabinet as 4948 and connect with two 1GB ports setup in a Trunk
3) Confirmed traffic flow all OK and could manage the 6610 by jumping across from devices on 4948 to the 6610
4) Setup VLANs on the trunk to carry all my traffic across to the 6610
5) Took first ESXI 7 host offline - this had previously had 4 x 1GB ports in a bond on a vswitch into the cisco and a seperate 1GB (onboard) management port
6) Added FS.COM QSFP breakout cable to the 6610, remove 4 x 1GB network adapter and replaced it with Intel 520DA-2
7) Move that ESXI management port (onboard Intel) from Cisco 4948 to 6610 and added into management VLAN and had it come back up into the Vsphere Vcentre server - so traffic flowing for that VLAN
8) Added VLANs to the 1/2/2 port for VMNetwork traffic and connected to the host to that and verifed comms were OK
9) Added a Storage VLAN (200) to the 1/2/3 port and attempted to connect that to the host - this is on a seperate vSwitch in ESXI with a VMKernel port and is to be used for NFS datastore and vMotion traffic. This VLAN is exclusively for that traffic and is not used on the 1st port on the adapter.

Thats when the problems started and i started going insane

The 2nd port refused to come up - so figured it must be a dud cable (they were purchased brand new but old stock) so i tried it on 1/2/4 and then 1/2/5 thinking it was a bad port all along.

As i did not have any production traffic on there at this stage i started a sequence of reboots between the Host and the switch - with no conclusive result that X action would re-enable the port.

So i went away and purchased another two of the FS.COM cables from the same vendor - with the same results.

At this stage i thought it may have been my old hosts (HP Elite 8300) were having problems with the new cards - so i brought forward the upgrade of my lab that i had planned for XMAS and started to deploy these newer Dell Optiplex 7050 units - exactly the same problem.

So to try and narrow it down i got some Dell 10G-BaseT transceivers and a couple of Intel 540T dual port cards and went through the identical problem.

So i decided it must be a dud switch in some fashion and purchased another two on EBAY from the US

One of them was the 6610 POE that i mentioned in some earlier posts - that was giving me random reboots so that is now pulled out - and the 2nd one was another 6610 - thats when i introduced it and connected it through an old SFP+ DAC cable that i had - the connection came up straight away and i started back down the rabbit burrow !

Thats where we are at now

I will try the previous posters suggest of firing up a Linux box and see where we get to (i already have one connected with an AOC cable to 1/3/8 on the first switch - but can not really play with that as that is the key box in the house with docker containers, media and the like.)

Craig

Craig

 

[Craig Curtin]

OK fired up a Debian 11 box and put one of the Dual port adapters into it. Left it with no VLAN config to start with

Initial testing

1) Connect to a port on the 6610 that is not "blocked" - comes up straight away and lights up the card
2) Connect to a port that is "disabled/blocked" and neither the switch nor the card think there is a connection there
3) Connect the same card to the other switch through the FS.COM cable and the ports come up straight away (with no VLAN)

So it does not look like Linux magically clears whatever is wrong with the port on the switch

I have just received 3 x Mellanox 312a adapters - so will flash one of those for ethernet and try it under Windows 10 and see what happens

Will report back once i have done some more testing on the Linux side also

Craig

 

[Mushishi]

@Craig Curtin not sure if you have checked this as I did try to read trough the posts while i was a work so was only able to skim them.

If you go into one of the interfaces that do not work from the cli and do a show interfaces do you then get this line on page 2 almost at the bottom:

GigabitEthernet1/1/2 is disabled, line protocol is down

(That was from a random port that I did disable on my 6450)

If you see that then try a enable to see if that brings the port up then.

If it have been tested then disregard my thought while I get more coffee.

 

[blue172]

Are all QSFP+ Ports usable on the 6610-24 or 6610-48 ? I need 4x 40G ETH. Is this possible ?

edit. NO found it on FCX / ICX6610 - Fohdeesha Docs

 

[itronin]

So it does not look like Linux magically clears whatever is wrong with the port on the switch

sorry I should have been more clear.

My motivation is to change your point of view - lest you continue to go crazy - cause it sounds like from your descriptions you are looking at the problem primarily from the network equipment POV.

My suggestion was meant to follow the same process as you do with your ESX host using a clean simpler config'ed host which implies that the switch has all ports up (starting point with previous changes undone). follow the same process with this new host. Connect the host, add the vlan. do you get a port block?

If not that implies that there may be something "special" about your existing host config and not the switch that is causing the problem...

If the clean host worked - then I'd consider (maybe even using the same host) spinning up ESXIt, leaving everything as default as you can - then follow the same method step by step, connect host, test, add vlan, test, maybe add more vlans, test - do you get a port block?

If not then start your "normal" post ESX installation process (to make this host very similiar to your others) and testing to see if you get a port block along the way... when/if you do - what steps had you just performed? back those out. test again.

My gut instinct says you've exhausted all the T/S switch side - maybe time to look at things from an OS POV?


Yes, what you had worked with your previous network configuration - but this is new (to you) gear - maybe some interactions with some features (you use and have configured host side) behave differently?

If only this was something simple like multicast! (HA!)

- edit -

if these steps don't work or are too much effort then it may also be time to repost your CURRENT switch config - or configs - post-change-that-blocks and pre - change-that-blocks - as well as screen shots of your ESX network configuration, vmks, vswitch, port groups, and NIC configurations. <-- lotta effort in this last ask - hoping that a clean system and step by step testing can clear this up rather than the painful here's what I have.

Still feels like an STP issue (only thing I've seen port which blocks that just don't go away easily is STP) but WHERE is it coming from if it is STP?

edit 2 - just making sure you saw this.

Enabling or disabling STP globally

docs.ruckuswireless.com

 

[Craig Curtin]

@Craig Curtin not sure if you have checked this as I did try to read trough the posts while i was a work so was only able to skim them.

If you go into one of the interfaces that do not work from the cli and do a show interfaces do you then get this line on page 2 almost at the bottom:

GigabitEthernet1/1/2 is disabled, line protocol is down

(That was from a random port that I did disable on my 6450)

If you see that then try a enable to see if that brings the port up then.

If it have been tested then disregard my thought while I get more coffee.

Thanks for taking the time to read and respond !

Yep i havegone through the enable/disable cycle - just trying to kick it in the guts in an way i can.

I have also cleared the whole config on the port - disabled it, then re-enabled it, connected a different device to it (so it should just come up as being in VLAN1 as the default VLAN) and no link at all.

The logs do not even acknowledge that something tries to connect.

Craig

 

[Craig Curtin]

sorry I should have been more clear.

My motivation is to change your point of view - lest you continue to go crazy - cause it sounds like from your descriptions you are looking at the problem primarily from the network equipment POV.

My suggestion was meant to follow the same process as you do with your ESX host using a clean simpler config'ed host which implies that the switch has all ports up (starting point with previous changes undone). follow the same process with this new host. Connect the host, add the vlan. do you get a port block?

If not that implies that there may be something "special" about your existing host config and not the switch that is causing the problem...

If the clean host worked - then I'd consider (maybe even using the same host) spinning up ESXIt, leaving everything as default as you can - then follow the same method step by step, connect host, test, add vlan, test, maybe add more vlans, test - do you get a port block?

If not then start your "normal" post ESX installation process (to make this host very similiar to your others) and testing to see if you get a port block along the way... when/if you do - what steps had you just performed? back those out. test again.

My gut instinct says you've exhausted all the T/S switch side - maybe time to look at things from an OS POV?


Yes, what you had worked with your previous network configuration - but this is new (to you) gear - maybe some interactions with some features (you use and have configured host side) behave differently?

If only this was something simple like multicast! (HA!)

- edit -

if these steps don't work or are too much effort then it may also be time to repost your CURRENT switch config - or configs - post-change-that-blocks and pre - change-that-blocks - as well as screen shots of your ESX network configuration, vmks, vswitch, port groups, and NIC configurations. <-- lotta effort in this last ask - hoping that a clean system and step by step testing can clear this up rather than the painful here's what I have.

Still feels like an STP issue (only thing I've seen port which blocks that just don't go away easily is STP) but WHERE is it coming from if it is STP?

edit 2 - just making sure you saw this.

Enabling or disabling STP globally

docs.ruckuswireless.com

Thanks mate - really appreciate you hanging in there.

Yep it definitely feels like an STP issue - and the only thing i can think of is something on the Cisco end as there is a Trunk between the two - but the Cisco is just chugging away and not reporting any ports being blocked - and is still allowing traffic across the link etc.

I do have my old ESXi hosts still sitting here - its not a lot of work to blow them away and reinstall from scratch and build up a simple ESXi config to do the testing - will start there - you are right that stepping back and seeing if i can reproduce it on a differnet OS is a good first step.

I will spend the day today cutting the original 6610 out of the link to the Cisco - so i can free it up to start from scratch and then work my way forward from there.

Will report in as i progress

[EDIT 1] Yes i saw the spanning tree stuff early in the piece - my initial assumption based on reading was that Spanning Tree was disabled globally in the L3 mode - and could then be controlled through VLANs.

In my case if i do

show 802-1w i get this



I have then manually disabled it on the 1/2/1 to 1/2/10 ports as well



Craig

 

[Mushishi]

Thanks for taking the time to read and respond !

Yep i havegone through the enable/disable cycle - just trying to kick it in the guts in an way i can.

I have also cleared the whole config on the port - disabled it, then re-enabled it, connected a different device to it (so it should just come up as being in VLAN1 as the default VLAN) and no link at all.

The logs do not even acknowledge that something tries to connect.

Craig

Well extra coffee for me at work tomorrow then. It just reminded me of the problem I had the first time I removed a LAG on my 6450 where only the Primary Port was active and the rest of the ports was disabled.

 

[itronin]

You're very welcome! I may need some cisco help in the future - will be sure to give your PM a jingle! ;-).

Thanks mate - really appreciate you hanging in there.

Yep it definitely feels like an STP issue - and the only thing i can think of is something on the Cisco end as there is a Trunk between the two - but the Cisco is just chugging away and not reporting any ports being blocked - and is still allowing traffic across the link etc.

I do have my old ESXi hosts still sitting here - its not a lot of work to blow them away and reinstall from scratch and build up a simple ESXi config to do the testing - will start there - you are right that stepping back and seeing if i can reproduce it on a differnet OS is a good first step.

I will spend the day today cutting the original 6610 out of the link to the Cisco - so i can free it up to start from scratch and then work my way forward from there.

Will report in as i progress

[EDIT 1] Yes i saw the spanning tree stuff early in the piece - my initial assumption based on reading was that Spanning Tree was disabled globally in the L3 mode - and could then be controlled through VLANs.

the only thing I can think of STP wise is if somehow you've bridged VLAN traffic inside ESX - like ESX management traffic (if its on your Cisco) to another vlan that is on your 10gb and going back to the cisco - but I would have expected the Cisco to shutdown the port when you were hooked up over there - then again maybe cisco's behavior is to shut down a VLAN on a trunk rather than all traffic? - somehow someway - I believe its possible - never explored it... I've done some odd stuff (internal vm to vm private networks and the like for nested storage back to the host) but not bridging.

might be worth getting some screenshots of your vswitches and port groups anyway - might be something that fresh eyes see? don't want to ask you to do extra work - but ya never know!