Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[itronin]

@Craig Curtin

if it walks like a duck quacks like a duck its a platypus. okay not a great analogy but maybe culturally appropriate... ? <-- I jest I jest (and guess)
Still smells like STP in some way almost like the switch is hearing itself from ESX? CDP? something really weird there for sure.

Dumb question:

Do you have ANOTHER PC / Server that you can bare metal a vlan capable OS on and try the same thing to see what happens?
if it works to me that would say something NOT right in ESX - would not explain how it worked on another mfg switch but ya never know...

Personally I think this is a better route if you can swing it than taking a scalpel to your back. but IDK.

 

[vangoose]

Still not making a lot of progress

I took a port that was fine 1/2/10 cabled it into a host that had previously been connected on another 6610 - port came up immediately on both the Host and Switch and looked fine

Then went and addedd it to a single VLAN as tagged and the port immediately dropped at the ESXI host and the switch shows it in blocking mode

View attachment 25426

View attachment 25427

Any ideas from the brains trust on this ? @Fodeesha - i have a kidney i can donate if you can work this out or even give me a hint

Craig

How many vlan have been configured and how stp is configured per vlan? And how your new switch is connected to the old switch?
I also think it's a stp issue.

Something like this?

6610----------old switch

  \                  /

    \             /

        host

 

[Craig Curtin]

@Craig Curtin

if it walks like a duck quacks like a duck its a platypus. okay not a great analogy but maybe culturally appropriate... ? <-- I jest I jest (and guess)
Still smells like STP in some way almost like the switch is hearing itself from ESX? CDP? something really weird there for sure.

Dumb question:

Do you have ANOTHER PC / Server that you can bare metal a vlan capable OS on and try the same thing to see what happens?
if it works to me that would say something NOT right in ESX - would not explain how it worked on another mfg switch but ya never know...

Personally I think this is a better route if you can swing it than taking a scalpel to your back. but IDK.

Yep i have another PC here that i could put Linux onto and start playing around with to see where that gets us - good suggestion - will fire up a machine and report back

Craig

 

[itronin]

Yep i have another PC here that i could put Linux onto and start playing around with to see where that gets us - good suggestion - will fire up a machine and report back

Craig

awesome!

though if you do decide to self remove a kidney please do take video and share!

 

[Craig Curtin]

How many vlan have been configured and how stp is configured per vlan? And how your new switch is connected to the old switch?
I also think it's a stp issue.

Something like this?

6610----------old switch

  \                  /

    \             /

        host

I have 6 VLANs configured
STP is off for all the VLANs

I have the old 6610 connected to the new one with a single SFP+ cable

This problem was happening before i introduced the 2nd 6610 - this problem was what made me introduce the 2nd switch so i could get an idea of what was happening and perform reboots etc without affecting my Linux NFS host that is connected to the 1/3/8 port on the original switch

So the sequence is something like this

1) Purchase and runup original 6610 - with a view to replace current Cisco 4948-10G
2) Deploy 6610 into same cabinet as 4948 and connect with two 1GB ports setup in a Trunk
3) Confirmed traffic flow all OK and could manage the 6610 by jumping across from devices on 4948 to the 6610
4) Setup VLANs on the trunk to carry all my traffic across to the 6610
5) Took first ESXI 7 host offline - this had previously had 4 x 1GB ports in a bond on a vswitch into the cisco and a seperate 1GB (onboard) management port
6) Added FS.COM QSFP breakout cable to the 6610, remove 4 x 1GB network adapter and replaced it with Intel 520DA-2
7) Move that ESXI management port (onboard Intel) from Cisco 4948 to 6610 and added into management VLAN and had it come back up into the Vsphere Vcentre server - so traffic flowing for that VLAN
8) Added VLANs to the 1/2/2 port for VMNetwork traffic and connected to the host to that and verifed comms were OK
9) Added a Storage VLAN (200) to the 1/2/3 port and attempted to connect that to the host - this is on a seperate vSwitch in ESXI with a VMKernel port and is to be used for NFS datastore and vMotion traffic. This VLAN is exclusively for that traffic and is not used on the 1st port on the adapter.

Thats when the problems started and i started going insane

The 2nd port refused to come up - so figured it must be a dud cable (they were purchased brand new but old stock) so i tried it on 1/2/4 and then 1/2/5 thinking it was a bad port all along.

As i did not have any production traffic on there at this stage i started a sequence of reboots between the Host and the switch - with no conclusive result that X action would re-enable the port.

So i went away and purchased another two of the FS.COM cables from the same vendor - with the same results.

At this stage i thought it may have been my old hosts (HP Elite 8300) were having problems with the new cards - so i brought forward the upgrade of my lab that i had planned for XMAS and started to deploy these newer Dell Optiplex 7050 units - exactly the same problem.

So to try and narrow it down i got some Dell 10G-BaseT transceivers and a couple of Intel 540T dual port cards and went through the identical problem.

So i decided it must be a dud switch in some fashion and purchased another two on EBAY from the US

One of them was the 6610 POE that i mentioned in some earlier posts - that was giving me random reboots so that is now pulled out - and the 2nd one was another 6610 - thats when i introduced it and connected it through an old SFP+ DAC cable that i had - the connection came up straight away and i started back down the rabbit burrow !

Thats where we are at now

I will try the previous posters suggest of firing up a Linux box and see where we get to (i already have one connected with an AOC cable to 1/3/8 on the first switch - but can not really play with that as that is the key box in the house with docker containers, media and the like.)

Craig

Craig

 

[Craig Curtin]

OK fired up a Debian 11 box and put one of the Dual port adapters into it. Left it with no VLAN config to start with

Initial testing

1) Connect to a port on the 6610 that is not "blocked" - comes up straight away and lights up the card
2) Connect to a port that is "disabled/blocked" and neither the switch nor the card think there is a connection there
3) Connect the same card to the other switch through the FS.COM cable and the ports come up straight away (with no VLAN)

So it does not look like Linux magically clears whatever is wrong with the port on the switch

I have just received 3 x Mellanox 312a adapters - so will flash one of those for ethernet and try it under Windows 10 and see what happens

Will report back once i have done some more testing on the Linux side also

Craig

 

[Mushishi]

@Craig Curtin not sure if you have checked this as I did try to read trough the posts while i was a work so was only able to skim them.

If you go into one of the interfaces that do not work from the cli and do a show interfaces do you then get this line on page 2 almost at the bottom:

GigabitEthernet1/1/2 is disabled, line protocol is down

(That was from a random port that I did disable on my 6450)

If you see that then try a enable to see if that brings the port up then.

If it have been tested then disregard my thought while I get more coffee.

 

[blue172]

Are all QSFP+ Ports usable on the 6610-24 or 6610-48 ? I need 4x 40G ETH. Is this possible ?

edit. NO found it on FCX / ICX6610 - Fohdeesha Docs

 

[itronin]

So it does not look like Linux magically clears whatever is wrong with the port on the switch

sorry I should have been more clear.

My motivation is to change your point of view - lest you continue to go crazy - cause it sounds like from your descriptions you are looking at the problem primarily from the network equipment POV.

My suggestion was meant to follow the same process as you do with your ESX host using a clean simpler config'ed host which implies that the switch has all ports up (starting point with previous changes undone). follow the same process with this new host. Connect the host, add the vlan. do you get a port block?

If not that implies that there may be something "special" about your existing host config and not the switch that is causing the problem...

If the clean host worked - then I'd consider (maybe even using the same host) spinning up ESXIt, leaving everything as default as you can - then follow the same method step by step, connect host, test, add vlan, test, maybe add more vlans, test - do you get a port block?

If not then start your "normal" post ESX installation process (to make this host very similiar to your others) and testing to see if you get a port block along the way... when/if you do - what steps had you just performed? back those out. test again.

My gut instinct says you've exhausted all the T/S switch side - maybe time to look at things from an OS POV?


Yes, what you had worked with your previous network configuration - but this is new (to you) gear - maybe some interactions with some features (you use and have configured host side) behave differently?

If only this was something simple like multicast! (HA!)

- edit -

if these steps don't work or are too much effort then it may also be time to repost your CURRENT switch config - or configs - post-change-that-blocks and pre - change-that-blocks - as well as screen shots of your ESX network configuration, vmks, vswitch, port groups, and NIC configurations. <-- lotta effort in this last ask - hoping that a clean system and step by step testing can clear this up rather than the painful here's what I have.

Still feels like an STP issue (only thing I've seen port which blocks that just don't go away easily is STP) but WHERE is it coming from if it is STP?

edit 2 - just making sure you saw this.

Enabling or disabling STP globally

docs.ruckuswireless.com

 

[Craig Curtin]

@Craig Curtin not sure if you have checked this as I did try to read trough the posts while i was a work so was only able to skim them.

If you go into one of the interfaces that do not work from the cli and do a show interfaces do you then get this line on page 2 almost at the bottom:

GigabitEthernet1/1/2 is disabled, line protocol is down

(That was from a random port that I did disable on my 6450)

If you see that then try a enable to see if that brings the port up then.

If it have been tested then disregard my thought while I get more coffee.

Thanks for taking the time to read and respond !

Yep i havegone through the enable/disable cycle - just trying to kick it in the guts in an way i can.

I have also cleared the whole config on the port - disabled it, then re-enabled it, connected a different device to it (so it should just come up as being in VLAN1 as the default VLAN) and no link at all.

The logs do not even acknowledge that something tries to connect.

Craig

 

[Craig Curtin]

sorry I should have been more clear.

My motivation is to change your point of view - lest you continue to go crazy - cause it sounds like from your descriptions you are looking at the problem primarily from the network equipment POV.

My suggestion was meant to follow the same process as you do with your ESX host using a clean simpler config'ed host which implies that the switch has all ports up (starting point with previous changes undone). follow the same process with this new host. Connect the host, add the vlan. do you get a port block?

If not that implies that there may be something "special" about your existing host config and not the switch that is causing the problem...

If the clean host worked - then I'd consider (maybe even using the same host) spinning up ESXIt, leaving everything as default as you can - then follow the same method step by step, connect host, test, add vlan, test, maybe add more vlans, test - do you get a port block?

If not then start your "normal" post ESX installation process (to make this host very similiar to your others) and testing to see if you get a port block along the way... when/if you do - what steps had you just performed? back those out. test again.

My gut instinct says you've exhausted all the T/S switch side - maybe time to look at things from an OS POV?


Yes, what you had worked with your previous network configuration - but this is new (to you) gear - maybe some interactions with some features (you use and have configured host side) behave differently?

If only this was something simple like multicast! (HA!)

- edit -

if these steps don't work or are too much effort then it may also be time to repost your CURRENT switch config - or configs - post-change-that-blocks and pre - change-that-blocks - as well as screen shots of your ESX network configuration, vmks, vswitch, port groups, and NIC configurations. <-- lotta effort in this last ask - hoping that a clean system and step by step testing can clear this up rather than the painful here's what I have.

Still feels like an STP issue (only thing I've seen port which blocks that just don't go away easily is STP) but WHERE is it coming from if it is STP?

edit 2 - just making sure you saw this.

Enabling or disabling STP globally

docs.ruckuswireless.com

Thanks mate - really appreciate you hanging in there.

Yep it definitely feels like an STP issue - and the only thing i can think of is something on the Cisco end as there is a Trunk between the two - but the Cisco is just chugging away and not reporting any ports being blocked - and is still allowing traffic across the link etc.

I do have my old ESXi hosts still sitting here - its not a lot of work to blow them away and reinstall from scratch and build up a simple ESXi config to do the testing - will start there - you are right that stepping back and seeing if i can reproduce it on a differnet OS is a good first step.

I will spend the day today cutting the original 6610 out of the link to the Cisco - so i can free it up to start from scratch and then work my way forward from there.

Will report in as i progress

[EDIT 1] Yes i saw the spanning tree stuff early in the piece - my initial assumption based on reading was that Spanning Tree was disabled globally in the L3 mode - and could then be controlled through VLANs.

In my case if i do

show 802-1w i get this



I have then manually disabled it on the 1/2/1 to 1/2/10 ports as well



Craig

 

[Mushishi]

Thanks for taking the time to read and respond !

Yep i havegone through the enable/disable cycle - just trying to kick it in the guts in an way i can.

I have also cleared the whole config on the port - disabled it, then re-enabled it, connected a different device to it (so it should just come up as being in VLAN1 as the default VLAN) and no link at all.

The logs do not even acknowledge that something tries to connect.

Craig

Well extra coffee for me at work tomorrow then. It just reminded me of the problem I had the first time I removed a LAG on my 6450 where only the Primary Port was active and the rest of the ports was disabled.

 

[itronin]

You're very welcome! I may need some cisco help in the future - will be sure to give your PM a jingle! ;-).

Thanks mate - really appreciate you hanging in there.

Yep it definitely feels like an STP issue - and the only thing i can think of is something on the Cisco end as there is a Trunk between the two - but the Cisco is just chugging away and not reporting any ports being blocked - and is still allowing traffic across the link etc.

I do have my old ESXi hosts still sitting here - its not a lot of work to blow them away and reinstall from scratch and build up a simple ESXi config to do the testing - will start there - you are right that stepping back and seeing if i can reproduce it on a differnet OS is a good first step.

I will spend the day today cutting the original 6610 out of the link to the Cisco - so i can free it up to start from scratch and then work my way forward from there.

Will report in as i progress

[EDIT 1] Yes i saw the spanning tree stuff early in the piece - my initial assumption based on reading was that Spanning Tree was disabled globally in the L3 mode - and could then be controlled through VLANs.

the only thing I can think of STP wise is if somehow you've bridged VLAN traffic inside ESX - like ESX management traffic (if its on your Cisco) to another vlan that is on your 10gb and going back to the cisco - but I would have expected the Cisco to shutdown the port when you were hooked up over there - then again maybe cisco's behavior is to shut down a VLAN on a trunk rather than all traffic? - somehow someway - I believe its possible - never explored it... I've done some odd stuff (internal vm to vm private networks and the like for nested storage back to the host) but not bridging.

might be worth getting some screenshots of your vswitches and port groups anyway - might be something that fresh eyes see? don't want to ask you to do extra work - but ya never know!

 

[AndroidCat]

@Craig Curtin
Have you tried with MTU of 1500 as well? I can see you have it set to 10k. I know it's layer 3, but worth trying with defaults.

 

[Craig Curtin]

@Craig Curtin
Have you tried with MTU of 1500 as well? I can see you have it set to 10k. I know it's layer 3, but worth trying with defaults.

I am only using the 9000 bytes on a single VLAN and on a single vswitch in ESXI (for the storage and vmotion traffic - that is running across VLAN 200) - reluctant to drop that back yet (but it may come to that !) as that will mean changing the Linux stuff also which is used elsewhere.

Now that i have freed up a 6610 (as of today) i can do some more testing without disrupting everything. So i will add that to the list - first thing i intend to do is try the spare 6610 with the current config to see if having it disconnected from everything else makes a difference

Then i can start on the ESXI and Linux side

might be worth getting some screenshots of your vswitches and port groups anyway - might be something that fresh eyes see? don't want to ask you to do extra work - but ya never know!

Yep i agree - will start on those tonight and put together another post

Craig

 

[Craig Curtin]

Ok now i know it must be something i am doing - here is the scenario for some background and tell me what i have done wrong

1) 6450 this time not a 6610
2) Had to do some work at a clients last night so took a 6450 in with me to test on their Dell R730 server - running ESXi 7.03 with an add-in Intel 540T-2
3) One port of the 540T is used in cross over direct to a linux box that acts as their backup repository - this has been working fine for 18 months or so.
4) I ran up the 6450 as per the Fodeesha docs - but stopped after i had updated the firmware etc - but before adding licenses etc
5) Put the 6450 into the same rack as the server and connected a Dell 10G-BaseT transceiver into the 1/2/3 port (just picked that port at random) and connected it to the 2nd unused port on the Intel card.
6) Immediately the port came up and showed activity and i went into Vsphere and looked at the NIC on the server and it was showing as 10GB full duplex connection - no vswitch etc configured at this point and no VLANs
7) I then went through the rest of the doc and finished the config on the 6450 and applied the licenses, enabled ssh login, disabled web login etc etc
8) did a reload for the new licenses to take effect and the switch came back up - connected to their live network (flat single netgear switch with only a single default vlan 1) - could then access the switch through SSH across the network and everything appeared to be OK - BUT the 10G NIC has now gone down on ESXi and is shown as down on the 6450 login also.

So to summarise at this point - no changes in VLANs, no config on the ESXi and the port has gone into down state and showing as blocked

So i then thought maybe the port did not get activated automatically at 10GB once licensed so went in and set all 4 ports to 10G full and turned off spanning tree.

Still will not come up

Now this server was built from the same ESXi image CD as my Lab machines so is using all the same drivers etc - here are a a heap of screenshots for anyone that has the time to look

So here is the adapters on the server - VMNIC 5 is the 2nd port that is in crossover to the Linux box, VMNIC4 is the 1st port that is now attached to the 6450

Not added to a vSwitch etc at this point



on the 6450 end





Show run on 6450

SSH@afsydney-6450-1#show run
Current configuration:
!
ver 08.0.30uT313
!
stack unit 1
module 1 icx6450-48p-poe-port-management-module
module 2 icx6450-sfp-plus-4port-40g-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
hostname afsydney-6450-1
ip dhcp-client disable
ip dns server-address 192.168.0.1
ip route 0.0.0.0/0 192.168.0.1
!
no telnet server
username root password .....
!
!
clock summer-time
clock timezone gmt GMT+11
!
!
ntp
disable serve
server 192.168.0.1
server 216.239.35.0
server 216.239.35.4
!
!
no web-management http
!
!
!
interface ethernet 1/2/1
no spanning-tree
!
interface ethernet 1/2/2
no spanning-tree
!
interface ethernet 1/2/3
no spanning-tree
!
interface ethernet 1/2/4
no spanning-tree
!
interface ve 1
ip address 192.168.0.240 255.255.255.0
!
!
!
!
end

SSH@afsydney-6450-1#

Show Media



Any ideas ?

Craig

 

[itronin]

Ok now i know it must be something i am doing - here is the scenario for some background and tell me what i have done wrong



Any ideas ?

Craig

Questions:

What DAC, or Optics are you using for *your* tests ?
Different from your client's optics?
What Optics vendor server side does your client use? I see SR 300M for the connection so assuming multi mode fiber/optics..

I know you said your lab has Melly CX3 - those are for sure non vendor locked SFP+

I know that some intel SFP+ cards (I thought just 520's) are vendor locked for DAC's (maybe optics too?)
Is that a DELL OEM NIC card? do you know if is vendor locked for DAC/optics?
I'd hope that is not the case with DELL stuff but I figured I'd ask - so obvious I'm sure this is not an issue for you but I do have to ask.

No screenshot but did you perform a show license ot make sure that they applied correctly?
My assumption is yes and as soon as the 10Gbe was enabled it got shutdown

Is this a DELL OEM ESXI image? If so in your lab I'd download a non OEM vanilla image - that said. I have absolutely used a DELL OEM 6.7 ESXI install connected to both a 6610 *and* 7250 and had no issues with Melly CX3's... Both Cisco DAC's and Brocade SR optics.


Asking questions to make sure that a separate issue didn't pop up into the middle of this.

 

[itronin]

4) I ran up the 6450 as per the Fodeesha docs - but stopped after i had updated the firmware etc - but before adding licenses etc
5) Put the 6450 into the same rack as the server and connected a Dell 10G-BaseT transceiver into the 1/2/3 port (just picked that port at random) and connected it to the 2nd unused port on the Intel card.


Any ideas ?

Craig

I believe 10gb ports 1 and 3 are stacking. Looking at your config... stacking enabled?
That could be the problem in this particular test.

edit - total guess on my part - don't have a 6450 - just read about when I was deciding what to buy... could be totally wrong guess.

FWIW on my 6610 I have to explicitly set speed and duplex on my FP 10Gbe ports. Don't know if that is the same on the 6450 but should
look something like:

!
interface ethernet 1/2/1
speed-duplex 10G-full
!
interface ethernet 1/2/2
speed-duplex 10G-full
!
interface ethernet 1/2/3
speed-duplex 10G-full
!
interface ethernet 1/2/4
speed-duplex 10G-full

edit - and using interfaces where /3/ becomes /2/ I simply changed it in the code section.

edit - further reading - looks like stacking is NOT enabled by default on 6450 and the default speed for the SFP+ ports is 10Gbe so the commands I put in are unnecessary.

 

[AndroidCat]

I believe 10gb ports 1 and 3 are stacking. Looking at your config... stacking enabled?
That could be the problem in this particular test.

edit - total guess on my part - don't have a 6450 - just read about when I was deciding what to buy... could be totally wrong guess.

FWIW on my 6610 I have to explicitly set speed and duplex on my FP 10Gbe ports. Don't know if that is the same on the 6450 but should
look something like:

!
interface ethernet 1/2/1
speed-duplex 10G-full
!
interface ethernet 1/2/2
speed-duplex 10G-full
!
interface ethernet 1/2/3
speed-duplex 10G-full
!
interface ethernet 1/2/4
speed-duplex 10G-full

edit - and using interfaces where /3/ becomes /2/ I simply changed it in the code section.

edit - further reading - looks like stacking is NOT enabled by default on 6450 and the default speed for the SFP+ ports is 10Gbe so the commands I put in are unnecessary.

In 6610 ports 1/2/x do not require any speed and duplex setting, but ports 1/3/x do require "speed-duplex 10G-full".
I don't know 6450.

 

[Craig Curtin]

Questions:

What DAC, or Optics are you using for *your* tests ?
Different from your client's optics?
What Optics vendor server side does your client use? I see SR 300M for the connection so assuming multi mode fiber/optics..

I know you said your lab has Melly CX3 - those are for sure non vendor locked SFP+

I know that some intel SFP+ cards (I thought just 520's) are vendor locked for DAC's (maybe optics too?)
Is that a DELL OEM NIC card? do you know if is vendor locked for DAC/optics?
I'd hope that is not the case with DELL stuff but I figured I'd ask - so obvious I'm sure this is not an issue for you but I do have to ask.

No screenshot but did you perform a show license ot make sure that they applied correctly?
My assumption is yes and as soon as the 10Gbe was enabled it got shutdown

Is this a DELL OEM ESXI image? If so in your lab I'd download a non OEM vanilla image - that said. I have absolutely used a DELL OEM 6.7 ESXI install connected to both a 6610 *and* 7250 and had no issues with Melly CX3's... Both Cisco DAC's and Brocade SR optics.


Asking questions to make sure that a separate issue didn't pop up into the middle of this.

On this latest one with the 6450 i was using a Dell 10GB-BaseT in the 1/2/3 slot - i do have some Dell DAC cables and could take them in and attach to the onboard SFP+ (two of these on the 730XD) and try a different port on the 6450

This Dell DAC was attached to the Intel 540T which is the native twisted pair and it was a genuine non-oem adapter (as much as you can be sure nowadays with these)

I have just received the Mellanox cards late this week - have not put them into production or testing anywhere - started flashing them last night for Ethernet mode - they are the dual SFP+ units - 312a i believe (top of my head)

I think ALL of my adapters to this point have been based on the Intel 82599 chipset - be they single port DAC, Dual Port DAC or Dual Port RJ45

Yes a lot of the Intel cards are locked for Fibre Optics but not for DACs from what i have been able to find in my searching over the last week

Yes i did check the licenses and they show the Fodeesha ones with the HackThePlanet license codes for both licenses.

Yes this is a Dell OEM ESXi download/install - on all of the Optiplexs in my home lab - however the HP/Compaqs that i started with were using a HP image with 7.01 ESXi - but still all using the same IXGBEN drivers

Earlier in the testing i did download the very latest ESXi certified ixgben drivers (this was on the HP/Compaqs) (top of my head they were September release)

The one common factor in all of this is that all of the cards have been using the IXGBEN drivers - so i would like to try a Mellanox to see what happens.

So the plan today is to get a vanilla ESXi image and put that onto one of the HP/Compaqs and try that with both the Intel and Mellanox cards on the now spare 6610 to see if i can create a reproducible issue and a path to resolve and recreate it

Craig