Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[AndroidCat]

@Craig Curtin
Have you tried with MTU of 1500 as well? I can see you have it set to 10k. I know it's layer 3, but worth trying with defaults.

 

[Craig Curtin]

@Craig Curtin
Have you tried with MTU of 1500 as well? I can see you have it set to 10k. I know it's layer 3, but worth trying with defaults.

I am only using the 9000 bytes on a single VLAN and on a single vswitch in ESXI (for the storage and vmotion traffic - that is running across VLAN 200) - reluctant to drop that back yet (but it may come to that !) as that will mean changing the Linux stuff also which is used elsewhere.

Now that i have freed up a 6610 (as of today) i can do some more testing without disrupting everything. So i will add that to the list - first thing i intend to do is try the spare 6610 with the current config to see if having it disconnected from everything else makes a difference

Then i can start on the ESXI and Linux side

might be worth getting some screenshots of your vswitches and port groups anyway - might be something that fresh eyes see? don't want to ask you to do extra work - but ya never know!

Yep i agree - will start on those tonight and put together another post

Craig

 

[Craig Curtin]

Ok now i know it must be something i am doing - here is the scenario for some background and tell me what i have done wrong

1) 6450 this time not a 6610
2) Had to do some work at a clients last night so took a 6450 in with me to test on their Dell R730 server - running ESXi 7.03 with an add-in Intel 540T-2
3) One port of the 540T is used in cross over direct to a linux box that acts as their backup repository - this has been working fine for 18 months or so.
4) I ran up the 6450 as per the Fodeesha docs - but stopped after i had updated the firmware etc - but before adding licenses etc
5) Put the 6450 into the same rack as the server and connected a Dell 10G-BaseT transceiver into the 1/2/3 port (just picked that port at random) and connected it to the 2nd unused port on the Intel card.
6) Immediately the port came up and showed activity and i went into Vsphere and looked at the NIC on the server and it was showing as 10GB full duplex connection - no vswitch etc configured at this point and no VLANs
7) I then went through the rest of the doc and finished the config on the 6450 and applied the licenses, enabled ssh login, disabled web login etc etc
8) did a reload for the new licenses to take effect and the switch came back up - connected to their live network (flat single netgear switch with only a single default vlan 1) - could then access the switch through SSH across the network and everything appeared to be OK - BUT the 10G NIC has now gone down on ESXi and is shown as down on the 6450 login also.

So to summarise at this point - no changes in VLANs, no config on the ESXi and the port has gone into down state and showing as blocked

So i then thought maybe the port did not get activated automatically at 10GB once licensed so went in and set all 4 ports to 10G full and turned off spanning tree.

Still will not come up

Now this server was built from the same ESXi image CD as my Lab machines so is using all the same drivers etc - here are a a heap of screenshots for anyone that has the time to look

So here is the adapters on the server - VMNIC 5 is the 2nd port that is in crossover to the Linux box, VMNIC4 is the 1st port that is now attached to the 6450

Not added to a vSwitch etc at this point



on the 6450 end





Show run on 6450

SSH@afsydney-6450-1#show run
Current configuration:
!
ver 08.0.30uT313
!
stack unit 1
module 1 icx6450-48p-poe-port-management-module
module 2 icx6450-sfp-plus-4port-40g-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
hostname afsydney-6450-1
ip dhcp-client disable
ip dns server-address 192.168.0.1
ip route 0.0.0.0/0 192.168.0.1
!
no telnet server
username root password .....
!
!
clock summer-time
clock timezone gmt GMT+11
!
!
ntp
disable serve
server 192.168.0.1
server 216.239.35.0
server 216.239.35.4
!
!
no web-management http
!
!
!
interface ethernet 1/2/1
no spanning-tree
!
interface ethernet 1/2/2
no spanning-tree
!
interface ethernet 1/2/3
no spanning-tree
!
interface ethernet 1/2/4
no spanning-tree
!
interface ve 1
ip address 192.168.0.240 255.255.255.0
!
!
!
!
end

SSH@afsydney-6450-1#

Show Media



Any ideas ?

Craig

 

[itronin]

Ok now i know it must be something i am doing - here is the scenario for some background and tell me what i have done wrong



Any ideas ?

Craig

Questions:

What DAC, or Optics are you using for *your* tests ?
Different from your client's optics?
What Optics vendor server side does your client use? I see SR 300M for the connection so assuming multi mode fiber/optics..

I know you said your lab has Melly CX3 - those are for sure non vendor locked SFP+

I know that some intel SFP+ cards (I thought just 520's) are vendor locked for DAC's (maybe optics too?)
Is that a DELL OEM NIC card? do you know if is vendor locked for DAC/optics?
I'd hope that is not the case with DELL stuff but I figured I'd ask - so obvious I'm sure this is not an issue for you but I do have to ask.

No screenshot but did you perform a show license ot make sure that they applied correctly?
My assumption is yes and as soon as the 10Gbe was enabled it got shutdown

Is this a DELL OEM ESXI image? If so in your lab I'd download a non OEM vanilla image - that said. I have absolutely used a DELL OEM 6.7 ESXI install connected to both a 6610 *and* 7250 and had no issues with Melly CX3's... Both Cisco DAC's and Brocade SR optics.


Asking questions to make sure that a separate issue didn't pop up into the middle of this.

 

[itronin]

4) I ran up the 6450 as per the Fodeesha docs - but stopped after i had updated the firmware etc - but before adding licenses etc
5) Put the 6450 into the same rack as the server and connected a Dell 10G-BaseT transceiver into the 1/2/3 port (just picked that port at random) and connected it to the 2nd unused port on the Intel card.


Any ideas ?

Craig

I believe 10gb ports 1 and 3 are stacking. Looking at your config... stacking enabled?
That could be the problem in this particular test.

edit - total guess on my part - don't have a 6450 - just read about when I was deciding what to buy... could be totally wrong guess.

FWIW on my 6610 I have to explicitly set speed and duplex on my FP 10Gbe ports. Don't know if that is the same on the 6450 but should
look something like:

!
interface ethernet 1/2/1
speed-duplex 10G-full
!
interface ethernet 1/2/2
speed-duplex 10G-full
!
interface ethernet 1/2/3
speed-duplex 10G-full
!
interface ethernet 1/2/4
speed-duplex 10G-full

edit - and using interfaces where /3/ becomes /2/ I simply changed it in the code section.

edit - further reading - looks like stacking is NOT enabled by default on 6450 and the default speed for the SFP+ ports is 10Gbe so the commands I put in are unnecessary.

 

[AndroidCat]

I believe 10gb ports 1 and 3 are stacking. Looking at your config... stacking enabled?
That could be the problem in this particular test.

edit - total guess on my part - don't have a 6450 - just read about when I was deciding what to buy... could be totally wrong guess.

FWIW on my 6610 I have to explicitly set speed and duplex on my FP 10Gbe ports. Don't know if that is the same on the 6450 but should
look something like:

!
interface ethernet 1/2/1
speed-duplex 10G-full
!
interface ethernet 1/2/2
speed-duplex 10G-full
!
interface ethernet 1/2/3
speed-duplex 10G-full
!
interface ethernet 1/2/4
speed-duplex 10G-full

edit - and using interfaces where /3/ becomes /2/ I simply changed it in the code section.

edit - further reading - looks like stacking is NOT enabled by default on 6450 and the default speed for the SFP+ ports is 10Gbe so the commands I put in are unnecessary.

In 6610 ports 1/2/x do not require any speed and duplex setting, but ports 1/3/x do require "speed-duplex 10G-full".
I don't know 6450.

 

[Craig Curtin]

Questions:

What DAC, or Optics are you using for *your* tests ?
Different from your client's optics?
What Optics vendor server side does your client use? I see SR 300M for the connection so assuming multi mode fiber/optics..

I know you said your lab has Melly CX3 - those are for sure non vendor locked SFP+

I know that some intel SFP+ cards (I thought just 520's) are vendor locked for DAC's (maybe optics too?)
Is that a DELL OEM NIC card? do you know if is vendor locked for DAC/optics?
I'd hope that is not the case with DELL stuff but I figured I'd ask - so obvious I'm sure this is not an issue for you but I do have to ask.

No screenshot but did you perform a show license ot make sure that they applied correctly?
My assumption is yes and as soon as the 10Gbe was enabled it got shutdown

Is this a DELL OEM ESXI image? If so in your lab I'd download a non OEM vanilla image - that said. I have absolutely used a DELL OEM 6.7 ESXI install connected to both a 6610 *and* 7250 and had no issues with Melly CX3's... Both Cisco DAC's and Brocade SR optics.


Asking questions to make sure that a separate issue didn't pop up into the middle of this.

On this latest one with the 6450 i was using a Dell 10GB-BaseT in the 1/2/3 slot - i do have some Dell DAC cables and could take them in and attach to the onboard SFP+ (two of these on the 730XD) and try a different port on the 6450

This Dell DAC was attached to the Intel 540T which is the native twisted pair and it was a genuine non-oem adapter (as much as you can be sure nowadays with these)

I have just received the Mellanox cards late this week - have not put them into production or testing anywhere - started flashing them last night for Ethernet mode - they are the dual SFP+ units - 312a i believe (top of my head)

I think ALL of my adapters to this point have been based on the Intel 82599 chipset - be they single port DAC, Dual Port DAC or Dual Port RJ45

Yes a lot of the Intel cards are locked for Fibre Optics but not for DACs from what i have been able to find in my searching over the last week

Yes i did check the licenses and they show the Fodeesha ones with the HackThePlanet license codes for both licenses.

Yes this is a Dell OEM ESXi download/install - on all of the Optiplexs in my home lab - however the HP/Compaqs that i started with were using a HP image with 7.01 ESXi - but still all using the same IXGBEN drivers

Earlier in the testing i did download the very latest ESXi certified ixgben drivers (this was on the HP/Compaqs) (top of my head they were September release)

The one common factor in all of this is that all of the cards have been using the IXGBEN drivers - so i would like to try a Mellanox to see what happens.

So the plan today is to get a vanilla ESXi image and put that onto one of the HP/Compaqs and try that with both the Intel and Mellanox cards on the now spare 6610 to see if i can create a reproducible issue and a path to resolve and recreate it

Craig

 

[Craig Curtin]

I believe 10gb ports 1 and 3 are stacking. Looking at your config... stacking enabled?
That could be the problem in this particular test.

edit - total guess on my part - don't have a 6450 - just read about when I was deciding what to buy... could be totally wrong guess.

FWIW on my 6610 I have to explicitly set speed and duplex on my FP 10Gbe ports. Don't know if that is the same on the 6450 but should
look something like:

!
interface ethernet 1/2/1
speed-duplex 10G-full
!
interface ethernet 1/2/2
speed-duplex 10G-full
!
interface ethernet 1/2/3
speed-duplex 10G-full
!
interface ethernet 1/2/4
speed-duplex 10G-full

edit - and using interfaces where /3/ becomes /2/ I simply changed it in the code section.

edit - further reading - looks like stacking is NOT enabled by default on 6450 and the default speed for the SFP+ ports is 10Gbe so the commands I put in are unnecessary.

Yep - i tried the speed-duplex commands on the 6450 last night and they were accepted but do not show up in the config anywhere.

Yes i believe that stacking on the 6450 is something that has to be enabled rather than defaulted to (again just from my reading and looking at the default config as generated)

Craig

 

[itronin]

In 6610 ports 1/2/x do not require any speed and duplex setting, but ports 1/3/x do require "speed-duplex 10G-full".
I don't know 6450.

I edited what I posted to match the module number on the 6450. correct on 6610 x/2/x does not require speed setting.

 

[AndroidCat]

@Craig Curtin
I found the original explanation about spanning tree, the one I had mentioned 2 pages before (which did not help you).
Quote:


STP is off by default with the l3 OS image (off globally and off per vlan), but remains "ON" per port, meaning ports are eligible to participate in an STP topology, but one hasn't been defined globally so they don't perform STP. They may still be blocking BPDU's though, so try turning off STP on each port connected to something related to sonos crap (every port connected to an access point and every port connected to a wired sonos device)

enable
conf t
interface ethernet 1/1/5
no spanning-tree

 

[Mushishi]

@Craig Curtin

I do have a R720 and the 6450 at home for my homelab but I am using the broadcom 10GB nic module and not a intel nic in it but if you want i can look into testing it with a ESXi image this weekend to see if I have the same problem.

I have listed the config I use on my 6450 just if something can help and i have STP enabled, and I do also have 10k MTU enabled, and I don't have any issuses with my 10g ports.

SSH@beefchunk(config)#show run
Current configuration:
!
ver 08.0.30uT313
!
stack unit 1
  module 1 icx6450-48-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
!
global-stp
!
!
lag TrueNAS dynamic id 2047
 ports ethernet 1/1/33 to 1/1/36
 primary-port 1/1/33
 lacp-timeout long
 deploy
 port-name "TrueNAS Docker" ethernet 1/1/33
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
 spanning-tree
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
jumbo
enable password-display
hostname beefchunk
ip dhcp-client disable
ip dns domain-list moonisp.dk
ip dns server-address 10.10.10.2 10.10.10.5
ip route 0.0.0.0/0 10.10.10.1
!
logging host 10.10.10.4
logging enable rfc5424
no telnet server
username root password ..........
snmp-server community ............
!
!
clock summer-time
clock timezone gmt GMT+03
!
!
ntp
 disable serve
 server 10.10.10.2
!
!
web-management https
web-management refresh port-statistic 10
web-management refresh front-panel 10
!
!
!
interface ethernet 1/1/25
 port-name Proxmox
!
interface ethernet 1/1/33
 port-name TrueNAS Docker
!
interface ethernet 1/1/47
 port-name Unraid
!
interface ethernet 1/1/48
 port-name Unraid
!
interface ethernet 1/2/1
 port-name TrueNAS
!
interface ethernet 1/2/2
 port-name Enterprise
!
interface ethernet 1/2/3
 port-name Dell
!
interface ve 1
 ip address 10.10.10.250 255.255.255.0
!
!
!
!
!
lldp run
!
!
!
!
end

And my show media:

Port 1/2/1:  Type : 10GE Twinax   1m (SFP +)
Port 1/2/2:  Type : 10GE SR 300m (SFP +)
Port 1/2/3:  Type : 10GE Twinax   5m (SFP +)
Port 1/2/4:  Type : EMPTY

 

[Craig Curtin]

@Craig Curtin

I do have a R720 and the 6450 at home for my homelab but I am using the broadcom 10GB nic module and not a intel nic in it but if you want i can look into testing it with a ESXi image this weekend to see if I have the same problem.

I have listed the config I use on my 6450 just if something can help and i have STP enabled, and I do also have 10k MTU enabled, and I don't have any issuses with my 10g ports.

SSH@beefchunk(config)#show run
Current configuration:
!
ver 08.0.30uT313
!
stack unit 1
  module 1 icx6450-48-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
!
global-stp
!
!
lag TrueNAS dynamic id 2047
ports ethernet 1/1/33 to 1/1/36
primary-port 1/1/33
lacp-timeout long
deploy
port-name "TrueNAS Docker" ethernet 1/1/33
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
spanning-tree
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
jumbo
enable password-display
hostname beefchunk
ip dhcp-client disable
ip dns domain-list moonisp.dk
ip dns server-address 10.10.10.2 10.10.10.5
ip route 0.0.0.0/0 10.10.10.1
!
logging host 10.10.10.4
logging enable rfc5424
no telnet server
username root password ..........
snmp-server community ............
!
!
clock summer-time
clock timezone gmt GMT+03
!
!
ntp
disable serve
server 10.10.10.2
!
!
web-management https
web-management refresh port-statistic 10
web-management refresh front-panel 10
!
!
!
interface ethernet 1/1/25
port-name Proxmox
!
interface ethernet 1/1/33
port-name TrueNAS Docker
!
interface ethernet 1/1/47
port-name Unraid
!
interface ethernet 1/1/48
port-name Unraid
!
interface ethernet 1/2/1
port-name TrueNAS
!
interface ethernet 1/2/2
port-name Enterprise
!
interface ethernet 1/2/3
port-name Dell
!
interface ve 1
ip address 10.10.10.250 255.255.255.0
!
!
!
!
!
lldp run
!
!
!
!
end

And my show media:

Port 1/2/1:  Type : 10GE Twinax   1m (SFP +)
Port 1/2/2:  Type : 10GE SR 300m (SFP +)
Port 1/2/3:  Type : 10GE Twinax   5m (SFP +)
Port 1/2/4:  Type : EMPTY

Thanks for that - would really appreciate it if you could do some testing with ESXi just to put my mind at ease - i setup ESxi 7.01 yesterday on one of my Compaq/HP units and attached it to the 6610 (spare) and will start testing and screen shotting today to see if i can get to the bottom of it. (I also have 2 x 6450s here that i will do the same testing with once i am sure i have a working config.

I have a couple of leftover R710s here that i can also fire up - but they do not have onboard 10GB so will have to put in an Intel or Mellanox card for that testing

looks like i can going to have a big weekend of testing !

on your 6450 the 1/2/2 that reports as 10GE SR is that a twisted pair module as well or just some short range fibre ?

Craig

 

[Craig Curtin]

@Craig Curtin
I found the original explanation about spanning tree, the one I had mentioned 2 pages before (which did not help you).
Quote:


STP is off by default with the l3 OS image (off globally and off per vlan), but remains "ON" per port, meaning ports are eligible to participate in an STP topology, but one hasn't been defined globally so they don't perform STP. They may still be blocking BPDU's though, so try turning off STP on each port connected to something related to sonos crap (every port connected to an access point and every port connected to a wired sonos device)

enable
conf t
interface ethernet 1/1/5
no spanning-tree

Hmm - that is an interesting find as it does state the ports are blocked (can not understand why still though)

I will give the explicit no spanning-tree and see how i go

Craig

 

[Craig Curtin]

Hmm - that is an interesting find as it does state the ports are blocked (can not understand why still though)

I will give the explicit no spanning-tree and see how i go

Craig

Just confirming i am running with the no spanning-tree (per port) for the 10GB ports already - here is my currently running config on the live system - currently with 2 x Dell Hosts to 10GB fibre and the 3rd one waiting to connect for me to solve this problem (or for some more DAC cables to arrive)

SSH@6610-Basement#show run
Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-48-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack disable
!
!
!
lag ToCisco dynamic id 1
ports ethernet 1/1/37 to 1/1/38
primary-port 1/1/37
deploy
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 100 name data-100 by port
tagged ethe 1/1/37 to 1/1/38 ethe 1/2/4 to 1/2/5 ethe 1/3/1 to 1/3/4 ethe 1/3/8
router-interface ve 100
!
vlan 101 name iot-101 by port
tagged ethe 1/1/37 to 1/1/38 ethe 1/2/4 to 1/2/5 ethe 1/3/1 ethe 1/3/8
!
vlan 102 name crypto-102 by port
tagged ethe 1/1/1 to 1/1/2 ethe 1/2/4 to 1/2/5 ethe 1/3/8
!
vlan 200 name storage-200 by port
tagged ethe 1/1/37 to 1/1/38 ethe 1/2/3 ethe 1/2/5 ethe 1/2/7 ethe 1/2/9 ethe 1/3/7 to 1/3/8
router-interface ve 200
!
vlan 202 name prosis-202 by port
tagged ethe 1/1/1 to 1/1/3 ethe 1/1/37 to 1/1/38 ethe 1/2/4 to 1/2/5 ethe 1/3/1 to 1/3/2 ethe 1/3/8
router-interface ve 202
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
jumbo
enable aaa console
hostname 6610-Basement
ip dhcp-client disable
ip dns server-address 172.16.100.1
ip route 0.0.0.0/0 172.16.100.1
!
logging host 172.16.100.26
no telnet server
username root password .....
!
!
!
!
ntp
disable serve
server 172.16.100.1
!
!
no web-management http
!
!
!
!
!
!
!
interface ethernet 1/1/1
port-name ESXI-Host1 - Onboard Management port
!
interface ethernet 1/1/2
port-name ESXI-Host2 - Onboard Management port
!
interface ethernet 1/1/3
port-name ESXI-Host-3 - Onboard Mangement port
!
interface ethernet 1/2/1
no spanning-tree
!
interface ethernet 1/2/2
no spanning-tree
!
interface ethernet 1/2/3
port-name Host-2-I520d-Port-2-VMNIC2
no spanning-tree
!
interface ethernet 1/2/4
port-name Host-3-I520s-VMNIC1
no spanning-tree
!
interface ethernet 1/2/5
port-name Host-1-I520s-VMNIC1-Bottom-Slot
no spanning-tree
!
interface ethernet 1/2/6
no spanning-tree
!
interface ethernet 1/2/7
port-name Host-1-I520s-VMNCI3-Top-Slot
no spanning-tree
!
interface ethernet 1/2/8
no spanning-tree
!
interface ethernet 1/2/9
port-name Host-3-I520-Port-2-VMNIC2
no spanning-tree
!
interface ethernet 1/2/10
port-name Host-2-I520d-Port-1-VMNIC1
no spanning-tree
!
interface ethernet 1/3/7
port-name Mediastore-Fibre
speed-duplex 10G-full
no spanning-tree
!
interface ethernet 1/3/8
speed-duplex 10G-full
no spanning-tree
!
interface ve 1
ip address 192.168.1.254 255.255.255.0
!
interface ve 100
ip address 172.16.100.254 255.255.255.0
!
interface ve 200
ip address 172.16.200.254 255.255.255.0
!
interface ve 202
ip address 192.168.202.254 255.255.255.0
!
!
!
!
!
!
!
!
!
end

SSH@6610-Basement#


Craig

 

[Mushishi]

Thanks for that - would really appreciate it if you could do some testing with ESXi just to put my mind at ease - i setup ESxi 7.01 yesterday on one of my Compaq/HP units and attached it to the 6610 (spare) and will start testing and screen shotting today to see if i can get to the bottom of it. (I also have 2 x 6450s here that i will do the same testing with once i am sure i have a working config.

I have a couple of leftover R710s here that i can also fire up - but they do not have onboard 10GB so will have to put in an Intel or Mellanox card for that testing

looks like i can going to have a big weekend of testing !

on your 6450 the 1/2/2 that reports as 10GE SR is that a twisted pair module as well or just some short range fibre ?

Craig

Not a problem at all. I had been thinking about giving ESXi a test again as I had not used it after they removed support for socket 1366 cpu's so I was not able to use a newer version on my old HP DL380 server.

After looking at the spare 10G nics i have at home i did see that i had a extra ConnectX-3 card that i can test with together with the Broadcom 57800S daughter card I have in the system just to test a nic that you have also tested with.

If you want also i can change the password on my config and give you a copy if i can get it to save to a tftp server. Just note i have the 48 port non-poe not sure if that will work on a poe/24-port switch.

Regarding 1/2/2 yes that is a fiber module as my desktop is nowhere near my servers/switch. I did not even think about the SR might mean short range.

 

[Zackey_TNT]

Hey everyone.
My Brocade ICX 6610 keeps randomly rebooting itself every so often (like once a month)
Can anyone please advise how to review the logs to find the problem? Or share if you have had the same experience.
It has 1 fan and 1 PSU.

 

[Rttg]

Had a similar random reboot a few days ago, and @AndroidCat pointed me on the right direction (I spun up a syslog-NRG container to consume the logs)

You need to stream your logs to external syslog server.
For instance:

logging host <IP>  udp-port 1514
logging enable rfc5424

 

[itronin]

Hey everyone.
My Brocade ICX 6610 keeps randomly rebooting itself every so often (like once a month)
Can anyone please advise how to review the logs to find the problem? Or share if you have had the same experience.
It has 1 fan and 1 PSU.

Is your ICX 6610 plugged into a UPS? If so, mfg and model please.

 

[Mushishi]

@Craig Curtin Just did a quick test here in the afternoon with a fresh install of the Dell ESXi. Forgot to test the the ConnectX though but the broadcom port from the intergraded module did work fine.

I did not setup any VM's and my management network was on the 2x1G ports on the broadcom. Just to see if it would disable the port as soon as i removed it from a vSwitch. I do also not run vsphere on it just the buildin webinterface.

 

[rootpeer]

I would like to ask for some help, as I have no experience with L3 switching (total noob).

I have two different sites that are connected via a pair of Ubiquiti Nanobeam APs. Both sites have a pfSense router in place and are using different LAN subnets and are in general separate from each other.

The point of the Ubiquiti bridge was to use the DSL of site-1 as a backup internet connection for site-2. Thus, I have connected the Ubiquiti AP at site-1 directly to pfSense-1 on a separate interface and the other AP at site-2 also directly to pfSense-2 as a WAN interface. That lets me access site-1's internet and LAN from site-2 but not the opposite.

Now the situation has arisen that I would like to be able to access both LANs from both locations, regardless of if pfSense-1 or pfSense-2 are online or not, basically access static servers on either LAN (VLAN 1) from either LAN (VLAN 1) in case one or both pfSense routers are online.

I have Ruckus ICX6450s at both locations.

My question is, is there a configuration recipe that would allow me to connect the Nanobeams to the switches, then use the L3 functionality to route between the native (V)LANs of each switch?

Edit: Is the following thought process correct?:

1. Connect the nanobeams to the switches on a new VLAN (e.g. VLAN 800)
2. Set up VEs (VE 80) on VLAN 800 and tag or untag the interfaces where the Nanobeams are connected
3. Set static route on site-1: LAN-IP-ADDRESS-SPACE of site-2 via VE 80
4. Set static route on site-2: LAN-IP-ADDRESS-SPACE of site-1 via VE 80
5. Profit