Had a similar random reboot a few days ago, and @AndroidCat pointed me on the right direction (I spun up a syslog-NRG container to consume the logs)
Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
@Craig Curtin Just did a quick test here in the afternoon with a fresh install of the Dell ESXi. Forgot to test the the ConnectX though but the broadcom port from the intergraded module did work fine.
I did not setup any VM's and my management network was on the 2x1G ports on the broadcom. Just to see if it would disable the port as soon as i removed it from a vSwitch. I do also not run vsphere on it just the buildin webinterface.
I did not setup any VM's and my management network was on the 2x1G ports on the broadcom. Just to see if it would disable the port as soon as i removed it from a vSwitch. I do also not run vsphere on it just the buildin webinterface.
I would like to ask for some help, as I have no experience with L3 switching (total noob).
I have two different sites that are connected via a pair of Ubiquiti Nanobeam APs. Both sites have a pfSense router in place and are using different LAN subnets and are in general separate from each other.
The point of the Ubiquiti bridge was to use the DSL of site-1 as a backup internet connection for site-2. Thus, I have connected the Ubiquiti AP at site-1 directly to pfSense-1 on a separate interface and the other AP at site-2 also directly to pfSense-2 as a WAN interface. That lets me access site-1's internet and LAN from site-2 but not the opposite.
Now the situation has arisen that I would like to be able to access both LANs from both locations, regardless of if pfSense-1 or pfSense-2 are online or not, basically access static servers on either LAN (VLAN 1) from either LAN (VLAN 1) in case one or both pfSense routers are online.
I have Ruckus ICX6450s at both locations.
My question is, is there a configuration recipe that would allow me to connect the Nanobeams to the switches, then use the L3 functionality to route between the native (V)LANs of each switch?
Edit: Is the following thought process correct?:
I have two different sites that are connected via a pair of Ubiquiti Nanobeam APs. Both sites have a pfSense router in place and are using different LAN subnets and are in general separate from each other.
The point of the Ubiquiti bridge was to use the DSL of site-1 as a backup internet connection for site-2. Thus, I have connected the Ubiquiti AP at site-1 directly to pfSense-1 on a separate interface and the other AP at site-2 also directly to pfSense-2 as a WAN interface. That lets me access site-1's internet and LAN from site-2 but not the opposite.
Now the situation has arisen that I would like to be able to access both LANs from both locations, regardless of if pfSense-1 or pfSense-2 are online or not, basically access static servers on either LAN (VLAN 1) from either LAN (VLAN 1) in case one or both pfSense routers are online.
I have Ruckus ICX6450s at both locations.
My question is, is there a configuration recipe that would allow me to connect the Nanobeams to the switches, then use the L3 functionality to route between the native (V)LANs of each switch?
Edit: Is the following thought process correct?:
- Connect the nanobeams to the switches on a new VLAN (e.g. VLAN 800)
- Set up VEs (VE 80) on VLAN 800 and tag or untag the interfaces where the Nanobeams are connected
- Set static route on site-1: LAN-IP-ADDRESS-SPACE of site-2 via VE 80
- Set static route on site-2: LAN-IP-ADDRESS-SPACE of site-1 via VE 80
- Profit
Last edited:
Good one thanks - really appreciate the effort.
OK i have advanced a bit further i think - still have not stood up the Mellanox cards yet - that is the job for this afternoon.
However what i have discovered (i think) is that i might have a DAC compatibiity issue - as far as i knew i thought that a QSFP to SFP+ breakout cable would be a passive device - but i have tried the BIOS update process for the 82599 based Intel cards and i seem to be makig a bit of progress with this.
The thing that first put me onto it - was in the testing i stood up a Linux host with a dual port 520-da2 in it and connected that with a SFP+ AOC 10 metre cable back to my switch - came up the first time and did not think anything of it - then did some VLANning and it stayed up - but then tried a reboot and it would not come back up - went into DMESG and found the card had disabled the port because it did not like the SFP+ module.
So found the method using Ethtool to patch the BIOS on the card - https://forums.servethehome.com/ind...m-to-unlock-all-sfp-transceivers.24634/page-1 and performed that - and then it was working fine.
So thought this card was worth a try in the ESXi servers now it was patched and known to work - so moved it across and it appears to have brought up both links OK
Not definitive yet and have to perform more testing with the VLANs etc to see what happens - will also try a Mellanox card in the same box and let you know
Craig
@Craig Curtin well that seems like to be good news. In regards to your mellanox cards could they be OEM branded and then locked to that OEM's modules.? I faintly remember that mine was branded to someone but it is a few years ago i bought that card, and i know that i flashed it with stock firmware as some of the first.
The Intel X520 i have in my desktop was a sun branded card that allowed unsupported modules OOB. I did check with ethtool.
The Intel X520 i have in my desktop was a sun branded card that allowed unsupported modules OOB. I did check with ethtool.
edit: Ok so got a lot of it straightened out, I think part due to order of operations and flipping between the web ui, which sucks for this, and cli.
vlan and dhcp is working for vlan200 now on assigned ports, I just need to get my ubiquity ap to properly worth with the vlan for the guest network only.
But yeah, if anyone else reading this has issues with setting up vlans, ignore the web ui entirely except for a visual check after.
vlan and dhcp is working for vlan200 now on assigned ports, I just need to get my ubiquity ap to properly worth with the vlan for the guest network only.
But yeah, if anyone else reading this has issues with setting up vlans, ignore the web ui entirely except for a visual check after.
Last edited:
OK so a couple of things
1) Post up your config
2) I assume based on what you have stated that you have a spare interface in OpnSense you can plug into the switch for the 200 VLAN ?
3) So the steps on the switch would be
a) Create VLAN 200 - login as root etc, then en, then conf t, then VLAN 200
b) add the ports to the VLAN - the least disruptive way is to have a spare port on OpnSense and plug it into one port and the AP into the other
c) lets say opnsense in port 1/1/34 and the ubiquiti is 1/1/35
d) so as per step a above we should still be in VLAN 200 config
e) type tag e 1/1/34 - this will remove it from VLAN 1 and put it into VLAN 200 as a tagged port only
f) then type tag e 1/1/35 - same as above
4) At this stage you have two devices that will have to support VLAN tagging and will only have access to VLAN 200
5) on Opensense you need to make sure the interface that is attached to port 1/1/34 has a tagged VLAN 200 defined on it
6) You need to assign it a valid IP address in the 200 subnet and then make sure the DHCP server is applied to that interface and is giving out IP addresses relevant to that subnet
7) You need to make sure the AP has a valid IP in the subnet and that you have turned off the DHCP server on the AP and it is bridging between the wireless interface and the LAN and it is putting devices into the 200 VLAN when they are successfully authenticated
Craig
The Brocade are meant to be one of the more flexible in terms of which coding they support - but it appears to be difficult to get down to a deep level to actually confirm if it is accepted - beyond doing a show media - i am going through some problems (see my posts above) that i "think" may be related to transceiver issues and coding - but the show media commands are not complaining about any of the devices so it is just a feeling at this stage
Craig
Nope still have not had the Mellanoxs in there yet - all testing done to this point with INtel 520DA-2 and Intel 540T@Craig Curtin well that seems like to be good news. In regards to your mellanox cards could they be OEM branded and then locked to that OEM's modules.? I faintly remember that mine was branded to someone but it is a few years ago i bought that card, and i know that i flashed it with stock firmware as some of the first.
The Intel X520 i have in my desktop was a sun branded card that allowed unsupported modules OOB. I did check with ethtool.
Hopefully - time permitting - will be onto the mellanox to start with this afternoon
The Mellanox all the took the flash to IB/ETH dual mode with no problem (not that i plan on using IB mode)
Craig
OK so i have just removed one of the intel dual port 520da-2 adapters and have added in a Mellanox CX3 - and it looks like similar problems.
So to recap
ESXI 7.03
Mellanx CX312a dual port adapter - cross flashed to IB/ETH mode with latest avaiable from Nvidia site.
6610 - connecting it to 2 x Arista QSFP to SFP+ breakout cables - one breaking out 1/2/2 to 1/2/5 and the other 1/2/7 to 1/2/10
now had two different architecture machines - HP/Compaq Elite 8300 and no Dell Optiplex 7050
Tried both Intel 520da-2 and now Mellanox CX312a dual port.
Connected up and powered up host
One port came straight back up = 1/2/3 - attached it to the vSwitch and traffic flowing
2nd port - now registers as up on the switch - but down at the vNIC level in vsphere.
Any ideas now ?
Craig
Last edited:
@Craig Curtin I do remember some problems to get my Mellenox cards to bring the interface up because they would default to IB and not Ethernet.
I can see that i did follow @fohdeesha guide here: https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015 to flash a stock firmware on my card, and to force it to Ethernet only and not IB.
I'm at work right now so I can't boot up the ESXi install i did to see if ESXi have the mlxconfig command but if it don't then do like i think i did and use a ubuntu live disk and try this command.
#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
I will give it a quick test tonight after work if i am not totally busted when I get home.
I can see the Mellnox do also have a guide on howto do that on ESXi as it was a bit diffrent. Note that the path will depend on the adapter so check what you have in /dev/mst
I can see that i did follow @fohdeesha guide here: https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015 to flash a stock firmware on my card, and to force it to Ethernet only and not IB.
I'm at work right now so I can't boot up the ESXi install i did to see if ESXi have the mlxconfig command but if it don't then do like i think i did and use a ubuntu live disk and try this command.
#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
I will give it a quick test tonight after work if i am not totally busted when I get home.
I can see the Mellnox do also have a guide on howto do that on ESXi as it was a bit diffrent. Note that the path will depend on the adapter so check what you have in /dev/mst
Last edited:
got a couples of questions/problems.
1) though i can talk to the switch managment(running on port 1 and managment port) i cant reach none of the devices connected directly to the switch
2) does have usb to serial cable but i do have NAS that run a tftp server, how do i change these commands to use telnet or ssh please
setenv serverip 192.168.1.8
setenv image_name ICX64xx/ICX64R08030u.bin
setenv uboot ICX64xx/kxz10105.bin
update_primary
update_uboot
1) though i can talk to the switch managment(running on port 1 and managment port) i cant reach none of the devices connected directly to the switch
2) does have usb to serial cable but i do have NAS that run a tftp server, how do i change these commands to use telnet or ssh please
setenv serverip 192.168.1.8
setenv image_name ICX64xx/ICX64R08030u.bin
setenv uboot ICX64xx/kxz10105.bin
update_primary
update_uboot
What do you mean change the commands - those commands are what you will type in when attached through serial/usb to the management.
Presumably - although you have not stated - you have a 6450 ?
Then you need to attach a console cable with either a serial or usb connection to your PC - presumably you are running windows.
You would then use a program such as putty (free) to connect to the serial/com port that is attached to the switch and follow through all the steps in the doco as per the OP
Craig
Yep the cards i have are the dual 10GB (CX312a) SFP+ cards. I have flashed them - but i believe it was a dual stack - i will go back and try one of them as just ethernet (drop the IB) and see if that makes any difference.
Will report back
Craig
so change them so I can use shh/telnet over IP ethernet connection(or is that not possible) and yes 6450 sorry for missing out on a useful bit of info.
I don't know who needs to hear this but it was a pleasant surprise that my new (to me) Brocade 6450 can power 2 unifi pro APs, a voip desk phone, and one of those grandstream cordless base stations while running fanless. Temps bumped 2C and stayed there for 24h now.
My rack is in the basement and i have nothing else hot racked against the 6450, so ymmv.
My rack is in the basement and i have nothing else hot racked against the 6450, so ymmv.
Yes if you follow the OP doc at the start of ths thread it does the following
1) Gets you to attach through the console port and enable the network for your environment (i.e. your IP addressing scheme)
2) It then has you attach to a network cable to the management port on your switch and set it up to receive the various files for firmare updating
3) You then enable TFTP on your NAS device and tel the switch to go out and get the files over the management network
4) Once this is completed - still using your console cable you then add the switch to your live network and move the ethernet connection from the management port to the main switch ports - at which point you should be able to telnet/ssh/web into the switch
Craig
dont have a console cable but forgot a the most basic check, to see if it was already running the latest version.
still cant connect to any NTP server, ping external IP (can do dns lookups), all devices can see the switch but any device connected to the router can not see anything that connected directly to the switch