Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Craig Curtin]

@Craig Curtin Just did a quick test here in the afternoon with a fresh install of the Dell ESXi. Forgot to test the the ConnectX though but the broadcom port from the intergraded module did work fine.

I did not setup any VM's and my management network was on the 2x1G ports on the broadcom. Just to see if it would disable the port as soon as i removed it from a vSwitch. I do also not run vsphere on it just the buildin webinterface.

View attachment 25563

Good one thanks - really appreciate the effort.

OK i have advanced a bit further i think - still have not stood up the Mellanox cards yet - that is the job for this afternoon.

However what i have discovered (i think) is that i might have a DAC compatibiity issue - as far as i knew i thought that a QSFP to SFP+ breakout cable would be a passive device - but i have tried the BIOS update process for the 82599 based Intel cards and i seem to be makig a bit of progress with this.

The thing that first put me onto it - was in the testing i stood up a Linux host with a dual port 520-da2 in it and connected that with a SFP+ AOC 10 metre cable back to my switch - came up the first time and did not think anything of it - then did some VLANning and it stayed up - but then tried a reboot and it would not come back up - went into DMESG and found the card had disabled the port because it did not like the SFP+ module.

So found the method using Ethtool to patch the BIOS on the card - https://forums.servethehome.com/ind...m-to-unlock-all-sfp-transceivers.24634/page-1 and performed that - and then it was working fine.

So thought this card was worth a try in the ESXi servers now it was patched and known to work - so moved it across and it appears to have brought up both links OK

Not definitive yet and have to perform more testing with the VLANs etc to see what happens - will also try a Mellanox card in the same box and let you know

Craig

 

[Mushishi]

@Craig Curtin well that seems like to be good news. In regards to your mellanox cards could they be OEM branded and then locked to that OEM's modules.? I faintly remember that mine was branded to someone but it is a few years ago i bought that card, and i know that i flashed it with stock firmware as some of the first.

The Intel X520 i have in my desktop was a sun branded card that allowed unsupported modules OOB. I did check with ethtool.

 

[Junction Runner]

edit: Ok so got a lot of it straightened out, I think part due to order of operations and flipping between the web ui, which sucks for this, and cli.

vlan and dhcp is working for vlan200 now on assigned ports, I just need to get my ubiquity ap to properly worth with the vlan for the guest network only.

But yeah, if anyone else reading this has issues with setting up vlans, ignore the web ui entirely except for a visual check after.

 

[Cobra0101]

Does anyone know how compatible the SFP+ ports are on the Brocade ICX6450 i.e. are they coded/password to stop 3rd party devices being used?

 

[Craig Curtin]

edit: Ok so got a lot of it straightened out, I think part due to order of operations and flipping between the web ui, which sucks for this, and cli.

vlan and dhcp is working for vlan200 now on assigned ports, I just need to get my ubiquity ap to properly worth with the vlan for the guest network only.

But yeah, if anyone else reading this has issues with setting up vlans, ignore the web ui entirely except for a visual check after.

OK so a couple of things

1) Post up your config
2) I assume based on what you have stated that you have a spare interface in OpnSense you can plug into the switch for the 200 VLAN ?
3) So the steps on the switch would be

a) Create VLAN 200 - login as root etc, then en, then conf t, then VLAN 200
b) add the ports to the VLAN - the least disruptive way is to have a spare port on OpnSense and plug it into one port and the AP into the other
c) lets say opnsense in port 1/1/34 and the ubiquiti is 1/1/35
d) so as per step a above we should still be in VLAN 200 config
e) type tag e 1/1/34 - this will remove it from VLAN 1 and put it into VLAN 200 as a tagged port only
f) then type tag e 1/1/35 - same as above

4) At this stage you have two devices that will have to support VLAN tagging and will only have access to VLAN 200
5) on Opensense you need to make sure the interface that is attached to port 1/1/34 has a tagged VLAN 200 defined on it
6) You need to assign it a valid IP address in the 200 subnet and then make sure the DHCP server is applied to that interface and is giving out IP addresses relevant to that subnet
7) You need to make sure the AP has a valid IP in the subnet and that you have turned off the DHCP server on the AP and it is bridging between the wireless interface and the LAN and it is putting devices into the 200 VLAN when they are successfully authenticated

Craig

 

[Craig Curtin]

Does anyone know how compatible the SFP+ ports are on the Brocade ICX6450 i.e. are they coded/password to stop 3rd party devices being used?

The Brocade are meant to be one of the more flexible in terms of which coding they support - but it appears to be difficult to get down to a deep level to actually confirm if it is accepted - beyond doing a show media - i am going through some problems (see my posts above) that i "think" may be related to transceiver issues and coding - but the show media commands are not complaining about any of the devices so it is just a feeling at this stage

Craig

 

[Craig Curtin]

@Craig Curtin well that seems like to be good news. In regards to your mellanox cards could they be OEM branded and then locked to that OEM's modules.? I faintly remember that mine was branded to someone but it is a few years ago i bought that card, and i know that i flashed it with stock firmware as some of the first.

The Intel X520 i have in my desktop was a sun branded card that allowed unsupported modules OOB. I did check with ethtool.

Nope still have not had the Mellanoxs in there yet - all testing done to this point with INtel 520DA-2 and Intel 540T

Hopefully - time permitting - will be onto the mellanox to start with this afternoon

The Mellanox all the took the flash to IB/ETH dual mode with no problem (not that i plan on using IB mode)

Craig

 

[Craig Curtin]

Nope still have not had the Mellanoxs in there yet - all testing done to this point with INtel 520DA-2 and Intel 540T

Hopefully - time permitting - will be onto the mellanox to start with this afternoon

The Mellanox all the took the flash to IB/ETH dual mode with no problem (not that i plan on using IB mode)

Craig

OK so i have just removed one of the intel dual port 520da-2 adapters and have added in a Mellanox CX3 - and it looks like similar problems.

So to recap

ESXI 7.03
Mellanx CX312a dual port adapter - cross flashed to IB/ETH mode with latest avaiable from Nvidia site.
6610 - connecting it to 2 x Arista QSFP to SFP+ breakout cables - one breaking out 1/2/2 to 1/2/5 and the other 1/2/7 to 1/2/10
now had two different architecture machines - HP/Compaq Elite 8300 and no Dell Optiplex 7050
Tried both Intel 520da-2 and now Mellanox CX312a dual port.

Connected up and powered up host

One port came straight back up = 1/2/3 - attached it to the vSwitch and traffic flowing
2nd port - now registers as up on the switch - but down at the vNIC level in vsphere.












Any ideas now ?

Craig

 

[Mushishi]

@Craig Curtin I do remember some problems to get my Mellenox cards to bring the interface up because they would default to IB and not Ethernet.

I can see that i did follow @fohdeesha guide here: https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015 to flash a stock firmware on my card, and to force it to Ethernet only and not IB.

I'm at work right now so I can't boot up the ESXi install i did to see if ESXi have the mlxconfig command but if it don't then do like i think i did and use a ubuntu live disk and try this command.


#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

I will give it a quick test tonight after work if i am not totally busted when I get home.

I can see the Mellnox do also have a guide on howto do that on ESXi as it was a bit diffrent. Note that the path will depend on the adapter so check what you have in /dev/mst

https://support.mellanox.com/s/article/MLNX2-117-6558kn

 

[Cobra0101]

got a couples of questions/problems.

1) though i can talk to the switch managment(running on port 1 and managment port) i cant reach none of the devices connected directly to the switch

2) does have usb to serial cable but i do have NAS that run a tftp server, how do i change these commands to use telnet or ssh please

setenv serverip 192.168.1.8
setenv image_name ICX64xx/ICX64R08030u.bin
setenv uboot ICX64xx/kxz10105.bin
update_primary
update_uboot

 

[Craig Curtin]

got a couples of questions/problems.

1) though i can talk to the switch managment(running on port 1 and managment port) i cant reach none of the devices connected directly to the switch

2) does have usb to serial cable but i do have NAS that run a tftp server, how do i change these commands to use telnet or ssh please

setenv serverip 192.168.1.8
setenv image_name ICX64xx/ICX64R08030u.bin
setenv uboot ICX64xx/kxz10105.bin
update_primary
update_uboot

What do you mean change the commands - those commands are what you will type in when attached through serial/usb to the management.

Presumably - although you have not stated - you have a 6450 ?

Then you need to attach a console cable with either a serial or usb connection to your PC - presumably you are running windows.

You would then use a program such as putty (free) to connect to the serial/com port that is attached to the switch and follow through all the steps in the doco as per the OP

Craig

 

[Craig Curtin]

@Craig Curtin I do remember some problems to get my Mellenox cards to bring the interface up because they would default to IB and not Ethernet.

I can see that i did follow @fohdeesha guide here: https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015 to flash a stock firmware on my card, and to force it to Ethernet only and not IB.

I'm at work right now so I can't boot up the ESXi install i did to see if ESXi have the mlxconfig command but if it don't then do like i think i did and use a ubuntu live disk and try this command.


#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

I will give it a quick test tonight after work if i am not totally busted when I get home.

I can see the Mellnox do also have a guide on howto do that on ESXi as it was a bit diffrent. Note that the path will depend on the adapter so check what you have in /dev/mst

https://support.mellanox.com/s/article/MLNX2-117-6558kn

Yep the cards i have are the dual 10GB (CX312a) SFP+ cards. I have flashed them - but i believe it was a dual stack - i will go back and try one of them as just ethernet (drop the IB) and see if that makes any difference.

Will report back

Craig

 

[Cobra0101]

What do you mean change the commands - those commands are what you will type in when attached through serial/usb to the management.

Presumably - although you have not stated - you have a 6450 ?

Then you need to attach a console cable with either a serial or usb connection to your PC - presumably you are running windows.

You would then use a program such as putty (free) to connect to the serial/com port that is attached to the switch and follow through all the steps in the doco as per the OP

Craig

so change them so I can use shh/telnet over IP ethernet connection(or is that not possible) and yes 6450 sorry for missing out on a useful bit of info.

 

[baskethammer]

I don't know who needs to hear this but it was a pleasant surprise that my new (to me) Brocade 6450 can power 2 unifi pro APs, a voip desk phone, and one of those grandstream cordless base stations while running fanless. Temps bumped 2C and stayed there for 24h now.

My rack is in the basement and i have nothing else hot racked against the 6450, so ymmv.

 

[Craig Curtin]

so change them so I can use shh/telnet over IP ethernet connection(or is that not possible) and yes 6450 sorry for missing out on a useful bit of info.

Yes if you follow the OP doc at the start of ths thread it does the following

1) Gets you to attach through the console port and enable the network for your environment (i.e. your IP addressing scheme)
2) It then has you attach to a network cable to the management port on your switch and set it up to receive the various files for firmare updating
3) You then enable TFTP on your NAS device and tel the switch to go out and get the files over the management network
4) Once this is completed - still using your console cable you then add the switch to your live network and move the ethernet connection from the management port to the main switch ports - at which point you should be able to telnet/ssh/web into the switch

Craig

 

[Cobra0101]

Yes if you follow the OP doc at the start of ths thread it does the following

1) Gets you to attach through the console port and enable the network for your environment (i.e. your IP addressing scheme)
2) It then has you attach to a network cable to the management port on your switch and set it up to receive the various files for firmare updating
3) You then enable TFTP on your NAS device and tel the switch to go out and get the files over the management network
4) Once this is completed - still using your console cable you then add the switch to your live network and move the ethernet connection from the management port to the main switch ports - at which point you should be able to telnet/ssh/web into the switch

Craig

dont have a console cable but forgot a the most basic check, to see if it was already running the latest version.

still cant connect to any NTP server, ping external IP (can do dns lookups), all devices can see the switch but any device connected to the router can not see anything that connected directly to the switch

 

[Craig Curtin]

dont have a console cable but forgot a the most basic check, to see if it was already running the latest version.

still cant connect to any NTP server, ping external IP (can do dns lookups), all devices can see the switch but any device connected to the router can not see anything that connected directly to the switch

Well you are going to need to post your config up here - we are not psychic - presuming you have your root level password on the switch to dump the config. If you do not then you are wasting your time and need to get a console cable.

Craig

 

[Cobra0101]

Well you are going to need to post your config up here - we are not psychic - presuming you have your root level password on the switch to dump the config. If you do not then you are wasting your time and need to get a console cable.

Craig

thanks solved the problem I forgot to set the default gateway.

Hopefully a quick question, if the switch is running in router mode can some of the interfaced still behave like a switch(layer 2)
from reading the manual don't think this is possible as it says that if using layer 3 modes you can only have 1 IP per subnet. Is there any way to get around this limitation? Want to use some of the ports for my main subnet which is also on my main router. So 4 SPF+ and 8 rj45s working on my main subnet and then 8 ports for 8 VLANs/subnets and the remaining 8 as spares.

 

[Craig Curtin]

thanks solved the problem I forgot to set the default gateway.

Hopefully a quick question, if the switch is running in router mode can some of the interfaced still behave like a switch(layer 2)

Yes as long as you do not assign a routing interface to the VLAN in question then the ports will all act as Layer 2 and you will have to do offboard routing.

Craig

 

[Damo]

Can someone link a compatible MM-SFP+ transceiver for these switches on Amazon.co.uk or Ebay.co.uk