Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Craig Curtin

Member
Jun 18, 2017
103
20
18
60
Nope still have not had the Mellanoxs in there yet - all testing done to this point with INtel 520DA-2 and Intel 540T

Hopefully - time permitting - will be onto the mellanox to start with this afternoon

The Mellanox all the took the flash to IB/ETH dual mode with no problem (not that i plan on using IB mode)

Craig
OK so i have just removed one of the intel dual port 520da-2 adapters and have added in a Mellanox CX3 - and it looks like similar problems.

So to recap

ESXI 7.03
Mellanx CX312a dual port adapter - cross flashed to IB/ETH mode with latest avaiable from Nvidia site.
6610 - connecting it to 2 x Arista QSFP to SFP+ breakout cables - one breaking out 1/2/2 to 1/2/5 and the other 1/2/7 to 1/2/10
now had two different architecture machines - HP/Compaq Elite 8300 and no Dell Optiplex 7050
Tried both Intel 520da-2 and now Mellanox CX312a dual port.

Connected up and powered up host

One port came straight back up = 1/2/3 - attached it to the vSwitch and traffic flowing
2nd port - now registers as up on the switch - but down at the vNIC level in vsphere.


1669186412649.png

1669186563273.png

1669186620707.png

1669187022517.png

1669187069384.png

Any ideas now ?

Craig
 
Last edited:

Mushishi

New Member
Apr 26, 2016
15
6
3
45
Turku, Finland
@Craig Curtin I do remember some problems to get my Mellenox cards to bring the interface up because they would default to IB and not Ethernet.

I can see that i did follow @fohdeesha guide here: https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015 to flash a stock firmware on my card, and to force it to Ethernet only and not IB.

I'm at work right now so I can't boot up the ESXi install i did to see if ESXi have the mlxconfig command but if it don't then do like i think i did and use a ubuntu live disk and try this command.


#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

I will give it a quick test tonight after work if i am not totally busted when I get home.

I can see the Mellnox do also have a guide on howto do that on ESXi as it was a bit diffrent. Note that the path will depend on the adapter so check what you have in /dev/mst

 
Last edited:
  • Like
Reactions: itronin

Cobra0101

New Member
Nov 22, 2022
10
0
1
got a couples of questions/problems.

1) though i can talk to the switch managment(running on port 1 and managment port) i cant reach none of the devices connected directly to the switch

2) does have usb to serial cable but i do have NAS that run a tftp server, how do i change these commands to use telnet or ssh please

setenv serverip 192.168.1.8
setenv image_name ICX64xx/ICX64R08030u.bin
setenv uboot ICX64xx/kxz10105.bin
update_primary
update_uboot
 

Craig Curtin

Member
Jun 18, 2017
103
20
18
60
got a couples of questions/problems.

1) though i can talk to the switch managment(running on port 1 and managment port) i cant reach none of the devices connected directly to the switch

2) does have usb to serial cable but i do have NAS that run a tftp server, how do i change these commands to use telnet or ssh please

setenv serverip 192.168.1.8
setenv image_name ICX64xx/ICX64R08030u.bin
setenv uboot ICX64xx/kxz10105.bin
update_primary
update_uboot
What do you mean change the commands - those commands are what you will type in when attached through serial/usb to the management.

Presumably - although you have not stated - you have a 6450 ?

Then you need to attach a console cable with either a serial or usb connection to your PC - presumably you are running windows.

You would then use a program such as putty (free) to connect to the serial/com port that is attached to the switch and follow through all the steps in the doco as per the OP

Craig
 

Craig Curtin

Member
Jun 18, 2017
103
20
18
60
@Craig Curtin I do remember some problems to get my Mellenox cards to bring the interface up because they would default to IB and not Ethernet.

I can see that i did follow @fohdeesha guide here: https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015 to flash a stock firmware on my card, and to force it to Ethernet only and not IB.

I'm at work right now so I can't boot up the ESXi install i did to see if ESXi have the mlxconfig command but if it don't then do like i think i did and use a ubuntu live disk and try this command.


#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

I will give it a quick test tonight after work if i am not totally busted when I get home.

I can see the Mellnox do also have a guide on howto do that on ESXi as it was a bit diffrent. Note that the path will depend on the adapter so check what you have in /dev/mst

Yep the cards i have are the dual 10GB (CX312a) SFP+ cards. I have flashed them - but i believe it was a dual stack - i will go back and try one of them as just ethernet (drop the IB) and see if that makes any difference.

Will report back

Craig
 

Cobra0101

New Member
Nov 22, 2022
10
0
1
What do you mean change the commands - those commands are what you will type in when attached through serial/usb to the management.

Presumably - although you have not stated - you have a 6450 ?

Then you need to attach a console cable with either a serial or usb connection to your PC - presumably you are running windows.

You would then use a program such as putty (free) to connect to the serial/com port that is attached to the switch and follow through all the steps in the doco as per the OP

Craig
so change them so I can use shh/telnet over IP ethernet connection(or is that not possible) and yes 6450 sorry for missing out on a useful bit of info.
 

baskethammer

New Member
Nov 4, 2022
6
1
3
I don't know who needs to hear this but it was a pleasant surprise that my new (to me) Brocade 6450 can power 2 unifi pro APs, a voip desk phone, and one of those grandstream cordless base stations while running fanless. Temps bumped 2C and stayed there for 24h now.

My rack is in the basement and i have nothing else hot racked against the 6450, so ymmv.
 
  • Like
Reactions: Silly Valley Serf

Craig Curtin

Member
Jun 18, 2017
103
20
18
60
so change them so I can use shh/telnet over IP ethernet connection(or is that not possible) and yes 6450 sorry for missing out on a useful bit of info.
Yes if you follow the OP doc at the start of ths thread it does the following

1) Gets you to attach through the console port and enable the network for your environment (i.e. your IP addressing scheme)
2) It then has you attach to a network cable to the management port on your switch and set it up to receive the various files for firmare updating
3) You then enable TFTP on your NAS device and tel the switch to go out and get the files over the management network
4) Once this is completed - still using your console cable you then add the switch to your live network and move the ethernet connection from the management port to the main switch ports - at which point you should be able to telnet/ssh/web into the switch

Craig
 

Cobra0101

New Member
Nov 22, 2022
10
0
1
Yes if you follow the OP doc at the start of ths thread it does the following

1) Gets you to attach through the console port and enable the network for your environment (i.e. your IP addressing scheme)
2) It then has you attach to a network cable to the management port on your switch and set it up to receive the various files for firmare updating
3) You then enable TFTP on your NAS device and tel the switch to go out and get the files over the management network
4) Once this is completed - still using your console cable you then add the switch to your live network and move the ethernet connection from the management port to the main switch ports - at which point you should be able to telnet/ssh/web into the switch

Craig
dont have a console cable but forgot a the most basic check, to see if it was already running the latest version.

still cant connect to any NTP server, ping external IP (can do dns lookups), all devices can see the switch but any device connected to the router can not see anything that connected directly to the switch
 

Craig Curtin

Member
Jun 18, 2017
103
20
18
60
dont have a console cable but forgot a the most basic check, to see if it was already running the latest version.

still cant connect to any NTP server, ping external IP (can do dns lookups), all devices can see the switch but any device connected to the router can not see anything that connected directly to the switch
Well you are going to need to post your config up here - we are not psychic - presuming you have your root level password on the switch to dump the config. If you do not then you are wasting your time and need to get a console cable.

Craig
 

Cobra0101

New Member
Nov 22, 2022
10
0
1
Well you are going to need to post your config up here - we are not psychic - presuming you have your root level password on the switch to dump the config. If you do not then you are wasting your time and need to get a console cable.

Craig
thanks solved the problem I forgot to set the default gateway.

Hopefully a quick question, if the switch is running in router mode can some of the interfaced still behave like a switch(layer 2)
from reading the manual don't think this is possible as it says that if using layer 3 modes you can only have 1 IP per subnet. Is there any way to get around this limitation? Want to use some of the ports for my main subnet which is also on my main router. So 4 SPF+ and 8 rj45s working on my main subnet and then 8 ports for 8 VLANs/subnets and the remaining 8 as spares.
 
Last edited:

Craig Curtin

Member
Jun 18, 2017
103
20
18
60
thanks solved the problem I forgot to set the default gateway.

Hopefully a quick question, if the switch is running in router mode can some of the interfaced still behave like a switch(layer 2)
Yes as long as you do not assign a routing interface to the VLAN in question then the ports will all act as Layer 2 and you will have to do offboard routing.

Craig
 

Damo

Active Member
Sep 7, 2022
128
35
28
Can someone link a compatible MM-SFP+ transceiver for these switches on Amazon.co.uk or Ebay.co.uk
 

rootpeer

Member
Oct 19, 2019
86
17
8
I have a weird issue and I need some help.

I have two ICX6450 switches connected to each other.
SW1 has VLAN1 on 10.1.0.0/24 and SW2 has VLAN1 on 10.2.0.0/24.
Both switches are connected via VLAN1012 on 10.1.2.0/24.
Both switches have ve's on VLAN1012 and VLAN1.
Both switches have static routes configured for each other's VLAN1 subnet via their "partner's" VLAN1012 ve. I will post the config below.

I am trying to route from SW1 VLAN1 to SW2 VLAN2. Using a host on SW1, I can ping some hosts on SW2 but not others. Traceroute does not help at all. The host is configured with a static IP and the SW1 VLAN1 ve IP as its gateway for this troubleshooting session.

SW1:
Code:
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
ve1        Up      N/A     N/A  N/A   None  N/A N/A  N/A 609c.9f79.4b20                 
ve1012     Up      N/A     N/A  N/A   None  N/A N/A  N/A 609c.9f79.4b20   

###################################################

interface ve 1                                                   
 ip address 10.1.0.2 255.255.255.0
!
interface ve 1012
 ip address 10.1.2.240 255.255.255.0

################################################

        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          10.1.0.1        ve 1          1/1           S    6d11h
2       10.1.0.0/24        DIRECT          ve 1          0/0           D    6d11h
3       10.1.2.0/24        DIRECT          ve 1012       0/0           D    5d21h
4       10.2.0.0/24        10.1.2.241      ve 1012       1/1           S    5d21h

SW2:
Code:
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
ve1        Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2465.ea40                 
ve1012     Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2465.ea40

#################################################

interface ve 1
 ip address 10.2.0.2 255.255.255.0
!
interface ve 1012
 ip address 10.1.2.241 255.255.255.0

#################################################

        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          10.2.0.1        ve 1          1/1           S    27m30s
2       10.1.0.0/24        10.1.2.240      ve 1012       1/1           S    5d21h
3       10.1.2.0/24        DIRECT          ve 1012       0/0           D    7d0h 
4       10.2.0.0/24        DIRECT          ve 1          0/0           D    25d4h


traceroute to reachable SW2 host from SW1 host:
Code:
$ traceroute 10.2.0.20
traceroute to 10.2.0.20 (10.2.0.20), 30 hops max, 60 byte packets
 1  _gateway (10.1.0.2)  1.420 ms  2.451 ms  1.737 ms
 2  10.1.2.241 (10.1.2.241)  219.946 ms  219.910 ms  219.886 ms
 3  10.2.0.20 (10.2.0.20)  219.929 ms  219.911 ms  219.877 ms
traceroute to an unreachable SW2 host from SW1 host:
Code:
$ traceroute 10.2.0.38
traceroute to 10.2.0.38 (10.2.0.38), 30 hops max, 60 byte packets
 1  _gateway (10.1.0.2)  1.229 ms  2.380 ms  1.677 ms
 2  10.1.2.241 (10.1.2.241)  2.658 ms  8.522 ms  8.492 ms
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
traceroute to a second unreachable SW2 host from SW1 host:
Code:
$ traceroute 10.2.0.4
traceroute to 10.2.0.4 (10.2.0.4), 30 hops max, 60 byte packets
 1  _gateway (10.1.0.2)  7.037 ms  7.712 ms  8.369 ms
 2  10.1.2.241 (10.1.2.241)  5.443 ms  5.417 ms  5.392 ms
 3  10.1.2.116 (10.1.2.116)  6.600 ms  6.576 ms  6.550 ms
This last one is particularly confusing to me because 10.1.2.116 is a pfSense interface on VLAN1012 at SW2 location. Why is the SW2 trying to forward the packet to the pfSense interface instead of the actual host at 10.2.0.4?

Please help!
 

Craig Curtin

Member
Jun 18, 2017
103
20
18
60
I have a weird issue and I need some help.

I have two ICX6450 switches connected to each other.
SW1 has VLAN1 on 10.1.0.0/24 and SW2 has VLAN1 on 10.2.0.0/24.
Both switches are connected via VLAN1012 on 10.1.2.0/24.
Both switches have ve's on VLAN1012 and VLAN1.
Both switches have static routes configured for each other's VLAN1 subnet via their "partner's" VLAN1012 ve. I will post the config below.

I am trying to route from SW1 VLAN1 to SW2 VLAN2. Using a host on SW1, I can ping some hosts on SW2 but not others. Traceroute does not help at all. The host is configured with a static IP and the SW1 VLAN1 ve IP as its gateway for this troubleshooting session.

SW1:
Code:
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
ve1        Up      N/A     N/A  N/A   None  N/A N/A  N/A 609c.9f79.4b20                
ve1012     Up      N/A     N/A  N/A   None  N/A N/A  N/A 609c.9f79.4b20  

###################################################

interface ve 1                                                  
ip address 10.1.0.2 255.255.255.0
!
interface ve 1012
ip address 10.1.2.240 255.255.255.0

################################################

        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          10.1.0.1        ve 1          1/1           S    6d11h
2       10.1.0.0/24        DIRECT          ve 1          0/0           D    6d11h
3       10.1.2.0/24        DIRECT          ve 1012       0/0           D    5d21h
4       10.2.0.0/24        10.1.2.241      ve 1012       1/1           S    5d21h

SW2:
Code:
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
ve1        Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2465.ea40                
ve1012     Up      N/A     N/A  N/A   None  N/A N/A  N/A cc4e.2465.ea40

#################################################

interface ve 1
ip address 10.2.0.2 255.255.255.0
!
interface ve 1012
ip address 10.1.2.241 255.255.255.0

#################################################

        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          10.2.0.1        ve 1          1/1           S    27m30s
2       10.1.0.0/24        10.1.2.240      ve 1012       1/1           S    5d21h
3       10.1.2.0/24        DIRECT          ve 1012       0/0           D    7d0h
4       10.2.0.0/24        DIRECT          ve 1          0/0           D    25d4h


traceroute to reachable SW2 host from SW1 host:
Code:
$ traceroute 10.2.0.20
traceroute to 10.2.0.20 (10.2.0.20), 30 hops max, 60 byte packets
1  _gateway (10.1.0.2)  1.420 ms  2.451 ms  1.737 ms
2  10.1.2.241 (10.1.2.241)  219.946 ms  219.910 ms  219.886 ms
3  10.2.0.20 (10.2.0.20)  219.929 ms  219.911 ms  219.877 ms
traceroute to an unreachable SW2 host from SW1 host:
Code:
$ traceroute 10.2.0.38
traceroute to 10.2.0.38 (10.2.0.38), 30 hops max, 60 byte packets
1  _gateway (10.1.0.2)  1.229 ms  2.380 ms  1.677 ms
2  10.1.2.241 (10.1.2.241)  2.658 ms  8.522 ms  8.492 ms
3  * * *
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
traceroute to a second unreachable SW2 host from SW1 host:
Code:
$ traceroute 10.2.0.4
traceroute to 10.2.0.4 (10.2.0.4), 30 hops max, 60 byte packets
1  _gateway (10.1.0.2)  7.037 ms  7.712 ms  8.369 ms
2  10.1.2.241 (10.1.2.241)  5.443 ms  5.417 ms  5.392 ms
3  10.1.2.116 (10.1.2.116)  6.600 ms  6.576 ms  6.550 ms
This last one is particularly confusing to me because 10.1.2.116 is a pfSense interface on VLAN1012 at SW2 location. Why is the SW2 trying to forward the packet to the pfSense interface instead of the actual host at 10.2.0.4?

Please help!
I am no expert on the Brocade - but having different IP subnets on the same VLAN is just asking for trouble in other implementations - not sure why you would try and do that as even if it does work it is just asking for confusion

Personally i never use the VLAN 1 (default VLAN) for any traffic and change it to a different number as part of my switch setup

Craig
 

rootpeer

Member
Oct 19, 2019
86
17
8
I am no expert on the Brocade - but having different IP subnets on the same VLAN is just asking for trouble in other implementations - not sure why you would try and do that as even if it does work it is just asking for confusion

Personally i never use the VLAN 1 (default VLAN) for any traffic and change it to a different number as part of my switch setup

Craig
The switches are at separate locations connected via a pair of Ubiquiti antennas. The two VLAN1s are not connected at L2, thus the need to route between them through VLAN1012 at L3. So, even if they both have the same VLAN tag, they are two separate networks. I don't see how them having the same tag is relevant here, they just need to forward the packets to the next hop that is statically assigned.
 

vangoose

Active Member
May 21, 2019
326
106
43
Canada
The switches are at separate locations connected via a pair of Ubiquiti antennas. The two VLAN1s are not connected at L2, thus the need to route between them through VLAN1012 at L3. So, even if they both have the same VLAN tag, they are two separate networks. I don't see how them having the same tag is relevant here, they just need to forward the packets to the next hop that is statically assigned.
You just need to configure route on each switch to so it knows where to forward traffics to. Your client only needs 1 default gateway, no need to insert custom routes.
 

ramicio

Member
Nov 30, 2022
78
14
8
Hello. I recently got an ICX-6610-24p switch, and I am having trouble with the 40g ports. I bought an Intel XL710-QDA1 NIC, and my QSFP+ DAC cable is one from FS, just "generic." I am using Ubuntu Server 20.04. I did the tutorial for licensing and whatnot and everything seems to work. 10g front ports work. On the back I have the QSFP+ cable plugged into the top left of the 4 ports. I cannot get an IP address via DHCP. It, however, does show up as a client on my router (where I have static DHCP stuff set). Pings to and fro just fine if I use an internal IP address. Go to ping a site like google, and it can't even get any DNS record for that. I updated the firmware of the NIC. No change. I can set it to a static IP address and no change. ifconfig shows almost as many RX errors as there are RX packets. None on the TX side of things. If I plug it into the breakout ports, nothing (even though I followed the tutorial and that's not in the config anymore). If I try the other proper 40g port (bottom left), I get no link, whatsoever. Should I be barking up the tree to the switch seller (fried ports), or is there more I should be trying?

Thank you.