Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

DRAGONKZ

Member
Apr 9, 2018
71
9
8
37
I’m just hoping to get some help getting my stack running, using the 2 x 40Gb ports between 2 ICX-6610-48P units.

I originally planned for 2 units, but due the having no luck getting people to ship to AUS I proceeded with 1 unit for a while, but my second has finally arrived.

My first is configured and working with all required vlans and ve and lags...etc, and I’ve updated the firmware and license on the second and configured a basic ve as part of that process.

I’ve plugged a 40Gb DAC between 1/2/1 ports on both and am trying to follow the guide linked in the first post.

None of the commands in the guide for the second unit work.

Am I right in assuming that a “normal” stack using multiple cables needs to be created first, and then the guide basically blows that config away and configures it differently... I also assume 2 cables between units are required to form a stack?

Ive tried to run the normal stack setup commands with just the one cable plugged in but nothing happens.

Any ideas where I’m going wrong? (Unless it’s just a case of set up a normal stack first with 2 cables as mentioned)

Thanks!
 

Mithril

Member
Sep 13, 2019
96
19
8
Can anyone with a 6610 and a Rev B or C power supply comment on noise with 1 fan tray VS 2? My 6610-24 came with 1 fan tray but if it makes a big noise difference (aka "do the fans spin slower for the same load") I'm going to pick up a second fan tray. Also whats the part number I should search for?
 

itronin

Active Member
Nov 24, 2018
377
232
43
Denver, Colorado
I have 2 6610's with dual rev b and dual fans. YMMV as my hearing may not be as good as yours. I find 2 PSU's and 2 fans quieter than a single PSU and single fan. I have not payed attention to fan spin up or down for load. they tend to be pretty constant in my environment so someone else may need to comment.

ICX-FAN10-E or ICX-FAN10-I (Egress or Ingress)

I'd verify the pictures to make sure the stickers match the listing and airflow direction you require. If it were me I'd probably also contact the seller to verify too.

FWIW prices seem to be coming down on the 6610's again. I've seen listings with dual fans and dual psu's for ~199 you may find that it is cheaper to get what you need buying a whole switch if you need more than 1 fan and/or PSU. A number of the listings are spec'ing the psu's now as rev a and b so the sellers are getting more sophisticated. You could also get lucky though and score a rev b psu too...
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
1,855
1,647
113
29
fohdeesha.com
I’m just hoping to get some help getting my stack running, using the 2 x 40Gb ports between 2 ICX-6610-48P units.

I originally planned for 2 units, but due the having no luck getting people to ship to AUS I proceeded with 1 unit for a while, but my second has finally arrived.

My first is configured and working with all required vlans and ve and lags...etc, and I’ve updated the firmware and license on the second and configured a basic ve as part of that process.

I’ve plugged a 40Gb DAC between 1/2/1 ports on both and am trying to follow the guide linked in the first post.

None of the commands in the guide for the second unit work.

Am I right in assuming that a “normal” stack using multiple cables needs to be created first, and then the guide basically blows that config away and configures it differently... I also assume 2 cables between units are required to form a stack?

Ive tried to run the normal stack setup commands with just the one cable plugged in but nothing happens.

Any ideas where I’m going wrong? (Unless it’s just a case of set up a normal stack first with 2 cables as mentioned)

Thanks!
you need to set up a normal stack first so the second switch becomes a member, eg conf t, stack enable, exit, stack secure-setup on the master switch, then wait while the slave reboots 30 times, eventually "show stack" should show both switches
 

Fallen Kell

New Member
Mar 10, 2020
11
4
3
FWIW prices seem to be coming down on the 6610's again. I've seen listings with dual fans and dual psu's for ~199 you may find that it is cheaper to get what you need buying a whole switch if you need more than 1 fan and/or PSU. A number of the listings are spec'ing the psu's now as rev a and b so the sellers are getting more sophisticated. You could also get lucky though and score a rev b psu too...
I can attest to the $199 price range. It depends on the options you look for, but I picked my 6610-24-i up for $175+shipping, which put it about $200. Unfortunately, mine was rev A PSUs, but it was dual PSU and dual fans, which is quieter (according to the documentation) than a single fan/PSU setup (as long as you connect power to both PSUs).

On another note, thanks to this thread, I got my server connected via 40gbe last night. Biggest problem I was having was getting working drivers for the Mellanox connectx-3 VPI card I had, as there was no direct support for Xen Cloud Platform (xcp-ng) 8.0. Once I finally figured out that it was mostly a RHEL/Centos 7.x system, I was able to hack up the Mellanox installer to support it properly (the kernel was throwing me as well as the name, so I was thinking it was based on 8.x).

Now I just need to test it all out some more and get my wifi/router connected via 10gbe with proper VLANs (I am running into issues on the router side with VLAN trunk for the SFP+ port).
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,855
1,647
113
29
fohdeesha.com
IBiggest problem I was having was getting working drivers for the Mellanox connectx-3 VPI card I had, as there was no direct support for Xen Cloud Platform (xcp-ng) 8.0
Huh? I run xcp-ng everywhere (on the dev team technically), and use nothing but connectx-3's, they're supported natively and the mellanox driver has been packaged natively forever (like, since xenserver 6.0). Are you sure you don't mean the MST/flashing tools? those are indeed a pain, I just use a debian live cd to get the thing updated and configured out of VPI mode (which can cause issues, sometimes centos will try to stick it in infiniband mode, but if you use mellanox firmware tools to take the ports out of auto, its saved in the card's eeprom and it'll never happen again). Anyway, they show right up in xen natively
 

Fallen Kell

New Member
Mar 10, 2020
11
4
3
Huh? I run xcp-ng everywhere (on the dev team technically), and use nothing but connectx-3's, they're supported natively and the mellanox driver has been packaged natively forever (like, since xenserver 6.0). Are you sure you don't mean the MST/flashing tools? those are indeed a pain, I just use a debian live cd to get the thing updated and configured out of VPI mode (which can cause issues, sometimes centos will try to stick it in infiniband mode, but if you use mellanox firmware tools to take the ports out of auto, its saved in the card's eeprom and it'll never happen again). Anyway, they show right up in xen natively
My install of xcp-ng 8.0 did not come with the drivers installed to detect the card. There were no included mlx commands with the OS install and I was not seeing them in the yum repos that were configured out of the box (this is my first xcp-ng system so I am not very familiar with it, but I am very familiar with RHEL/Centos considering I have been admin'ing them for 20 years now...). I got the drivers installed first, but once I had them, I was able to see that my card had the ports set in VPI mode, so I then compiled and installed the MST/flashing tools as well. Those were actually easier then the drivers (well, mainly because I better understood what the underlying OS was after smacking my head against it for the drivers). I then got the ports set to ethernet mode and the system finally detected the card and ports properly as network devices (and my 6610 saw the connection).
 
  • Like
Reactions: tommybackeast

fohdeesha

Kaini Industries
Nov 20, 2016
1,855
1,647
113
29
fohdeesha.com
they're absolutely there, I would have thrown a fit a year ago if they weren't included as it's all I use:

Code:
[root@xen-home-01 ~]# cat /etc/issue
XCP-ng Host 8.0.0

[root@xen-home-01 ~]# lsmod | grep -i mlx
mlx4_en               155648  0
vxlan                  45056  1 mlx4_en
mlx4_core             331776  1 mlx4_en

[root@xen-home-01 ~]# lspci | grep -i mellanox
41:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
I'm quite active on the xcp-ng forum as well, I've probably answered 10+ threads relating to CX3 cards, mostly configuration issues, but never had a single report of a driver missing entirely. You won't have any mlx commands as those are part of MST/tools, not the driver, you'll need to either install the tools separately, or install the giant driver blob from mellanox's site which also includes them (but like I said it's a hassle, I just use a debian live cd to configure the thing and never touch it again)
 
  • Like
Reactions: tommybackeast

fohdeesha

Kaini Industries
Nov 20, 2016
1,855
1,647
113
29
fohdeesha.com
What probably happened now that I think about it, since you never updated/flashed the card with MST first and told it to go into ethernet mode instead of VPI, it probably booted in infiniband mode, which will never load the ethernet driver (because it shows up as an infiniband device)
 
  • Like
Reactions: tommybackeast

Fallen Kell

New Member
Mar 10, 2020
11
4
3
What probably happened now that I think about it, since you never updated/flashed the card with MST first and told it to go into ethernet mode instead of VPI, it probably booted in infiniband mode, which will never load the ethernet driver (because it shows up as an infiniband device)
That makes sense. That said, most of the mlx tools/commands are included in the drivers, not the MST (which is why I just assumed it had no drivers included). The MST simply has MST, not the mlx commands.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,855
1,647
113
29
fohdeesha.com
OK yeah it must have just been in infiniband mode, just checked on a fresh 8.0 install and it's definitely there, you should have the same files on your install (the manual driver install shouldn't have touched them as it should have installed those under /lib/modules/4.19.0+1/extra/):

Code:
[18:16 R320-MGMT kernel]# modinfo /lib/modules/4.19.0+1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko
filename:       /lib/modules/4.19.0+1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko
version:        4.0-0
license:        Dual BSD/GPL
description:    Mellanox ConnectX HCA low-level driver
author:         Roland Dreier
srcversion:     2DE9EA324DE8A2DEE857C93

[18:20 R320-MGMT kernel]# modinfo /lib/modules/4.19.0+1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko
filename:       /lib/modules/4.19.0+1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko
version:        4.0-0
license:        Dual BSD/GPL
description:    Mellanox ConnectX HCA Ethernet driver
author:         Liran Liss, Yevgeny Petrilin
srcversion:     4417EB14A4203408F8CD076
depends:        mlx4_core,devlink
 
  • Like
Reactions: tommybackeast

fohdeesha

Kaini Industries
Nov 20, 2016
1,855
1,647
113
29
fohdeesha.com
@fohdeesha Any thoughts on the question above? Haven't been able to find any info in this thread.

everything I could find in the install guide mentions that they're probably normal ports, they just don't have PoE and are full duplex only. So unless you plan on plugging in stuff from 1990 you're probably fine. says they should show up as 1/2/1 and 1/2/2
 
  • Like
Reactions: tommybackeast

CorvetteGS

Member
Jan 20, 2014
35
4
8
Atlanta, GA
everything I could find in the install guide mentions that they're probably normal ports, they just don't have PoE and are full duplex only. So unless you plan on plugging in stuff from 1990 you're probably fine. says they should show up as 1/2/1 and 1/2/2
Thanks a lot for the quick response! Also, when you refer to the guide, do you mean this: ICX7150 - Fohdeesha Docs I did read through it but didn't see any references to the uplink ports. If it's another guide, could you please link the reference?
 

epicurean

Active Member
Sep 29, 2014
642
35
28
trying to load L3 firmware onto the icx6450, but getting "TFTP to flash error code 5". I have disabled the private, public and domain network firewall. still same error. Is there something else I need to do?
 

infoMatt

Active Member
Apr 16, 2019
148
53
28
trying to load L3 firmware onto the icx6450, but getting "TFTP to flash error code 5". I have disabled the private, public and domain network firewall. still same error. Is there something else I need to do?
Which TFTP software are you using?
Does it show any log/connection attempt/download statistics?

Does the switch have an IP on a VE on the right VLAN? Can you ping your PC/TFTP server from the switch itself?

Sorry for this bunch of questions, but it's what I will check in the same situation...
 

jzeus

New Member
Jan 22, 2017
18
2
3
After a failed flash attempt on icx7150, the next boot says the flash is to be reformated and device keys deleted. The following command fails:

Code:
SSH@ICX7150-C12 Router(config)#dm verify-device-certs
Commencing sanity check for device certs ...
Verifying TPM files ...
Failed: Check TCSD_PS Files
The following shows the key file is missing:

Code:
SSH@ICX7150-C12 Router(config)#dm read_device_cert_and_key 0
Error: read_private_key_from_tpm, Private key file ../opt/tpm/mfg-wrapped-key.pem does not exists...!!
Anyway to fix this?
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,855
1,647
113
29
fohdeesha.com
I think you just need to generate new certs, can't remember the command, might require zeroing the flash completely from u-boot then flashing a fresh image so it boots empty and does all the initial setup. might have time to look later tonight
 
  • Like
Reactions: tommybackeast

fohdeesha

Kaini Industries
Nov 20, 2016
1,855
1,647
113
29
fohdeesha.com
found these looking through the list of hidden dev commands, looks like you can generate some using your favorite keypair generation tool and import them using scp (I'm guessing it won't do it over tftp because it's not secure):

Code:
copy scp flash <> <> device-private-key
copy scp flash <> <> device-certificate
I'm guesing the <> <> are placeholders for scp server IP and filename. do a ctrl+f for cert in this list you'll see a bunch to try https://fohdeesha.com/data/other/brocade/FastIron-Hidden.txt