Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Fallen Kell

Member
Mar 10, 2020
57
23
8
FWIW prices seem to be coming down on the 6610's again. I've seen listings with dual fans and dual psu's for ~199 you may find that it is cheaper to get what you need buying a whole switch if you need more than 1 fan and/or PSU. A number of the listings are spec'ing the psu's now as rev a and b so the sellers are getting more sophisticated. You could also get lucky though and score a rev b psu too...
I can attest to the $199 price range. It depends on the options you look for, but I picked my 6610-24-i up for $175+shipping, which put it about $200. Unfortunately, mine was rev A PSUs, but it was dual PSU and dual fans, which is quieter (according to the documentation) than a single fan/PSU setup (as long as you connect power to both PSUs).

On another note, thanks to this thread, I got my server connected via 40gbe last night. Biggest problem I was having was getting working drivers for the Mellanox connectx-3 VPI card I had, as there was no direct support for Xen Cloud Platform (xcp-ng) 8.0. Once I finally figured out that it was mostly a RHEL/Centos 7.x system, I was able to hack up the Mellanox installer to support it properly (the kernel was throwing me as well as the name, so I was thinking it was based on 8.x).

Now I just need to test it all out some more and get my wifi/router connected via 10gbe with proper VLANs (I am running into issues on the router side with VLAN trunk for the SFP+ port).
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
IBiggest problem I was having was getting working drivers for the Mellanox connectx-3 VPI card I had, as there was no direct support for Xen Cloud Platform (xcp-ng) 8.0
Huh? I run xcp-ng everywhere (on the dev team technically), and use nothing but connectx-3's, they're supported natively and the mellanox driver has been packaged natively forever (like, since xenserver 6.0). Are you sure you don't mean the MST/flashing tools? those are indeed a pain, I just use a debian live cd to get the thing updated and configured out of VPI mode (which can cause issues, sometimes centos will try to stick it in infiniband mode, but if you use mellanox firmware tools to take the ports out of auto, its saved in the card's eeprom and it'll never happen again). Anyway, they show right up in xen natively
 

Fallen Kell

Member
Mar 10, 2020
57
23
8
Huh? I run xcp-ng everywhere (on the dev team technically), and use nothing but connectx-3's, they're supported natively and the mellanox driver has been packaged natively forever (like, since xenserver 6.0). Are you sure you don't mean the MST/flashing tools? those are indeed a pain, I just use a debian live cd to get the thing updated and configured out of VPI mode (which can cause issues, sometimes centos will try to stick it in infiniband mode, but if you use mellanox firmware tools to take the ports out of auto, its saved in the card's eeprom and it'll never happen again). Anyway, they show right up in xen natively
My install of xcp-ng 8.0 did not come with the drivers installed to detect the card. There were no included mlx commands with the OS install and I was not seeing them in the yum repos that were configured out of the box (this is my first xcp-ng system so I am not very familiar with it, but I am very familiar with RHEL/Centos considering I have been admin'ing them for 20 years now...). I got the drivers installed first, but once I had them, I was able to see that my card had the ports set in VPI mode, so I then compiled and installed the MST/flashing tools as well. Those were actually easier then the drivers (well, mainly because I better understood what the underlying OS was after smacking my head against it for the drivers). I then got the ports set to ethernet mode and the system finally detected the card and ports properly as network devices (and my 6610 saw the connection).
 
  • Like
Reactions: tommybackeast

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
they're absolutely there, I would have thrown a fit a year ago if they weren't included as it's all I use:

Code:
[root@xen-home-01 ~]# cat /etc/issue
XCP-ng Host 8.0.0

[root@xen-home-01 ~]# lsmod | grep -i mlx
mlx4_en               155648  0
vxlan                  45056  1 mlx4_en
mlx4_core             331776  1 mlx4_en

[root@xen-home-01 ~]# lspci | grep -i mellanox
41:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
I'm quite active on the xcp-ng forum as well, I've probably answered 10+ threads relating to CX3 cards, mostly configuration issues, but never had a single report of a driver missing entirely. You won't have any mlx commands as those are part of MST/tools, not the driver, you'll need to either install the tools separately, or install the giant driver blob from mellanox's site which also includes them (but like I said it's a hassle, I just use a debian live cd to configure the thing and never touch it again)
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
What probably happened now that I think about it, since you never updated/flashed the card with MST first and told it to go into ethernet mode instead of VPI, it probably booted in infiniband mode, which will never load the ethernet driver (because it shows up as an infiniband device)
 
  • Like
Reactions: tommybackeast

Fallen Kell

Member
Mar 10, 2020
57
23
8
What probably happened now that I think about it, since you never updated/flashed the card with MST first and told it to go into ethernet mode instead of VPI, it probably booted in infiniband mode, which will never load the ethernet driver (because it shows up as an infiniband device)
That makes sense. That said, most of the mlx tools/commands are included in the drivers, not the MST (which is why I just assumed it had no drivers included). The MST simply has MST, not the mlx commands.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
OK yeah it must have just been in infiniband mode, just checked on a fresh 8.0 install and it's definitely there, you should have the same files on your install (the manual driver install shouldn't have touched them as it should have installed those under /lib/modules/4.19.0+1/extra/):

Code:
[18:16 R320-MGMT kernel]# modinfo /lib/modules/4.19.0+1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko
filename:       /lib/modules/4.19.0+1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko
version:        4.0-0
license:        Dual BSD/GPL
description:    Mellanox ConnectX HCA low-level driver
author:         Roland Dreier
srcversion:     2DE9EA324DE8A2DEE857C93

[18:20 R320-MGMT kernel]# modinfo /lib/modules/4.19.0+1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko
filename:       /lib/modules/4.19.0+1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko
version:        4.0-0
license:        Dual BSD/GPL
description:    Mellanox ConnectX HCA Ethernet driver
author:         Liran Liss, Yevgeny Petrilin
srcversion:     4417EB14A4203408F8CD076
depends:        mlx4_core,devlink
 
  • Like
Reactions: tommybackeast

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
@fohdeesha Any thoughts on the question above? Haven't been able to find any info in this thread.

everything I could find in the install guide mentions that they're probably normal ports, they just don't have PoE and are full duplex only. So unless you plan on plugging in stuff from 1990 you're probably fine. says they should show up as 1/2/1 and 1/2/2
 
  • Like
Reactions: tommybackeast

CorvetteGS

Member
Jan 20, 2014
40
5
8
Atlanta, GA
everything I could find in the install guide mentions that they're probably normal ports, they just don't have PoE and are full duplex only. So unless you plan on plugging in stuff from 1990 you're probably fine. says they should show up as 1/2/1 and 1/2/2
Thanks a lot for the quick response! Also, when you refer to the guide, do you mean this: ICX7150 - Fohdeesha Docs I did read through it but didn't see any references to the uplink ports. If it's another guide, could you please link the reference?
 

epicurean

Active Member
Sep 29, 2014
785
80
28
trying to load L3 firmware onto the icx6450, but getting "TFTP to flash error code 5". I have disabled the private, public and domain network firewall. still same error. Is there something else I need to do?
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
trying to load L3 firmware onto the icx6450, but getting "TFTP to flash error code 5". I have disabled the private, public and domain network firewall. still same error. Is there something else I need to do?
Which TFTP software are you using?
Does it show any log/connection attempt/download statistics?

Does the switch have an IP on a VE on the right VLAN? Can you ping your PC/TFTP server from the switch itself?

Sorry for this bunch of questions, but it's what I will check in the same situation...
 

jzeus

New Member
Jan 22, 2017
19
4
3
After a failed flash attempt on icx7150, the next boot says the flash is to be reformated and device keys deleted. The following command fails:

Code:
SSH@ICX7150-C12 Router(config)#dm verify-device-certs
Commencing sanity check for device certs ...
Verifying TPM files ...
Failed: Check TCSD_PS Files
The following shows the key file is missing:

Code:
SSH@ICX7150-C12 Router(config)#dm read_device_cert_and_key 0
Error: read_private_key_from_tpm, Private key file ../opt/tpm/mfg-wrapped-key.pem does not exists...!!
Anyway to fix this?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
I think you just need to generate new certs, can't remember the command, might require zeroing the flash completely from u-boot then flashing a fresh image so it boots empty and does all the initial setup. might have time to look later tonight
 
  • Like
Reactions: tommybackeast

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
found these looking through the list of hidden dev commands, looks like you can generate some using your favorite keypair generation tool and import them using scp (I'm guessing it won't do it over tftp because it's not secure):

Code:
copy scp flash <> <> device-private-key
copy scp flash <> <> device-certificate
I'm guesing the <> <> are placeholders for scp server IP and filename. do a ctrl+f for cert in this list you'll see a bunch to try https://fohdeesha.com/data/other/brocade/FastIron-Hidden.txt
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
I've got a 6610 on the way, before I start messing with it I'd like to dump/save everything I can as backups (licences, firmware), feel free to just point out the previous posts that cover that
as for backing up firmware, there's no need as it's all available freely on ruckus' website, it will most likely come with an ancient version anyhow. as for backing up licenses, you're going to need to follow my guide with either a lot of patience or a PPC JTAG unit, they weren't really meant to be extracted off the switch: Extracting Licenses - Fohdeesha Docs

as for the fans, don't bother, no quieter fans that move the same air in the same form factor, and re-thermal compounding everything will gain 1c, maybe 2c if you're lucky
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Unrelated Question : was just watching a youtube video on 100GBe Switches. I"m certainly not buying one; but was curious about their noise : If you have experience with one, are they as insanely loud and power hungry as one might guess? What price range are they in? $10,000? 5000?
it totally depends on the generation of switch, the first few 100gbe switches indeed sucked a lot of power (300w+) and put out a lot of air/heat. The later gens are much closer to switches in this thread power/heat wise, somewhere in the 200w range. You can get a lot of older used 100gbE chassis on ebay for ~2000, the edge-core 7512-32X for example. models this cheap are the early gen 100gb ASICs and you can expect something stupid like ~350w power draw with nothing plugged in and incredibly loud fans to remove 350w worth of heat

The "catch" with most of this stuff also is they're designed for running your network OS of choice, meaning they come with a bootloader and nothing else. You're expected to provide your own OS to run, like Cumulus, which has its own license costs on top of everything else. there's free/open source NOS solutions like microsoft's SONIC, but hardware support varies wildly between all the NOS products
 
  • Like
Reactions: tommybackeast

tommybackeast

Active Member
Jun 10, 2018
286
105
43
it totally depends on the generation of switch, the first few 100gbe switches indeed sucked a lot of power (300w+) and put out a lot of air/heat. The later gens are much closer to switches in this thread power/heat wise, somewhere in the 200w range. You can get a lot of older used 100gbE chassis on ebay for ~2000, the edge-core 7512-32X for example. models this cheap are the early gen 100gb ASICs and you can expect something stupid like ~350w power draw with nothing plugged in and incredibly loud fans to remove 350w worth of heat

The "catch" with most of this stuff also is they're designed for running your network OS of choice, meaning they come with a bootloader and nothing else. You're expected to provide your own OS to run, like Cumulus, which has its own license costs on top of everything else. there's free/open source NOS solutions like microsoft's SONIC, but hardware support varies wildly between all the NOS products
Thank you for satisfying some basic curiosity I had re 100GBe Switches.
 

CorvetteGS

Member
Jan 20, 2014
40
5
8
Atlanta, GA
I am having trouble updating my ICX7150-C12P to the latest PoE firmware per the @fohdeesha guide. I had no issues upgrading the bootrom and primary images. I noticed upon reload after the 8.0.80e upgrade, the switch did an automatic PoE firmware upgrade which appears to be successful, see below:
Code:
PoE Info: Current Firmware version 2.1.0, Recommended Firmware version 2.1.1,  Upgrade Required.
PoE Info: PoE FW upgrade is required. Auto upgrade will start now.
Firmware version from File: 2.1.1
PoE Warning: Upgrading firmware in slot 1....DO NOT SWITCHOVER OR POWER DOWN THE UNIT.
PoE Info: FW Download on slot 1...sending download command...
PoE Info: FW Download on slot 1...TPE response received.
PoE Info: FW Download on slot 1...sending erase command...
PoE Info: FW Download on slot 1...erase command...accepted.
PoE Info: FW Download on slot 1...erasing firmware memory...
PoE Info: FW Download on slot 1...erasing firmware memory...completed
PoE Info: FW Download on slot 1...sending program command...
PoE Info: FW Download on slot 1...sending program command...accepted.
PoE Info: FW Download on slot 1...programming firmware...takes around 5 minutes....
U1-MSG: PoE Info: Firmware Download on slot 1.....10 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....20 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....30 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....40 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....50 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....60 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....70 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....80 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....90 percent completed.
U1-MSG: PoE Info: Firmware Download on slot 1.....100 percent completed.
PoE Info: FW Download on slot 1...programming firmware...completed.
PoE Info: FW Download on slot 1...upgrading firmware...completed. Module will be reset.
PoE Info: Resetting in slot 1....
PoE Info: Resetting module in slot 1....completed.
PoE Info: Programming Ruckus defaults.....
PoE Info: Programming Ruckus defaults. Step 1: Writing port defaults on module in slot 1....
PoE Info: Programming Ruckus Defaults: Step 2: Writing PM defaults on module in slot 1.
PoE Info: Programming Ruckus defaults. Step 3: Writing user byte 0xf4 on module in slot 1.
PoE Info: Programming Ruckus defaults. Step 4: Saving settings on module in slot 1.
PoE Info: Programming Ruckus defaults....completed.
PoE Info: PoE module 1 of Unit 1 initialization is done.
When I get to the section of upgrading the PoE firmware in the guide, immediately the TFTP server shows a transfer is initiated but no data is transferred and the server log shows ACK timeout. See below:
Code:
Connection received from 192.168.1.55 on port 1027 [30/03 17:46:44.647]
Read request for file <icx7xxx_poe_02.1.1.b002.fw>. Mode octet [30/03 17:46:44.648]
OACK: <blksize=2048,tsize=159316,> [30/03 17:46:44.648]
Using local port 61045 [30/03 17:46:44.648]
TIMEOUT waiting for Ack block #1  [30/03 17:47:03.652]
Is the firmware that auto-upgraded on reload the same as the one I'm trying to update to per the guide? They are both 2.1.1, but the file fohdeesha provides on his site has b002 appended to the end. Not sure if there is a difference.

Thanks for the input.

*EDIT*
Resolved via an alternate method, see post.
 
Last edited: