Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Raice]

Does anybody know if these switches support PFC and RDMA? Is it possible to create RoCE lossless network for storage using it?

 

[klui]

Does anybody know if these switches support PFC and RDMA?

@fohdeesha knows and that's been answered already.

https://forums.servethehome.com/index.php?search/42162165/&q=pfc&t=post&o=relevance&c[thread]=21107

 

[Anubis]

So, thanks to this thread I now have two 6450-24p sitting on my desk, waiting to be configured
I've gone through the guide and upgraded them both, and I'm at the step where I intend to stack the two switches.
As I understand it, one cable should be enough, but does it matter if that cable is from port 1 -> port 3 or can it be from port 1 -> port 1?

 

[fohdeesha]

the port to port doesn't matter, however if I remember right, the default config only has ports 1 and 2 set for stacking. (or it could be 1 and 3, I can't remember, it'll be one of the first lines in "show run"). If you want to use a port that's not specified there you'll have to change the stack ports

 

[Anubis]

"show run" displayed ports 1 and 3 as stack ports, so I connected port 1 ->port 1, created the stack and ran "no stack-port 1/2/3" and "no stack-port 2/2/3". I'm guessing that should make port 3 available as a regular 10G port?

 

[infoMatt]

Just a quick message to say that, after reading all messages of this thread, and thanks to all the information that @fohdeesha and all other valuable user have provided, I've toss the coin and I am an owner of an ICX6450-24P!

That thing is really well made! I haven't got much time to tinker with, but I've managed to update it to the latest release (thanks to the guide linked on the first page) and apply a very basic config, just to keep it usable.
To me though it seems a bit loud, louder than say a new-ish Cisco 2960x, even with the fans spun down; to have a reference, I can hear it from 2 rooms in the quiet of the night in a all-brick Euro house (with the doors opened, btw).
I need to try and oil the fans, maybe they're simply dry (EDIT: done -- no appreciable change in noise profile); otherwise it doesn't matter, I'd buy two new fans, not the end of the world.

After about two hours, a "show chassis" shows that it was at 46°C with the room at 28-29, so no heat concerns... with the hand, I could barely feel any heat exhausted from the fans, in comparison with the room air around it.

Now I have to study the Brocade syntax, and start thinking about the network that I want to build... maybe I'll tear the config apart and restart from scratch a couple of times as I change my mind we'll see!

For me, we've found the "unicorn switch", that's for sure.

 

[AngryLatinChanting]

Aww man, just discovered this fantastic thread yesterday... 6610-48P is definitely my next hot item to get. Long time lurker, but I tend to stay in my lanes unless I am buying something out of the norm. I kinda regret that now, because this thread saved me from getting the Microtik crs309-1g-8s+in and being forever salty about the L3 speeds.

But looks like I showed up 1 month late to the party, since the licensing favours @fohdeesha was doing for the community had to (of course) be taken advantage of and thus ruined for the rest of us. Nor do I want to pay $100+ more to sellers on principle alone, but even without the price increase this switch is pushing close to $500 CAD all in. Considered the 7250 just to avoid dealing with this licensing crap, but they cost just as much or more, and I'd take the hit to much needed 10g connectivity by not having the qsfp+ breakout

Guess I'm going to have to be extra patient and wait a few months for a good deal to pop up. I just hope the main wave of old stock isn't coming to a close. Prices have definitely gone up since the start of this thread!

 

[infoMatt]

But looks like I showed up 1 month late to the party, since the licensing favours @fohdeesha was doing for the community had to (of course) be taken advantage of and thus ruined for the rest of us.

Please read the thread 'till the end... you've just missed the latest news about the licensing for the community.

Guess I'm going to have to be extra patient and wait a few months for a good deal to pop up. I just hope the main wave of old stock isn't coming to a close. Prices have definitely gone up since the start of this thread!

Well... being patient and snap the deals when they come up it's always a good strategy, unless you specifically need the switch right now.

I've waited just over two leeeeengthy months as I've missed the opportunity to make an offer to a ICX 6450-24P, the only one as far as I could tell sold by an EU ebayer at a right price (~175 EUR + ship.), as it went out of stock.. just some weeks ago the seller added some other to the auction, and so it is now on my desk

 

[AngryLatinChanting]

Please read the thread 'till the end... you've just missed the latest news about the licensing for the community.

Oof, just went back and saw the post on page 100. I didn't miss it, just read it out of order apparently as I was probably bouncing around the thread last night before I decided to go in order today. Is this what you were referring to or something else? Apologies if I'm missing something obvious.

One more question for either @fohdeesha or anyone else that knows, but do these licenses only enable the 10g SPF+ functionality or also the advanced routing capability?

 

[kroy]

@AngryLatinChanting there are two. One enables the PoD (ports) and one enables advanced routing.

 

[Dawg10]

@AngryLatinChanting Where in Canada? AB here; local would be nice.

If you're interested, I'm planning on selling a 6610-24P, two Mellanox CX353A 40Gbe cards and two Mellanox FDR 40Gbe cables, all working together under ESXi 6.7. There's also a 6450-24 I no longer use.

I'm now set up with the servers on 10G DAC into a 7250-24, 10G fiber over to a 6450-48P supporting the 1G environment.

FYI & fair warning: 40G works but requires a lot of tuning to get the speeds up (as in: max out the performance/ power draw of all attached devices. Last winter it kept the garage warm but it's too much in the summer).

 

[AngryLatinChanting]

@AngryLatinChanting Where in Canada? AB here; local would be nice.

If you're interested, I'm planning on selling a 6610-24P, two Mellanox CX353A 40Gbe cards and two Mellanox FDR 40Gbe cables, all working together under ESXi 6.7. There's also a 6450-24 I no longer use.

I'm now set up with the servers on 10G DAC into a 7250-24, 10G fiber over to a 6450-48P supporting the 1G environment.

FYI & fair warning: 40G works but requires a lot of tuning to get the speeds up (as in: max out the performance/ power draw of all attached devices. Last winter it kept the garage warm but it's too much in the summer).

Toronto but curious; PM'ed.

 

[infoMatt]

Oof, just went back and saw the post on page 100. I didn't miss it, just read it out of order apparently as I was probably bouncing around the thread last night before I decided to go in order today. Is this what you were referring to or something else?

Yep, I was referring to that post @fohdeesha has decided to continue giving licences to the forum members (thank you!), so you're not screwed up

There are two types of licences, one to enable the PoD (Port on Demand) at 10G, and another one to enable advanced routing.

 

[eleitl]

Verily, it boots:

# picocom /dev/ttyS0
picocom v1.7

port is : /dev/ttyS0
flowcontrol : none
baudrate is : 9600
parity is : none
databits are : 8
escape is : C-a
local echo is : no
noinit is : no
noreset is : no
nolock is : no
send_cmd is : sz -vv
receive_cmd is : rz -vv
imap is :
omap is :
emap is : crcrlf,delbs,

Terminal ready
$
ICX Boot Code Version 7.3.02 (grz07302)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 1fef
BOOT INFO: verify flash files........
BOOT INFO: load image from primary copy...

platform type = 8
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
.....................
.....
Starting Main Task ...CPSS DxCh Version: cpss3.4 release
Pre Parsing Config Data ...

Parsing Config Data ...
stacking is enable. optical monitoring for stacking ports 1/2/1, 1/2/6 is not available.


SYSLOG: <14>0d00h00m09s:ESC-SW1-BldgC System: Cold start

SYSLOG: <15>0d00h00m09s:ESC-SW1-BldgC DHCPC: starting dhcp client service on 65 port(s)
Copyright (c) 1996-2011 Brocade Communications Systems, Inc.
UNIT 1: compiled on Apr 25 2012 at 16:45:34 labeled as FCXR07300c
(6811631 bytes) from Primary fcxr07300c.bin
SW: Version 07.3.00cT7f3
Boot-Monitor Image size = 370555, Version:07.3.02T7f5 (grz07302)
HW: Stackable ICX6610-24
==============================
============================================
UNIT 1: SL 1: ICX6610-24 24-port Management Module
Serial #: xxxxxxxxxxx
License: ICX6610_PREM_ROUTER_SOFT_PACKAGE (LID: xxxxxxxxxxx)
P-ENGINE 0: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
512 MB DRAM
STACKID 1 system uptime is 9 seconds
The system : started=cold start
My stack unit ID = 1, bootup role = active

........................
ESC-SW1-BldgC>
Power supply 1 detected.
Power supply 1 is up.

SYSLOG: <14>0d00h01m33s:ESC-SW1-BldgC System: Stack unit 1 Power supply 1 is up

Power supply 2 detected.
Power supply 2 is up.

SYSLOG: <14>0d00h01m33s:ESC-SW1-BldgC System: Stack unit 1 Power supply 2 is up

SYSLOG: <12>0d00h01m40s:ESC-SW1-BldgC NTP: Server 10.20.144.11 failed to respond

SYSLOG: <12>0d00h01m55s:ESC-SW1-BldgC NTP: Server 10.20.144.11 failed to respond

SYSLOG: <12>0d00h02m10s:ESC-SW1-BldgC NTP: Server 10.20.144.11 failed to respond

SYSLOG: <12>0d00h02m25s:ESC-SW1-BldgC NTP: Server 10.20.144.11 failed to respond

SYSLOG: <12>0d00h02m40s:ESC-SW1-BldgC NTP: Server 10.20.144.11 failed to respond

SYSLOG: <15>0d00h02m52s:ESC-SW1-BldgC DHCPC: Stopped DHCP Client service

SYSLOG: <12>0d00h02m55s:ESC-SW1-BldgC NTP: Server 10.20.144.11 failed to respond

SYSLOG: <12>0d00h03m10s:ESC-SW1-BldgC NTP: Server 10.20.144.11 failed to respond

 

[explosivelobster]

Sorry if this has already been asked, but is it possible to reverse the physical mounting (and therefore airflow direction) of the fans in the ICX6610 to convert an -I to an -E variant and vice versa?

 

[infoMatt]

Sorry if this has already been asked, but is it possible to reverse the physical mounting (and therefore airflow direction) of the fans in the ICX6610 to convert an -I to an -E variant and vice versa?

Preface: I don't own that model, so take this with a grain of salt...
Reversing airflow is as difficult as reversing the fan inside it's tray... but logically the switch will continue to believe that the flow is still acting accordingly to the fan/PSU identifier, and it still refuses to boot if you mix a believed-E fan with an actual -I PSU, as it thinks that you're crossing flows, impairing cooling capabilities.

If you do open the tray to reverse the fan, you'll have to do it also inside the PSU, and stick with just that components mounted on the chassis... the moment you'll add a stock fan/PSU tray at best you'll cross flow, at worse the switch will shut itself down because it thinks you're crossing air flows.

 

[Thorzeen]

I am still looking to trade 2 ICX6610 1000w "E" for 2 "I" also fan trays. Just throwing it out there

 

[kroy]

Any downside to upgrading to 08.0.91 on the 7250? I'd love to have the Ansible magic.

 

[fohdeesha]

8080 is the stable codetrain, 8090 less so and 8091 is bleeding edge/expect bugs. Bad enough bugs you'd encounter them in home use? Not sure, worth a try if you're up to it

 

[kroy]

@fohdeesha thanks for the confirmation. I wasn’t really sure what differences there might have been. I might risk it for the Ansible capabilities.

Thanks again for all the help with these devices.