Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[AllenB]

Got my ICX6610-48P fully nine days ago, literally *on the day* of moving into a new house. Now finally spending some quality time in the (still very messy, stuff on the floor) home office getting set up. What a great piece of used kit!

My main desktop is Ubuntu Linux 18.04 and as some of you probably have, I ran into issues with the old SSH server on the switch. For anyone who's not familiar with getting around it, here's what I did to generate and install a public key, then connect to the switch:

On Linux: (with IP 192.168.1.10/24)
cd ~/.ssh
ssh-keygen -t rsa
ssh-keygen -e -f id_rsa.pub >id_rsa_ssh2.pub # convert to SSH2 format, demanded by the switch
cp id_rsa_ssh2.pub /tftpboot

On ICX6610: (with IP 192.168.1.99/24)
ip ssh pub-key-file tftp 192.168.1.10 id_rsa_ssh2.pub
ip ssh key-authentication yes

On Linux:
ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 -i ~/.ssh/id_rsa 192.168.1.99
# I set up an alias like this in .bashrc
alias icx='ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 -i ~/.ssh/id_rsa 192.168.1.99'

Hope this comes in handy for someone.

Cheers,
Allen

 

[sean]

Another option to the alias is editing your ~/.ssh/config so a plain ssh 192.168.1.99 does the right thing.

 

[fohdeesha]

almost forgot to mention @anlin @Skud and anyone else who bought a 7250 - the 7250 (and most of the icx7xxx series) uses an annoying mini usb connector for the serial port, and it's still rs232 serial too so you can't plug it into a usb port.

You will need one of these - Brocade Console Cable Mini-USB (Male) to RJ45 (Male) 8ft Serial PN 50-1000122-01 | eBay

the ICX7150-C12P has a regular RJ45 serial port

 

[Skud]

almost forgot to mention @anlin @Skud and anyone else who bought a 7250 - the 7250 (and most of the icx7xxx series) uses an annoying mini usb connector for the serial port, and it's still rs232 serial too so you can't plug it into a usb port.

You will need one of these - Brocade Console Cable Mini-USB (Male) to RJ45 (Male) 8ft Serial PN 50-1000122-01 | eBay

the ICX7150-C12P has a regular RJ45 serial port

Ooooo... thanks for the heads up. That’s pretty annoying they did that. I’ve done some quick looking for a pinout but haven’t seen anything.

I have a bunch of mini-USBs I could hack and put an 8P8C/RJ45 keystone there to make it work with my standard console cable.

Riley

 

[fohdeesha]

If you really want to hack something up (I am not responsible if you fry the serial port):

 

[Skud]

If you really want to hack something up (I am not responsible if you fry the serial port):

Awesome! Thanks.

 

[anlin]

almost forgot to mention @anlin @Skud and anyone else who bought a 7250 - the 7250 (and most of the icx7xxx series) uses an annoying mini usb connector for the serial port, and it's still rs232 serial too so you can't plug it into a usb port.

You will need one of these - Brocade Console Cable Mini-USB (Male) to RJ45 (Male) 8ft Serial PN 50-1000122-01 | eBay

the ICX7150-C12P has a regular RJ45 serial port

@fohdeesha Good to know, even though it is annoying to have to buy yet another proprietary cable. How essential to the setup process is having a console cable? Are pins 1 and 4 meant to be left unconnected or pulled to the ground?

@Skud please let me know if you get console access working with a hacked mini usb cable. I don't see a reason why it wouldn't work, but it's always nice to get confirmation.

 

[Skud]

@fohdeesha Good to know, even though it is annoying to have to buy yet another proprietary cable. How essential to the setup process is having a console cable? Are pins 1 and 4 meant to be left unconnected or pulled to the ground?

@Skud please let me know if you get console access working with a hacked mini usb cable. I don't see a reason why it wouldn't work, but it's always nice to get confirmation.

Will do. I won’t get mine until later next week.

I would say they are to be left unconnected. Pin 1 on a standard mini USB is VCC so I suspect they left this disconnected on purpose. Probably because there are a lot of people who would assume that this is like the Cisco USB ports where it’s actually a built in serial converter. They would plug in a standard usb cable and put 5v on pin 1.

Pin 4 is the ID pin on a standard mini USB. Not used in this case. I’m going to leave it disconnected.

 

[anlin]

Will do. I won’t get mine until later next week.

I would say they are to be left unconnected. Pin 1 on a standard mini USB is VCC so I suspect they left this disconnected on purpose. Probably because there are a lot of people who would assume that this is like the Cisco USB ports where it’s actually a built in serial converter. They would plug in a standard usb cable and put 5v on pin 1.

Pin 4 is the ID pin on a standard mini USB. Not used in this case. I’m going to leave it disconnected.

This was my thinking as well. It's unfortunate Brocade didn't take a page from Cisco's book and just stick an FTDI/USB-UART chip in there, would have made things much easier. My switch probably won't arrive for at least another week and a half, so plenty of time to figure out what to do about the console cable.

 

[dwright1542]

Ah, so it was YOU who outbid me at the last second

Oh, well, probably good I didn't win the auction. My wife likely wouldn't be too happy with me when those showed up on my doorstep...

I had a hunch that STH people were bidding on this, so I stayed away. Glad my hunch was right.

 

[fohdeesha]

@fohdeesha Good to know, even though it is annoying to have to buy yet another proprietary cable. How essential to the setup process is having a console cable?

100% essential

This was my thinking as well. It's unfortunate Brocade didn't take a page from Cisco's book and just stick an FTDI/USB-UART chip in there, would have made things much easier. My switch probably won't arrive for at least another week and a half, so plenty of time to figure out what to do about the console cable.

strangely the rest of the ICX7xxx series have exactly this, even the little 7150 (USB-C on the front, which is an embedded serial adapter). Not sure why they omitted the 7250

 

[fohdeesha]

if anyone remembers, a couple months ago I posted how to get into the hidden linux system underneath https://forums.servethehome.com/index.php?threads/brocade-icx6450-icx6610-etc.21107/#post-196452

Well, 2 months later, in this release for the ICX7150, they added a root password and now both local and telnet access require authentication!

Didn't take long to find a way past it though

 

[TrumanHW]

How they managed to get so few CFM per dB while managing to keep the fans 8400 rpm.
(That's what IP is for though.)

Buy a QSFP+ switch that's successfully had OEM fans replaced with quiet, aftermarket fans?

 

[N1h1l1ty]

I'm assuming

M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/2 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/2
------------------------------------------------------------------
M:9 L:0 - chow_qsfp_read, qsfp 3, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 3, port 1/2/7 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/7


is a safe bet that the QSFP to SFP+ DAC breakout I got is no bueno? Any possible way of troubleshooting this further?

EDIT: Upon booting looks like I'm seeing activity on the links and they do show up as up in the switch. Initially when I plugged them in the switch was on and I got nothing, so I reloaded. I guess might be a case of it not liking the modules being hotplugged?

 

[fohdeesha]

hotplugging is definitely supported, are you on the latest software? Which breakout cable is it? never seen those errors on the qsfp ports before

Unrelated: got the ICX6450 dev bootloader running on one of my switches: hastebin

 

[N1h1l1ty]

Yep, on ver 08.0.30 straight from the docs page - looking up the serial for these QSFP breakouts, looks like its this guy here:

5 Meter QSFP+ 4-SFP+ DAC Splitter Cable

They seem to be running nicely now, I managed to get connectivity to the switch, but I'll need to test it hotplugging

 

[N1h1l1ty]

So definitely does not like hotplugging - getting nothing after replugging the DAC on the client end - same error upon reloading the switch

EDIT: Tinkered further, and after another reload its now playing nice with hotplugging on the client end. Not *exactly* sure what the deal is - I did have to manually enable the interface again, though I'm guessing this is more my user error than the switch doing something silly. That being said, its reporting the correct config and state for 1/2/7 so it seems to be ... fine?

 

[fohdeesha]

that's...odd. everyone else in the thread (including myself) have been hotplugging qsfp no problem (in fact, my zfs store would never come online if it couldn't bring up the qsfp interface well after the switch boots)

do you have optics or another dac you could try?

 

[N1h1l1ty]

Unfortunately I don't have another QSFP breakout to test out but if it proves a problem I may pick up one up to test out. Front panel SFP+ ports seem to be fine with Cisco branded optics however.

EDIT: I suppose I should try cold booting the switch and see what happens - Let me do that and report in.

EDIT2: Cold start - same error, but link is brought online automatically, and hotplugging does seem to work though it takes *forever*

EDIT3: Well, things did not go as well as I had hoped. It appears that something is not playing nice with the QSFP breakouts and I'm getting perfect bandwidth on some links but gigabit speeds and crazy jitter on others. I need to do more testing but it's definitely not a duplex mismatch as I double checked all the clients, and I can't think of anything that would cause that on the switch - I checked with the clients hooked up to my 3750E via 10G and everything was hunky dory, so I'm going to operate under the suspicion that these QSFP breakouts are just not playing nice with the ICX6610 - Ordered another QSFP breakout from amazon set up for Cisco, should be here in the afternoon and I can hopefully use that to troubleshoot further. If that one doesn't play nice, I probably screwed up somewhere along the way when flashing the firmware, though everything else seems to work fine. Worst comes to worst I'll return these and the amazon one and go buy the Dell ones you suggested in the first place @fohdeesha

Unfortunately I get the niggling suspicion that me cheaping out and buying these instead of the Dell QSFP breakout is the cause of the issue

 

[N1h1l1ty]

So reporting in - I've got another QSFP cable from Amazon, this time Cisco, and its throwing the same error. So it does not look like the cables are the cause of the issue - Any ideas what I could try? Possibly re-flashing firmware?

EDIT: Re-flashed firmware, have not removed the stack units yet but it is no longer throwing an error on reload.