Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

tony&

New Member
Sep 12, 2018
5
1
3
Thanks to you I had extra motives to get the switch up and running. I read through Brocade's pinout and made an attempt to map out the RJ45 and DB9 which indeed worked, so now I have everything up and running. But yes, the fan is still sounding though.

>show chassis

The stack unit 1 chassis info:


Power supply 1 (NA - AC - Regular) present, status ok

Power supply 2 not present


Fan ok, speed (auto): [[1]]<->2


Fan speed switching temperature thresholds:

1 -> 2 @ 69 deg-C

1 <- 2 @ 64 deg-C


Sensor B Temperature Readings:

Current temperature : 40.5 deg-C

Sensor A Temperature Readings:

Current temperature : 50.0 deg-C

Warning level.......: 66.0 deg-C

Shutdown level......: 76.0 deg-C
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,668
113
31
fohdeesha.com
Well that's not good, it seems yours has the same failure mode as @u238 's did 2 pages back - where it thinks/is trying to send out speed 1 (4.5v) but something in the PWM driver is busted and it still sends the full 12v - I believe he returned his switch. I wonder if you both bought from the same sellers lot? Never seen that failure mode in my life and now two people in this thread have it, strange.

it doesn't happen to have a small black round brocade QR sticker near/on the front does it?
 
  • Like
Reactions: tommybackeast

tony&

New Member
Sep 12, 2018
5
1
3
Indeed sounds similar to what @u238 experienced. No QR stickers visible. This is one of 4 that went up in mid september from an UK Ebay supplier.
 
  • Like
Reactions: fohdeesha

mixmansc

Member
Feb 15, 2016
47
27
18
It seems to me that it might also be this....

"Power supply 2 not present"

That is in the post log tony has a few posts up.

I know on this 6610 I have if you do not have BOTH power supplies plugged in then the fans stay at full speed.

When I started looking for a 6610 I did find a few with only one power supply but when I saw the price of the power supplied I started looking more until I found one with both included.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,668
113
31
fohdeesha.com
It always says that on the 6450, since it only has one integrated/permenant power supply. It has a big molex connector on the back for their weird icx6400 DC power distro thing, but I've never seen anyone use them (that's what it's referring to as not present)
 

Thomas H

Member
Dec 2, 2017
63
25
8
51
I had time to put together ICX 6450 and HP 649281-B21 onto my servers. The HP 649281-B21 has been cross-flashed to latest Mellanox MCX354A-FCBT firmware. Popped them into my proxmox servers and reconfigured my network settings. Used iperf to check out 1G and 10G network speed tests as follows:
root@pve1352:~# iperf -c 192.168.1.23
------------------------------------------------------------
Client connecting to 192.168.1.23, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.24 port 51132 connected with 192.168.1.23 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.09 GBytes 936 Mbits/sec
root@pve1352:~# iperf -c 10.0.10.23
------------------------------------------------------------
Client connecting to 10.0.10.23, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.10.24 port 34174 connected with 10.0.10.23 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 10.9 GBytes 9.38 Gbits/sec
root@pve1352:~#
However, I am having problems trying to use the second port as 40GbE/56Gb InfiniBand. I have tried many combinations and it does not come up. That is no IP address and no LED status light on the nic. See attached screenshot of proxmox admin console. Note enp1s0d1 is not active. I don't know if I have a bad cable (NetApp X6558-R6 from eBay), unsupported setup (e.g., can't mix 10GbE and 56Gb IB), configuration error, etc.
 

Attachments

Last edited:

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
241
43
These switches are Ethernet/IP only, no IB or IPoIB support.
This right here. IB is a completely different (competing) protocol from Ethernet. The only way you'll be able to use IB on the second port is with an IB/VPI switch or direct point to point link between two cards. You'll need to run a subnet manager on one (or both) machines if you're linking point to point or using an unmanaged IB switch.

Also, don't expect 56Gb E/IB without a Mellanox FDR rated cable.
 

u238

Member
Aug 11, 2018
40
10
8
Thanks to you I had extra motives to get the switch up and running. I read through Brocade's pinout and made an attempt to map out the RJ45 and DB9 which indeed worked, so now I have everything up and running. But yes, the fan is still sounding though.

>show chassis

The stack unit 1 chassis info:


Power supply 1 (NA - AC - Regular) present, status ok

Power supply 2 not present


Fan ok, speed (auto): [[1]]<->2


Fan speed switching temperature thresholds:

1 -> 2 @ 69 deg-C

1 <- 2 @ 64 deg-C


Sensor B Temperature Readings:

Current temperature : 40.5 deg-C

Sensor A Temperature Readings:

Current temperature : 50.0 deg-C

Warning level.......: 66.0 deg-C

Shutdown level......: 76.0 deg-C
Ok, weird. This is the exact same issue I have. Even the temperature thresholds are identical, which I noticed are different from several other people's results.
 

u238

Member
Aug 11, 2018
40
10
8
Well that's not good, it seems yours has the same failure mode as @u238 's did 2 pages back - where it thinks/is trying to send out speed 1 (4.5v) but something in the PWM driver is busted and it still sends the full 12v - I believe he returned his switch. I wonder if you both bought from the same sellers lot? Never seen that failure mode in my life and now two people in this thread have it, strange.

it doesn't happen to have a small black round brocade QR sticker near/on the front does it?
I haven't returned the switch yet. I don't have any QR sticker and it looks like we got the switch from totally different places. I bought mine from a US seller.
 
  • Like
Reactions: tony&

Thomas H

Member
Dec 2, 2017
63
25
8
51
These switches are Ethernet/IP only, no IB or IPoIB support.
This right here. IB is a completely different (competing) protocol from Ethernet. The only way you'll be able to use IB on the second port is with an IB/VPI switch or direct point to point link between two cards. You'll need to run a subnet manager on one (or both) machines if you're linking point to point or using an unmanaged IB switch.

Also, don't expect 56Gb E/IB without a Mellanox FDR rated cable.
Thanks for clarification!

@arglebargle In your post #302, you used a NetApp cable for 40GbE back-to-back on two servers. That's what I was trying to accomplish. So I got this NetApp X6558-R6 External SAS Cable QSFP-QSFP 2 Meters 112-00177 cable at eBay. Hooked it up as follows:

Server A <-> HP 649281-B21 <-> NetApp X6558-R6 cable <-> HP 649281-B21 <-> Server B

See picture attached. Notice LED status lights are off for port 2 with QSFP-QSFP cable.
 

Attachments

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
241
43
Thanks for clarification!

@arglebargle In your post #302, you used a NetApp cable for 40GbE back-to-back on two servers. That's what I was trying to accomplish. So I got this NetApp X6558-R6 External SAS Cable QSFP-QSFP 2 Meters 112-00177 cable at eBay. Hooked it up as follows:

Server A <-> HP 649281-B21 <-> NetApp X6558-R6 cable <-> HP 649281-B21 <-> Server B

See picture attached. Notice LED status lights are off for port 2 with QSFP-QSFP cable.
Make sure OpenSM is running -- I didn't even get link lights until I had a subnet manager running. There's also ibstat to tell you what's going on with your links too.
 

tommybackeast

Active Member
Jun 10, 2018
290
111
43
How do I do "this" ? ...... Brocade 6450 ........ I would like to set up Brocade 6450's Port 20 to be a port that only has LAN access but no WAN access. So that way, whatever I plug in there will be on the LAN, but cannot have any internet access.

Is this possible?
 

explosivelobster

New Member
Jul 28, 2018
24
2
3
What does show media display for your cable? I had some random QSFP+ cables, some worked and some didn't. The ones that work for me show this:

Code:
SSH@ICX6610-24-Router>sh med e 1/2/6
Port   1/2/6:Type  : 40GBASE-Passive Copper
Vendor Name: Amphenol         Serial Num: APF13270034F27  Revision: F
EDIT: Sorry hadn't read the previous posts properly and didn't realise you were doing a direct connection, do you have any other cables you could try?

Thanks for clarification!

@arglebargle In your post #302, you used a NetApp cable for 40GbE back-to-back on two servers. That's what I was trying to accomplish. So I got this NetApp X6558-R6 External SAS Cable QSFP-QSFP 2 Meters 112-00177 cable at eBay. Hooked it up as follows:

Server A <-> HP 649281-B21 <-> NetApp X6558-R6 cable <-> HP 649281-B21 <-> Server B

See picture attached. Notice LED status lights are off for port 2 with QSFP-QSFP cable.
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,668
113
31
fohdeesha.com
How do I do "this" ? ...... Brocade 6450 ........ I would like to set up Brocade 6450's Port 20 to be a port that only has LAN access but no WAN access. So that way, whatever I plug in there will be on the LAN, but cannot have any internet access.

Is this possible?
Definitely possible & easy using ACL's, I can post quick instructions when I have more free time tonight or tomorrow.

Arris (Ruckus's parent company) finally got back to me about my request for source code 9000 months ago, they've began publishing: Ruckus ICX 7650 - Browse Files at SourceForge.net

Once the ICX6450 source is up I will probably compile a new u-boot image with a lot of the useful commands unhidden (i2c read/write etc) granted this probably won't be useful for anyone here but me
 

tommybackeast

Active Member
Jun 10, 2018
290
111
43
Definitely possible & easy using ACL's, I can post quick instructions when I have more free time tonight or tomorrow.

Arris (Ruckus's parent company) finally got back to me about my request for source code 9000 months ago, they've began publishing: Ruckus ICX 7650 - Browse Files at SourceForge.net

Once the ICX6450 source is up I will probably compile a new u-boot image with a lot of the useful commands unhidden (i2c read/write etc) granted this probably won't be useful for anyone here but me
Thank you and I shall await you having the time, much appreciated.

I knew Ruckus bought part of Brocade, but did not know Arris owns Ruckus....

that's cool you'll compile new code for us 6450 users, thank you (once the source code is available)
 

dwright1542

Active Member
Dec 26, 2015
370
73
28
48
Anyone know how to turn on CDP advertising? I have it in listen mode "cdp run", but I want it to send info to vsphere.
 

Dawg10

Associate
Dec 24, 2016
220
114
43
Anyone know how to turn on CDP advertising? I have it in listen mode "cdp run", but I want it to send info to vsphere.
From the Command Reference:

cdp enable

Example:

device(config)# interface ethernet 1/1/1
device(config-if-e1000-1/1/1)# cdp enable
 

nezach

Active Member
Oct 14, 2012
208
118
43
I would like the switch to have access to NTP server, but can't figure out how to configure default gateway when running layer 3 software. I have router interface on VLAN and can access local subnet, but not anything else.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,668
113
31
fohdeesha.com
I would like the switch to have access to NTP server, but can't figure out how to configure default gateway when running layer 3 software. I have router interface on VLAN and can access local subnet, but not anything else.
Code:
enable
have a seat in my conf t chair
ip route 0.0.0.0/0 192.168.1.1
ip dns server-address 8.8.8.8
write mem