Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

jzeus

New Member
Jan 22, 2017
17
1
3
Thanks to this forum and fohdeesha, I have installed my converted 649281-B21(now a MCX354A-FCBT) Network card in my QNAP NAS and connected it at 40GbE via the Brocade ICX6610-48P. I am using the NetApp 112-00178 X6559-R6 External SAS Cable to connect the two devices.

View attachment 10316

View attachment 10317

Still running tests on performance, but so far everything is working perfectly. Contemplating enabling Jumbo frames, but reading some of the replies on this thread, not sure it is worth the hassle.
Does 40G work 1/2/1 too?
 

Zervun

Member
Feb 2, 2019
44
6
8
Oregon
Awesome thread find thanks for all your work, longtime lurker and just registered - I just ordered one of these on ebay (48p-e) and am going to get a second one here soon. Upgrading from an aruba s2500
 
  • Like
Reactions: fohdeesha

maes

Member
Nov 11, 2018
64
30
18
@fohdeesha do you have a specific model number on these? Are you getting the HP branded ones for this cheap or Mellanox ones?

Thanks!
Seeing them in the high 20s for the MCX311A-XCAT version, depending on the bracket size. I've seen them lower before, too.
 

SuperServer

New Member
Feb 1, 2019
15
1
3
So prior to purchasing all of this I would like to run this by everyone to ensure i get the correct supplies for my upgrade. Basic setup is...

ICX6610 in rack in garage
Server in rack in garage


Connecting these 2 at 40G

2X
Avago 40GB QSFP+ AFBR-83PDZ Transceiver Parallel Ethernet InfiniBand 850nm 40G 654323094729 | eBay

1X
https://www.amazon.com/FiberCablesD...ture_keywords_four_browse-bin:7800973011&th=1

1x (install in server)
HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21, Full Profile | eBay


Then ICX6610 in rack in garage
PC in livingroom (65 feet)

Connecting these 2 at 10G

1X
https://www.amazon.com/FiberCablesD...ture_keywords_four_browse-bin:7800973011&th=1

2X
https://www.amazon.com/FiberCablesD...ture_keywords_four_browse-bin:7800973011&th=1

1X (install in PC)
HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21, Full Profile | eBay

I am pretty sure this is correct, if anyone has a better recommendation for any reason please let me know or for cheaper etc.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,816
1,582
113
29
fohdeesha.com
those 40gb transceivers require MTP or MPO fiber, not LC. On top of that they need to be reversed, type B MPO/MTP, which is harder to find. more info here: https://forums.servethehome.com/ind...icx6450-icx6610-etc.21107/page-23#post-201367

if they're that close, just use DACs: Brocade 40G-QSFP-C-0101 40 GBPS Direct-Attached QSFP+ TO QSFP+ Copper Cable 1M | eBay

there's much cheaper DACs under infiniband listings if I remember right, someone else will link them

for longer distances active DACs are generally cheapest IBM 59Y1930 30M QLogic Infiniband QDR QSFP Optical Cable | eBay
 

Zervun

Member
Feb 2, 2019
44
6
8
Oregon
Are cisco DAC cables and Finisar SFP+s friendly with the 6610? I had ordered a few of those those for the s2500 and wondering if they would work with the non 40g ports.
 

SuperServer

New Member
Feb 1, 2019
15
1
3
those 40gb transceivers require MTP or MPO fiber, not LC. On top of that they need to be reversed, type B MPO/MTP, which is harder to find. more info here: https://forums.servethehome.com/ind...icx6450-icx6610-etc.21107/page-23#post-201367

if they're that close, just use DACs: Brocade 40G-QSFP-C-0101 40 GBPS Direct-Attached QSFP+ TO QSFP+ Copper Cable 1M | eBay

there's much cheaper DACs under infiniband listings if I remember right, someone else will link them

for longer distances active DACs are generally cheapest IBM 59Y1930 30M QLogic Infiniband QDR QSFP Optical Cable | eBay
This would be the correct one...

https://www.amazon.com/Jeirdus-MPO-...ds=mtp+20m&dpPl=1&dpID=51s0ye88QbL&ref=plSrch

The cheaper version that would to be flipped would be this one.

https://www.amazon.com/GOWOS-Standa...=AC_SX236_SY340_QL65&keywords=mtp+cable&psc=1
 

Zervun

Member
Feb 2, 2019
44
6
8
Oregon
So I went to order that, and the seller said the following

There was a receiving error with the listing. We have listed the 40G-QSFP-C-0101, which would be ruckus p/n : 58-0000033-01, and we have pictured a 40G-QSFP-QSFP-C-0101; ruckus p/n: 58-0000041-01

Trying to google and figure it out, it appears that maybe the 41-01 is active while 33-01 is passive. Which one should I order? I read that the actives go longer distances, but I'm not sure if an active one could act passively if need be.

I was going to get 2 - one to pair up with stacking the 2x6610s switches, and one going to a Mellenox Connect 3 (IBM flashed) into a x720xd
 
Last edited:

Zervun

Member
Feb 2, 2019
44
6
8
Oregon
looking up that part number it's still just the regular qsfp to qsfp twinax DAC, will work fine
I went ahead and requested the 58-0000041-01 from the seller. Figured if anything it would have the "active" capability if needed. If you zoom in on the cable it does say "active" in the pictures. Pictures I have found with the 58-0000033-01 do say "passive" on them.

I found a few articles on passive/active.

What's the Difference between Active and Passive Twinax Cable?
Active vs. Passive Direct Attach Cables (DAC)

Even though this cable is short (5m) I figured it being active might be better.
 

kyled

New Member
Jan 25, 2019
5
0
1
Has anyone swapped the fans in an ICX7250? I've tried a few and they all stop spinning once the switch boots