Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

klui

Active Member
Feb 3, 2019
161
67
28
QSFP+ and SFP+ are different standards.
It's just a lot of acronyms that are very confusing. Kinda like 802.11g, a, n, ac, ax. They should really rename them.

Because the SFP and QSFP form factors are common place in data centers newer data rates have piggybacked on these two for newer standards when possible.

SFP (1G); SFP+ (10G); SFP28 (25G, 'cuz it maxes out at 28Gbps)
QSFP (4 x 1G); QSFP+ (4 x 10G; 40G [+56G for Mellanox]); QSFP28 (100G; 4 x 28G)
OSFP (400G)
QSFP-DD (400G)

They're backward compatible. For example, QSFP28 to SFP28 breakouts should (ours at work does) support 40G to 4 x 10G. But QSFP+/SFP+ most likely will not support 100G/25G/4 X 25G.
 

Schut

New Member
Apr 13, 2020
9
2
3
I've got an ICX-6610-48P on the way along with some mikrotik S+RJ10 SFPs that I'm planning on using to create a 10G link over CAT6 to a Mikrotik CRS305-1G-4S+IN that will sit on my desk. The final piece of my puzzle is the link from the switch to the desktop. From this thread it seems like the best bet is a ConnectX-3 (is that still the case?). I'm just not sure what I need to buy for the QSFP (??) to Mikrotik. I'd prefer to go fiber. If anyone could point me to the correct parts and cable I need to do 10G from the Mikrotik to the Desktop I'd appreciate it.
Just to be completely clear, the Mikrotik CRS305-1G-4S+IN you say you want to connect to your desktop only has SFP+ ports on it. It does not have QSFP or QSFP+ ports on it, so those aren't relevant.

You have these choices as I see it:

1a) Another copper run to a copper card:
* A copper 10gbe card (I've been using Intel X540-T2s, and even used on EBay where I get them, they are pricey.)
* Another mikrotik S+RJ10 or similar for the SFP+ port on the CRS305-1G-4S+IN.
* A Cat6a copper cable

1b) Another copper run to an SFP+ card:
* A dual or single port SFP+ card. (see 2a below for details)
* A pair of mikrotik S+RJ10 or similar for the SFP+ port on the CRS305-1G-4S+IN and the SFP+ NIC card
* A Cat6a copper cable

2a) A fiber run:
* A dual or single port SFP+ card. I've been using Solarflare SFN7122Fs which go for $30-$40 on Ebay, but there's also the ConnectX-3 which I have no personal experience with, and there are surely others as well.
* Two 10GBASE-SR SFP+ fiber transceivers. I've been using the house branded ones from fs.com (Cisco SFP-10G-SR Compatible 10GBASE-SR SFP+ Transceiver Module), but I am certain you can save here if you shop around and know what you are doing. I'm just getting started on 10gb so I wanted something I knew would work. Probably $6-$20/transceiver, but the experts here should feel free to set me straight.
* An appropriate length of fiber cable. I've bought different lengths of these for short (<50 foot) fiber runs: OM4 50/125 LC-LC Multimode Fiber Optic Cable Duplex 5m (16ft) - Aqua. Again, you can probably save here, but I'm a novice so I went with fs.com to be sure I'd get something I was confident would work.

2b) A DAC run:
* A dual or single port SFP+ card. (same as 2a; see above.)
* A appropriate DAC (Direct Attached Copper) cable. For example, here's a 10ft cable from fs.com: Brocade 10G-SFPP-TWX-0301 10G SFP+ DAC Twinax Cable. This replaces both fiber transceivers and the fiber cable in 2a.

2a & 2b are the best choices in my book. If your switch is a good distance away in your room - say you have a large room - fiber may be the only practical option of the two - DAC cables are length limited. But if your CRS305-1G-4S+IN switch is very close to your desktop, the 2b (DAC) option is likely best since it will be cheaper and is very simple to understand and use. 1b may be a bit cheaper than 1a, but either will likely be more expensive than either 2a or 2b. I would do Fiber (2a) before I would do 1a unless I had to use copper for some reason, like for example, pre-existing Copper wiring.

In my server room I have been using a mix of all of these, and I'll admit to buying some stuff without cost being my overriding concern, just so I can see how it all works.
 
  • Like
Reactions: safrax and hmw

Rand__

Well-Known Member
Mar 6, 2014
4,555
904
113
There are only two 'newer' options:
25G in x8 or 100G in x16 slots...

Unless you want 100 or features that only newer cards aupport (RocE v2) the CX3 are fine
 

hmw

Active Member
Apr 29, 2019
199
61
28
I'll admit to buying some stuff without cost being my overriding concern, just so I can see how it all works.
I find myself buying 3x the requisite number of required patch cables just so that I can make the rack look pretty :eek: Or read the Great Deals section and blindly buy stuff off eBay

This forum is VERY addictive for that

:D:D
 

hmw

Active Member
Apr 29, 2019
199
61
28
Unless you want 100 or features that only newer cards aupport (RocE v2) the CX3 are fine
I have CX-3 and ESXi 6.7, will probably upgrade to ESXi 7 at some point in which case I would invest in a CX-4 or even CX4-Lx - just so I can use the SR-IOV features
 

vangoose

Active Member
May 21, 2019
263
69
28
Canada
I have CX-3 and ESXi 6.7, will probably upgrade to ESXi 7 at some point in which case I would invest in a CX-4 or even CX4-Lx - just so I can use the SR-IOV features
If you have old platform, like SM X9, sr-iov doesn't work on CX-4 LX cards. It probably looks for sr-iov bios option that doesn't exist. If bios has specific sr-iov option, it will work since it's working on H11 board.

All other cards I have, Intel, Qlogic 57810, Marvell QL41000/45000 all work fine.
 

nerdalertdk

Fleet Admiral
Mar 9, 2017
151
63
28
::1
Hi all

So i startede to design my new lab, and i'm trying to get the Brocade 7250 in to GNS3
have anyone successful done that ?
 
May 1, 2020
37
8
8
I've got an ICX-6610-48P on the way along with some mikrotik S+RJ10 SFPs that I'm planning on using to create a 10G link over CAT6 to a Mikrotik CRS305-1G-4S+IN that will sit on my desk. The final piece of my puzzle is the link from the switch to the desktop. From this thread it seems like the best bet is a ConnectX-3 (is that still the case?). I'm just not sure what I need to buy for the QSFP (??) to Mikrotik. I'd prefer to go fiber. If anyone could point me to the correct parts and cable I need to do 10G from the Mikrotik to the Desktop I'd appreciate it.
I recently grabbed myself this HP card (it's actually a rebranded ConnectX-3 and there are instructions on this forum for how to flash it to Mellanox firmware) and this QSFP+ to SFP+ adapter.

Since it sounds like your desktop will be very close to the switch, then you could get a set of these for the desktop and a DAC cable like this (note: this one is only .5m long, so depending on the distance you might need to find a longer one) and that should be sufficient to get you connected.
 

nerdalertdk

Fleet Admiral
Mar 9, 2017
151
63
28
::1
No, but I have 4 * 7250 in a stack. If you have any question, just ask maybe I can help. I really like 7250.
Well it's more for playing with it, Testing some stuff out since I haven't found a 7250 yet (I'm located in EU, so sadly there is longer between 7250 at a good price)

Going to look at MLAG vs stacking, STP and maybe VRRP at some point
 

EngineerNate

Member
Jun 3, 2017
61
10
8
31
Hi all! Joining the ICX club. Just scored an ICX7250 on eBay from a seller with a good rating out of Montreal. Also picked up the icx7750-rmk kit to replace the rack ears (where do they go on these switches?!? Never seen a piece of gear so often sold without!).

With the USB/serial cable included in the rmk kit work with the 7250? That'd be a nice bonus considering it's cost.

Second, how much trouble is reversing the airflow on these? I like my ToR switch facing the back. I couldn't find a 6610 in the "I" config with PoE or without missing psus/blocking plates or I would have likely fine that route.

It'll be replacing a Nortel/Avaya 4526GTX-PWR as my top of rack switch. Should be a pretty decent capability upgrade. The Nortel will move upstairs to the other end of a fiber link up drive future PoE cameras. I'll only have 2x fiber runs up there so it's 2x xfp ports won't be a huge limitation and it's nice and efficient/quiet with modded fans already.

Cheers!
Nathan
 

LodeRunner

Member
Apr 27, 2019
50
26
18
I don't know what the port on the 7750 is, but on the 7250, the micro-USB is wired as serial. Do not connect it to a regular PC USB port. I sacrificed a USB cable, wired it to a RJ-45 keystone and then plugged my Cisco serial cable into that. The wire order is in thread somewhere.
 

EngineerNate

Member
Jun 3, 2017
61
10
8
31
Yes, the adapter cable is mini or micro USB (can't remember which) to rj45 with an rj45 to serial adapter included. I already have a USB to serial cable. If it's micro/mini mismatch only the adapter should be cheap.
 

Spearfoot

Member
Apr 22, 2015
65
25
18
62
I recently picked up an ICX7150-C12P on Ebay. It managed to boot once and then all it would do is flash the leds once when you plugged it in and that was about it. Found the reddit thread recommending this PS SL Power LB130S56K 56V 2.32 130W Power Supply 689192481516 | eBay. FYI they don't go for 30 bucks any more :)

After receiving the replacement supply and removing the old supply, I slid off the plastic mounting retention from the pins (they are tight but it can be done) and flipped them so the output cables were correct. I also took the old AC input retention plastic and installed it on the new power supply so the ac input cable fit correctly.

This fixed the booting issue. After booting with installed firmware 08060d the console log continuously spewed POE power errors. Since this is still the non-ufi based firmware, I had to upgrade to the non-ufi 08090b first and install the new bootloader. After a reboot the POE firmware was upgraded automatically. Error messages were now gone. I then upgraded to the ufi version of 08090b and rebooted. This allowed me to install the latest 08092b ufi only firmware and the 7150 has been working great the last couple of days.

Looking at the original power supply, it seems heat dissipation was an afterthought with all the conductive thermal pads added along with all the shrink wrapped metal bars that were on top of the mosfets(didn't look up the part number so don't quote me on this) to increase contact to the aluminum frame of the original power suppply. I have actually never seen thermal pads this thick before. Really bad design.

Thanks to this forum and @fohdeesha 's posts on these switches which are a hell of a deal.

--MikeP
I had the same experience this morning! Bought an ICX7150-C12P 32 days ago on eBay... this morning it suddenly died. When I tested it I found that the front panel LED's briefly flash when it's plugged in, same as you. So now I'm off to find a new power supply. Thanks for the link.
 
  • Like
Reactions: mpolter

EngineerNate

Member
Jun 3, 2017
61
10
8
31
Does anyone know if it's a problem to reverse airflow in the 7250? I just read in the spec sheet that only the G model is back to side, the others are side to back.
 

EngineerNate

Member
Jun 3, 2017
61
10
8
31
I just noticed in the photos of the unit I bought (had to zoom in) that the metal plate that covers one section of the RJ45 plugs is damaged. How concerned should I be about this?


 

Schut

New Member
Apr 13, 2020
9
2
3
I just noticed in the photos of the unit I bought (had to zoom in) that the metal plate that covers one section of the RJ45 plugs is damaged. How concerned should I be about this?
I would test jacks 21 & 22, but if they link and pass traffic, I wouldn't give it a second thought.
 

EngineerNate

Member
Jun 3, 2017
61
10
8
31
Seller offered to send me a different unit without the damage at the same cost rather than refund, I accepted. I offered to purchase the other unit (listed at a slightly higher cost due to license differences) outright after a refund on the first unit but I guess he figured just swapping the two was easier.

All's well that ends well, assuming the other unit functions fully. :)
 

nickf1227

Member
Sep 23, 2015
71
40
18
29
To my friends here in this thread, I hope one of you has an answer for me.
Are there any reasonably priced and more purpose built alternatives to the mini-usb to db-9 adapters out there?
I literally have to take one of USB mini to rj-45s, convert it to DB9 like in this picture:

I really miss using my Aruba/HP/Cisco either DB-9 to USB or RJ-45 to USB adapters....one nice meter long cable that doesnt have to adapt to 3 different connectors to work...

The resulting setup costs around $40 at minimum and is about 5 meters long which is insane..

s-l1600.jpg

Then to one of these adapters to get it back to USB to my laptop:
s-l1600 (1).jpg