Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Noppadet]

Awesome, thanks. I’m alright with the power draw but hopefully it’s not crazy loud. I currently have my rack in the living room. Spec sheet says 47db so I guess I’ll find out what that means. The 6610 would have been better I think but oh well.

I went ahead and snagged some 40gb qsfp+ cables/modules/NICs since it was reasonable on eBay (~$150 for 3x cards, 3x modules, 2x cables). Hopefully the NICs will work with Promox/Unraid.

It’ll definitely be something to grow into considering I’ve been using an ISP router with a hodgepodge of cat5 and cat5e cables.

Stuff I bought (hopefully I picked this stuff correctly):
3x https://www.ebay.com/itm/Brocade-ICX7400-1X40GQ-1-Port-40-GbE-QSFP-Module/132908345279
2x https://www.ebay.com/itm/HP-656089-001-Infiniband-10-40GB-DP-NIC-649281-B21-Full-Profile/183999048540 (will flash)
1x https://www.ebay.com/itm/649281-B21-661685-001-656089-001-HP-544QSFP-10-40GB-adapter-low-profile/392542482955
1x (2 pack 1m) https://www.ebay.com/itm/Lot-2-Brocade-40G-QSFP-C-0101-40GE-QSFP-Direct-Attached-Cable-1m-ICX-stacking/333158541715

2x vm hosts currently 1x proxmox 1x unraid

 

[fohdeesha]

you might want to cancel one of those 40gbE modules, the 48 port 7450 only supports two of them at once:

details: https://fohdeesha.com/data/other/brocade/icx7450-installguide.pdf

weird limitations like that, costing more than a 6610 with less connectivity regardless of config and a couple other reasons is why I skipped the icx7450 in my main post

 

[Noppadet]

Damn you have a wealth of knowledge. Hopefully the seller will modify and I can save a bit. Thanks (again) for the info.

Looks like I need to familiarize myself with that doc. I can’t wait to get this stuff installed.

 

[Rand__]

weird limitations like that, costing more than a 6610 with less connectivity regardless of config and a couple other reasons is why I skipped the icx7450 in my main post

Maybe you could add them and mention this (or more generic) - at this point they are common enough that they are cheaply available every now and then and they sound nice if one does not know all the nitty gritty details you do

 

[eduncan911]

I know it was posted in this thread before; but, I was unable to find it.

Here's the pinout for the ICX 7250 and 7450 serial cable. It's really only 3 soldering points from an old mini-USB cable you may have.

Ruckus ICX 7250 Switch Technical Specifications
^- scroll down towards the bottom

Serial port specifications (pinout mini-USB)
Pin - Signal - Description
--------------------------------------
1 - Reserved - Not used
2 - UART_RX - Receive data by Ruckus ICX 7250
3 - UART_TX - Transmit data by Ruckus ICX 7250
4 - Reserved - Not used
5 - GND - Logic ground

Serial port specifications (protocol)
Parameter - Value
----------------
Baud - 9600 bps
Data bits - 8
Parity - None
Stop bits - 1
Flow control - None

Also, created a PR against @fohdeesha 's github repo so we don't have to keep looking it up.

Add 7250/7450 Serial Pinout Details by eduncan911 · Pull Request #5 · Fohdeesha/lab-docu

 

[camper8080]

I run the Sunon KDE1204PKV3-MS ones, I have all 3 in there, and I live in Dubai. It survived last summer where the ambient 100+ F room temp was no problem at all for the switch..

Thanks for sharing. Did the noise level drop after replacing the stock fans?

 

[SRussell]

I finally finished reading this entire thread: I think I clocked around 6 hours. I have numerous notes about what to read next to better understand most of what is discussed here.

@fohdeesha Do you have an Amazon wishlist? I don't have a Brocade switch but I feel I should do something given how much of your time and knowledge you have shared.

 

[fohdeesha]

heh no need, just contribute back if you ever find yourself able

 

[kapone]

Can I give my Amazon wishlist?

 

[Dave Corder]

I found an extra pair of ICX6610 two-post rack ears while cleaning this weekend. Going to make an official for-sale post later, unless someone here wants to PM me first...

 

[csementuh]

I have a stupid question: Does it make any sense to stack 2x switches that are on opposite sides of a building, or in out buildings? A core switch and a secondary one. As in to be able to admin them both at once since they'd likely share common stuff like VLANs and network settings.

Or does stacking only make sense for uses where the switches are sitting in the same rack, closet or close proximity and serving the same type of general purpose LAN stuff?

Do you strictly speaking need the 10G uplink for the garage, or could you get away with gigabit? 6430s are absurdly cheap these days, so that might be a option. I don't know as I'd try getting one for just the PoE board as they likely don't match up right, but if all you need is power and not throughput...

Thanks for the recommendation! I run a small business out of the garage, so while nothing earth shattering I'd like to keep it 10G to help with moving files around. I'm basically an overkill guy. 10G really isn't needed for anything I do but since I'm going to waste the time and energy to install fiber between the locations, and buy $15 in Brocade SPF+'s, I guess I might as well do 10G lol.

I'll have to see if someone can identify the POE board from a 6430 and maybe it's a match?

That would be pretty cool! But in reality I'm likely better off buying a 24 port POE 6450 off of eBay and trying to sell my 'POE broken' 48 port 6450 instead. I don't need 48 ports in the garage and I never mind saving some power and heat with the equipment.

I know it was posted in this thread before; but, I was unable to find it.

Here's the pinout for the ICX 7250 and 7450 serial cable. It's really only 3 soldering points from an old mini-USB cable you may have.

Cool, good info! Wish I had it a few weeks back. For mine I was impatient and got all McGuyver on it lol.

Cut off mini-USB cable to CAT6 keystone jack to normal RJ45 terminal serial adapter to USB serial adapter to Putty on laptop. It took about 15 mins of experimentation to get the pinout matched 100% but fortunately worked!

 

[Diavuno]

Hello!
Just picked up a ICX 6610-48P for $120 deliered!

Setting it up now, what's the prefered SPF+ to 10GBE copper?

I See Patrick has been doing quite a few reviews of such modules... I'm just curious what everyone's preferred module is?

Also, has anyone tried the 10Gtek QSPF+ breakout?
https://****/39dAu2o

 

[juey]

Mikrotik S+RJ10

 

[fohdeesha]

Is this now the longest thread in all of STH history?

 

[i386]

Yes
2nd: https://forums.servethehome.com/index.php?threads/intel-xeon-e5-2670-deal-and-price-tracking.7133/
3rd: https://forums.servethehome.com/index.php?threads/es-xeon-discussion.5031/
I think

 

[nthu9280]

This marks post #3000 (reply #2999).

 

[arglebargle]

I know it was posted in this thread before; but, I was unable to find it.

Here's the pinout for the ICX 7250 and 7450 serial cable. It's really only 3 soldering points from an old mini-USB cable you may have.

Ruckus ICX 7250 Switch Technical Specifications
^- scroll down towards the bottom

Serial port specifications (pinout mini-USB)
Pin - Signal - Description
--------------------------------------
1 - Reserved - Not used
2 - UART_RX - Receive data by Ruckus ICX 7250
3 - UART_TX - Transmit data by Ruckus ICX 7250
4 - Reserved - Not used
5 - GND - Logic ground

Serial port specifications (protocol)
Parameter - Value
----------------
Baud - 9600 bps
Data bits - 8
Parity - None
Stop bits - 1
Flow control - None

Also, created a PR against @fohdeesha 's github repo so we don't have to keep looking it up.

Add 7250/7450 Serial Pinout Details by eduncan911 · Pull Request #5 · Fohdeesha/lab-docu

I just did something like this two days ago as well.

I took an 8P8C (RJ45) keystone and attached it to a donor micro USB cable so I could use the same USB Cisco-compatible console cable I use for the rest of my switches.

From memory pins 3 and 6 on the keystone correspond to white/green on the donor mini USB cable, and pins 4 or 5 correspond to ground (USB black.) Double check the RX/TX order before you punch down, I don't remember which was which on the cisco cable off hand.

#DongleLyfe

 

[sash]

Not sure if this is the correct thread to post this... I've installed MCX354A-QCBT (followed excellent conversion guide) on two ESXi servers with similar hardware. One has Supermicro X10SLM-F board, the other Asus P9D-M board. CPUs are also similar: One is E3-1246 v3 the other is E3-1240 v3. VMs have 2 vCPUs and 4 to 8 gigs of RAM depending on the OS type. Running iperf3 tests yields weird results. When I test windows VMs I cannot get over 10G speed even when they are on the same host. When using ubuntu, I get decent speed on the same host, but pretty low when VMs are on different hosts. Any ideas where to look for a bottleneck?

 

[RoachedCoach]

Hi, I'm looking at purchasing a ICX 7250-48P and I found one I like, but the listing notes 8 x 1GbE SFP+ Ports - and I was reading elsewhere these are upgradable to 10Gbe. I went through a lot of these posts but I'm still a bit confused - is that a hardware upgrade, or is that some sort of license?

Sorry if this is covered, there's a lot here to sift through, haha. I don't want to get in over my head.

 

[infoMatt]

Hi, I'm looking at purchasing a ICX 7250-48P and I found one I like, but the listing notes 8 x 1GbE SFP+ Ports - and I was reading elsewhere these are upgradable to 10Gbe. I went through a lot of these posts but I'm still a bit confused - is that a hardware upgrade, or is that some sort of license?

Sorry if this is covered, there's a lot here to sift through, haha. I don't want to get in over my head.

It's a software license upgrade, if you don't have it and you configure the ports as a 10G-fullduplex they will go in a ERR-Disabled state.
On the ICX7xxx series the licences are honor based, so basically you just ask the switch to activate the 10G port saying "yes, I'll buy the licence, I promise"; more details in the first page of this thread.