Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[PGlover]

In some of my bedrooms I have smaller switches like the TP-LINK 6 port switch because I don't have enough network ports available in that room. The switch is connected to a port on the ICX6610. Is there anything special I need to configure on the ICX6610 to accommodate other switches connected to the ICX6610? Any global or port specific configuration?

 

[nev_neo]

Are those switches L2/L3 capable switches ?
if they are, i'd think you would need to configure the VLAN tagging.
If not, then just the port the switches are connecting to on the ICX would need to be on the proper untagged VLAN.

 

[N1h1l1ty]

@nev_neo they are full L3

@PGlover if you're just trying to get extra connectivity, there's nothing you need to configure - all the traffic from the "dumb" switch will land on that port and travel untagged on the native VLAN that you have configured (I imagine should be 1 by default) - that being said, if you're configuring VLANs, you need to make sure that port is set to access whichever VLAN you want the devices on that dumb switch to travel on (tagging, as nev_neo explained above)

@fohdeesha - quick question - I probably missed it in the thread (was reading at like, 4 in the morning) but do the QSFP ports on the 6610 need to be licensed to use SFP+ breakout? Or do they work out of the box?

 

[fohdeesha]

Hi all, I’ve been enjoying this thread for a while now. Had decided to take the 10Gb+ plunge at home and started browsing eBay for likely candidates. Spotting an ICX6610 and getting curious quickly led here.

Licensing looks like the only hurdle. @fohdeesha, you’ve done dozens of folks a solid on this. Can you spare one more or, alternately, can someone let me know what licensing to look for prior to buying? It seems there’s some danger of effectively buying a brick if licensing can’t be solved.

Thanks!

whoever needs lics please pm me instead of posting in here

qsfp ports do not need lics, but pm me for them anyway

 

[N1h1l1ty]

So thanks to this thread, I've pulled the trigger on a Brocade ICX6610-24P.

On my short list of switches to replace my (limited 10 gbe connectivity problem) Cisco 3750E were the Cisco N3K, two HP JCA100A (still mulling trying this one out), HP JC102A (likewise still mulling pulling the trigger on a PSU-less one for 100 bucks on ebay) and the Dell 8024 (too pricey). I looked over my potential future port counts and thought about what I need *right now* and I decided that for the time being I can't really justify spending the cash on the N3K when I can just as easily get everything linked up with redundancy to the Brocade and still have some cash left over - (ok, more like a lot of cash left over. Saved 450$ + the shipping to Canada which would have been unsightly for the N3K).

It also leaves plenty of room for potential future expansion with the QSFP breakouts, since I could even pick up a 24 port ICX6450 non-poe in the immediate future to stack with the 6610 for a total of 48 ports of 1gbe connectivity (24 ports PoE, more than enough for me), and 16(18 with the ICX 6450?) of 10gbe SFP+.

 

[e97]

Can anyone confirm these will work for 40Gb with ICX6610: **LOT OF 2** AVAGO QSFP+ 40Gbe TRANSCEIVER AFBR-79EIDZ | eBay

 

[fohdeesha]

they will work but these are better

Avago 40GB QSFP+ AFBR-79EQPZ Transceiver Parallel Ethernet InfiniBand 850nm 40G 654323094729 | eBay

 

[e97]

they will work but these are better

Avago 40GB QSFP+ AFBR-79EQPZ Transceiver Parallel Ethernet InfiniBand 850nm 40G 654323094729 | eBay

In what way? I looked at the specs on the datasheet and they look identical. Other than being New vs Used @ $7.50/ea

 

[fohdeesha]

way better handles and harder to get stuck

 

[e97]

Ah forgot about usability! good point.

Then, I'll probably go with the ones you mentioned earlier in the thread: Brocade XBR-000232 QSFP 57-1000267-01 4*16Gb SWL 100m for DCX8510 EMC | eBay

The price is about the same and they are listed as new. my only concern is the datasheet doesn't say 40Gb or Infiniband support..

 

[fohdeesha]

they're fibre channel, but work perfectly fine. I've used both, and would go with the avago's just to save my fingers

I wouldn't worry about new vs used for optics, it's a serial to LED converter in a metal box, not much to wear out

 

[koisama]

qsfp ports do not need lics

So even in the worst possible case ICX6610 will have 8 10Gb SFP+ via QSFP breakout cables?

 

[fohdeesha]

So even in the worst possible case ICX6610 will have 8 10Gb SFP+ via QSFP breakout cables?

Yes, but again, just pm me for lics

 

[u238]

Possibly a dumb question, but if I were to stack a pair of 6450s with 2 10gb ports, can I use the remaining 2 10gb ports regularly?

 

[fohdeesha]

Possibly a dumb question, but if I were to stack a pair of 6450s with 2 10gb ports, can I use the remaining 2 10gb ports regularly?

yes. you can also just stack with one port on each switch and have 6x left over

 

[svtkobra7]

I'm not sure how I overlooked this thread so many times, but I believe the Brocade ICX 6450-24 + 4 x Mikrotik S+RJ10 will integrate perfectly with my 2 servers which both have 2 x 10 Gb RJ45 ports built in. Amazingly, it looks like this can be accomplished for less than $350! And solidly meets all of my requirements, without any competition in that price range.

I figured it would be prudent to detail my Current State and Proposed Future State (below) to ensure I'm not missing something, so I'd definitely appreciate your kind review and mention of any areas where my proposal is flawed, or "optimizations" which I may have missed. Thanks in advance and sorry for the long post.

Current State

• ESXi-01 (6.7U1) Host (SMCI 6027R-E1R12T w/ X9DRH-7TF)
  • GLAN1 [Intel X540] => Cat6 => ISP Uplink (pfSense "WAN" Interface)
  • GLAN2 [Intel X540] => Cat6 => Cisco SG300-20 (pfSense "LAN" Interface)
  • IPMI LAN => Cat6 => Cisco SG300-20
• ESXi-02 (6.7U1) Host (SMCI 6027R-E1R12T w/ X9DRH-7TF)
  • GLAN1 & GLAN2 [Intel X540] => Cat6 => Cisco SG300-20
  • IPMI LAN => Cat6 => Cisco SG300-20
• Workstation
  • 1 Gb => Cat5e => Cisco SG300-20 (unable to upgrade drop to this being a condo)
• Misc
  • ESXi-01 / GLAN1 connected directly to ISP Uplink so all traffic goes through firewall
  • I was unable to successfully configure VLANs on the Cisco SG300-20 (see Objective)

Future State - Requirements / Objectives

• Requirements
  • Switch Features = Plenty of 1 Gb ports, 4+ 10GBASE-T ports (or SFP+), doesn't produce too much noise / heat, cost efficiency
    
  • Switch Physical Characteristics = Less than 11" Deep
    • Constrained by vertical rack, oriented sideways, which from "posts" to wall = 12"; however, Datasheet, Page 9 states "All 24-port models: 1.7 in. (H) × 17.44 in. (W) × 9.45 in. (D)," so I should be OK here.
    • See Figure 1.
• Primary Objective
  • Achieve 10 Gb connectivity between ESXi Hosts
  • pfSense HA: (1) ISP Uplink <=> Switch Port X; (2) ESXi-01 Host <=> Switch Port Y; (3) ESXi-02 Host <=> Switch Port Z; where Switch Ports X, Y, Z = VLAN 100
• Secondary Objective
  • 10 Gb connectivity between ESXi Hosts and Workstation

Future State - Additional components

• 1 x Brocade ICX 6450-24
  • Question: Is this about the best price I will find => Brocade ICX6450-24 @ $125 ???
• 4 x Mikrotik S+RJ10
  • Question: Is this about the best price I will find => MikroTik S+RJ10 @ $51 ???
• 2 x SMCI AOC-SG-I2 (2 x 1 Gb ports)
  • Comment: These can be had for 2 x SMCI AOC-SG-I2 @ $20 and considering their proposed use, I believe they will get the job done, but if you have a better option, please let me know.
• 1 x 10 Gb NIC for Workstation
  • Comment: Undecided as to which, I wouldn't mind guidance here.
• @fohdeesha 's help!

Proposed Future State - "Architecture"

• ESXi-01 Host
  • GLAN1 [Intel X540] => Cat6 => S+RJ10 | 6450-24 | SFP+ Port 1/4
  • GLAN2 [Intel X540] => Cat6 => S+RJ10 | 6450-24 | SFP+ Port 2/4
  • IPMI LAN => Cat6 => 6450-24 | RJ45 Port = VLAN 200
  • SMCI AOC-SG-I2
    • Port 1 => Cat6 => 6450-24 | RJ45 Port = pfSense "WAN" Interface = VLAN 100
    • Port 2 => Cat6 => 6450-24 | RJ45 Port = ESXi Management = VLAN 300
• ESXi-01 Host
  • GLAN1 [Intel X540] => Cat6 => S+RJ10 | 6450-24 | SFP+ Port 3/4
  • GLAN2 [Intel X540] => Cat6 => 6450-24 | RJ45 Port
  • IPMI LAN => Cat6 => 6450-24 | RJ45 Port = VLAN 200
  • SMCI AOC-SG-I2
    • Port 1 => Cat6 => 6450-24 | RJ45 Port = pfSense "WAN" Interface = VLAN 100
    • Port 2 => Cat6 => 6450-24 | RJ45 Port = ESXi Management = VLAN 300
• Workstation
  • 10 Gb NIC => Cat5e => S+RJ10 | ICX 6450-24 | SFP+ Port 4/4

Figure 1

 

[N1h1l1ty]

This is probably lazy of me - is there a verdict for cheapest transceiver that just *works* with the ICX6610? Will it play nice with Cisco SR ones? (Those seem to be easiest to find / cheapest on fleabay)

EDIT: Nvm the Brocades seem to be cheap enough where I might as well buy those. Brocade 57-0000075-01 10GbE Ethernet SR 850nm XBR-000180 10G-SFPP-SR SFP+ 8904725111763 | eBay

 

[fohdeesha]

literally anything works

with brocade branded optics, you can enable optical monitoring (to get optical link measurements)

 

[u238]

yes. you can also just stack with one port on each switch and have 6x left over

Nice. I might go that route.

 

[N1h1l1ty]

Went with two of these: QSFP+ to 4-SFP+ Passive Direct Attach Cable 5m 602630005 | eBay

Pretty good deal, seller accepted 12$ for each of them.