Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,824
113
29
fohdeesha.com
If you look at the bottom of the 3rd page, it says, "Two 10/100/1000 Ethernet ports with 802.3af/at PoE."

My understanding is that it can do a lower performance mode to be compatible with old switches (802.3af mode). I want high performance.
it's just saying it's backwards compatible with an AT power source (which it should be, it's in the AF spec). I just double checked the full user manual and start guide off of Ruckus' site, the r700 does not negotiate or support AT power. It even directly says "The R700 is a Class 3 device", and even the DC input jack is rated for~15 watts tops

AT power support was introduced with the R710 and R610, those are the ones that reduce features if on AF - "Note that if 802.3af PoE is used to power the R610, the feature set is reduced, as follows: • 2.4 GHz radio chains reduced to 2x3 (2-chain transmit at 21dBm aggregate power, 3-chain receive)" - when this happens it also displays a notification in the UI
 

tommybackeast

Active Member
Jun 10, 2018
258
85
28
Given the high level of network knowledge on this thread, excuse a most basic question re VLAN (something I know nothing about).

I have a Brocade 6450-24P all setup in the basement. On the 1st floor is the TV Media Center . I have one CAT6 cable from the Brocade to a Keystone wall plate at the TV Media Center; and CAT6 from wall to a very old Dell 2708 8-port 1GB Switch. The TV Media Center's Smart TV, Tivo, Apple TV, old WDTV are all connected to this old Dell Switch within the Media Center cabinet.

There is a NAS in the basement next to the Brocade 6450 and the NAS has video,photo,audio files I play from the TV Media Center.

I would like to create a VLAN for the TV Media Center but yet still allows internal LAN access to the files on the LAN NAS.

Can I do this; or must the TV, Tivo, Apple TV, WDTV be CAT6 connected each with their own cable back to the Brocade?
 

billc.cn

Member
Oct 6, 2017
41
7
8
The Dell 2708 is a managed switch, so you can set up different VLANs for different ports, and have them all go over the same cable (i.e. trunk) to your basement switch. This is the primary use case of managed switches.

If this is your first time dealing with VLANs, you should remember to be conscious about which switch port is passing tagged traffic and which ones are untagged.
 

tommybackeast

Active Member
Jun 10, 2018
258
85
28
The Dell 2708 is a managed switch, so you can set up different VLANs for different ports, and have them all go over the same cable (i.e. trunk) to your basement switch. This is the primary use case of managed switches.

If this is your first time dealing with VLANs, you should remember to be conscious about which switch port is passing tagged traffic and which ones are untagged.
For complex topics, I prefer asking humans vs. google.

Can you explain "tagged and untagged" pertaining to VLAN, thank u
 

billc.cn

Member
Oct 6, 2017
41
7
8
Tagged means there's an 802.1q header in each Ethernet frame to indicate the VLAN id, which allows multiple VLANs to share the same trunk connection. Most consumer devices can only handle untagged traffic.

Thus you will have to configure the switch to do the tagging/untagging. You should configure only one VLAN to be switched to a port connected to such a device and enable untagging. You also have to set untagged traffic coming into that port to be tagged with the same VLAN id.

Image from Mikrotik Wiki:
 
  • Like
Reactions: leebo_28

PGlover

Active Member
Nov 8, 2014
472
58
28
54
For some reason, ports 1/2/2 - 1/2/5 is not active using the following parts.. If there something special I need to do?

What 10g network adapter are folks using? I am using the Intel x520-DA2 10g nework adapters in all my servers. I am wondering if the Brocade QSFP and SFP+ adapters are not compatible with the Intel x520-DA2 10g network adapters.
 

tommybackeast

Active Member
Jun 10, 2018
258
85
28
I had asked a question re Mellanox PCI cards recently; and now that I've finally found the Dell Server I want , I have a followup question.

Brocade 6450-24p

Dell T620 Server (which will be located right next to the Brocade 6450)

I need to buy a 10gig PCIe card and cables to connect the Dell Server to Brocade switch.

Earlier, I had received an answer to buy this $16 ConnectX-2 card;

671798-001 COMPATIBLE 10GB MELLANOX CONNECTX-2 PCIe 10GBe ETHERNET NIC | eBay

but I've seen people here mention the ConnectX-3 card, this one

Mellanox MCX311A-XCAT CX311A ConnectX-3 Network 10GbE SFP+ Both Low+Full Profile | eBay

so does the ConnectX-3 card offer any benefits to me given my above hardware usage? thanks



"The Intel card will cost you about $35 more than a Mellanox Connect-X2 and shouldn't function any differently in practice. The only reason I'd buy an Intel 10Gb card is for BSD, the drivers on Windows and Linux are flawless (someone chime in about ESXi please ESXi reported fine as well.)

This is the $16 Mellanox card that most people are buying if you just want one port:
671798-001 COMPATIBLE 10GB MELLANOX CONNECTX-2 PCIe 10GBe ETHERNET NIC | eBay

Mellanox is great about compatibility with dacs and optics, you shouldn't have any trouble using most cables or transceivers. I use these:
MCP2104-X001B Mellanox 1M 10G SFP+ Passive Direct Attach Copper Twinax Cable | eBay"
 

buckminst

New Member
Jul 20, 2018
8
5
3
Utah
I had asked a question re Mellanox PCI cards recently; and now that I've finally found the Dell Server I want , I have a followup question.

Brocade 6450-24p

Dell T620 Server (which will be located right next to the Brocade 6450)

I need to buy a 10gig PCIe card and cables to connect the Dell Server to Brocade switch.

Earlier, I had received an answer to buy this $16 ConnectX-2 card;

671798-001 COMPATIBLE 10GB MELLANOX CONNECTX-2 PCIe 10GBe ETHERNET NIC | eBay

but I've seen people here mention the ConnectX-3 card, this one

Mellanox MCX311A-XCAT CX311A ConnectX-3 Network 10GbE SFP+ Both Low+Full Profile | eBay

so does the ConnectX-3 card offer any benefits to me given my above hardware usage? thanks
The ConnectX-3 is a newer card revision and supported under current versions of Mellanox's WinOF drivers It's also a PCIe 3.0 x4 card, versus the ConnectX-2's PCIe 2.0 x8. That does give you more flexibility in regards to what slots you can install the CX3 into for the same bandwidth usage on the PCIe bus, but other than those points, there's really not too much difference if you're getting the Ethernet-only cards like the CX311A.

That said, I'm using ConnectX-2s in my FreeNAS 11 box, my R710 running Proxmox, and an HP server running Windows Server 2016 with no issues (though since I'm just using the stock drivers and not WinOF, I don't have the advanced features enabled on it like VLAN tagging) and they came right up on my ICX6610-48 on the untagged VLAN I set up for my storage network.
 
  • Like
Reactions: tommybackeast

buckminst

New Member
Jul 20, 2018
8
5
3
Utah
I finally got my ICX6610-48 set up (shoutouts and thanks to Fohdeesha for all the help!), and am definitely happy with my purchase so far. I sadly lost the PSU lottery and got the revA firmware versions, so I had to plug both PSUs in to avoid it sounding like a jet engine, but I don't think I can differentiate its fan noise over that of my R710's at this point. It's certainly wonderful finally being on a more modern managed switch OS (catOS is a bummer), and the copious amounts of 10GbE ports are enabling some things I've wanted to mess around with for a while.

Currently, my setup involves the following hardware:

Brocade ICX6610-48-E Switch
2 Finisar FTLF8519P3BNL 1GbE SFPs (1 in the Brocade, 1 in my Meraki MS220-8P)
2 Ubiquiti UF-MM-1G 1GbE SFPs (1 in the Brocade, 1 in my Meraki MS220-8P [unused currently])
2 Avago AFBR-703SDZ 10GbE SFP+s (1 in the Brocade, 1 in my primary use PC's Mellanox ConnectX-3)
2 Intel FTLX8571D3BCV-IT Dual-Mode 1GbE/10GbE SFP+s (1 in the Brocade, 1 in my gaming PC's Mellanox ConnectX-3)
1 Monoprice 2M Cisco QSFP-4SFP10G-CU2M Compatible QSFP+ to 4SFP+ Breakout Cable (3 connections in use, all to Mellanox ConnectX-2 cards)

Of note:
The dreaded Monoprice QSFP breakout cable slid in nice and smooth in the 40GbE breakout port I installed it in, and didn't really seem like it was binding or anything, but I haven't tried removing it. I should mention, my 1/2/2 port works _just fine_.

The Intel dual-mode SFP+ module I'm using is reporting as 1GbE even with the port set to 10GbE, but this may just be due to the link not being in use right now as I'm waiting on the OM3 fiber cable to hook it up with.

Rev. A PSUs are annoying :p

I had a hard time getting my VLANs working for a while, until I realized I needed to add them on my Meraki MX64 first. Why that was needed when the traffic shouldn't even be hitting it directly must be a Meraki thing. But it all works now!
 
  • Like
Reactions: fohdeesha

PGlover

Active Member
Nov 8, 2014
472
58
28
54
The ConnectX-3 is a newer card revision and supported under current versions of Mellanox's WinOF drivers It's also a PCIe 3.0 x4 card, versus the ConnectX-2's PCIe 2.0 x8. That does give you more flexibility in regards to what slots you can install the CX3 into for the same bandwidth usage on the PCIe bus, but other than those points, there's really not too much difference if you're getting the Ethernet-only cards like the CX311A.

That said, I'm using ConnectX-2s in my FreeNAS 11 box, my R710 running Proxmox, and an HP server running Windows Server 2016 with no issues (though since I'm just using the stock drivers and not WinOF, I don't have the advanced features enabled on it like VLAN tagging) and they came right up on my ICX6610-48 on the untagged VLAN I set up for my storage network.
Is the dual port 10g model of the ConnectX-3 adapter model # MCX312A? Please advise as I am trying to find some dual 10g adapters to replace my Intel x520-da2 adapters.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Is the dual port 10g model of the ConnectX-3 adapter model # MCX312A? Please advise as I am trying to find some dual 10g adapters to replace my Intel x520-da2 adapters.
Here's the spec sheet for the CX3 EN cards:
http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX3_EN_Card.pdf

You're looking for the MCX312A-XCBT.

The ConnectX-3 is a newer card revision and supported under current versions of Mellanox's WinOF drivers It's also a PCIe 3.0 x4 card, versus the ConnectX-2's PCIe 2.0 x8. That does give you more flexibility in regards to what slots you can install the CX3 into for the same bandwidth usage on the PCIe bus, but other than those points, there's really not too much difference if you're getting the Ethernet-only cards like the CX311A.

That said, I'm using ConnectX-2s in my FreeNAS 11 box, my R710 running Proxmox, and an HP server running Windows Server 2016 with no issues (though since I'm just using the stock drivers and not WinOF, I don't have the advanced features enabled on it like VLAN tagging) and they came right up on my ICX6610-48 on the untagged VLAN I set up for my storage network.
There's an official feature set difference too. Officially the CX3s support SR-IOV and RDMA/RoCE v1 while the CX2 supports neither. Both are reported to work fine on CX2s with some minor firmware mods though.

I used a pair of CX2s on Win7/10 and Linux for years without a single issue, I run a mix of CX2 and CX3 10 & 40GbE cards now.
 
Last edited:
  • Like
Reactions: tommybackeast

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,824
113
29
fohdeesha.com
just imagine an intel nic, but better. kind of like that

Also, with the $75+ price of the 10gbE only cards, it's cheaper to spend $35 on the 40gbE versions like this: HP 649281-B21 IB FDR/EN 10/40Gb 2P 544QSFP Adapter 656089-001 | eBay

or short bracket version: HP 649281-B21 Infiniband 10/40GB Dual Port Adapter Low Profile Bracket | eBay

then a couple QSFP+ to SFP+ adapters so you can use SFP+ in them GENUINE 655902-001 655874-B21 HP MELLANOX QSFP/SFP ADAPTER 886111656585 | eBay

you'll still come in under or at the $75 price of the 10gbE only cards, but now you have cards already in your systems that are ready to do 40gbE if you ever upgrade your switches/network
 
Last edited:

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Be aware of airflow with the 40GbE CX3s. I forgot that server hardware expects a minimum level of airflow/ventilation when I upgraded to a CX3 40Gb card in my quiet desktop and crashed pretty consistently after some hours until I screwed a fan onto it.

At least I found a use for that 6.3CFM Sunon fan that I never could get working in my 6450, heh.
 

PGlover

Active Member
Nov 8, 2014
472
58
28
54
just imagine an intel nic, but better. kind of like that

Also, with the $75+ price of the 10gbE only cards, it's cheaper to spend $35 on the 40gbE versions like this: HP 649281-B21 Infiniband FDR/EN 10/40GB 2-Port 544QSFP ADAPTER 656089-001 ,New | eBay

or short bracket version: HP 649281-B21 Infiniband 10/40GB Dual Port Adapter Low Profile Bracket | eBay

then a couple QSFP+ to SFP+ adapters so you can use SFP+ in them GENUINE 655902-001 655874-B21 HP MELLANOX QSFP/SFP ADAPTER 886111656585 | eBay

you'll still come in under or at the $75 price of the 10gbE only cards, but now you have cards already in your systems that are ready to do 40gbE if you ever upgrade your switches/network
So are those HP 40g adapters the same as the Mellanox MCX314X model? Additionally, can they be flashed/upgraded with the Mellanox firmware?