Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
235
43
Those are OEM rebadged MCX354A-FCBT, they can be flashed over to standard mellanox firmware pretty easily.
 

PGlover

Active Member
Nov 8, 2014
477
63
28
55
just imagine an intel nic, but better. kind of like that

Also, with the $75+ price of the 10gbE only cards, it's cheaper to spend $35 on the 40gbE versions like this: HP 649281-B21 Infiniband FDR/EN 10/40GB 2-Port 544QSFP ADAPTER 656089-001 ,New | eBay

or short bracket version: HP 649281-B21 Infiniband 10/40GB Dual Port Adapter Low Profile Bracket | eBay

then a couple QSFP+ to SFP+ adapters so you can use SFP+ in them GENUINE 655902-001 655874-B21 HP MELLANOX QSFP/SFP ADAPTER 886111656585 | eBay

you'll still come in under or at the $75 price of the 10gbE only cards, but now you have cards already in your systems that are ready to do 40gbE if you ever upgrade your switches/network
Just purchased the two 40g adapters and a couple QSFP+ to SFP+ adapters.. I will read over the flashing instructions. Any tips on the flashing?
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
235
43

buckminst

New Member
Jul 20, 2018
8
5
3
Utah
Be aware of airflow with the 40GbE CX3s. I forgot that server hardware expects a minimum level of airflow/ventilation when I upgraded to a CX3 40Gb card in my quiet desktop and crashed pretty consistently after some hours until I screwed a fan onto it.

At least I found a use for that 6.3CFM Sunon fan that I never could get working in my 6450, heh.
I wonder if that might have something to do with why my gaming rig kept crashing with the CX3 in it. Although it's the 10GbE-only one, and I have decent airflow in the case. Maybe I need more. I thought Windows was just being really picky or that it didn't like the particular version of WinOF I'd installed for it (was trying the Win10 180x build of WinOF 5.50)
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
235
43
I wonder if that might have something to do with why my gaming rig kept crashing with the CX3 in it. Although it's the 10GbE-only one, and I have decent airflow in the case. Maybe I need more. I thought Windows was just being really picky or that it didn't like the particular version of WinOF I'd installed for it (was trying the Win10 180x build of WinOF 5.50)
Yeah, it took me several days to figure out what was going on. I couldn't reproduce the crash with any combination of loads but if I left it on for a long period of time I'd come back to find it hung. I screwed a spare 40mm fan into the heatsink and I haven't had an issue since.

The weird thing is that my NAS (CentOS) uses the same card and I haven't had any problems with it, it's been running stably since I installed the card a couple of weeks ago. The airflow isn't much better in that machine either.
 

PGlover

Active Member
Nov 8, 2014
477
63
28
55
Follow @fohdeesha's link above: https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015

It's really straightforward, the Mellanox firmware tools are great. I had no problem cross flashing two OEM CX3 cards and then modding a handful of CX2's to support RDMA/SR-IOV later.
Are these the commands to flash the card on a Windows OS machine?

#Backup the cards current board definition file just in case
flint -d /dev/mst/mt4099_pci_cr0 dc orig_firmware.ini

#Flash the FCBT image to the card
flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn

use it in config commands to configure the card
#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

I will need to download the Mellanox WinMFT software package

http://www.mellanox.com/page/management_tools
 
Last edited:

i386

Well-Known Member
Mar 18, 2016
2,654
773
113
32
Germany
the commands/tools are from the mellanox firmware tools (mft), they are available for different operations sytems
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
235
43
Does anyone have a starter guide to working with fastiron, or even better a set of lab exercises, that they can recommend? I've skimmed through a huge amount of material but I learn best when I can follow along with known projects to bootstrap. I'm willing to pay for course material if someone can recommend something good and not super expensive. These 6450s are my first enterprise switches and while I understand most of what I'm doing the system layout is completely new to me.

Right now I'm looking to do basic segmentation and L3 routing; I'd also like to setup a couple of LAGGs.
 
  • Like
Reactions: tommybackeast

fohdeesha

Kaini Industries
Nov 20, 2016
2,271
2,341
113
30
fohdeesha.com
I highly recommend Terry Henry's youtube channel - he's a fastiron engineer at brocade/ruckus and posts a LOT of micro tutorials you can follow along. Some of the newer videos apply to fastiron v8080 only, but 95% of them apply to the 8030 codetrain the switches in this thread run. The brocadecampus youtube link above is a smaller channel that just occasionally reposts some of terry's videos

main channel - Terry Henry

VLAN basics:



a couple videos on inter-vlan routing:


Video on LACP:



Video on named ACL's (which is the ACL type you want to use, the old cisco-numbered ACL types are being retired thank god):

 
Last edited:

tommybackeast

Active Member
Jun 10, 2018
290
110
43
I highly recommend Terry Henry's youtube channel - he's a fastiron engineer at brocade/ruckus and posts a LOT of micro tutorials you can follow along. Some of the newer videos apply to fastiron v8080 only, but 95% of them apply to the 8030 codetrain the switches in this thread run. The brocadecampus youtube link above is a smaller channel that just occasionally reposts some of terry's videos

main channel - Terry Henry

VLAN basics:



a couple videos on inter-vlan routing:


Video on LACP:



Video on named ACL's (which is the ACL type you want to use, the old cisco-numbered ACL types are being retired thank god):

Thank you. For myself (a newbie), good videos are so helpful
 
  • Like
Reactions: fohdeesha

tommybackeast

Active Member
Jun 10, 2018
290
110
43
The Dell 2708 is a managed switch, so you can set up different VLANs for different ports, and have them all go over the same cable (i.e. trunk) to your basement switch. This is the primary use case of managed switches.

If this is your first time dealing with VLANs, you should remember to be conscious about which switch port is passing tagged traffic and which ones are untagged.
I asked my question poorly: If there is a single Cat6 going from the Brocade 6450's Port#10 and going to an 8port Switch at the TV Media Center; and my desire is to put all the TV related devices (TV, Stereo, Tivo, Apple TV) onto its own VLAN.

Can I then take Brocade 6450's port 10 and VLAN that one port?
 

tommybackeast

Active Member
Jun 10, 2018
290
110
43
The ConnectX-3 is a newer card revision and supported under current versions of Mellanox's WinOF drivers It's also a PCIe 3.0 x4 card, versus the ConnectX-2's PCIe 2.0 x8. That does give you more flexibility in regards to what slots you can install the CX3 into for the same bandwidth usage on the PCIe bus, but other than those points, there's really not too much difference if you're getting the Ethernet-only cards like the CX311A.

That said, I'm using ConnectX-2s in my FreeNAS 11 box, my R710 running Proxmox, and an HP server running Windows Server 2016 with no issues (though since I'm just using the stock drivers and not WinOF, I don't have the advanced features enabled on it like VLAN tagging) and they came right up on my ICX6610-48 on the untagged VLAN I set up for my storage network.
This is more a Dell Server T620 question than a Brocade question.

If I get a Mellanox ConnectX-3 dual SFP+ card and insert it into Dell T620 server; and run two (2) TwinAx 10gig cables back to the Brocade 6450 - do I gain anything interesting ? plan to try ESXi and Proxmox on the Dell T620 server as I"ve never run either one.
 

tommybackeast

Active Member
Jun 10, 2018
290
110
43
just imagine an intel nic, but better. kind of like that

Also, with the $75+ price of the 10gbE only cards, it's cheaper to spend $35 on the 40gbE versions like this: HP 649281-B21 Infiniband FDR/EN 10/40GB 2-Port 544QSFP ADAPTER 656089-001 ,New | eBay

or short bracket version: HP 649281-B21 Infiniband 10/40GB Dual Port Adapter Low Profile Bracket | eBay

then a couple QSFP+ to SFP+ adapters so you can use SFP+ in them GENUINE 655902-001 655874-B21 HP MELLANOX QSFP/SFP ADAPTER 886111656585 | eBay

you'll still come in under or at the $75 price of the 10gbE only cards, but now you have cards already in your systems that are ready to do 40gbE if you ever upgrade your switches/network
I am shortly getting a Dell T620 Tower Server, and the manual for it says I can use "Mellanox® ConnectX®-3 dual-port 40GbE QSFP+" ; on ebay I see
Dell 3CYRK Mellanox MCX384A-BCAA ConnectX-3 Dual port QSFP 40Gb/s Mezzanine card | eBay

Since Tower Servers and 40GB cards are new for me, I am used to seeing gold connectors on the bottom of PCIe cards; which this card seemingly lacks.

Just wished to confirm I was getting the correct card, thank you (will then be getting QSFP= to SFP+ adapters as I have a brocade 6450 and not 6610; but like the idea of future-proofing.
 

buckminst

New Member
Jul 20, 2018
8
5
3
Utah
This is more a Dell Server T620 question than a Brocade question.

If I get a Mellanox ConnectX-3 dual SFP+ card and insert it into Dell T620 server; and run two (2) TwinAx 10gig cables back to the Brocade 6450 - do I gain anything interesting ? plan to try ESXi and Proxmox on the Dell T620 server as I"ve never run either one.
You could use LACP to do link aggregation, or you could have 1 direct 10GbE connection to 2 other boxes, other than that you don't really gain a whole lot from having a dual slot card over the single-slot. It depends on your intended uses and how much bandwidth you think you'll be pushing total, to how many other systems.
 
  • Like
Reactions: tommybackeast

buckminst

New Member
Jul 20, 2018
8
5
3
Utah
I am shortly getting a Dell T620 Tower Server, and the manual for it says I can use "Mellanox® ConnectX®-3 dual-port 40GbE QSFP+" ; on ebay I see
Dell 3CYRK Mellanox MCX384A-BCAA ConnectX-3 Dual port QSFP 40Gb/s Mezzanine card | eBay

Since Tower Servers and 40GB cards are new for me, I am used to seeing gold connectors on the bottom of PCIe cards; which this card seemingly lacks.

Just wished to confirm I was getting the correct card, thank you (will then be getting QSFP= to SFP+ adapters as I have a brocade 6450 and not 6610; but like the idea of future-proofing.
Mezzanine cards are typically used in blade enclosures, your T620 would more need something like this:

Mellanox ConnectX-3 MCX354A-FCBT Dual Port FDR 56.6 Gb/s Infiniband + 40GbE

It's possible you might be able to use the card to replace the embedded Ethernet on the T620, but chances are you're better off getting a PCIe card anyway.
 
  • Like
Reactions: tommybackeast