Cost effective 40Gbe and 10Gbe - Brocade ICX6610 to FreeBSD, VMWare ESXi, and Windows

[1310nm]

I have been scrubbing the forum looking for good and cost effective choices to connect a ICX6610 to FreeBSD (TrueNAS) at 40gbe. Is the best cost effective approach to use a Mellanox MCX354A-QCBT (cross-flashed to MCX354A-FCBT)? Or is this solution too old now. I'm having a hard time finding a "OEM Equivalent" matrix for these cards.

If I need to run 10gbe from ICX6610 to a Windows 10 desktop, is there a "modern" cost effective 10GBASE-T solution that is supported well on the 6610 and Windows side?

My final connection is ICX6610 to VMWare ESXi at 40gbe, this solution may be the same as to TrueNAS, but having the benefit of SR-IOV would be preferable. The Mellanox ConnectX-3/ConnectX-4/ConnectX-5/ConnectX-6 all support this I believe.

 

[StevenDTX]

HP 544QSFP / MCX354A-FCBT still work great. They work just fine under ESXi 7.0. I also have a couple of MCX416A-BCAT that work great.

 

[1310nm]

Thanks for the input. Is there a "OEM Equivalent" matrix for these MCX354A cards? They are not listed in the master thread on "OEM Equivalents" and a forum search and google search did not turn-up anything super useful.

 

[Fallen Kell]

I bought HP 544QSFP MCX354A-FCBT cards which I connected to my 6610. One system is running pfsense, and the other is running XCP-ng which hosts a freenas vm for my storage. I originally paid $30 shipped for them at the time, but the price varies.

 

[1310nm]

I ordered two MCX354A-QCBT cards which I will have to flash to FCBT. Are there preferred DACS cables for these?

Does anyone have a suggestion on the 10GBASE-T solution? I am thinking that I will get a couple SFP+'s based on the Marvell 88x33X0 chip set from this list here. I just need a good low cost SFP+ NIC or a 10GBase-T NIC for the Windows machine (good choices?).

 

[Fallen Kell]

One of the DACs I got on Amazon from 10GTek. The other was from fs.com. I don't think the 6610 or the mellanox card is very picky.

As for cheap 10Gbase-t cards, well they don't really exist yet. There are so many counterfeits out there that the used market is very suspect (assume anything that is shipping from China is a fake). And the price of the transceivers to go from QSFP+/SFP+ to 10Gbase-t make the cost of going optical just as expensive as finding a 10Gbase-t card. I bought a few Dell branded X540-t2 cards a couple years ago when I managed to find a few for under $100.

 

[firworks]

If you see that the card is marked / listed as ConnectX-3 does that mean it will be flashable to the Mellanox 40gb firmware? Does one port vs two matter for that? Most of them I see are listed as 10gb. I'm also trying to connect a 6610 to a few different systems and was searching to find the current best deal / suggestion for 40gb cards.

 

[Fallen Kell]

No, not all ConnectX-3 cards are 40gb. You will typically want a VPI card:
MCX353A-QCBT
MCX354A-QCBT
MCX353A-FCBT
MCX354A-FCBT

Or ConnectX-3 Pro cards:
MCX313A-BCCT
MCX314A-BCCT

All the other ConnectX-3 cards are 10gbe or infiniband only and can not be used for ethernet networking.

As for single port vs dual port, it really doesn't matter a whole lot with the above 40gb cards. The PCI-e bus can not provide the full bandwidth to run dual ports. The main reason for the cards with two ports is for network redundancy purposes (i.e. one port is connected to one switch and the other to a second switch, this way if a switch fails, the system is still able to reach the network over the other switch). At 10gbe, the dual ports do have full bandwidth, and thus, it is useful for bonding/teaming to allow for more bandwidth.

 

[firworks]

I found two MCX354A-QCBT Dual Port on eBay for a decent price and have them installed in my servers. I also found this post which has a guide for how to flash them to the FCBT firmware, but unfortunately it doesn't work.

sudo flint -d /dev/mst/mt4099_pciconf0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3
.4.752.bin -allow_psid_change burn

    Current FW version on flash:  2.9.4260
    New FW version:               2.42.5000


    You are about to replace current PSID on flash - "MT_1020120018" with a different PSID - "MT_1090120019".
    Note: It is highly recommended not to change the PSID.

Do you want to continue ? (y/n) [n] : y
-E- Burning FS2 image failed: Device/Image mismatch: FW image file cannot be programmed to device ConnectX-3 A0, it is intended for: ConnectX-3 A1 only

Am I missing something? I've seen people talk about crossflashing these Mellanox cards all over the forum and I've never seen anyone say it only worked with specific hardware/firmware revisions. If so it seems like a complete gamble to try to get these cards second hand.

Edit: I just looked and this card has a Rev. A2 sticker on the back of it. The other card I received has a Rev. A6 sticker and I'm trying that one now to see if I can cross flash it.

 

[firworks]

Little update on the above. I've now got 3 of these MCX354A-QCBT cards and the other two flashed perfectly following the directions from that other thread. The one that won't flash and says it's an A0 even though it's labeled A2 also has a small "Engineering Sample" sticker on it. So I wonder if maybe the firmware on it is different in some way or otherwise causing problems. So I guess if someone comes across this in the future check for that. I'm just considering that card a write off and part of the learning process.

 

[iZohanX]

I bought HP 544QSFP MCX354A-FCBT cards which I connected to my 6610. One system is running pfsense, and the other is running XCP-ng which hosts a freenas vm for my storage. I originally paid $30 shipped for them at the time, but the price varies.

I'm eyeing out a few MCX354A-QCBT cards myself for a similar setup to what you're running.

If you don't mind, can I probe you for some questions? (these might be "noob" questions, but I'd sound stupid and be sure than make a huge mistake).

Is it possible to flash the firmware of the cards on a windows machine?
If I do flash them on one machine and then carry them over to a different OS type (say from windows to a FreeBSD/ESXi) do I need to re-flash them? Or does flashing "burns" the firmware onto the card until otherwise re-flashed?
Do you have them hooked up to the 6610 at 40gig using the stacking ports at the back? As in, the link speed, it's at 40gigs?

OP, if you're able to answer these as well, I'd apreciate any input I can get =)

 

[Fallen Kell]

I'm eyeing out a few MCX354A-QCBT cards myself for a similar setup to what you're running.

If you don't mind, can I probe you for some questions? (these might be "noob" questions, but I'd sound stupid and be sure than make a huge mistake).

Is it possible to flash the firmware of the cards on a windows machine?
If I do flash them on one machine and then carry them over to a different OS type (say from windows to a FreeBSD/ESXi) do I need to re-flash them? Or does flashing "burns" the firmware onto the card until otherwise re-flashed?
Do you have them hooked up to the 6610 at 40gig using the stacking ports at the back? As in, the link speed, it's at 40gigs?

OP, if you're able to answer these as well, I'd apreciate any input I can get =)

I don't know about flashing on a windows system, as I flashed them from freenas (after compiling and installing the full suite of tools). I think there are easier ways to flash from some linux distro's which you wouldn't need to compile everything.

Once they have been flashed, it carries over, so no need to do it again if you move the hardware from one system that you use to flash them into another system that will run them.

Yes, I have them hooked up to the two ports in the back that are officially labeled as for stacking, but if you read the first page of the monster Brocade ICX switch thread and follow through to the instructions on the 6610, you will see how to convert those ports to be used as normal 40 QSFP+ ports which will register at up to 40gbps speeds (which is what mine do). Please note that only 2 of the ports will work like this for the 6610, the other two can only be used as breakout ports for 4x10gbps links using the various breakout cables that will provide 1xQSFP+ -> 4xSFP+ connections.

 

[fohdeesha]

I'm eyeing out a few MCX354A-QCBT cards myself for a similar setup to what you're running.

If you don't mind, can I probe you for some questions? (these might be "noob" questions, but I'd sound stupid and be sure than make a huge mistake).

Is it possible to flash the firmware of the cards on a windows machine?
If I do flash them on one machine and then carry them over to a different OS type (say from windows to a FreeBSD/ESXi) do I need to re-flash them? Or does flashing "burns" the firmware onto the card until otherwise re-flashed?
Do you have them hooked up to the 6610 at 40gig using the stacking ports at the back? As in, the link speed, it's at 40gigs?

OP, if you're able to answer these as well, I'd apreciate any input I can get =)

use this post for flashing, you can follow it from either linux or windows. On windows, you'll just need to download mellanox firmware tools. You can also just temporarily boot a PC from a debian live ISO, then you can follow the post exactly (like, copy/pasting every command) then boot back into windows when you're done. Everything is saved on the card, even the bootrom and port mode settings, so you can moe it from machine to machine and it will keep them https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015

 

[fohdeesha]

Little update on the above. I've now got 3 of these MCX354A-QCBT cards and the other two flashed perfectly following the directions from that other thread. The one that won't flash and says it's an A0 even though it's labeled A2 also has a small "Engineering Sample" sticker on it. So I wonder if maybe the firmware on it is different in some way or otherwise causing problems. So I guess if someone comes across this in the future check for that. I'm just considering that card a write off and part of the learning process.

the actual ASIC on those rev0 cards is slightly different (can only handle the slower clock of the QCBT firmware), which is why it won't let you flash the FCBT firmware. You must have pretty bad luck, I've only seen one or two other instances of people coming across those rev0 sample cards