Brocade 1020 CNA 10GbE PCIe Cards

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

soapbox

New Member
Feb 24, 2014
17
0
1
If I put one of these Brocade 1010/1020 cards in each of my two Win7/8 computers with SSD drives in them, and connect the two cards directly with Brocade active twinax cable (probably 3m is enough for me), will I be able to transfer files between the two computers at 10Gbit (or as much as SSDs in computers performance will be - around 500-600 MB/s)?

Does this kind of connection require the same kind of setup as with ethernet (IPs, workgroups, sharing), or is it different in any way?
 

jingjing

Banned
Nov 23, 2013
22
0
0
I'll let you know if they work when they get here. So far the cables are installed and happy with Win 7 and 8. Like s0lid, I had no luck with Ubuntu so far, and no love on FreeBSD.
 

kachunkachunk

New Member
Feb 27, 2014
12
3
3
Hey everyone. I got this working in Ubuntu 13.04 and 13.10, but it definitely felt hacky. I'm sure that I'm not doing this the right way, but it works.

My guess is that udev and/or the inbox driver for the brocade adapter is specifying the firmware file ctfw-3.2.1.0.bin, explicitly. Even if you put a newer driver in the /lib/firmware directory, it still doesn't load; it's insistent on 3.2.1.0, which is not only out of date, but not specifically available from Brocade on their site, as far as I can tell. This said, it's not crazy to do this. From experience, adapter drivers and firmware operate in lockstep and you shouldn't really mix the release levels up. In this case I didn't have much choice. And perhaps it has everything to do with my occasional problems with the adapter, maybe.


So. If you were so inclined, what you can try is getting ctfw-3.2.1.1 from one of Brocade's firmware packages, and copying that to the /lib/firmware directory. It still won't load, and insists on 3.2.1.0, however. To address this, I symlinked ctfw-3.2.1.1.bin to ctfw-3.2.1.0.bin in /lib/firmware. After unloading/reloading bna, it works.

Despite this, the system won't boot with the adapter loaded. I put together a hacky init script set up to basically run `rmmod bna; modprobe bna` one more time somewhere during boot.
A better explanation: When you/boot runs `modprobe bna` and it fails, it stays in that broken-loaded state. Running modprobe again won't do anything new for you, as it's "already loaded" in a sense. Unload it, then load it agian).
After this, you should see the Brocade network ports in ifconfig afterwards.

If you're using the FCoE part of the adapter, do the same with cbfw and ct2fw, like above.

I noticed that 14.04 (Trusty Tahr)'s linux-firmware package contains some firmware files of interest:
/lib/firmware/ct2fw-3.2.1.1.bin
/lib/firmware/ct2fw-3.2.3.0.bin
/lib/firmware/ctfw-3.2.1.1.bin
/lib/firmware/ctfw-3.2.3.0.bin
This suggests to me that the inbox driver has been updated and everything should just work out of the box. I'll post my experience soon - that is, if the daily I'm installing from right now doesn't break my system.


The best way to address all of this was to find a way to compile the Brocade driver and not to use the inbox one, however their noarch/generic Linux driver won't compile on Ubuntu. I never really figured it out - I feel it's a fundamental problem/difference between a build environment on RedHat/CentOS/Fedora, compared to what Ubuntu is doing. You can't install the RPM via Alien for the same reasons.
To name build issue, it's the replacement of /bin/sh with dash (https://wiki.ubuntu.com/DashAsBinSh) on Ubuntu. Despite reverting this back to /bin/sh, or changing some part of the compiling process to use /bin/bash instead, I didn't have much more luck. There's still some pretty elementary broken stuff going on for compiling and I really just don't have the technical know-how to get past that.


Oh, and regarding cables, I'm using some direct-connect Active Twinax cables+SFPs. Provided you're using fairly recent firmware, you can use third-party SFPs, not just Brocade's.


Edit: 14.04 successfully loads its inbox driver and a newer firmware was included, no tweaks or hacks needed. It's still behind what Brocade supplies, but I'll take a working solution!
It also broke my bonding config, but restarting networking solves that. Oh, Ubuntu. https://bugs.launchpad.net/ubuntu/+source/ifenslave-2.6/+bug/1280366
Edit 2: If you try a fresh 14.04 install and the firmware is missing, install the package linux-firmware.
 
  • Like
Reactions: spazoid

Jeggs101

Well-Known Member
Dec 29, 2010
1,529
241
63
WOW that is a GREAT guide. Thank you for sharing all of that! The new firmware + 3rd party twinax thing is super too.
 

RTM

Well-Known Member
Jan 26, 2014
956
359
63
Anyone been having any luck with FCoE in a NIC-NIC setup?
I have been trying to get it to work, but have had to make do with iSCSI for now.

Seems like the FCoE interface will not come up, because DCB will not come up, status is 'Protocol error'.
From what I understand DCB is only possible when connected to a switch, so maybe it is not a valid configuration?
 

mrkrad

Well-Known Member
Oct 13, 2012
1,244
52
48
FCoE doesn't really work V2V (direct), it is meant to be converted using a switch to FC (ala cisco nexus).

DCBx is required for many nic's to ensure no packets are lost, it is a requirement from just about everyone - You might check out ESOS to see if they have come up with a FCoE target yet! But honestly the hardwrae ISCSI/FCoE is so specific to vendor/SAN that it is the reason it hasn't really taken off!

DCB also helps ISCSI and part of the specification is the only way you can create CLASS OF SERVICE for per-vlan flow control, you know having 4 vlan's with port-based flow control just isn't going to work out so hot!
 

firenity

Member
Jun 29, 2014
51
8
8
Before the server crash there used to be alot more information in this thread, maybe we can get it started again?

I'm particularly interested in supported SFP+ transceivers. Are third party transceivers supported at all? Is there a compatibility list or something?
 

firenity

Member
Jun 29, 2014
51
8
8
3rd post of this thread. There's some info now.
Thanks.

I remember someone mentioning they got the cards working with 3rd party transceivers/cables after flashing the latest (?) firmware - I suppose that wasn't you?
 

kachunkachunk

New Member
Feb 27, 2014
12
3
3
Here's a model that I assert will work - I have three or four Cisco SFP-H10GB-ACU7M active twinax cables (these come with transceivers terminating both ends). The 7M part is just the length. Here's a data sheet for Cisco: Cisco 10GBASE SFP Modules Data Sheet - Cisco
Axiom has a bunch of them too, but I don't know if they work. Probably, though. As mentioned before, third-party ones (non-Brocade) should work on any current firmware levels since 2013.

Also to note Brocade was recently acquired by Q-Logic: QLogic buys Brocade’s converged network adapter business
My hope is they'll maybe deliver a little better for the Debian-based distributions, but it's doubtful. Perhaps in whatever CNA lineup they decide to put out later!


Edit: Just re-read the OP and post #3. Definitely note that third party SFPs are now functional at least on the 1010/1020s, but maybe not *supported*.
Further, I've got this working on Windows 8 and then on 8.1. Support there was trivial, just use the Server 2012 or R2 package.

Ubuntu 14.04 support is native, but already dated as of this post. I also noticed frequent memory allocation failures were logged from the Brocade bna module during file transfers, made worse/common especially when using jumbo frames. I never figured it out, and it never killed transmissions or connectivity... but it doesn't give me the warm and fuzzies (also spams the logs pretty bad). I've since gone to regular 1514 MTU.
These days, I'm noticing DPC/interrupt behavior seems to be different than what Windows can tolerate, and this is triggering DPC watchdog violation (timeout) BSODs. Great. So I'm about to update my Windows drivers to a more recent revision to see if it goes away. Fingers crossed.
 
Last edited:

firenity

Member
Jun 29, 2014
51
8
8
Thanks for the update, kachunkachunk.

Which 3rd party SFP+ transceivers did you use?

Please keep us updated on whether the new drivers fix the issues you've been experiencing.
 

Kristian

Active Member
Jun 1, 2013
347
84
28
I would like to ask your assistance on how to do a Firmware upgrade on these CNA's.
I have received mine yesterday. And I was hoping to be able to do the Firmware upgrade via the Multi-Boot Firmware LiveCD (ISO)
( QLogic Driver Download )

Booting the device suceeded. I think I would need a command line parameter to do the firmware upgrade.
Unfortunatelly I don't know how.

From what I have read in the manual the firmware should automatically be updated while installing the recent drivers.
This does not happen and so my CISCO SFP-H10GB-CU3M does not work...

Edit: Firmware upgrade was successful: Firmware is now 3.2.3.1 (09/23/2013)
But the Cisco SFP is not working as of now.


Can anybody help?
Help would be greatly appreciated.
 
Last edited:

firenity

Member
Jun 29, 2014
51
8
8
From what I have read in the manual the firmware should automatically be updated while installing the recent drivers.
This does not happen and so my CISCO SFP-H10GB-CU3M does not work...

Edit: Firmware upgrade was successful: Firmware is now 3.2.3.1 (09/23/2013)
But the Cisco SFP is not working as of now.
It looks like the Cisco SFP-H10GB-CU3M is in fact a passive direct attach cable. The Brocade 1010/1020 cards only work with active DACs.
 
  • Like
Reactions: Kristian

MiniKnight

Well-Known Member
Mar 30, 2012
3,072
973
113
NYC
Yea the Active DAC was key in making these usable. Cables cost more but these are cheap cards.
 

Kristian

Active Member
Jun 1, 2013
347
84
28
It looks like the Cisco SFP-H10GB-CU3M is in fact a passive direct attach cable. The Brocade 1010/1020 cards only work with active DACs.
So the problem is right in front of the Computer: me...
Shoul have read more carefully, probably then I would have noticed the missing "A" for active... :-/
 

Kristian

Active Member
Jun 1, 2013
347
84
28
@kachunkachunk: Thanks for mentioning the Cisco SFP-H10GB-ACU7M! Its working flawlessly!

I would really like to see these cards working with freeNas. But that does not seem to happen any time soon.
Neither native nor in ESXI via the vmxnet3 driver.

Any experience if those cards will work with openfiler or unraid? or anything similar (that does not need hours and hours and hours so setup and is self explaining)
 

firenity

Member
Jun 29, 2014
51
8
8
My 1020 cards arrived today.

Unfortunately third party SFP+ transceivers do NOT work, not even on the newst firmware/BIOS (3.2.4.0).
 

scint

New Member
Jul 19, 2014
5
0
1
hi, long time lurker. Anyway I just purchased one 1020 with SR SFP+ for $45, will notify how well this one works. Note this one comes with low height profile, does anyone know where I can get a full height profile for this card?