Brocade 1020 CNA 10GbE PCIe Cards

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
s0lid

s0lid

Active Member
Feb 25, 2013
259
35
28
Tampere, Finland
Debian tested.
This should work also on any other debian like distro as is and other distros that use Linux kernel. You may need to check if the bna driver is compiled as module.
Code: 
uname -a
Linux Morannon 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u1 x86_64 GNU/Linux
The workaround:
Code: 
wget http://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git/tree/ctfw-3.2.3.0.bin
wget http://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git/tree/ct2fw-3.2.3.0.bin
wget http://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git/tree/cbfw-3.2.3.0.bin
cp ctfw-3.2.3.0.bin /lib/firmware/
cp ct2fw-3.2.3.0.bin /lib/firmware/
cp cbfw-3.2.3.0.bin /lib/firmware/
cd /lib/firmware
ln -s ctfw-3.2.3.0.bin ctfw.bin
ln -s ct2fw-3.2.3.0.bin ct2fw.bin
ln -s cbfw-3.2.3.0.bin cbfw.bin
modprobe -r bna
modprobe bna
dmesq after reloading the bna module.
Code: 
[  435.774656] Brocade 10G Ethernet driver - version: 3.0.2.2
[  435.774678] bnad_pci_probe : (0xffff88021674a000, 0xffffffffa04988c0) PCI Func : (2)
[  435.778004] bna 0000:01:00.2: firmware: agent loaded ctfw.bin into memory
[  435.778053] bna 0000:01:00.2: setting latency timer to 64
[  435.778153] bar0 mapped to ffffc90016a80000, len 262144
[  435.778206] bna 0000:01:00.2: irq 52 for MSI/MSI-X
[  435.778212] bna 0000:01:00.2: irq 53 for MSI/MSI-X
[  435.778218] bna 0000:01:00.2: irq 54 for MSI/MSI-X
[  435.778224] bna 0000:01:00.2: irq 55 for MSI/MSI-X
[  435.778229] bna 0000:01:00.2: irq 56 for MSI/MSI-X
[  435.778234] bna 0000:01:00.2: irq 57 for MSI/MSI-X
[  436.176936] bnad_pci_probe : (0xffff88021674b000, 0xffffffffa04988c0) PCI Func : (3)
[  436.176962] bna 0000:01:00.3: setting latency timer to 64
[  436.177047] bar0 mapped to ffffc90016b00000, len 262144
[  436.177094] bna 0000:01:00.3: irq 58 for MSI/MSI-X
[  436.177100] bna 0000:01:00.3: irq 59 for MSI/MSI-X
[  436.177106] bna 0000:01:00.3: irq 60 for MSI/MSI-X
[  436.177111] bna 0000:01:00.3: irq 61 for MSI/MSI-X
[  436.177117] bna 0000:01:00.3: irq 62 for MSI/MSI-X
[  436.177122] bna 0000:01:00.3: irq 63 for MSI/MSI-X
lshw:
Code: 
root@Morannon:~# lshw -class network
  *-network:0 DISABLED
  description: Ethernet interface
  product: 1010/1020/1007/1741 10Gbps CNA
  vendor: Brocade Communications Systems, Inc.
  physical id: 0.2
  bus info: pci@0000:01:00.2
  logical name: eth5
  version: 01
  serial: 00:05:33:90:a7:0a
  width: 64 bits
  clock: 33MHz
  capabilities: pm msix pciexpress vpd bus_master cap_list rom ethernet physical fibre
  configuration: autonegotiation=off broadcast=yes driver=bna driverversion=3.0.2.2 firmware=3.2.3.0 latency=0 link=no multicast=yes port=fibre
  resources: irq:16 memory:f7840000-f787ffff memory:f7904000-f7907fff memory:f7500000-f75fffff
  *-network:1 DISABLED
  description: Ethernet interface
  product: 1010/1020/1007/1741 10Gbps CNA
  vendor: Brocade Communications Systems, Inc.
  physical id: 0.3
  bus info: pci@0000:01:00.3
  logical name: eth6
  version: 01
  serial: 00:05:33:90:a7:0b
  width: 64 bits
  clock: 33MHz
  capabilities: pm msix pciexpress vpd bus_master cap_list rom ethernet physical fibre
  configuration: autonegotiation=off broadcast=yes driver=bna driverversion=3.0.2.2 firmware=3.2.3.0 latency=0 link=no multicast=yes port=fibre
  resources: irq:16 memory:f7800000-f783ffff memory:f7900000-f7903fff memory:f7400000-f74fffff
 
s0lid

s0lid

Active Member
Feb 25, 2013
259
35
28
Tampere, Finland
Little update after compiling new kernel for the system:
Code: 
root@Morannon:~# uname -a
Linux Morannon 3.15.6morannon.1 #2 SMP Sun Jul 20 18:31:22 EEST 2014 x86_64 GNU/Linux
lshw:
Code: 
lshw -class network
  *-network:0 DISABLED
  description: Ethernet interface
  product: 1010/1020/1007/1741 10Gbps CNA
  vendor: Brocade Communications Systems, Inc.
  physical id: 0.2
  bus info: pci@0000:01:00.2
  logical name: eth5
  version: 01
  serial: 00:05:33:90:a7:0a
  width: 64 bits
  clock: 33MHz
  capabilities: pm msix pciexpress vpd bus_master cap_list rom ethernet physical fibre
  configuration: autonegotiation=off broadcast=yes driver=bna driverversion=3.2.23.0 firmware=3.2.3.0 latency=0 link=no multicast=yes port=fibre
  resources: irq:16 memory:f7840000-f787ffff memory:f7904000-f7907fff memory:f7500000-f75fffff
  *-network:1 DISABLED
  description: Ethernet interface
  product: 1010/1020/1007/1741 10Gbps CNA
  vendor: Brocade Communications Systems, Inc.
  physical id: 0.3
  bus info: pci@0000:01:00.3
  logical name: eth6
  version: 01
  serial: 00:05:33:90:a7:0b
  width: 64 bits
  clock: 33MHz
  capabilities: pm msix pciexpress vpd bus_master cap_list rom ethernet physical fibre
  configuration: autonegotiation=off broadcast=yes driver=bna driverversion=3.2.23.0 firmware=3.2.3.0 latency=0 link=no multicast=yes port=fibre
  resources: irq:16 memory:f7800000-f783ffff memory:f7900000-f7903fff memory:f7400000-f74fffff
Newer Driver is present than in the stock 3.2.0 kernel.
 
S

Sukeshini

New Member
Jul 15, 2014
2
0
1
43
kachunkachunk said:
Hey everyone. I got this working in Ubuntu 13.04 and 13.10, but it definitely felt hacky. I'm sure that I'm not doing this the right way, but it works.

My guess is that udev and/or the inbox driver for the brocade adapter is specifying the firmware file ctfw-3.2.1.0.bin, explicitly. Even if you put a newer driver in the /lib/firmware directory, it still doesn't load; it's insistent on 3.2.1.0, which is not only out of date, but not specifically available from Brocade on their site, as far as I can tell. This said, it's not crazy to do this. From experience, adapter drivers and firmware operate in lockstep and you shouldn't really mix the release levels up. In this case I didn't have much choice. And perhaps it has everything to do with my occasional problems with the adapter, maybe.


So. If you were so inclined, what you can try is getting ctfw-3.2.1.1 from one of Brocade's firmware packages, and copying that to the /lib/firmware directory. It still won't load, and insists on 3.2.1.0, however. To address this, I symlinked ctfw-3.2.1.1.bin to ctfw-3.2.1.0.bin in /lib/firmware. After unloading/reloading bna, it works.

Despite this, the system won't boot with the adapter loaded. I put together a hacky init script set up to basically run `rmmod bna; modprobe bna` one more time somewhere during boot.
A better explanation: When you/boot runs `modprobe bna` and it fails, it stays in that broken-loaded state. Running modprobe again won't do anything new for you, as it's "already loaded" in a sense. Unload it, then load it agian).
After this, you should see the Brocade network ports in ifconfig afterwards.

If you're using the FCoE part of the adapter, do the same with cbfw and ct2fw, like above.

I noticed that 14.04 (Trusty Tahr)'s linux-firmware package contains some firmware files of interest:
/lib/firmware/ct2fw-3.2.1.1.bin
/lib/firmware/ct2fw-3.2.3.0.bin
/lib/firmware/ctfw-3.2.1.1.bin
/lib/firmware/ctfw-3.2.3.0.bin
This suggests to me that the inbox driver has been updated and everything should just work out of the box. I'll post my experience soon - that is, if the daily I'm installing from right now doesn't break my system.


The best way to address all of this was to find a way to compile the Brocade driver and not to use the inbox one, however their noarch/generic Linux driver won't compile on Ubuntu. I never really figured it out - I feel it's a fundamental problem/difference between a build environment on RedHat/CentOS/Fedora, compared to what Ubuntu is doing. You can't install the RPM via Alien for the same reasons.
To name build issue, it's the replacement of /bin/sh with dash (DashAsBinSh - Ubuntu Wiki) on Ubuntu. Despite reverting this back to /bin/sh, or changing some part of the compiling process to use /bin/bash instead, I didn't have much more luck. There's still some pretty elementary broken stuff going on for compiling and I really just don't have the technical know-how to get past that.


Oh, and regarding cables, I'm using some direct-connect Active Twinax cables+SFPs. Provided you're using fairly recent firmware, you can use third-party SFPs, not just Brocade's.


Edit: 14.04 successfully loads its inbox driver and a newer firmware was included, no tweaks or hacks needed. It's still behind what Brocade supplies, but I'll take a working solution!
It also broke my bonding config, but restarting networking solves that. Oh, Ubuntu. https://bugs.launchpad.net/ubuntu/ source/ifenslave-2.6/ bug/1280366
Edit 2: If you try a fresh 14.04 install and the firmware is missing, install the package linux-firmware.
Click to expand...
Hi,

Tried... still not able to install it. Is there any other solution?

Regards,
Sukeshini
 
S

scint

New Member
Jul 19, 2014
5
0
1
firenity said:
You guys reckon these transceivers would be compatible with the 1020 CNAs?
Click to expand...
Can't answer if that transceiver works or not, but on mine they have the following brocade part #: 57-0000075-01. Searching on ebay they are going for $30 to $40 each.

Interesting thing is on my cards, four of the transceivers were made in China, while the other four Malaysia. They all have the same part # though.
 
S

scint

New Member
Jul 19, 2014
5
0
1
s0lid said:
Little update after compiling new kernel for the system:
Code: 
root@Morannon:~# uname -a
Linux Morannon 3.15.6morannon.1 #2 SMP Sun Jul 20 18:31:22 EEST 2014 x86_64 GNU/Linux
lshw:
Code: 
lshw -class network
  *-network:0 DISABLED
  description: Ethernet interface
  product: 1010/1020/1007/1741 10Gbps CNA
  vendor: Brocade Communications Systems, Inc.
  physical id: 0.2
  bus info: pci@0000:01:00.2
  logical name: eth5
  version: 01
  serial: 00:05:33:90:a7:0a
  width: 64 bits
  clock: 33MHz
  capabilities: pm msix pciexpress vpd bus_master cap_list rom ethernet physical fibre
  configuration: autonegotiation=off broadcast=yes driver=bna driverversion=3.2.23.0 firmware=3.2.3.0 latency=0 link=no multicast=yes port=fibre
  resources: irq:16 memory:f7840000-f787ffff memory:f7904000-f7907fff memory:f7500000-f75fffff
  *-network:1 DISABLED
  description: Ethernet interface
  product: 1010/1020/1007/1741 10Gbps CNA
  vendor: Brocade Communications Systems, Inc.
  physical id: 0.3
  bus info: pci@0000:01:00.3
  logical name: eth6
  version: 01
  serial: 00:05:33:90:a7:0b
  width: 64 bits
  clock: 33MHz
  capabilities: pm msix pciexpress vpd bus_master cap_list rom ethernet physical fibre
  configuration: autonegotiation=off broadcast=yes driver=bna driverversion=3.2.23.0 firmware=3.2.3.0 latency=0 link=no multicast=yes port=fibre
  resources: irq:16 memory:f7800000-f783ffff memory:f7900000-f7903fff memory:f7400000-f74fffff
Newer Driver is present than in the stock 3.2.0 kernel.
Click to expand...
So I just received my four cards today, followed your guide and they are recognized by CentOS 6.5 without any issues. Now just need to get some LC cables to test it out.
 
MiniKnight

MiniKnight

Well-Known Member
Mar 30, 2012
3,072
973
113
NYC
Has anyone tried with CentOS 7 yet? I'd be curious to know if future support is going to be possible
 
D

DarkOrb

New Member
Jul 24, 2014
10
4
3
38
Wellington, New Zealand
firenity said:
You guys reckon these transceivers would be compatible with the 1020 CNAs?
Click to expand...
I've a couple of them turning up next week along with the cards, so I can test and let you know if you still need the information.

@MiniKnight Unfortunately I can't test connectivity (the SPF+/LC cables haven't been delivered yet), but it's detected on a clean install without issue on CentOS 7 - paste of the lshw below using only what was in the install media.

Code: 
  *-network:0
       description: Ethernet interface
       product: 1010/1020/1007/1741 10Gbps CNA
       vendor: Brocade Communications Systems, Inc.
       physical id: 0.2
       bus info: pci@0000:02:00.2
       logical name: enp2s0f2
       version: 01
       serial: 00:05:33:48:d0:46
       width: 64 bits
       clock: 33MHz
       capabilities: pm msix pciexpress vpd bus_master cap_list rom ethernet physical fibre
       configuration: autonegotiation=off broadcast=yes driver=bna driverversion=3.2.23.0r firmware=3.2.3.0 latency=0 link=no multicast=yes port=fibre
       resources: irq:18 memory:fe300000-fe33ffff memory:fe3b4000-fe3b7fff memory:fe000000-fe0fffff
  *-network:1
       description: Ethernet interface
       product: 1010/1020/1007/1741 10Gbps CNA
       vendor: Brocade Communications Systems, Inc.
       physical id: 0.3
       bus info: pci@0000:02:00.3
       logical name: enp2s0f3
       version: 01
       serial: 00:05:33:48:d0:47
       width: 64 bits
       clock: 33MHz
       capabilities: pm msix pciexpress vpd bus_master cap_list rom ethernet physical fibre
       configuration: autonegotiation=off broadcast=yes driver=bna driverversion=3.2.23.0r firmware=3.2.3.0 latency=0 link=no multicast=yes port=fibre
       resources: irq:18 memory:fdfc0000-fdffffff memory:fe3b0000-fe3b3fff memory:fde00000-fdefffff
 
Last edited:
R

RTM

Well-Known Member
Jan 26, 2014
956
359
63
Zankza said:
Today finally the packages arrived.

Excited, but only to find out the drivers links in this thread is no longer valid, does anybody have off site download?
Click to expand...
The part of Brocade that was making these CNA's were acquired by Qlogic, so you can find the drivers on their site.
 
D

DarkOrb

New Member
Jul 24, 2014
10
4
3
38
Wellington, New Zealand
Zankza said:
As soon the LVM was about to be finished, it all of sudden maxed out at 10Gbe.
Click to expand...
Shared nothing or shared storage migration? As far as I recall the last part it copies in a shared nothing is the memory, it will be able to max out the connection copying from memory where as the disk copy is limited and you'll end up with a graph like yours.
 
D

DarkOrb

New Member
Jul 24, 2014
10
4
3
38
Wellington, New Zealand
firenity said:
You guys reckon these transceivers would be compatible with the 1020 CNAs?
Click to expand...
I can confirm those do work. Here is a screenshot from the Host Connectivity Manager tool (running on Server 2012, though shouldn't make a difference for support).


I've got link up on both the SFP+ modules installed on that server, though given that my other server is sulking I don't have any transfer tests (plugged the ports into each other heh) so I can't do any speed testing :(

Also, I found them really easy to deal with for ordering. They shipped a little later than they said they would but I used TNT (about the same cost as airmail for shipping to New Zealand, go figure) and it got here today after shipping last Thursday so only 1 working day late after all.

Hope that helps.
 
Last edited:
  • Like
Reactions: crazyduck
M

minddragon

New Member
Aug 4, 2014
14
0
1
51
Hey guys,

So I'm trying Nexenta 4.01 and I can't get more than 3 gigabits per second. I tried contacting QLogic for support. They do not support Nexenta for any of their products. I was completely shocked that QLogic refuses to support something that should be supported -- Oracle Solaris. But there is no support for Oracle Solaris 5.11 (OpenSolaris). After about five emails, the QLogic support representative said that I have to use a supported Operating System in order to receive support. Their supported operating systems do not include Oracle Solaris 5.11 (OpenSolaris).

So these 10 gigabit cards are now paperweights as I cannot figure out how to get more than 3 gigabits per second from a simple IPERF test. Has anyone else on here gotten the Qlogic Brocade 1020 cards to do more than 3 gigabits per second on an IPERF test?

Thanks,

MD
 
s0lid

s0lid

Active Member
Feb 25, 2013
259
35
28
Tampere, Finland
I got 9.3Gbps between 2 centos boxes using the brocade 10gbase-sr lamps and om3 fiber.
Just MTU9000 and no extra settings on iperf side.
 
s0lid

s0lid

Active Member
Feb 25, 2013
259
35
28
Tampere, Finland
Got them with the cards, they're 100% genuine brocade transceivers. Fiber is just something I found from my fiber box, happened to be OM3. OM1 also works and did about 50m run of it this summer at certain lanparty.
 
You must log in or register to reply here.
Top Bottom