Brocade 1020 CNA 10GbE PCIe Cards

[TuxDude]

The transceiver seems to have connections for two fibers, in what they call a duplex arrangement. Do I need both?

You need a single cable with duplex ends. If you are shopping at monoprice, then pretty much any cable on this page will work: LC/LC, Multi Mode, Duplex - Monoprice.com

While a OM1 (62.5/125) cable will technically work over short distances, I would recommend going with OM2 or OM3 (both are 50/125, OM2 is orange, OM3 aqua). Besides going longer distances with better cables, the only real restriction is that you can't mix cable diameters in a single run (eg. when using fiber-optic patch panels), so 62.5 micron OM1 can't be mixed with 50 micron OM2/3/4

 

[mattlach]

Appreciate the help guys!

 

[mattlach]

So the answer is: YES!

I was able to dremel one of my Brocade 1020 so it fits in a PCI-E 2 x4 slot and its working perfectly

As far as bandwidth goes for this. PCIe 2.0 has 500MB/s full duplex per lane. 8x is thus 4000MB/s or 32Gbit/s 4x is 16Gbit/s

Thus my assumption would be that 4x at PCIe 2.0 would be sufficient for one 10Gbit port, but not for two.

There is likely some overhead, but I doubt it is the 60% it would take to reach 16Gbit/s for a single port.

I would just melt out the backstop on the PCIe4x slot on the motherboard with a soldering iron (or cut it with a dremel) and cover the unused leads on the PCIe card sticking out hte back of the port with electrical tape (to make sure they don't short to something) rather than cutting down the expansion card, as that permanently makes it 4x. No turning back.

I have done this successfully in the past with other cards.

 

[markarr]

These brocades are much cheaper than most motherboards. I would be too worried about mb warranty to modify the pcie slot.

 

[mattlach]

These brocades are much cheaper than most motherboards. I would be too worried about mb warranty to modify the pcie slot.

They aren't as cheap anymore. Since the hobby/home server market discovered them prices have shot up. Cheapest I found the 1020 for on eBay recently was $185, which is why I bought the single port 1010's for $39 each.

I don't know about you guys, but I got my new old stock dual socket Supermicro 8XDTE for $150 on eBay.

Besides, the alteration to the motherboard is minor and simple and might not even be very noticeable unless you are looking for it.

 

[markarr]

For lga 1366 you can find lots of motherboards cheap. You get into lga 2011 / 2011-3 the mb prices start climbing quickly, then you start talking about $300+ motherboards. The price has come back down on the cards found about a dozen for under 100.

 

[mattlach]

They aren't as cheap anymore. Since the hobby/home server market discovered them prices have shot up. Cheapest I found the 1020 for on eBay recently was $185, which is why I bought the single port 1010's for $39 each.

Of course today now that I have already ordered mine, there are more on eBay cheaper again...

I didn't need the second port anyway I guess

 

[PigLover]

As far as bandwidth goes for this. PCIe 2.0 has 500MB/s full duplex per lane. 8x is thus 4000MB/s or 32Gbit/s 4x is 16Gbit/s

Thus my assumption would be that 4x at PCIe 2.0 would be sufficient for one 10Gbit port, but not for two.

There is likely some overhead, but I doubt it is the 60% it would take to reach 16Gbit/s for a single port.

I would just melt out the backstop on the PCIe4x slot on the motherboard with a soldering iron (or cut it with a dremel) and cover the unused leads on the PCIe card sticking out hte back of the port with electrical tape (to make sure they don't short to something) rather than cutting down the expansion card, as that permanently makes it 4x. No turning back.

I have done this successfully in the past with other cards.

Agree with your analysis - but needs to be tested to confirm results. For example, the same analysis should apply to the Intel-based dual 10gbe cards. But real world testing of these cards when plugged into 'PCIe 2.0 x4 in x8' slots on Supermicro MBs results in terrible performance, well below 5gbps throughput on a single 10gbe link. Same card in a proper x8 slot on the same MB, same CPUs, gets full bandwith.

No idea why this is. I didn't bother to root cause it. But the result was quite repeatable across multiple different systems. Something about these cards got quite upset about running with only x4 lanes.

Not even pretending to say these Brocade cards won't work well with only x4. Just reminding folks that YMMV, paper analysis does not always pan out in practice and test, test, test.

 

[mattlach]

Agree with your analysis - but needs to be tested to confirm results. For example, the same analysis should apply to the Intel-based dual 10gbe cards. But real world testing of these cards when plugged into 'PCIe 2.0 x4 in x8' slots on Supermicro MBs results in terrible performance, well below 5gbps throughput on a single 10gbe link. Same card in a proper x8 slot on the same MB, same CPUs, gets full bandwith.

No idea why this is. I didn't bother to root cause it. But the result was quite repeatable across multiple different systems. Something about these cards got quite upset about running with only x4 lanes.

Not even pretending to say these Brocade cards won't work well with only x4. Just reminding folks that YMMV, paper analysis does not always pan out in practice and test, test, test.

Fair enough!

Maybe we can get Kristian to comment on his results?

 

[Kristian]

unfortunatly not. That is because 5 minutes ago I requested a RMA number for my A1SAM.
As the BR-1020's are not working in my Asus P9A-I I am left with just one device with 10GB at the moment.
So I can't be of any assistance until I get a new A1SAM.
I will try to do some tests when the board is there.
Perhaps in the meantime you could tell me what software, test methods/software would be appropriate for the results you would like to know.

 

[mattlach]

unfortunatly not. That is because 5 minutes ago I requested a RMA number for my A1SAM.
As the BR-1020's are not working in my Asus P9A-I I am left with just one device with 10GB at the moment.
So I can't be of any assistance until I get a new A1SAM.
I will try to do some tests when the board is there.
Perhaps in the meantime you could tell me what software, test methods/software would be appropriate for the results you would like to know.

Ahh.

I haven't received mine yet, but I can do some testing when I get mine (and the transcievers and cable).

My SuperMicro X8DTE, has all 8x slots physically, 4 of them 8x PCIe 2.0 electrically as well, but one 4x PCIe 2.0 electrically and one 4x PCIe 1.1 electrically.

I could move it from slot to slot and do a iperf and see how it changes.

 

[mrkrad]

Most folks run 1 port as primary, and 1 port as standby-failover - with a second nic in a server! This allows oversubscribing of the pci-e bus since no 1 card will be pushing two ports at once!

 

[mattlach]

Most folks run 1 port as primary, and 1 port as standby-failover - with a second nic in a server! This allows oversubscribing of the pci-e bus since no 1 card will be pushing two ports at once!

Sounds reasonable as long as they don't suffer from the same problem PigLover mentioned the Intel 10gbit cards suffer from @4x

 

[itguy82]

I am trying to get two of the 1020 cards working point to point between two lenovo servers. One is the SAN (running OmniOS in Esxi), and the other I want to boot off a LUN on the SAN.

In Napp-It running on Omni OS I get: Error: Failed to open fcoe device

It also shows the link state as down. Any ideas? I am using a Brocade branded active twinax cable and the link light is green on both of the cards.

 

[mattlach]

So, people are not having any issues in Ubuntu anymore?

I am trying to use this card in a Linux Mint 17.1 box (based off of Ubuntu 14.04) and am having the same "Can't locate firmware ctfw-3.2.1.0.bin" as mentioned earlier in this thread for ubuntu 13.04 and 13.10.

Hey everyone. I got this working in Ubuntu 13.04 and 13.10, but it definitely felt hacky. I'm sure that I'm not doing this the right way, but it works.

My guess is that udev and/or the inbox driver for the brocade adapter is specifying the firmware file ctfw-3.2.1.0.bin, explicitly. Even if you put a newer driver in the /lib/firmware directory, it still doesn't load; it's insistent on 3.2.1.0, which is not only out of date, but not specifically available from Brocade on their site, as far as I can tell. This said, it's not crazy to do this. From experience, adapter drivers and firmware operate in lockstep and you shouldn't really mix the release levels up. In this case I didn't have much choice. And perhaps it has everything to do with my occasional problems with the adapter, maybe.


So. If you were so inclined, what you can try is getting ctfw-3.2.1.1 from one of Brocade's firmware packages, and copying that to the /lib/firmware directory. It still won't load, and insists on 3.2.1.0, however. To address this, I symlinked ctfw-3.2.1.1.bin to ctfw-3.2.1.0.bin in /lib/firmware. After unloading/reloading bna, it works.

Despite this, the system won't boot with the adapter loaded. I put together a hacky init script set up to basically run `rmmod bna; modprobe bna` one more time somewhere during boot.
A better explanation: When you/boot runs `modprobe bna` and it fails, it stays in that broken-loaded state. Running modprobe again won't do anything new for you, as it's "already loaded" in a sense. Unload it, then load it agian).
After this, you should see the Brocade network ports in ifconfig afterwards.

If you're using the FCoE part of the adapter, do the same with cbfw and ct2fw, like above.

I noticed that 14.04 (Trusty Tahr)'s linux-firmware package contains some firmware files of interest:
/lib/firmware/ct2fw-3.2.1.1.bin
/lib/firmware/ct2fw-3.2.3.0.bin
/lib/firmware/ctfw-3.2.1.1.bin
/lib/firmware/ctfw-3.2.3.0.bin
This suggests to me that the inbox driver has been updated and everything should just work out of the box. I'll post my experience soon - that is, if the daily I'm installing from right now doesn't break my system.


The best way to address all of this was to find a way to compile the Brocade driver and not to use the inbox one, however their noarch/generic Linux driver won't compile on Ubuntu. I never really figured it out - I feel it's a fundamental problem/difference between a build environment on RedHat/CentOS/Fedora, compared to what Ubuntu is doing. You can't install the RPM via Alien for the same reasons.
To name build issue, it's the replacement of /bin/sh with dash (DashAsBinSh - Ubuntu Wiki) on Ubuntu. Despite reverting this back to /bin/sh, or changing some part of the compiling process to use /bin/bash instead, I didn't have much more luck. There's still some pretty elementary broken stuff going on for compiling and I really just don't have the technical know-how to get past that.


Oh, and regarding cables, I'm using some direct-connect Active Twinax cables+SFPs. Provided you're using fairly recent firmware, you can use third-party SFPs, not just Brocade's.


Edit: 14.04 successfully loads its inbox driver and a newer firmware was included, no tweaks or hacks needed. It's still behind what Brocade supplies, but I'll take a working solution!
It also broke my bonding config, but restarting networking solves that. Oh, Ubuntu. Bug #1280366 “Bonding mode ignored” : Bugs : ifenslave package : Ubuntu
Edit 2: If you try a fresh 14.04 install and the firmware is missing, install the package linux-firmware.

Like above, I created the symlink in /lib/firmware from /vat/lib/ctfw-3.2.1.0.bin to /vat/lib/ctfw-3.2.1.1.bin This still doesn't work on boot, causing a LOOONG "waiting for ethernet interface" boot.

The funny part is, if I then proceed to remove the bna module (modprobe -r bna) and then reload it (modprobe bna) it works (and grabs a TON of IRQ's)!

[  259.460717] Brocade 10G Ethernet driver - version: 3.2.21.1
[  259.460732] bnad_pci_probe : (0xffff880beb97a000, 0xffffffffa131b0c0) PCI Func : (1)
[  259.462683] bar0 mapped to ffffc90022000000, len 262144
[  259.462911] bna 0000:03:00.1: irq 88 for MSI/MSI-X
[  259.462916] bna 0000:03:00.1: irq 89 for MSI/MSI-X
[  259.462919] bna 0000:03:00.1: irq 90 for MSI/MSI-X
[  259.462923] bna 0000:03:00.1: irq 91 for MSI/MSI-X
[  259.462926] bna 0000:03:00.1: irq 92 for MSI/MSI-X
[  259.462930] bna 0000:03:00.1: irq 93 for MSI/MSI-X
[  259.462933] bna 0000:03:00.1: irq 94 for MSI/MSI-X
[  259.462936] bna 0000:03:00.1: irq 95 for MSI/MSI-X
[  259.462940] bna 0000:03:00.1: irq 96 for MSI/MSI-X
[  259.462943] bna 0000:03:00.1: irq 97 for MSI/MSI-X
[  259.462946] bna 0000:03:00.1: irq 98 for MSI/MSI-X
[  259.462950] bna 0000:03:00.1: irq 99 for MSI/MSI-X
[  259.462953] bna 0000:03:00.1: irq 100 for MSI/MSI-X
[  259.462956] bna 0000:03:00.1: irq 101 for MSI/MSI-X
[  259.950972] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready

No idea why it would work when manually using modprobe but not on boot.

Does anyone have any suggestions?

Also, does anyone know if it is possible to write a firmware to these WITHOUT the boot rom, like you can with the IBM M1015's that everyone uses for storage on here? The added boot times of the bootrom are VERY long on this card and are driving me absolutely insane.

Much obliged,
Matt

 

[mattlach]

Edit: 14.04 successfully loads its inbox driver and a newer firmware was included, no tweaks or hacks needed. It's still behind what Brocade supplies, but I'll take a working solution!
It also broke my bonding config, but restarting networking solves that. Oh, Ubuntu. Bug #1280366 “Bonding mode ignored” : Bugs : ifenslave package : Ubuntu
Edit 2: If you try a fresh 14.04 install and the firmware is missing, install the package linux-firmware.

Yeah, so this definitely isn't the case for me in Mint 17.01. Even though it is based on Ubuntu 14.04, I get the same issue as in 13.04 and 13.10. Linux firmware is installed and at latest rev...

 

[mattlach]

Yeah, so this definitely isn't the case for me in Mint 17.01. Even though it is based on Ubuntu 14.04, I get the same issue as in 13.04 and 13.10. Linux firmware is installed and at latest rev...

So it looks like this may be a problem with my install, probably due to how I upgraded it.

Mint recommends against doing distribution upgrades, and insists instead on clean installs. (IMHO the biggest downside to Mint, otherwise it would be perfect)

I upgraded it from 16 ->17 -> 17.1 Something was obviously left behind.

When I booted the 17.1 mint live install CD, the Brocade just worked, like others reported, but my install behaves like Ubuntu 13.04 and 13.10.

Seems like it might be time for a fresh install.

 

[mattlach]

So, does anyone know if the BCU command utility can be used to completely disable the Option ROM on these?

The options are a little convoluted and difficult to understand.

I'm using this card in a Asus P9X79 Workstation board which seems to lack option rom disabling in bios, and this card has the longest option rom configuration screens I have ever encountered, and whats worse, there are two of them, one for the NIC portion and one for the HBA portion. Driving myself crazy when I am rebooting and testing...

 

[Entz]

IIRC you can turn off the BIOS for the HBA portion to speed it up a bit but yeah they are very slow. Haven't seen an option to disable it entirely though, super slow, worse when you have a couple of them in a machine =/

 

[mattlach]

IIRC you can turn off the BIOS for the HBA portion to speed it up a bit but yeah they are very slow. Haven't seen an option to disable it entirely though, super slow, worse when you have a couple of them in a machine =/

Yeah, for their originally intended uses in servers, this would have been fine. I mean, how often do you reboot a production server?

I decided to put one in my workstation though in order to get speeds to my storage server closer to that of local drives, and it is a little annoying.

The irony of it is, my server motherboard (Supermicro X8DTE) has bios options to disable the loading of Option ROM's on a per slot basis, I believe. My Workstation board (Asus P9X79 WS) where this is really bothering me does not. It has an option to hide the Option ROM screen behind a graphical boot screen, but it still takes the same time