Beware - Emulex 10GbE Virtual Fabric Adapter II x8 PCI-E slot

[Kaytro]

I managed to get the cards working without needing to reset the port on startup.

Update the cards to the 11.x firmware from broadcoms website and then disable all of the DCBx features from the One command CNA manager. Reboot and using the CNA manager, disable then re-enable the port. Verify all of the DCBx features are disabled. Reboot and check to see if the ports are online.

 

[mytime34]

I bought a pair of these and I am able to enable the port in windows with CNA, but if I unplug or reboot the link goes down. I have updated to 11.4 fw and it gives me lots of options, but no option for link enabled. Is there any work around?
I am also trying this card in unraid and it is detected and showing be2net driver, but power on or power down of the port does nothing. I may have to upgrade to a different card.

In windows when it is enabled I am getting 990mbs transfer between 2 computers via DAC or fiber, I was hoping to bond then for higher throughput.

 

[Sickculture]

Anybody get a IBM Emulex 49Y7942 working in VMWare ESXI 6.7? The card show with elnex driver and ports are green with signal. But i can't connect.

 

[mariog]

Forcing the Interface Speed to 10GB instead of default also gives a link with the manager under Windows.

Ethtool cannot set the speed, is there another tool for linux tot set the speed?

 

[SirGreedy]

I am just here to say: thank you so much my dear sir!

After so much frustration with no link signal at all, I bumped into your thread eventually. I was using the newest version of Emulex OneCommand Manager (and now they renamed it to "HBA manager"), and just simply cannot find my 49Y7942 card.

The "elxocm-windows-x64-10.0.803.9-1" included in your Google Drive folder is the savior. After unable/enable the ports, just as what you have described, the link is on!!

I am so happy and so grateful.

Hmmm ok had another play tonight

thought I would try and find an older firmware, specifically from IBM for these cards.... not the Emulex firmware that they generally direct you to.

found a different firmware,

cards originally were

1st card was firmware version 4.2.412
2nd card was older at 3.x.xx

updated both originally with the emulex firmware ver 10.0.803.31

Both cards and transceivers wouldn't get a link

so now I found IBM Firmware 10.2.261.36-1

applied that in windows, connected port 0 to port 1 and got a link up

applied the same firmware to the 2nd card, installed ubuntu 12.04.........connect cable between the two servers (port 0 on each) and back to the link down status

fiddled around in the bios and changed the port status's to disabled, then back again to enabled........boom 10gbps link

exit bios and reboot....link light still on (good sign)

boot into ubuntu 12.04 live usb.......link is still up.

quick ping and a samba copy test to the windows box.......good it's working

so now installing ubuntu 12.04 properly to see what happens

pretty happy, even if it's only testing at present.... and lots to learn and configure yet. (kind of wish I found this before I forked out for the DAC cables that are on route....but meh I still need one more cable at minimum)

total cost so far

2 x IBM 49Y7942 = $100 + $8 postage
3 x SFP+ Transceivers = $31.63 + $6.15 postage
2 x LC>LC Cables Free already owned them

Total $145.78

3 x DAC Cables still to arrive $155.71 >>> more than the working lot above

Might have to offload a couple of the DAC cables to recoup some costs

 

[EasyRhino]

So this super old thread was super useful to me.

I recently picked up two of the Emulex oce11102-n-xi cards for $15 each off ebay.

I only put them in x16 slots so I didn't have any physical fit problems.

I DID have the problem of them showing "link down" and no lights on when I plugged in two cards via a DAC cable that everyone else has had. And disabling/enabling (or forcing a speed in esxi) worked to enable it.

I almost wonder if the problem could be in SFP+ compatibility? Because even in the ctrl-p bios screen it shows as "link down". And this IBM page made it sound like the cards are really picky on their SFP+ compatibility:

Support list matrix for SFP and Twinax DAC cables - Converged Network Adapter

Link flapping, Light Emitting Diodes (LEDs) not illuminating, and other connectivity issues may occur when installing untested, unsupported Small Form-factor Pluggable (SFP) transceivers and Twinax/Direct Attached Copper (DAC) cables in Converged NetworkAdapters (CNA).

www.ibm.com

I managed to get the cards working without needing to reset the port on startup.
Update the cards to the 11.x firmware from broadcoms website and then disable all of the DCBx features from the One command CNA manager. Reboot and using the CNA manager, disable then re-enable the port. Verify all of the DCBx features are disabled. Reboot and check to see if the ports are online.

I wish I could get this working, but mine still shows link down even when I disable the DCBx and LLDP settings. Basically, for my windows desktop, if it goes to sleep or if I shut down/reboot it, then I'm going to show link down until I do a disable/enable.

 

[gtcdma]

I've been desperately trying to get one of these NICs (49Y7942) to work on Ubuntu 18.04 and had the same "link down" issue.
Tried different firmware versions and settings, but no luck.

Until I suddenly found out that running the ethtool test activates the link:

sudo ethtool -t enp1s0f0
sudo ethtool -t enp1s0f1

So I've added this command to /etc/network/interfaces and now the link comes up at boot as expected:
auto enp1s0f0
iface enp1s0f0 inet dhcp
pre-up ethtool -t enp1s0f0

It's on IBM Firmware 10.2.261.36-1, be2net driver version 12.0.0.0.

It works fine with a passive DAC cable (Arista CAB-SFP-SFP-2M) as well as Cisco SFP-10G-SR
I haven't noticed any performance issues so far.

 

[antenagora]

I've been desperately trying to get one of these NICs (49Y7942) to work on Ubuntu 18.04 and had the same "link down" issue.
Tried different firmware versions and settings, but no luck.

Until I suddenly found out that running the ethtool test activates the link:

sudo ethtool -t enp1s0f0
sudo ethtool -t enp1s0f1

So I've added this command to /etc/network/interfaces and now the link comes up at boot as expected:
auto enp1s0f0
iface enp1s0f0 inet dhcp
pre-up ethtool -t enp1s0f0

It's on IBM Firmware 10.2.261.36-1, be2net driver version 12.0.0.0.

It works fine with a passive DAC cable (Arista CAB-SFP-SFP-2M) as well as Cisco SFP-10G-SR
I haven't noticed any performance issues so far.

I can confirm this, even with firmware oc11-12.0.1280.0.ufi from:

HPE Support Center

support.hpe.com

rebooting gives me a "link up" but powering off the server gives me the usual "link down".

I wold like to use this adapters in truenas or freebsd but i am not able to find something like ethtool to make le nic reset and test the network speed.

Does anybody know if brcmhbacmd is available on *BSD ?

 

[EasyRhino]

This thread is super old, I'm just bumping for posterity with a tidbit of knowledge.

Here is a small and very, very basic batch file you can run at startup to enable the link . It's not optimal but it works.

First of all, the inspiration in this post of setting up task scheduler to bounce the card in Windows saved my butt last year, thanks Bevege!

However, I had a few modifications and additions to make to the task scheduler setup

1. If you use the latest onecommand CLI tools from Broadcom, the path has changed from Emulex to Broadcom. The new default location is this:

"C:\Program Files\Broadcom\Util\OCManager\BrcmHbaCmd.exe"

2. you don't actually have to make a batch file. You can have task schedule trigger two actions back to back. The program is the brcmhbacmd.exe and the arguments have the MAC and the 0 or 1 to disable and enable. two like this:


3. having a trigger of "at system startup" is great, but it doesn't handle the computer going to sleep and waking up, which also causes the connection to fail. Here's how to make a wake up trigger

1. trigger "on an event"
2. log is "system"
3. source is "Power-Troubleshooter"
4. Event ID is "1"
5. and, this part is crucial (don't know why) you need to enable "delay task for" for at least 30 seconds.

add that trigger in addition to system start, and you're more or less golden.

 

[EasyRhino]

So despite the quirky jankiness card, I've now been running two of these, one on Windows 10 and one with esxi 7.0 u2d, for over a year and a half and performance and reliability has been solid.

 

[repairman2be]

Thanks to everyone for their input and or suggestion.
What worked for me is the comment from user gtcdma

in my setup as follows:
ethtool -t enp68s0f0 - get the card to link up

/etc/network/interfaces

iface enp68s0f0 inet manual
mtu 9000
pre-up ethtool -t enp68s0f0


And the interface is turned on and link is up and running.
This is in Proxmox 7.x VE

I flashed the latest firmware from Emulex website which is 12.0.1317.2

I wish I could get rid of the IBM portion / firmware or whatever it is the card is using.

Anyway it is working now. Very frustrating indeed.

 

[EasyRhino]

regarding the bios / initial boot annoying thing, I've found that my supermicro motherboard allows me to selectively skip bioses in pci slots, so I can bypass that.

no trick like that on my consumer desktop motherboard

 

[mrjayviper]

what windows 10/11 drivers are you guys using? thanks

 

[EasyRhino]

what windows 10/11 drivers are you guys using? thanks

On Windows 10 I'm using the Microsoft driver. the date is 3/28/16 and version 11.0.274.8008.
Seems fine.
i had previously used the emulex downloaded ones from Broadcom, but there doesn't seem to be a need to.
The main quirk is it doesn't seem to support a jumbo MTU of 9000. Just 8222 or 9014.

 

[beta_init_n00b]

I think I a out of my depth here.
Having also bought a couple of these cards with the intention of connecting two computers directly via 10Gbe I am getting nowhere.
I tried to disable-enable the cards in the BIOS as has been suggested but keep getting link down. I've also tried to look into the other options suggested in this thread but as I am running W10 Pro it seems I am out of luck with regards to installing any of the utilities to control the cards.

Firmware is 11.1.215.0 and Windows driver is Microsoft 11.0.273.8008 (28 March 2016).

If anyone can point me in the direction of a solution I will be grateful. Happy to do my homework but cannot see the wood for the trees at the moment.

Thank you.

 

[EasyRhino]

For windows if you want a button to click to bring the link down/up, try to get the onecommand tools from Broadcoms site.

If if that works manually then I outlined how to automate it with windows task scheduler in post 49. I'm still using the card and it still works great. (Sometimes i forget about the 30 second delay)

 

[beta_init_n00b]

For windows if you want a button to click to bring the link down/up, try to get the onecommand tools from Broadcoms site.

If if that works manually then I outlined how to automate it with windows task scheduler in post 49. I'm still using the card and it still works great. (Sometimes i forget about the 30 second delay)

Thank you so much, @EasyRhino and @Bevege - cards are working now.
I downloaded the Emulex OneCommand CMA Manager and put together a task in Task Scheduler. In the beginning it did not work well as I enabled both ports on both (directly connected) cards even though only one port is currently in use.
After disabling the second port in the BIOS everything is working as it should.

One thing to note: Setting my task to start "at system startup" messed up iLok client (third party authorisation of various software). Switching to "user login (any user)" fixed that issue.

 

[shpitz461]

popped everything up here.... still uploading at present

emulex - Google Drive

Hi, I asked for access, I'm struggling getting my Emulex Oce11 card flashed. I tried booting up the iso package from Lenovo but it won't boot, getting cdrom not found error when booting in BIOS mode, in UEFI mode the USB drive gets stuck.

UEFI:

Thanks

 

[shpitz461]

Thank you so much, @EasyRhino and @Bevege - cards are working now.
I downloaded the Emulex OneCommand CMA Manager...

Hi, can you provide a link to the tool?
Thanks

 

[EasyRhino]

yes the broadcom support site is a dumpster fire to navigate.

you can search for downloads here:
Support Documents and Downloads (broadcom.com)

under "product family" you can search for "legacy ethernet network adapters", and then under "product name" you can search for key numbers like 11102.