Connectx-3 HP to Mellanox firmware help

[Marsh]

Look at Patrick forsale thread, he may still have the switch forsale.
Hopefully, Patrick would come along to answer your question about a "reasonably priced" switch.
But Patrick's reasonably priced $$ may be different than my $$.

Add:
Last week, I picked up 2 x Mellanox SX6015 switches ( not GBe switch ) for $150 each.
So I am not looking at 40 GBe switch for the time being.

 

[patchate]

Look at Patrick forsale thread, he may still have the switch forsale.
Hopefully, Patrick would come along to answer your question about a "reasonably priced" switch.
But Patrick's reasonably priced $$ may be different than my $$.

Add:
Last week, I picked up 2 x Mellanox SX6015 switches ( not GBe switch ) for $150 each.
So I am not looking at 40 GBe switch for the time being.

So sorry. I meant the QSFP+ DACs on eBay. Already have the switch

I guess I'm confused because there is no differentiation between Infiniband cables having different data rates (QDR/DDR/FDR) and QSFP+ DAC cables. For example, would an Infiniband FDR-10 cables work with ethernet switches having QSFP+ ports?

 

[i386]

Whoa... that was some deal.

Until last week many sellers had them listed for lower prices, and some of them accepted offers as low as 40$ per card (see the last posts in this thread: Mellanox ConnectX-3 MCX354A-FCBT/649281-B21 (Lot of 2) for $230 BO)

 

[sthsep]

Can someone upload the tgz files? It is not available anymore on the mellanox homepage.

 

[_alex]

this url works without the disclaimer that doesn't redirect the right way:

http://www.mellanox.com/downloads/firmware/ConnectX3-rel-2_40_5000.tgz

 

[sthsep]

Thank you.
And this is current the newest one:
http://www.mellanox.com/downloads/firmware/ConnectX3-rel-2_40_5030.tgz

 

[_alex]

Nice, i just was wondering if there might be a newer one and what it's address would be.
Will soon need to care about my 4 HP branded adapters, too.

 

[sthsep]

Well the flashing guid from @Marsh at the first page is working fine for me. Mine came with a firmware 2.11.1250 from 2013. They are looking like that they were never used.

 

[_alex]

flashing usually isn't that difficult, customizing fw if needed is a bit more interesting.
didn't have time to look closer on my cards than 'they work' after they arrived - hopefully next week is a bit more relaxed and offers more opportunities to play with my toys...

 

[SenH]

I crossflashed HP_0280210019 from eBay to ConnectX3-rel-2_40_5000 firmware as described in this thread but my card only works with the built-in Windows 10 drivers.

After installing MLNX_VPI_WinOF-5_35_All_win2016_x64 the ConnectX-3 Ethernet Adapter driver fails to load and the Port Type Configuration is greyed out. See attached screenshots.



Following messages in system log:
SingleFunc_2_0_0: Although Port #1 is configured as IB, the software sets it to Ethernet only. This occurs when the Ethernet link is already up due to one of the following: BMC is enabled or the firmware version installed in not up-to-date.

Port type registry value for device SingleFunc_2_0_0 could not be modified to value (PortType = ). Previous value will be set.

Mellanox ConnectX-3 VPI (MT04099) Network Adapter (PCI bus 2, device 0, function 0): SR-IOV cannot be enabled because FW does not support SR-IOV. In order to resolve this issue please re-burn FW, having added parameters related to SR-IOV support.

With built-in Windows 10 driver no error is logged, but I don't have the "Port Configuration" tab.



Anyone else encountered this? Does the firmware ini need to be changed? I noticed some differences between the extracted config.ini (via flint -dc) and the provided MCX354A-FCB_A2-A5.ini

 

[i386]

Please post the output of

flint -d mt4099_pci_cr0 query full

 

[SenH]

Here is the query after FW flash:

>flint -d mt4099_pci_cr0 query full
Image type:          FS2
FW Version:          2.40.5032
FW Release Date:     16.1.2017
MIC Version:         2.0.0
Config Sectors:      1
PRS Name:            cx3-1_MCX354A_fdr_09v.prs
Product Version:     02.40.50.32
Rom Info:            type=PXE version=3.4.747 devid=4099
Device ID:           4099
Description:         Node             Port1            Port2            Sys image
GUIDs:               0002c90300f2aff0 0002c90300f2aff1 0002c90300f2aff2 0002c90300f2aff3
MACs:                                     0002c9f2aff0     0002c9f2aff1
VSD:
PSID:                MT_1090120019

Query with stock HP firmware:

Image type:          FS2
FW Version:          2.10.2280
MIC Version:         1.2.0
Rom Info:            type=PXE version=3.3.650 devid=4099 proto=VPI
Device ID:           4099
Description:         Node             Port1            Port2            Sys image
GUIDs:               0002c90300f2aff0 0002c90300f2aff1 0002c90300f2aff2 0002c90300f2aff3
MACs:                                     0002c9f2aff0     0002c9f2aff1
VSD:                
PSID:                HP_0280210019

 

[i386]

Okay your firmware is up to date (2.40.5032, not available on mellanox' website but shipped in the winof package 5.35 fur2).

Googling "code 37 mellanox" lead me to the release notes for winof 5.35: http://www.mellanox.com/related-docs/prod_software/MLNX_VPI_WinOF_Release_Notes_Rev_5.35.pdf

The problems you described are the known issues #909595 & #951028. The workarounds suggest parsing the etl files fore the code 37 & disabling bmc (as the firmware is up to date).

 

[SenH]

I'm not sure if it's necessary to crossflash to Mellanox.

There is a fairly recent firmware (2.40.5030-Flex-3.4.746) for this card available via HP support. In combination with WinOF 5.22 driver everything is OK. No errors what so ever.

 

[SenH]

Just a small update, performance is good between Windows 10 LTSB (WinOF 5.22 driver) & Linux 4.10.8 (built-in mlx4_en module). I did not do any tuning except for setting the MTU to the max of 9614. CPU usage on Windows during transfers is around 10% (i7 6900K).

Linux sending to Windows:

iperf --client 10.0.0.2 --parallel 4 --window 512K
------------------------------------------------------------
Client connecting to 10.0.0.2, TCP port 5001
TCP window size:  416 KByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  6] local 10.0.0.1 port 48230 connected with 10.0.0.2 port 5001
[  4] local 10.0.0.1 port 48226 connected with 10.0.0.2 port 5001
[  5] local 10.0.0.1 port 48228 connected with 10.0.0.2 port 5001
[  3] local 10.0.0.1 port 48224 connected with 10.0.0.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec  11.4 GBytes  9.75 Gbits/sec
[  4]  0.0-10.0 sec  11.6 GBytes  9.94 Gbits/sec
[  5]  0.0-10.0 sec  11.5 GBytes  9.85 Gbits/sec
[  3]  0.0-10.0 sec  11.5 GBytes  9.89 Gbits/sec
[SUM]  0.0-10.0 sec  45.9 GBytes  39.4 Gbits/sec

Windows sending to Linux:

iperf --client 10.0.0.1 --parallel 4 --window 512K
------------------------------------------------------------
Client connecting to 10.0.0.1, TCP port 5001
TCP window size:  512 KByte
------------------------------------------------------------
[  4] local 10.0.0.2 port 51689 connected with 10.0.0.1 port 5001
[  5] local 10.0.0.2 port 51690 connected with 10.0.0.1 port 5001
[  3] local 10.0.0.2 port 51688 connected with 10.0.0.1 port 5001
[  6] local 10.0.0.2 port 51691 connected with 10.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  11.4 GBytes  9.82 Gbits/sec
[  5]  0.0-10.0 sec  11.6 GBytes  9.95 Gbits/sec
[  3]  0.0-10.0 sec  11.3 GBytes  9.70 Gbits/sec
[  6]  0.0-10.0 sec  11.4 GBytes  9.82 Gbits/sec
[SUM]  0.0-10.0 sec  45.7 GBytes  39.3 Gbits/sec