Connectx-3 HP to Mellanox firmware help

Marsh

Moderator
May 12, 2013
2,292
1,106
113
Look at Patrick forsale thread, he may still have the switch forsale.
Hopefully, Patrick would come along to answer your question about a "reasonably priced" switch.
But Patrick's reasonably priced $$ may be different than my $$.

Add:
Last week, I picked up 2 x Mellanox SX6015 switches ( not GBe switch ) for $150 each.
So I am not looking at 40 GBe switch for the time being.
 

patchate

Member
Jul 3, 2016
60
16
8
Look at Patrick forsale thread, he may still have the switch forsale.
Hopefully, Patrick would come along to answer your question about a "reasonably priced" switch.
But Patrick's reasonably priced $$ may be different than my $$.

Add:
Last week, I picked up 2 x Mellanox SX6015 switches ( not GBe switch ) for $150 each.
So I am not looking at 40 GBe switch for the time being.
So sorry. I meant the QSFP+ DACs on eBay. Already have the switch :)

I guess I'm confused because there is no differentiation between Infiniband cables having different data rates (QDR/DDR/FDR) and QSFP+ DAC cables. For example, would an Infiniband FDR-10 cables work with ethernet switches having QSFP+ ports?
 

sthsep

Member
Mar 7, 2016
72
10
8
Can someone upload the tgz files? It is not available anymore on the mellanox homepage.
 

_alex

Active Member
Jan 28, 2016
874
94
28
Bavaria / Germany
Nice, i just was wondering if there might be a newer one and what it's address would be.
Will soon need to care about my 4 HP branded adapters, too.
 

sthsep

Member
Mar 7, 2016
72
10
8
Well the flashing guid from @Marsh at the first page is working fine for me. Mine came with a firmware 2.11.1250 from 2013. They are looking like that they were never used.
 

_alex

Active Member
Jan 28, 2016
874
94
28
Bavaria / Germany
flashing usually isn't that difficult, customizing fw if needed is a bit more interesting.
didn't have time to look closer on my cards than 'they work' after they arrived - hopefully next week is a bit more relaxed and offers more opportunities to play with my toys...
 

SenH

New Member
Mar 28, 2017
6
1
1
I crossflashed HP_0280210019 from eBay to ConnectX3-rel-2_40_5000 firmware as described in this thread but my card only works with the built-in Windows 10 drivers.

After installing MLNX_VPI_WinOF-5_35_All_win2016_x64 the ConnectX-3 Ethernet Adapter driver fails to load and the Port Type Configuration is greyed out. See attached screenshots.

Mellanox Driver Error.PNG

Following messages in system log:
SingleFunc_2_0_0: Although Port #1 is configured as IB, the software sets it to Ethernet only. This occurs when the Ethernet link is already up due to one of the following: BMC is enabled or the firmware version installed in not up-to-date.

Port type registry value for device SingleFunc_2_0_0 could not be modified to value (PortType = ). Previous value will be set.

Mellanox ConnectX-3 VPI (MT04099) Network Adapter (PCI bus 2, device 0, function 0): SR-IOV cannot be enabled because FW does not support SR-IOV. In order to resolve this issue please re-burn FW, having added parameters related to SR-IOV support.

With built-in Windows 10 driver no error is logged, but I don't have the "Port Configuration" tab.

Windows Driver OK.PNG

Anyone else encountered this? Does the firmware ini need to be changed? I noticed some differences between the extracted config.ini (via flint -dc) and the provided MCX354A-FCB_A2-A5.ini
 
Last edited:

SenH

New Member
Mar 28, 2017
6
1
1
Here is the query after FW flash:
Code:
>flint -d mt4099_pci_cr0 query full
Image type:          FS2
FW Version:          2.40.5032
FW Release Date:     16.1.2017
MIC Version:         2.0.0
Config Sectors:      1
PRS Name:            cx3-1_MCX354A_fdr_09v.prs
Product Version:     02.40.50.32
Rom Info:            type=PXE version=3.4.747 devid=4099
Device ID:           4099
Description:         Node             Port1            Port2            Sys image
GUIDs:               0002c90300f2aff0 0002c90300f2aff1 0002c90300f2aff2 0002c90300f2aff3
MACs:                                     0002c9f2aff0     0002c9f2aff1
VSD:
PSID:                MT_1090120019
Query with stock HP firmware:
Code:
Image type:          FS2
FW Version:          2.10.2280
MIC Version:         1.2.0
Rom Info:            type=PXE version=3.3.650 devid=4099 proto=VPI
Device ID:           4099
Description:         Node             Port1            Port2            Sys image
GUIDs:               0002c90300f2aff0 0002c90300f2aff1 0002c90300f2aff2 0002c90300f2aff3
MACs:                                     0002c9f2aff0     0002c9f2aff1
VSD:                
PSID:                HP_0280210019
 

i386

Well-Known Member
Mar 18, 2016
2,040
539
113
31
Germany
Okay your firmware is up to date (2.40.5032, not available on mellanox' website but shipped in the winof package 5.35 fur2).

Googling "code 37 mellanox" lead me to the release notes for winof 5.35: http://www.mellanox.com/related-docs/prod_software/MLNX_VPI_WinOF_Release_Notes_Rev_5.35.pdf

The problems you described are the known issues #909595 & #951028. The workarounds suggest parsing the etl files fore the code 37 & disabling bmc (as the firmware is up to date).
 

SenH

New Member
Mar 28, 2017
6
1
1
I'm not sure if it's necessary to crossflash to Mellanox.

There is a fairly recent firmware (2.40.5030-Flex-3.4.746) for this card available via HP support. In combination with WinOF 5.22 driver everything is OK. No errors what so ever.
 
Last edited:

SenH

New Member
Mar 28, 2017
6
1
1
Just a small update, performance is good between Windows 10 LTSB (WinOF 5.22 driver) & Linux 4.10.8 (built-in mlx4_en module). I did not do any tuning except for setting the MTU to the max of 9614. CPU usage on Windows during transfers is around 10% (i7 6900K).

Linux sending to Windows:
Code:
iperf --client 10.0.0.2 --parallel 4 --window 512K
------------------------------------------------------------
Client connecting to 10.0.0.2, TCP port 5001
TCP window size:  416 KByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  6] local 10.0.0.1 port 48230 connected with 10.0.0.2 port 5001
[  4] local 10.0.0.1 port 48226 connected with 10.0.0.2 port 5001
[  5] local 10.0.0.1 port 48228 connected with 10.0.0.2 port 5001
[  3] local 10.0.0.1 port 48224 connected with 10.0.0.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec  11.4 GBytes  9.75 Gbits/sec
[  4]  0.0-10.0 sec  11.6 GBytes  9.94 Gbits/sec
[  5]  0.0-10.0 sec  11.5 GBytes  9.85 Gbits/sec
[  3]  0.0-10.0 sec  11.5 GBytes  9.89 Gbits/sec
[SUM]  0.0-10.0 sec  45.9 GBytes  39.4 Gbits/sec
Windows sending to Linux:
Code:
iperf --client 10.0.0.1 --parallel 4 --window 512K
------------------------------------------------------------
Client connecting to 10.0.0.1, TCP port 5001
TCP window size:  512 KByte
------------------------------------------------------------
[  4] local 10.0.0.2 port 51689 connected with 10.0.0.1 port 5001
[  5] local 10.0.0.2 port 51690 connected with 10.0.0.1 port 5001
[  3] local 10.0.0.2 port 51688 connected with 10.0.0.1 port 5001
[  6] local 10.0.0.2 port 51691 connected with 10.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  11.4 GBytes  9.82 Gbits/sec
[  5]  0.0-10.0 sec  11.6 GBytes  9.95 Gbits/sec
[  3]  0.0-10.0 sec  11.3 GBytes  9.70 Gbits/sec
[  6]  0.0-10.0 sec  11.4 GBytes  9.82 Gbits/sec
[SUM]  0.0-10.0 sec  45.7 GBytes  39.3 Gbits/sec
 
Last edited: