Dell C6100 and Windows Server 2012 no RDMA

Smalldog

Member
Mar 18, 2013
62
2
8
Goodyear, AZ
I wonder if we can order the sled that the motherboard sits on? I am going to take a motherboard off and see if there is a Dell sticker on it with a part number. The numbers etched on the bottom of the sled don't yield anything in a search.

I personally can use 8 of them.
 

PigLover

Moderator
Jan 26, 2011
2,969
1,280
113
Empty sleds show up from time to time on ebay. Haven't looked recently. They are usually over-priced...
 

seang86s

Member
Feb 19, 2013
147
12
18
So I got an MHQH29C-XTR which had firmware 2.7.xxxx on it. I put it in one of my C6100 nodes that also has a infiniband mezzanine card in it. I ran the WinOF VPI package for 2012 from here (Mellanox Products: Mellanox OFED for Windows (WinOF)) and it upgraded the MHQH29C-XTR to firmware to 2.10.720. It left the mezzanine card alone saying I need to obtain the firmware from the OEM. FWIW, the MHQH29C-XTR shows its RDMA capable while the Dell card is not.

The info on the cards are almost identical except the part number. I'm wondering if there is a way to modify the part number or the .MSI package to force it to flash the Dell card. It sure does seem to be exactly the same otherwise. Take a look:



Anyone with experience modifying MSIs? I'm willing to use one of my mezzanine cards as a guinea pig.
 

RimBlock

Member
Sep 18, 2011
788
8
18
Singapore
Cant really help on that one but I have also just got a MHQH29C-XTR delivered and installed the OFA OFED pachage from Windows SBS 2011 but it did not upgrade the firmware. I used the OFA version to try and see if the SRP miniport would work (it's removed from the Mellanox package). It does not work with this newer card either :(.

I came across a mailing list post on one of the OpenIB development mailing lists stating that SRP on Windows development had no one working on it and it was out of sync with other OS developments. The post was from 2008 but if that had not been rectified it would highlight why the SRP miniport is not working with non-Windows targets.

RB
 

seang86s

Member
Feb 19, 2013
147
12
18
Very frustrating. I wanted to do an all Windows 2012 Hyper-V cluster using the C6100 but it seems I will only get a fraction of the performance the hardware is capable of. Perhaps I will abandon Hyper-V and go ESXi.

That 2.10.720 firmware is nowhere on Mellanox's website. It's only in that Mellanox OFED MSI package in what appears to be a text file!
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
The raw firmware file is around 4MB and has a .mlx extension. If someone can dig up a .mlx file with the 2.10.720 firmware version, I can make a flashable firmware image out of it for the Dell Mezzanine cards.

Very frustrating. I wanted to do an all Windows 2012 Hyper-V cluster using the C6100 but it seems I will only get a fraction of the performance the hardware is capable of. Perhaps I will abandon Hyper-V and go ESXi.

That 2.10.720 firmware is nowhere on Mellanox's website. It's only in that Mellanox OFED MSI package in what appears to be a text file!
 

seang86s

Member
Feb 19, 2013
147
12
18
The raw firmware file is around 4MB and has a .mlx extension. If someone can dig up a .mlx file with the 2.10.720 firmware version, I can make a flashable firmware image out of it for the Dell Mezzanine cards.
PM me your email address. I grabbed all the files out of the MSI and have two .mlx files. One for the ConnectX-2 and the other for the ConnectX-3.

Or, if you want to try to grab them yourself, run the original package from Mellanox and leave it at the first dialog (IOW, don't click next). Look in your c:\users\username\appdata\local\temp directory. There should be two directories under that. One has about 4 or 5 files in it with a large .MSI. The original package creates that directory and automatically executes the .MSI. That .MSI in turn creates the second temp directory and you'll all the files for that package, including the .mlx files.
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
With the help of your tip, I have the files. Thanks!

Sunday evening, I'll take a stab at building a firmware from them.

PM me your email address. I grabbed all the files out of the MSI and have two .mlx files. One for the ConnectX-2 and the other for the ConnectX-3.

Or, if you want to try to grab them yourself, run the original package from Mellanox and leave it at the first dialog (IOW, don't click next). Look in your c:\users\username\appdata\local\temp directory. There should be two directories under that. One has about 4 or 5 files in it with a large .MSI. The original package creates that directory and automatically executes the .MSI. That .MSI in turn creates the second temp directory and you'll all the files for that package, including the .mlx files.
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
With the help of your tip, I have the files. Thanks!

Sunday evening, I'll take a stab at building a firmware from them.
Very preliminarily, I have some good news: I just burned new custom firmware version 2.10.720 to a Dell IB Mezzanine card and it now says that RDMA is enabled.

I will test, and - if it works - post detailed files and instructions.
 

Smalldog

Member
Mar 18, 2013
62
2
8
Goodyear, AZ
I know this is a little off topic, but how do you properly setup any of the Mellanox HCA's in ESXi to properly work with Windows Server 2012 as a guest OS?

I installed the drivers on ESXi, and I see the vmnic_ib0 and vmnic_ib1 adapters. I created a switch, added the nic to the switch, and added the switch to the VM.

Windows Server 2012 either sees it as 1gb connection, or as an unrecognized ethernet adapter.

Step by step instructions would be most helpful, from the the ESXi side and the guest OS side. I am not partial to Server 2012, so I could also go with Server 2008 or 2008 R2.

I am not doing this particular activity on the C6100, and the server lacks passthrough, so I was hoping to get it working at the ESXi level.

Thanks way in advance,
Jeff
 

RimBlock

Member
Sep 18, 2011
788
8
18
Singapore
I found that passthrough was not working for me with the Mellanox cards even with the latest patches for ESXi. Would be interested to hear if anyone has it working.

What type of NIC have to assigned to the VM. Is it an ET1000 ?.

I am using mine as a SAN connection to the ESXi host and them mount target datastores so any VMs on them benefit from the storage speed. From what you are saying, you wish to use them for network communication rather than specifically a storage fabric ?.
 

Smalldog

Member
Mar 18, 2013
62
2
8
Goodyear, AZ
I would like to use it either way, but I had not tried to use it as storage yet. I'll give that a whirl and see how it pans out.

Jeff

*EDIT* Ok, got it hooked up and working at as a storage controller to an iSCSI target on one of the nodes in the C6100 running Windows Server 2012.

I would like to see it working as a network adapter on the other port now. Has anyone done that yet?
 
Last edited:

Smalldog

Member
Mar 18, 2013
62
2
8
Goodyear, AZ
I found that passthrough was not working for me with the Mellanox cards even with the latest patches for ESXi. Would be interested to hear if anyone has it working.
I am pretty sure that I had passthrough working on the C6100. I have done so many scenarios over the past few weeks, I don't remember what did and didn't work, but I *think* that passthrough did. I'll try it again tonight.
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
Very preliminarily, I have some good news: I just burned new custom firmware version 2.10.720 to a Dell IB Mezzanine card and it now says that RDMA is enabled.

I will test, and - if it works - post detailed files and instructions.
I have RDMA working in Windows 2012 with the Dell c6100/c6145 Infiniband Mezzanine card. Instructions are right here.

The URL above has downloads and screenshots. Here is the text only version:

Steps to Create and Burn firmware version 2.10.720 to a Dell c61xx server Infiniband Mezzanine card for Windows 2012:

1) Install a Dell QDR Infiniband mezzanine card into your c6100 or c6145 node running Windows 2012 and then reboot.

2) Install the latest Mellanox driver software. At the time of this writing, the latest version is 4.2, and the Windows installer is named MLNX_VPI_win8_x64_4_2.exe. The driver will let you know that your firmware is out of date and will offer to update it for you. Unfortunately, the update will fail with the message that your card is an OEM version and that you need to contact the OEM to get the latest firmware. A quick visit to the c6100 driver download page shows that no updated version is currently available.

3) Install the Mellanox WinMFT software package. This install gives us some of the tools we need in order to create and flash the firmware. At the time of writing, the latest version is 2.7.2 and the installer is named WinMFT_x64_2_7_2.msi.

4) Now we need to retrieve some information from your card. At a Windows prompt, run the command mst status and look for the PCI ID of your card. For my card, the ID is mt26428_pci_cr0, as shown below; yours will likely be the same unless you have multiple cards installed. By the way, the number 26428 is the Device ID (a product identifier) for the Dell mezzanine card. You may notice that it's the same Device ID as one of the Mellanox ConnectX-2 dual-port QDR cards, an indication that our Dell card is indeed a standard Mellanox product, albeit with Dell specific firmware.

mst status command output image

5) Now that you know your card's PCI ID, we need to verify a few other card attributes. At the same command prompt, run the command flint -d <PCI ID> query (in most cases that will be flint -d mt26428_pci_cr0 query) and record your card's Board ID and note the firmware version. In the screenshot below, we see a Dell QDR card that has already been updated with firmware version 2.10.720. Your card will likely have firmware version 2.9.1000. The Board ID will be DEL09A0000009. If your board ID is different then these instructions will not work for you.

flint query results image

6) Download the correct raw firmware file into a folder - the version that I used is right here. This is a large text file with a .mlx extension. I used the one
named fw-ConnectX2-rel.mlx from the non-Dell version 4.4.2 Mellanox driver installer named MLNX_VPI_win8_x64_4_2.exe. If you don't want to download the version linked above, you can extract your own firmware file by starting the Mellanox Windows 2012 driver installer. Leave the installer running, and look in the folder c:\users\<username>\appdata\local\temp for a file with the .mlx extension (thanks to ServeTheHome member seang86s for the tip)

7) Download the Dell .ini file as well, and place it in exactly the same folder as the .mlx firmware file. My version is right here. If the files are not in the same folder then the commands that I share later will not work properly. The Dell .ini file is named after the Board ID, so you'll need the file named DEL09A0000009.ini. You can download the version linked above, or you can create your own by editing and then renaming the .ini file for the equivalent non-Dell Mellanox card which is also in one of the c:\users\<username>\appdata\local\temp folders and is named MHQH29C_A1-A2.ini. To make your own .ini file, edit the contents so that Name = DEL09A0000009 and PSID = DEL09A0000009. Change the file name to DEL09A0000009.ini

8) You are now ready to create the new firmware image and burn it to your card. These two steps take place with one command. From the prompt, enter the command mlxburn.exe -dev <PCI ID> -fw <firmware file path>, which in our case is mlxburn.exe -dev mt26428_pci_cr0 -fw fw-ConnectX2-rel.mlx

9) Assuming that you do not see any errors, reboot the server.

10) Verify the new firmware version. After the reboot, open a command prompt and again run the command flint -d <PCI ID> (which is still flint -d mt26428_pci_cr0 in most cases). Look at the firmware version to verify that it was updated.

11) Verify RDMA. Open a PowerShell window and enter Get-SmbServerNetworkInterface. Verify that your cards shows that it is RDMA capable. The screen shot below shows one of my Dell Infiniband cards. There are four entries total: two for each Infiniband port, with one for IPV4 and one for IPV6. In my setup one port is unused and shows up as 40Gbps while the other port is configured for IPoIB and shows up as 32Gbps. Both ports show up as RDMA capable.

Get-SmbServerNetworkInterface image

12) Continue setting up your SMB share. By default, Windows 2012 is configured to use RDMA, but if you have trouble, look at the many available how-to guides for multi-channel SMB file sharing.
 
Last edited:

seang86s

Member
Feb 19, 2013
147
12
18
dba,

Thanks for the work on this. I'm looking forward to getting RDMA set up with the Dell hardware. Have you done any benchmarking yet?
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
I have. Testing throughput between two c6100 nodes over QDR Infiniband:

With 2.9.1000 firmware (no RDMA): 1,958MB/Second
With 2.10.720 firmware (RDMA): 3,279MB/Second

The above results are actual file sharing performance - not synthetic IP throughput - and they were achieved without any fancy tuning. I even left the Windows firewall running for example.

dba,

Thanks for the work on this. I'm looking forward to getting RDMA set up with the Dell hardware. Have you done any benchmarking yet?
 
Last edited:

zane

Member
Aug 22, 2013
70
0
6
This did not work for me any idea why? I noticed that I do not have a board id so I used PSID instead. I also had to use the -conf flag in the upgrade cmd. Using the same hardware as you as far as I could tell.

C:\Program Files\Mellanox\WinMFT>mlxburn.exe -dev mt26428_pci_cr0 -fw fw-Connect
X2-rel.mlx
-I- Querying device ...
-E- Can't auto detect fw configuration file: {no files matched glob patterns "C:
/Program/*.ini C:/Program/*.INI"} {no files matched glob patterns "Files/Mellano
x/WinMFT/*.ini Files/Mellanox/WinMFT/*.INI"} . Please specify configuration (ini
) file using -conf flag .

C:\Program Files\Mellanox\WinMFT>mlxburn.exe -dev mt26428_pci_cr0 -fw fw-Connect
X2-rel.mlx -conf DEL09A0000009.ini
-I- Generating image ...

Current FW version on flash: 2.7.8100
New FW version: 2.10.720

Burning FW image without signatures - OK
Restoring signature - OK
-I- Image burn completed successfully.

AFTER I REBOOTED

C:\Users\Administrator.XXX>flint -d mt26428_pci_cr0 query
Image type: ConnectX
FW Version: 2.10.720
Device ID: 26428
Description: Node Port1 Port2 Sys image
GUIDs: 14feb59097fee724 14feb59097fee725 14feb59097fee726 14feb59097fe
e727
MACs: 14feb5fee725 14feb5fee726
VSD:
PSID: DEL09A0000009


PS C:\Users\Administrator.XXX> Get-SmbServerNetworkInterface

Scope Name Interface Index RSS Capable RDMA Capable Speed IpAddress
---------- --------------- ----------- ------------ ----- ---------
* 27 False True 10 Gbps 169.254.92.107
* 27 False True 10 Gbps fe80::a147:517c:...
* 28 False True 10 Gbps 169.254.48.63
* 28 False True 10 Gbps fe80::8188:9295:...
* 26 True False 1 Gbps 192.168.2.227
* 26 True False 1 Gbps fe80::41b7:93d9:...
* 23 False False 10 Gbps 169.254.165.229
* 23 False False 10 Gbps fe80::9010:2ef1:...
* 21 False False 10 Gbps 169.254.169.190
* 21 False False 10 Gbps fe80::1d1b:7e17:...
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
This did not work for me any idea why? I noticed that I do not have a board id so I used PSID instead. I also had to use the -conf flag in the upgrade cmd. Using the same hardware as you as far as I could tell.

C:\Program Files\Mellanox\WinMFT>mlxburn.exe -dev mt26428_pci_cr0 -fw fw-Connect
X2-rel.mlx
-I- Querying device ...
-E- Can't auto detect fw configuration file: {no files matched glob patterns "C:
/Program/*.ini C:/Program/*.INI"} {no files matched glob patterns "Files/Mellano
x/WinMFT/*.ini Files/Mellanox/WinMFT/*.INI"} . Please specify configuration (ini
) file using -conf flag .

...
All I know is that you burned firmware without having your .ini file sorted out. You "forced" it, which appears to have worked, and yet you don't have RDMA. I'd go back and fix the .ini file situation so that the burn command works the same way that it did in my blog article.

If you can, post your output from the mst status command and the flint -d mt26428_pci_cr0 query command. Also, where did you get your X2-rel.mlx file?
 
Last edited:

zane

Member
Aug 22, 2013
70
0
6
All I know is that you burned firmware without having your .ini file sorted out. You "forced" it, which appears to have worked, and yet you don't have RDMA. I'd go back and fix the .ini file situation so that the burn command works the same way that it did in my blog article.

If you can, post your output from the mst status command and the flint -d mt26428_pci_cr0 query command. Also, where did you get your X2-rel.mlx file?
I got all the files from the post. I guess I can go back and try to create my own firmware file.

PS C:\Users\Administrator.XXX> mst status
MST devices:
------------
mt26428_pci_cr0
mt26428_pciconf0


C:\Users\Administrator.XXX>flint -d mt26428_pci_cr0 query
Image type: ConnectX
FW Version: 2.10.720
Device ID: 26428
Description: Node Port1 Port2 Sys image
GUIDs: 14feb59097fee724 14feb59097fee725 14feb59097fee726 14feb59097fe
e727
MACs: 14feb5fee725 14feb5fee726
VSD:
PSID: DEL09A0000009
 
Last edited: