HP ConnectX-3 FDR/EN 2-Port Adapter $99

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

rum1k

New Member
Jan 10, 2014
8
0
0

mrkrad

Well-Known Member
Oct 13, 2012
1,244
52
48
these only fit the special gen8 slot iirc, they won't perform correctly anywhere else. might check on that first!
 

rum1k

New Member
Jan 10, 2014
8
0
0
these only fit the special gen8 slot iirc, they won't perform correctly anywhere else. might check on that first!
Sorry then for incorrect post.

I just found these FlexLOM slot are the PCIe 3.0 x8 or x16, But as i see, so i found a little informations.
 
Last edited:

TD_Trader

Member
Feb 26, 2013
63
7
8
Oye... I just purchased TWELVE of these cards almost a week ago! :(

(I bought 10 of them on January 22, 2014, and then bought two more on January 26th).

Then this morning I was doing a search on trying to find the latest drivers and stumbled upon this post written today/yesterday.

So these cards won't work in a normal PCIe slot? :(

Has anyone tried to get these cards to work in a standard PCIe slot?

I just bought NINE of them here last week: HP 649282 B21 Infiniband FDR En 10 40GB2P 544FLR QSFP Adapter Dual Port Spare | eBay

I also bought ONE of them here last week: HP 656090 001 Infiniband FDR En 544FLR QSFP 10 40GB 2 Port Adapter | eBay

Will these cards NOT work in a normal PCIe slot?
 

TD_Trader

Member
Feb 26, 2013
63
7
8
Well, one of the cards just showed up in the mail today. The other 11 cards are still being shipped (via UPS).

I removed the metal brackets from the card, and I'm about to stick it into a PowerEdge T110 server, but will the card to any damage to the server? Will these cards work in a normal PCIe slot?

I've been unable to find any detailed information about them.
 

TD_Trader

Member
Feb 26, 2013
63
7
8
The good news is that if you remove the metal bracket, and also remove the second blue plastic bracket (across the top), the cards do fit fine in a normal PCIe slot.


https://www.dropbox.com/s/xj3y4wjb2wcply4/2014-01-27%2019.00.28.jpg


https://www.dropbox.com/s/p9uxrru9kkiznv2/2014-01-27%2019.01.06.jpg


https://www.dropbox.com/s/hbnxdkm6pgsfji9/2014-01-27%2019.00.54.jpg

I just stuck one into a PowerEdge T110 server, and I powered it on. No smoke yet. :) :) :)


https://www.dropbox.com/sc/y9zg6u8cmxlz52u/yVHv_A6CmU

It doesn't smell like anything is burning... (just yet...)

ESXi 5.5 booted up fine, but I didn't see the card as being detected in ESXi 5.5.


https://www.dropbox.com/s/dx2tjbr9zhwu92g/Screenshot%202014-01-27%2018.49.17.png

The PowerEdge T110 doesn't have any x16 slots, and it seems that it's only an x8 card/adapter.

I have the Mellanox/HP 649282-B21 (MCX354A-FCBT) HCA in slot 1 (x8 slot)
I have a ServeRAID M5014/5015 (46M0916) in slot 2 (x8 slot)
I have a MHGH29-XTC (Rev X4) dual 20G ConnectX HCA in slot 3 (x4 slot)

Any suggestions/ideas as to how I can get the Mellanox/HP 649282-B21 HCA adapter to appear in ESXi 5.5.0?

Do I have to have a QSFP cable attached to one of the ports in order for it to appear in ESXi 5.5.0? Any ideas? Has anyone managed to get one of these working with ESXi 5.5.0?
 
Last edited:

lmk

Member
Dec 11, 2013
128
20
18
Any suggestions/ideas as to how I can get the Mellanox/HP 649282-B21 HCA adapter to appear in ESXi 5.5.0?

Do I have to have a QSFP cable attached to one of the ports in order for it to appear in ESXi 5.5.0? Any ideas? Has anyone managed to get one of these working with ESXi 5.5.0?
Hey,

I think it should still show the card even if it is not connected via cable. Tip: disconnect the DDR card and check if it still shows up, after a restart.

I'll write up another reply with some stuff you could check. I just have to install ESXi 5.5 first, so I'll reply shortly.
 

lmk

Member
Dec 11, 2013
128
20
18
Okay, get shell/console access, if you do not already have it.
>> press F2 at the ESXi screen on the ESXi server, you might have to press it twice to wake the server and get the credentials prompt
Login with account, wait for options to appear
Enable ESXi shell (and/or Enable SSH, if you will do the rest remotely)
Press escape twice to save and exit

If you are using console access and will not connect directly with SSH
>> press ALT+F1, this will switch the console on the ESXi server to the root login prompt
Login

If you are using remote SSH access
>> connect via SSH and login with root at prompt :)

---

Now that you are logged in via console or SSH, run "lspci"

You will have the server list all devices. Now, on my server with ESXi 5.5 freshly installed and using only the inbox drivers from VMware and Mellanox, a ConnectX-2 VPI card showed up as the second last item in that list.

Specifically, it was listed as: 0000:02:00.0 Serial bus controller: Mellanox Technologies MT26428 [ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s] [vmnic2]

Again, that was without setting anything up and just after a clean install. The driver that is included inbox by VMware and Mellanox is an ethernet only driver. To get Infiniband functionality the driver needs to be removed and regressed to an older version that includes Infiniband functionality.

Let me know if this is helping.
 

Aluminum

Active Member
Sep 7, 2012
431
46
28
I'm curious about these obviously due to price, but very cautious

The good news is that if you remove the metal bracket, and also remove the second blue plastic bracket (across the top), the cards do fit fine in a normal PCIe slot.


https://www.dropbox.com/s/xj3y4wjb2wcply4/2014-01-27%2019.00.28.jpg


https://www.dropbox.com/s/p9uxrru9kkiznv2/2014-01-27%2019.01.06.jpg


https://www.dropbox.com/s/hbnxdkm6pgsfji9/2014-01-27%2019.00.54.jpg

I just stuck one into a PowerEdge T110 server, and I powered it on. No smoke yet. :) :) :)


https://www.dropbox.com/sc/y9zg6u8cmxlz52u/yVHv_A6CmU

It doesn't smell like anything is burning... (just yet...)

ESXi 5.5 booted up fine, but I didn't see the card as being detected in ESXi 5.5.


https://www.dropbox.com/s/dx2tjbr9zhwu92g/Screenshot%202014-01-27%2018.49.17.png

The PowerEdge T110 doesn't have any x16 slots, and it seems that it's only an x8 card/adapter.

I have the Mellanox/HP 649282-B21 (MCX354A-FCBT) HCA in slot 1 (x8 slot)
I have a ServeRAID M5014/5015 (46M0916) in slot 2 (x8 slot)
I have a MHGH29-XTC (Rev X4) dual 20G ConnectX HCA in slot 3 (x4 slot)

Any suggestions/ideas as to how I can get the Mellanox/HP 649282-B21 HCA adapter to appear in ESXi 5.5.0?

Do I have to have a QSFP cable attached to one of the ports in order for it to appear in ESXi 5.5.0? Any ideas? Has anyone managed to get one of these working with ESXi 5.5.0?
It may fit in a pci-e slot but are the pin-outs even the same? Even the pin pitch could be different. (HP has a long history of proprietary bullshit) Line it up next to a real pci-e card and take a good look.

Just because it doesn't short out your motherboard doesn't mean its working, there are a lot of ground traces that don't care if a data line is in the wrong spot ;)

Does the card even light up? If it does it should show up in tools like lspci at the very least. The cable being plugged in is irrelevant, you should be able to detect the card and install drivers regardless.
 

lmk

Member
Dec 11, 2013
128
20
18
It may fit in a pci-e slot but are the pin-outs even the same? Even the pin pitch could be different. (HP has a long history of proprietary bullshit) Line it up next to a real pci-e card and take a good look.

Just because it doesn't short out your motherboard doesn't mean its working, there are a lot of ground traces that don't care if a data line is in the wrong spot ;)

Does the card even light up? If it does it should show up in tools like lspci at the very least. The cable being plugged in is irrelevant, you should be able to detect the card and install drivers regardless.
From what I have found online so far, HP apparently is using (or at the very least is calling/advertising it as) a proprietary/custom port/pin-out based on a PCIe x8/x16. I think the only way we will all find out is with someone with an actual card, like TD_Trader, confirming it.
 

mrkrad

Well-Known Member
Oct 13, 2012
1,244
52
48
well it is thought that it might run as pci-e x4 in a pci-e x8 3.0 slot - but I doubt it would perform that well in pci-e 2.0 at all :(
 

TD_Trader

Member
Feb 26, 2013
63
7
8
From what I have found online so far, HP apparently is using (or at the very least is calling/advertising it as) a proprietary/custom port/pin-out based on a PCIe x8/x16. I think the only way we will all find out is with someone with an actual card, like TD_Trader, confirming it.
Yep, and I can CONFIRM that it doesn't work. If you're looking to burn up a few thousands of dollars worth of cards, and destroy your mainboard... then just follow my simple steps in the other thread.

It'll probably be easier to just HEED MY ADVICE, and don't bother trying it. The pinouts are definitely different, and even though it LOOKS like a normal PCIe connector and it FITS (just like a normal PCIe card) into a standard PCIe socket, apparently the pinouts are VERY different, and yes it will probably damage the adapter card and/or destroy your mainboard.

I now have a Dell PowerEdge T110 with a destroyed mainboard, that after I shut it down, it won't post up properly. It just keeps continuously rebooting. So apparently after trying the THIRD ADAPTER CARD, something must have happened to the mainboard, but the computer mainboard is now toast.

The three adapter cards that I "tested" will end up in the trash, and the PowerEdge T110 will probably end up on E-bay (as scrap/parts). The processor and RAM are probably still good, but the motherboard is probably toast.

The other NINE adapter cards (which are still brand-new in the box and SEALED and unopened) I will probably post to E-bay and dump them. It was a VERY VERY expensive experiment, and the simple answer is "NO IT WON'T WORK, SO DO NOT NOT NOT TRY IT!!"
 

TD_Trader

Member
Feb 26, 2013
63
7
8
It may fit in a pci-e slot but are the pin-outs even the same? Even the pin pitch could be different. (HP has a long history of proprietary bullshit) Line it up next to a real pci-e card and take a good look.

Just because it doesn't short out your motherboard doesn't mean its working, there are a lot of ground traces that don't care if a data line is in the wrong spot ;)

Does the card even light up? If it does it should show up in tools like lspci at the very least. The cable being plugged in is irrelevant, you should be able to detect the card and install drivers regardless.
[VERY EXPENSIVE LESSON]
@Aluminium - Nah, don't worry... it'll completely short out your mainboard! ;-)
[/VERY EXPENSIVE LESSON]


The FULL step-by-step story can be found here: http://forums.servethehome.com/networking/3132-mellanox-connectx-3-fdr-en-56g-2-port-adapter-%7C-hp-649282-b21-mcx354a-fcbt.html


[/END OF STORY] :(
 

TD_Trader

Member
Feb 26, 2013
63
7
8
well it is thought that it might run as pci-e x4 in a pci-e x8 3.0 slot - but I doubt it would perform that well in pci-e 2.0 at all :(
Yeah, I thought the same EXACT thing... :(

Unfortunately, I thought that SAME exact thing BEFORE you did...

The moral of the story is: "IT DEFINITELY WON'T WORK, and HP REALLY HATES ME!"

I highly doubt that Mellanox/HP will replace my mainboard, or the three destroyed adapter cards. So, NO do NOT try this at home. It DEFINITELY won't work. :(

These cards are designed to work with the latest and newest late 2013/2014 HP Proliant Gen8 servers that have a Gen8 "FlexibleLOM" socket, which apparently looks EXACTLY like a PCIe 3.0 socket/card, but apparently the pin-outs are different, and the adapter card will smoke/destroy your mainboard. Apparently Mellanox/HP wired these adapter cards differently, so that they will smoke/destroy a normal mainboard.

So, no they definitely won't work in a normal PCIe slot, and please don't try this "experiment" at home unless you want to burn up a server mainboard, and also destroy a few thousands of dollars worth of adapter cards.

Read my other thread if you want the step-by-step sadness. :(

No happy endings here... :(
 

Darkytoo

Member
Jan 2, 2014
106
4
18
So i have read all your posts (sorry about your fried server) and I looked yesterday and they were tons of listing for that same card. I was going to start checking with them to make sure they change the listing to "flexlom only, not PCIe compliant" since they all said they were compatible with PCIe x8 slots. I check today, and all the listings disappeared, so it looks like you're not the only one. I re-read your listing you purchased and it does say "verify compatabiliy before purchase" and it doesn't say "PCIe x8 slot" which the other listings did say, so I doubt you could go after the sell for mis-representation. None of this changes your situation, but I think it's a safe educated guess that you are not the only one with a fried motherboard.
 

TD_Trader

Member
Feb 26, 2013
63
7
8
The SAD thing is that I spoke with SIX different HP sales reps prior to purchasing the cards, and they ALL said that the cards would work in a normal/standard PCIe 3.0 x8 slot.

(Apparently they were just reading exactly what it said off of their OWN website...)

Even the sales reps told me that the cards would DEFINITELY work and that they were just a standard PCIe 3.0 x8 adapter card (from Mellanox).

After purchasing the cards, and trying/testing it out, it appears that these cards DEFINITELY won't work in a standard PCIe 3.0 x8 slot, and HP really shouldn't have marketed these cards as "PCIe 3.0 x8" cards, and instead should have used a different mezzanine connector (and NOT a standard PCIe 3.0 interface), and even their sales staff/reps had no clue what they were even talking about, because EVERY HP sales rep that I spoke with said that the cards are standard PCIe 3.0 x8 and DO work in a standard PCIe 3.0 x8 slot.

Good old HP... :(

Either that or all those sellers are following the thread :)
Yep, probably! :(

Hopefully others will find this thread in the future, and will NOT try it (or listen to any of the stupid/misinformed/uneducated HP sales reps!)

Apparently the HP Sales Reps ONLY know how to read what is written on their website/computer screen, and for them to say that the cards/adapters ARE in fact PCIe 3.0 x8 adapters and WILL work in a normal PCIe 3.0 x8 slot, is just absurd. Even their sales reps had no idea that the pin-outs were different.
 

TD_Trader

Member
Feb 26, 2013
63
7
8
None of this changes your situation, but I think it's a safe educated guess that you are not the only one with a fried motherboard.
I was actually kinda shocked, because SIX different HP Sales Reps and TWO different people from Technical Sales, ALL told me the same exact thing (reading off their stupid computer screens). Which was the same exact thing that I read. I thought I even saw a post from 1 1/2 years ago, that suggested that these cards actually do work in a standard PCIe 3.0 x8 slot.

I think even PigLover commented on one of the posts (from about a year and a half ago).

Here: HP FlexLOM 10Gb NIC as a standard NIC - [H]ard|Forum

Nobody ever took the time to EDUCATE anyone over at HP, or even post a simple message saying "NOPE IT DOESN'T WORK".

The HP Sales reps still seem to think that it will work. I'm just SHOCKED at how ignorant/uneducated/misinformed the HP Sales Reps are, and I guess they were ALL under the assumption that standard PCIe 3.0 x8 meant standard PCIe 3.0 x8.

It would have been nice if HP would have actually posted technical specifications on their cards/adapters, so people can see/understand that the pin-outs are different. I did several searches, and posted several messages, and NOBODY seemed to have any information that these cards were NOT standard PCIe 3.0 x8 adapters and that they would NOT work in a standard PCIe 3.0 x8 slot.

Even I thought the SAME exact thing when I purchased them, and it wasn't until AFTER I tried them, that I realized that these cards are probably using some type of different/proprietary HP pin-out, but HP sure doesn't post any information/specifications, and their sales reps all seem CLUELESS, and all seem to be under the assumption that these are in fact STANDARD Mellanox/HP PCIe 3.0 x8 PC adapter cards.

I'm going to contact HP on Monday, and see what they have to say about it.

None of this changes your situation, but I think it's a safe educated guess that you are not the only one with a fried motherboard.
It's unfortunate.

Either that or all those sellers are following the thread :)
I wish some of the HP Sales Reps would actually FOLLOW this thread. :(

Lesson learned... NEVER trust anyone over at HP.

It's highly unlikely that anyone at HP will swap the remaining cards that I have for regular PCIe 3.0 x8 cards/adapters, so I'm probably just out a bunch of money, and now have a handful of destroyed equipment/servers.

I guess the moral of the story is, do NOT listen to any ignorant HP Sales Reps, and even if people tell you that they ARE in fact normal standard PCIe 3.0 x8 Mellanox adapter cards and WILL in fact work in a standard PCIe 3.0 x8 slot, they clearly won't.

[/LESSON LEARNED THE HARD WAY]
 

mrkrad

Well-Known Member
Oct 13, 2012
1,244
52
48
Well I would have told you they don't work :) - Just remember there are some really good hp sales rep's on this forum ;)

They are like the emulex nic's that are sold to IBM that have pci-e x9 slots - only get one slot to fit them into and they are cheap because of this. Sadly most folks sell these nic's as regular nic's and then tell you to check compatibility.

Hopefully the cards did not do any damage to your motherboards, they are supposed to function as x4 nic's in a normal slot (in theory) but that seems to be a theory that hasn't been tested (or you just tested it for us).