HP ConnectX-3 FDR/EN 2-Port Adapter $99

TD_Trader

Member
Feb 26, 2013
60
7
8
they are supposed to function as x4 nic's in a normal slot (in theory) but that seems to be a theory that hasn't been tested (or you just tested it for us).
Yes, I have been told that exact same thing, however that doesn't seem to be the case. The cards/adapters definitely don't work in a normal PCIe 3.0 slot. The pin-outs are definitely different, although HP doesn't give/post any type of documentation as to how they wired the pinouts on their Mellanox cards, or what the pinouts actually are/might be.

Nobody at HP even seems to know, and everyone I spoke with (from HP) said that they are in fact normal PCIe 3.0 x8 adapters (that will in fact work in a normal PCIe 3.0 x8 slot). Which clearly is not the case. Every HP rep that I spoke with was reading EXACTLY what was on their HP website, which wasn't very helpful and then trying to make "educated guesses" based on what was posted on the HP website, but even everyone at HP Sales and HP Technical Sales thought that standard "PCIe 3.0 x8" means "Standard PCIe 3.0 x8" and said that it should work.

Well, I can confirm that it doesn't work. I can also confirm that my Dell systems are now stuck in an endless rebooting loop. With lights 1/3/4 always on (meaning System Board/Hardware failure). So yes, the Dell server mainboards are probably toast.

Although I can't find ANY specific information on the Mellanox/HP 649282-B21 (MCX354A-FCBT) adapter card pinouts, it does seem that the pinouts are in fact different (not standard PCIe 3.0 x8 pinouts), and it does seem that the Dell mainboards (and possibly even the three adapter cards?) are definitely damaged.

It didn't occur to anyone at HP to just use a proprietary Mezzanine card connector/adapter if they are going to create proprietary mezzanine cards/adapters, instead of using a STANDARD PCIe 3.0 connector (that fits into a standard PCIe 3.0 slot) and then attempt to just rewire the pinouts? Which seems to create a LOT of confusion among customers, HP Sales staff, HP Technical Support Staff, and HP Engineers who don't even seem to know what the correct answer even is? (Everyone said that the cards are standard PCIe 3.0 x8 adapters and SHOULD work in a standard PCIe 3.0 x8 slot). One HP Sales/Technical rep told me to use an x16 PCIe 3.0 slot (even though it was only an x8 adapter card) because he said extra buffers were assigned to an x16 slot that would cause the card to perform better. (I'd never head of such a crazy thing, but coming from an HP Sales/Technical rep it's not that surprising).

It was a VERY EXPENSIVE experiment, but yes I can confirm that the adapter cards are definitely different (probably different pin-outs) and they do NOT work in a standard PCIe 3.0 slot, and it does seem that they can/might/will damage your mainboard (and/or possibly even the Mellanox/HP 649282-B21 / MCX354A-FCBT adapter card).

HP seems to believe that the cards will work in a normal PCIe 3.0 x8 slot. It would be extremely helpful if someone (outside of HP) could actually CONFIRM this. Based on what I have tried, the answer is "NO, they do NOT work in a normal PCIe 3.0 x8 slot". Also as far as I can tell "No, they do NOT seem to be wired as a standards-compliant PCIe 3.0 adapter". [If that were the case, then yes they probably would have worked in a normal PCIe 3.0 slot]

It would be nice if HP would at least post the hardware specifications (with exact pinouts) of the adapter cards. So people could at least check/read/see firsthand without having to rely on foolish/ignorant HP Sales Reps (that are only GUESSING based on what they are reading on their own HP Sales Website). HP really didn't do much to educate their Sales Reps, and if that's all they can do is read off of a computer screen (and tell me that they believe that they are in fact STANDARD PCIe x8 Mellanox Adapters and WILL work in a standard PCIe 3.0 slot). That doesn't score many points with "good HP sales reps". The EIGHT HP Sales Reps that I spoke with, all believed that the cards were in fact standard PCIe 3.0 x8 adapters and that they would/should in fact work in a standard PCIe 3.0 x8 slot.

Nobody at HP (not even on the technical side) seems to even know the pinouts (even after I contacted them and told them about the situation). They seemed shocked by it, and they don't believe that the pinouts are any different on their FlexLOM/non-flexLOM adapters, and two people insisted that they are wired exactly the same as a standard PCIe 3.0 x8 slot, so there shouldn't be any problems.

NOBODY can even come up with documentation showing the pinouts, and nobody at HP is even sure of what the pinouts even are. I'll probably have to contact Mellanox, because HP Technical Sales Reps seem fairly useless, and they don't seem to have any resources available to them (technical specifications/documentation) and I did a search on Mellanox's website and I can't seem to find anything on Mellanox website, it keeps referring me to HP Sales for documentation (because apparently it's an HP part number, not a Mellanox part number). It was built (by Mellanox) for HP.

I was hoping that maybe it was just non-standard HP firmware, but it seems now (after trying it) that it's probably different pinouts/wiring, but nobody at HP seems to even know/understand that. There doesn't seem to be ANY technical documentation on HP's website (or available to HP Sales/Technical Reps) so nobody can even send me a technical manual or ANYTHING on the adapters, showing the pinout wiring, or the FlexLOM specification.

I've asked FOURTEEN different times (since Friday) on HP's exact pinout specification for "FlexLOM" and nobody at HP seems to even know, or even have an answer. The simple answer is "HP doesn't KNOW". Their sales reps don't know, their Technical Sales reps don't know, and I don't think any of their HP engineers even know (since there doesn't seem to be any HP documentation written on/about it).

It would be extremely helpful if someone at HP could dig up a copy of an HP/Mellanox technical manual that showed the exact pinouts of the Mellanox/HP adapter, and the pinouts for the "FlexLOM" specification. Nobody at HP can seem to find anything on the Mellanox/HP 649282-B21 / MCX354A-FCBT adapter card pinouts or the FlexLOM specification.

well it is thought that it might run as pci-e x4 in a pci-e x8 3.0 slot - but I doubt it would perform that well in pci-e 2.0 at all
That's what every single HP Sales and HP Technical Sales rep has already told me so far. That it should/will run in a normal PCIe 3.0 x8 slot (possibly in an x8 or x4 mode).

Just remember there are some really good hp sales rep's on this forum.
I'm not doubting that, I'm sure there probably are some great HP Sales Reps out there, but UNFORTUNATELY I can probably read the HP website just as well as they can... ;-)

well it is thought that it might run as pci-e x4 in a pci-e x8 3.0 slot - but I doubt it would perform that well in pci-e 2.0 at all
That's what every single HP Sales and HP Technical Sales rep has already told me thus far. That it should/will run in a normal PCIe 3.0 x8 slot (possibly in an x8 or x4 mode).

It doesn't really help when HP's own website even says this:

PCI express compatibility - Compatible with x8 and x16 PCI Express slots
- When installed in a x8 or x16 slot, the adapter will operate at the x4 speed.


It would be extremely helpful if HP could could actually post the link to an HP/Mellanox Technical Manual that shows the exact wiring/pinouts of the Mellanox/HP 649282-B21 / MCX354A-FCBT adapter card pinouts and/or the FlexLOM specification. So that consumers can actually make an "informed" decision, based on a technical manual, instead of listening to HP Sales/Technical Reps (who don't seem to even know the exact pinouts of either the adapter or the FlexLOM specification).

PCIe Standard Pinouts can be found here: PCI Express - Wikipedia, the free encyclopedia

But HP claims that their cards are in fact STANDARD PCIe 3.0 x8 Adapters, but I've been unable to get a single Sales Rep or Technical Sales Rep (or even an engineer) to send me a document showing the exact pinouts of the Mellanox/HP 649282-B21 / MCX354A-FCBT adapter card and/or the FlexLOM specification.

HP should at least publish their pinouts in a technical paper (if they are in fact different than a standard PCIe 3.0 connector). HP should make that paper available to ALL HP Sales staff and HP Technical staff and HP Engineers (as well as HP Customers) so that we can actually see whether the pinouts are in fact the same (or not) and what damage the adapter card might/will cause to the motherboard.

Based on HP Sales and Technical Sale's "guessing" they've been wrong 22 out of 22 times so far.

It would be helpful if they could at least post some WRITTEN technical documents (if the HP/Mellanox adapters are in fact NOT PCIe 3.0 x8 standards compliant).

Sorry then for incorrect post.

I just found these FlexLOM slot are the PCIe 3.0 x8 or x16, But as i see, so i found a little informations.
Clearly since I'm not the ORIGINAL OP of this thread, and only stumbled upon this thread by ACCIDENT, that others must be quite confused by the "FlexLOM" PCIe 3.0 x8 standard as well.

I've spoke with 22 HP Reps during the past two weeks, and nobody is quite sure whether the adapters will (or will not) work, their original guess was that the adapters WOULD in fact work, but now after I have tried it, they are not quite sure and nobody at HP seems to know/understand why (is it a firmware issue? Can the cards be reflashed to work with a normal PCIe slot? Is it a mechanical pinout issue?)

It certainly doesn't help when HP's own website even says this:

PCI express compatibility - Compatible with x8 and x16 PCI Express slots
- When installed in a x8 or x16 slot, the adapter will operate at the x4 speed.


I still can't get anyone at HP to even come up with an answer, or provide any type of documentation on the technical specifications of the cards (pinouts, etc.) so that this can in fact be confirmed. HP Sales Reps seem to just be guessing and grabbing at straws and they will say ANYTHING to make a sale, but I'm just asking for WRITTEN technical documentation so that we can figure out whether these adapters are in fact PCIe 3.0 x8 standards-compliant, or whether they are not PCIe 3.0 x8 (standards compliant) adapters, and whether HP somehow modified the pinouts of the Mellanox/HP 649282-B21 / MCX354A-FCBT adapter cards (without publishing the technical details or the pinouts of their new "FlexLOM" standard). Which seems to be what is confusing people.

Apparently it's confusing enough that the OP published this thread (that I even happened to stumble upon while doing a search on the adapter card and PCIe compatibility).

Hopefully HP will come forward and post the technical specifications/details of their Mellanox/HP 649282-B21 / MCX354A-FCBT adapter cards and the pinouts on their "FlexLOM" PCIe 3.0 standard.

they are supposed to function as x4 nic's in a normal slot (in theory) but that seems to be a theory that hasn't been tested (or you just tested it for us).
Yep, as an HP Customer, I just love being HP's guinea pig. ;-)

Hopefully HP will come forward and post the written technical specifications/details of their Mellanox/HP 649282-B21 / MCX354A-FCBT adapter cards and the pinouts on their "FlexLOM" PCIe 3.0 standard, so we can actually figure out the WHO/HOW/WHAT/WHY as to why it didn't work. I'm guessing that HP probably rewired the PCIe pinouts, but it's still unclear (without any documentation) what those exact pinouts are (at this point). Nobody at HP seems to know, and hopefully someone can post the technical details/specifications/wiring schematics/diagrams or at least pinout details of the Mellanox/HP 649282-B21 / MCX354A-FCBT adapter cards and the HP "FlexLOM" PCIe 3.0 standard.

If it is a "standard" then there should be technical documents outlining that standard, and telling customers/engineers exactly what that standard is. HP should also choose a different adapter connector form factor instead of using a standard PCIe connector (and re-wiring the pins) to avoid future confusion (among Customers, Sales Reps and Technical Sales Reps).

Hopefully the cards did not do any damage to your motherboards, they are supposed to function as x4 nic's in a normal slot (in theory)
That's exactly what I was told by every HP Sales Reps that I spoke with.

but that seems to be a theory that hasn't been tested (or you just tested it for us).
That's NOT exactly what I was hoping to hear.

The PCIe pinout pitch and pinout spacing on the HP/Mellanox adapter cards seems to be identical. There doesn't seem to be any way for an HP Sales Rep (or Technical Sales Rep) or even an HP customer to visually tell (if HP did in fact re-wire the pins). Without having some WRITTEN HP Technical documentation explaining the detailed pinout diagrams for the adapter and technical documentation on the "FlexLOM" PCIe 3.0 "standard" pinout documentation.

Here: https://www.dropbox.com/s/jg4ht5xu5s2uwce/P1070431.jpg

Here: https://www.dropbox.com/sc/9ozamf8ta8z67d1/5qLRGf3YaS

Here: https://www.dropbox.com/sc/aqxkyzxrc7ev1ul/7h04CEI3z3

but that seems to be a theory that hasn't been tested (or you just tested it for us).
Well, it was a VERY EXPENSIVE test. I'd honestly prefer if HP was actually testing/publishing written technical documents (with technical hardware pinout specifications), instead of relying on their HP customers to scratch their heads and be the guinea pigs. :(
 

TD_Trader

Member
Feb 26, 2013
60
7
8
Well I would have told you they don't work :) - Just remember there are some really good hp sales rep's on this forum ;)
Well I wish you would have... :(

well it is thought that it might run as pci-e x4 in a pci-e x8 3.0 slot - but I doubt it would perform that well in pci-e 2.0 at all
What exactly is your definition of "thought"? Either they do (or DO NOT) work. I've been hearing everyone at HP saying the same thing, and that the cards DO (or "should") in fact work in a normal PCIe 3.0 slot, and that the cards are in fact PCIe 3.0 compliant and WILL work in standard PCIe 3.0 slot, but has this been tested by HP? Can HP at least post some technical documentation on the actual adapter card (including pinouts), as well as technical specifications on the "FlexLOM" pinout specification, so we (as customers) can determine whether or not this will in fact work (without destroying hardware or destroying our mainboards).

Is this a firmware issue? A physical hardware issue?

Hopefully the cards did not do any damage to your motherboards, they are supposed to function as x4 nic's in a normal slot (in theory) but that seems to be a theory that hasn't been tested (or you just tested it for us).
Yep, unfortunately it was just a "theory" but until I can get some written documentation from HP, it's very difficult to figure out exactly what has happened (without knowing the actual physical pinouts).

I'm guessing HP did something (either by re-wiring the pinouts, or some type of proprietary firmware? Is the HP/Mellanox firmware altering the way the card behaves?)

Is there a way to reflash these adapters with standard Mellanox firmware and will this solve the problems? (Or are these adapters PHYSICALLY hard-wired differently?) Is it a firmware issue/problem, or a mechanical/electrical issue?

I'd love to see some technical documentation from HP, so we can at least better understand what these Adapter cards are doing and how/what/why they are different, so we can at least understand what this "FlexLOM" is, and at least know (once and for all) whether these adapter cards are in fact PCIe 3.0 x8 compliant (or NOT).

Prior to me conducting this "test", there doesn't seem to be a whole lot of documentation, only speculation (by HP Sales Reps and HP Technical Sales Reps) that they believe that the adapter cards are in fact standard PCIe 3.0 adapter cards (even though nobody at HP can actually confirm that they have in fact actually stuck one into a machine to test/try it).

Well I would have told you they don't work :)
Well, of course you would have... ;-)

(After all the smoke in the room had cleared...)

Everytime I listen to an HP Sales Rep or HP Technical Rep, I just cross my fingers (and say a prayer) and HOPE that they are correct, but more often times than not, they seem to be WRONG.

Listening to an HP Sales Rep (or HP Technical Sales Rep) is like gambling. I just wish that HP had MUCH better documentation, and MUCH better training for their Sales Reps.

Or at least publish the technical documentation so that engineers can at least figure this stuff out without having to burn up/destroy equipment/hardware (by listening to misinformed HP Sales Reps).
 

TD_Trader

Member
Feb 26, 2013
60
7
8
It may fit in a pci-e slot but are the pin-outs even the same? Even the pin pitch could be different. (HP has a long history of proprietary bullshit) Line it up next to a real pci-e card and take a good look.
Yes, I posted pictures. The pins all seem to be located in the exact same spot/position/pitch, and the adapter card seems to have the exact same number of pins.

Eleven pins, and then an additional 38 pins (for a total of 49 pins). The adapter card even labels the pins B1 to B49.

The pin-pitch and spacing seems to match standard PCIe 3.0 spacing (EXACTLY) and I have lined a standard Mellanox adapter card (pins) up against the Mellanox/HP adapter pins, and the pins look identical.

I can't seem to find any WRITTEN documentation on HP's pinouts and whether they altered the physical pinouts, or what HP's "FlexLOM" pinout specification is. Nobody at HP seems to know. I've called and spoke with over 22+ HP Sales and HP Technical Sales Reps, but nobody over at HP seems to know the pinouts of the Mellanox/HP 649282-B21 adapter card. Everyone at HP that I spoke with seems to believe that it is a standard Mellanox Adapter card that is just "re-branded" with an HP part number. But from what I have seen/done/tested, that doesn't seem to be the case.

Either the hardware pins are in fact re-wired, or maybe there is some form of custom HP firmware on the adapter cards, I'm not exactly sure, but I've been unable to get/make the adapters work in a standard Dell Server.

Just because it doesn't short out your motherboard doesn't mean its working, there are a lot of ground traces that don't care if a data line is in the wrong spot ;)
Well, the Dell servers are continuously rebooting now. I'm also getting a 1/3/4 light error on the Dell Servers, which indicates a mainboard problem. (System mainboard failure).

So it seems that there is a good chance that these adapter cards are probably re-wired somehow, and/or do not comply with the PCIe 3.0 x8 standards.
Here: PCI Express - Wikipedia, the free encyclopedia

A Pinout diagram is VITAL in determining whether something will (or will NOT) work, and it just seems ODD that nobody at HP even has a Technical wiring/pinout diagram for the Mellanox/HP 649282-B21 adapter card.

The truth is, I don't think ANYONE at HP even knows what the pinout/wiring of the Mellanox/HP 649282-B21 adapter card even is. Based on every single person that I spoke with at HP, nobody seems to know. They keep telling me that it's just a standard PCIe 3.0 x8 interface and that the pinouts are exactly the same. (Which I don't believe to be the case, unless the first three adapters that I tried just happen to all be "bad").

At this point, after having motherboard failure/problems, I'm begging to believe that the hardware pinouts are in fact different, even though I can't get anyone at HP to confirm this, or give me written documentation/specifications on the Mellanox/HP 649282-B21 adapter card or even on the HP "FlexLOM" pinout specification.

Does the card even light up? If it does it should show up in tools like lspci at the very least. The cable being plugged in is irrelevant, you should be able to detect the card and install drivers regardless.
Nope, the card doesn't light up. Nothing. The card(s) are not being detected (by either Windows Server 2012 or ESXi 5.5.0) and the adapter cards do not show up in device manager (Windows 2012) or via an lspci (ESXi 5.5.0), so I'm guessing at this point, that the pinouts may/are probably different.

I'm just waiting to hear back from HP, as to what exactly the pinouts are (because NOBODY at HP seems to know at this point) and HP Sales and HP Technical Sales have absolutely no clue at this point.

Once HP provides some technical documentation (or posts it to their website) and begins to make this "public knowledge" then at least we'll know one way or another what the pinouts are and what damage was done, and whether these adapters can be used in a standard PCIe 3.0 x8 slot.

My guess at this point is probably not. Unless the pins are in fact wired exactly the same and it's just a HP custom firmware issue, but if that were the case, then I don't think I would be having the motherboard problems that I'm currently having. So I'm just waiting on a response from HP and hoping that someone at HP will provide me with the technical documentation on the Mellanox/HP 649282-B21 adapter card or even on the HP "FlexLOM" pinout specification.

Without the pinout diagrams from Mellanox/HP, its very difficult to say what exactly has happened and how bad the motherboard/mainboard damage may be.
 

TD_Trader

Member
Feb 26, 2013
60
7
8
It may fit in a pci-e slot but are the pin-outs even the same? Even the pin pitch could be different. (HP has a long history of proprietary bullshit) Line it up next to a real pci-e card and take a good look.

Just because it doesn't short out your motherboard doesn't mean its working, there are a lot of ground traces that don't care if a data line is in the wrong spot ;)
The PCIe pinout pitch and pinout spacing on the HP/Mellanox adapter cards seems to be identical. There doesn't seem to be any way for an HP Sales Rep (or Technical Sales Rep) or even an HP customer to visually tell (if HP did in fact re-wire the pinouts). Without having some HP WRITTEN Technical documentation explaining the detailed pinout diagrams for the Mellanox/HP 649282-B21 adapter and technical documentation on the "FlexLOM" PCIe 3.0 "standard" pinout documentation.

Here: https://www.dropbox.com/s/jg4ht5xu5s2uwce/P1070431.jpg

Here: https://www.dropbox.com/sc/9ozamf8ta8z67d1/5qLRGf3YaS

Here: https://www.dropbox.com/sc/aqxkyzxrc7ev1ul/7h04CEI3z3

Does the card even light up? If it does it should show up in tools like lspci at the very least. The cable being plugged in is irrelevant, you should be able to detect the card and install drivers regardless.
No, the adapter card does not light up. Nothing. It's not detected by the OS, and none of the first three adapter cards that I have tried/tested seem to work.

Either the first three HP/Mellanox adapters that I tried/tested are all bad (which I highly doubt) or the adapter card probably has re-wired pins and is not following the standard PCI Express (PCIe) connector pinout configuration.

Or there may be some custom/proprietary HP firmware on the adapter that is preventing it from functioning correctly in a standard/normal PCIe slot. It's hard/difficult to tell at this point, and NOBODY at HP seems to have any knowledge or answers. Everytime I call, their HP Sales and HP Technical Sales seem to just read whatever is written on their web page (which isn't much) and nobody seems to have any knowledge or answers at HP.

At this point, I guess I'll have to wait till someone at HP responds with a technical pinout diagram (or Technical hardware manual) on the Mellanox/HP 649282-B21 adapter and technical documentation on the "FlexLOM" PCIe 3.0 "standard" pinout documentation.

I sent an email to both Garth Fruge and Ken Bekampis (from HP/Mellanox) and hopefully they will be able to come up with some written technical documentation (with exact pinout specifications). So I can at least see what is going on, or what has happened.

After trying/testing, my guess at this point is that it's either a physical pinout issue or possible firmware issue (or a combination of the two).

If I can get WRITTEN technical documentation (from HP/Mellanox) confirming that the hardware pinouts are exactly the same (as standard PCIe specification), then it may just be a custom HP/Mellanox firmware issue. If I can get WRITTEN technical documentation (from HP/Mellanox) that the pinouts are in fact different, then it may be a physical hardware problem (and at least I can figure out how/what damage has probably been done to the physical hardware/mainboard).

As it stands, I just haven't been able to get much help from HP. HP's Sales Reps seem to be pretty clueless, and HP's Technical Sales Reps seem pretty clueless as well.

You'd think that if HP actually created/sold a piece of hardware, that they'd actually have technical documentation on that hardware (which does NOT seem to be the case).
 

mrkrad

Well-Known Member
Oct 13, 2012
1,244
52
48
You ever wonder if HP doesn't know? They sell everyone's NIC cards, every manufacturer there is, so it may be a sticker repackaging deal with something odd mellanox has done to the card.

Sorry to hear of your ordeal, but it is atypical to stuff specific cards from a gen8 HP into a DELL :). This is somewhere you would have to reach deep into engineering tier of support to find out why, and that is not going to be easily worked out without a serious contract!