Mezzanine and Backplane Connectors

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

b778

New Member
May 19, 2016
3
0
1
63
Hi all
I'm new to the site, so forgive me if I have missed finding previous discussion on this already.
I have a couple of HP BL460c (Gen1, I think) that I want to experiment with and I would like to work (in)directly with the mezzanine connectors and backplane port. So far all of my searches have come up empty.
Does anyone have information on pinouts or can you point me in the right direction? Alternatively, do you have thoughts on mezzanine to PCIe?
Thank you.
Bernard
 

Attachments

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
I guess the search is somewhat fruitless since the chassis is not really very expensive and those blades need shared power and cooling so really need to run in the chassis.

The first bl460c was G5 after that we have G6, G7 (using Xeon 56xx, this is the earliest I would waste any time on), then Gen8 and Gen9 which are e5 v1/v2 and v3/v4 respective)
 

b778

New Member
May 19, 2016
3
0
1
63
Thank you for your warm welcome and support, and a big hello from Australia.
A little back story - this came about as I was searching for parts for my very first project, which is a NAS. I am using an old IBM x346 which I am in the process of converting to SATA, when I came across these for $12 and $16 - so I thought, why not? Even if I can't use them the memory on board will return my investment. Cheap hardware seems hard to come by here and if we see it overseas, shipping almost always makes it inaccessible.
They fire up with just 12V and a local I/O SUV Diagnostic Cable gives me video out and USB access. Each has 2 Quad Core Xeon 5430 and enough HDD and RAM to run Linux. So that started me thinking about putting a switch between them to make a 16 core high performance cluster. From what I have read so far, the rear Molex connector is basically a bunch of Ethernet connections and possibly PCI (more reading needed) but at the moment, the pinout is indecipherable to me. On the other hand, I read that the mezzanine is a form of PCI, which opens up the possibility of cheap 3rd party cards with common connectors, but at added cost of the conversion from the mezzanine connector to PCIe as well as the third party card.
The only chassis I have seen so far was a three-hour drive away but I will keep looking and in the meantime see what more I can find. I've no specific purpose for this project at the moment, other than the challenge and learning though doing.
Once again, thanks for your support and for making me feel welcome.
 

s2600gl

Member
Apr 1, 2016
33
11
8
50
Finland
Don't know if you have stumbled upon this document in your searches:

http://pdfstream.manualsonline.com/d/daf0afed-5379-47a6-bb05-4dfabb13f4b3.pdf

It's the technology brief of the C3000 enclosure. It won't give you the pin out you're looking for, but it explains (briefly) the inner workings of these blades when they are paired together with the enclosure. The interesting stuff starts on page 13.

Seems you were quite right about the Ethernet / PCI stuff in the Molex connector. The midplane these blades connect to in the enclosure acts like a giant PCIe bus interconnecting the blades with each other and other modules (network / storage). But instead of only supporting memory-semantic protocols (PCIe, memory addresses in packet headers) this bus also supports network-semantic protocols (Ethernet/FC/Infiniband, network addresses in the packet headers). The interconnect modules dictates how you can mix and mash all together.

They're looking like mighty interesting beasts those C3000 / C7000. And it kind of tells why the blades tend to be dirt-cheap on their own (they really, really, really want to be paired with the enclosure and the interconnect modules).

I wish you luck on your journey, it's going to be a steep uphill climb. The information will be scarce (unless you have good friends at HP).
I have the utmost respect for people who take on the task to reverse-engineer and hack hardware, especially when it's done solely to educate oneself / others.
 

b778

New Member
May 19, 2016
3
0
1
63
Thanks very much for your kind words & encouragement!
I hadn't seen that and it is a big help.
I will keep you posted on my progress.
 

tim.yoshi

Member
Jun 25, 2017
40
0
6
36
Kiev, Ukraine
Sorry to resurrect old thread but have anybody had any progress on that? I'm in the kinda same boat. Want dirt cheap standalone server converted from old HP blade. So far I learned that it could be easily started with just 12V power and HP SUV cable which could provide VGA and USB. Also one could use usb network cards easily available everywhere. So basically this thing could easily be adapted to use as a regular PC. But the trick is to make iLO and embedded NIC to work also. As per my understanding one just have to know exact pinout of rear end of this server to make it all work as expected... So any info?
 

s2600gl

Member
Apr 1, 2016
33
11
8
50
Finland
Judging by the lack of updates I'd say that @b778 either a) took the three-hour drive to pick up the chassis or b) parted the blade
and headed on towards new adventures. My money is on option b, but I've been wrong before...

As for your question, if you're happy with running it as a "regular PC" it's going to be dirt cheap. But if you want to take use of the
mezzanine features of the blade (nic/fc/ib) the labor and expenses are going to start to build up. The midplane in these chassis (c7000 / c3000)
is a passive one (i.e. just a bunch of traces on a PCB) so you could theoretically just use wire instead.

But you'd still need the interconnect modules for the blades to be able to access "the outside world". And most probably also the administrator module to get all the settings done. By the time you have everything glued, taped and hacked together you probably wish that you would have picked up that chassis in the first place...

As @Evan said, those chassis aren't too pricey anyway and if you're lucky they can even be dirt cheap. Expect to pay a little premium for the
c3000, as they are smaller, quieter and hogs less power than the c7000. The c7000 OTOH is a 500lb monster and devours anything that you
throw at it so you might even find someone that will gladly give one to you for free.

But there's a caveat even if you manage to get the chassis and the blades for free. They are not going to sip power even at idle, they are
going to gobble it !. I had one of my chassis :)rolleyes:) populated with a single blade and just two psus and that thing drew 250ish watts just
sitting idle doing nothing (not even an OS running)... Granted, that was a FH Gen 1 blade, and I had all fans running.

Sorry for the long rant that probably didn't help you one bit (other than perhaps persuade you to get the chassis). Don't get me wrong,
these things are fun to tinker with and power up now and then (when the juice is cheap), but I would never consider running them 24/7.
 
  • Like
Reactions: Sleyk and tim.yoshi

tim.yoshi

Member
Jun 25, 2017
40
0
6
36
Kiev, Ukraine
@s2600gl thanks.
I'd be happy to pick up one C7000. I saw one on eBay as low as 38 bucks for the full enclosure. Without PSUs however, but with FANs and all other hardware that should be there. But here is the problem - I'm not in US. Shipping to my country this, as you say 500lb monster, could cost as an average old small sedan cost in US, so... no way at all :) I tried to find locally this thing, but no luck - only saw one or two complete machines and no parts at all.
Noise and power draw is not a problem for me.
So basically I'd need one midplate from either C7000 or C3000 to make total or partial pinout. I guess it will follow one simple logic, like pin A1 goes to pin A1 into interconnect BAY 1, Pin A2 goes to A1 into BAY2 and so on. I expect it to be easily interpolated into other bays and pins once initial tests with few dozen pins will be done. Firstly I thought that it could be as easy as insert bare copper wires into blades main connector appropriate socket pins and on the other end to crimp it with regular RJ45 connector to connect it's internal NIC to the outer world. Now, after reading all the data available in the internet I doubt it will be so easy. Likely one will need at least Ethernet switch module to allow server use it's internal NICs. All because I believe they are using something like dynamic pins switching, or something like that on electrical signal level. At least they want it to look like this from their brochures and documents. However counting number of pins in the main blade connector and figuring out possible expected combinations quite well correlates. So I still now expect that there is *maybe* a chance to stick bare copper into main connector :)
Anyways I'm already up for that adventure. One bare motherboard and SUV cable are on their way, right now found very cheap midplane board perfectly striped to the bare minimum - only plain PCB. So no need to pay extra for the shipping. Made a bid and hope it will be successful. The one thing that worries me is iLO. This thing could not accept working without fans signals and with absence of administration panel and control board. While they are not that expensive I doubt it will be economically viable to buy all that equipment for a very single board. However if one will make a cluster of few ghetto blades in da shitty housing of wood with one administrator control board and interconnect modules that could be quite cheap setup... Ok, only time will tell looks like.
 

s2600gl

Member
Apr 1, 2016
33
11
8
50
Finland
Aaah, those US prices...:(, no such luck at this side of the pond. But hang in there @tim.yoshi, you'll never know when one of those might just
fall into your lap. I was on-off hunting for a c7000 chassis for at least 2 years but had the same problem as you, shipping it would have
doubled it's price (on a good day). So I had pretty much abandoned the idea until earlier this year when I found one less than a hours drive from home.

Long story short, it turned out that the guy actually had 3 of these lying around and he made me an offer I just couldn't pass...
To add insult to injury, it wasn't even a week after this I got a message from a friend that a local company was getting rid of their old gear and
was literally giving it away. And I bet you can never guess what the gear was - yup, a c7000 chassis with ~ half a dozen blades :rolleyes:.

After a quick inventory, it turned out that one of the blades had 12 x 16Gb DDR3 modules in it => WIN :).

Hope it pans out for you with the midplane, I'm really curious to see if someone succeeds in making a "ghetto-chassis". If you want, I
could pull out everything but a blade, a network interconnect and a psu in one of my chassis to try and emulate the situation you would be in.
Just to see what errors that would generate and would there be a way to circumvent them.
 
  • Like
Reactions: tim.yoshi

tim.yoshi

Member
Jun 25, 2017
40
0
6
36
Kiev, Ukraine
@s2600gl wow! That is awesome proposal! Thanks, would be glade about that "virtual" experiment. I think it should be easily to emulate it's original fans tacho signal from just ordinary regular PC-like fans, and split this one signal to all the input tacho pins. Board should be fine with it and should not see the forgery. I know coz I successfully did this on another Dell server :) However administrator board, administrator insight display and iLO is what really bothers me... I would be very thankful if you could find a minimal boot configuration... I mean really minimal :)

Was lucky enough yesterday to win eBay auction for this bare midplane with dirt cheap price. Now I likely should find and buy appropriate interconnect module. By the way, could you advise? I'm not that familiar with all that blade topology at all. What interconnect module will I need to connect a blade to basic regular 100 Mbit ethernet router. I do not have any sophisticated network infrastructure. However I'd like to have both of embedded NICs active like a regular + reserve fail-safe channel scheme. Also I hope it is possible to set in BIOS to iLO "shared" connection with main NIC? So that I will could use only two Ethernet cables to connect blade to internet and to be able to connect into blade itself.

Very specific hardware. Very low odds I will find it locally with a reasonable price tag, however I set few alarms for the local searches...
 

s2600gl

Member
Apr 1, 2016
33
11
8
50
Finland
I think that the fan signal emulation might be a little trickier than that. The OA module is controlling both power, thermal and interconnect modules through a I2C bus. So to fake the presence of the fan you would need to tap into the I2C bus and generate the replies the OA is expecting.
I found another document describing the innards of the c7000 chassis:

https%3A%2F%2Fcommunity.hpe.com%2Fhpeb%2Fattachments%2Fhpeb%2Fbladescategory04%2F4014%2F1%2FTechnologies%2520in%2520the%2520HP%2520BladeSystem%2520c7000%2520Enclosure.pdf

OTOH it seems that there's no I2c communication between the blades and the OA, and since you are already bypassing the psus they might
not care that you have no fans. So the question might be, will the interconnect modules work without the OA ?
That same picture shows that the iLO is separately routed so you could probably not share it over the NICs.

As for interconnect modules, the HP 1GB Ethernet Pass-through module would be enough. Mind you, though, that module is a fixed speed
1Gbit meaning it cannot auto negotiate to 100Mbit if that is the speed of your network. You would need something in-between to sort that out.

Here's another page with tons of information about the OA, might or might not be useful:

HP BladeSystem Onboard Administrator User Guide
 
  • Like
Reactions: tim.yoshi

tim.yoshi

Member
Jun 25, 2017
40
0
6
36
Kiev, Ukraine
I think that the fan signal emulation might be a little trickier than that. The OA module is controlling both power, thermal and interconnect modules through a I2C bus. So to fake the presence of the fan you would need to tap into the I2C bus and generate the replies the OA is expecting.
Wow, that came unexpectedly. Will look more on it.

The link was somehow broken, but I found a way to extrract this document. Thanks, will be reading it.


OTOH it seems that there's no I2c communication between the blades and the OA, and since you are already bypassing the psus they might not care that you have no fans. So the question might be, will the interconnect modules work without the OA ?
Well, it all is just my assumptions based on what I've seen on the internet. My blade is in the mail yet somewhere beyond the sea. But what I noticed - they all disable onboard iLO with jumpers config in order to make it boot without warnings. So yeah, that's the question how enabled iLO will look at absent fans, PSU's, enclosure, "sea of sensors" and so on. Gotta pleasure it with some faked signals likely :)




As for interconnect modules, the HP 1GB Ethernet Pass-through module would be enough. Mind you, though, that module is a fixed speed
1Gbit meaning it cannot auto negotiate to 100Mbit if that is the speed of your network. You would need something in-between to sort that out.
This "something" should be something like a gigabit switch or router, don't it? I mean ordinary SOHO 100MBit router will not work it out? Also will HP Cisco blade switches do the job also? Like this one: HP Cisco Catalyst | 410916-B21 | WS-CBS3020-HPQ | 432904-001 | 882780367401 | eBay they are cheaper overall and to my understanding could provide basic (and even not basic) switching and all negotiating functions. Another words if everything will go out smooth with the blades I will be able to connect few blades with this one switch to the internet with just one 100MBit RJ45 cable, right?

That same picture shows that the iLO is separately routed so you could probably not share it over the NICs.
Not that good, but OTOH it could be easier to "extract" bare copper pinout for "bare copper" tap-in to iLO from onboard connector to real world. But what worries me is that they are using, as they say in documentation "Serializer/Deserializer" on every physical signal connection to the blade. Reading further about it (SerDes - Wikipedia) I realize that this is a tricky part - if that is true to my current understanding, one could in no way tap into connector. One would need properly configured deserializer part to work properly with signals. So basically what comes from it's mighty connector is not a few direct Ethernet lines, few direct PCI-e lines, etc, but a bunch of "SOMETHING encoded" that should be "decoded" on the other end. They are decreasing number of physical lines needed with that. But this deserialiser should be inside the pass-thru or switch interconnect modules for sure. So theoretically even if it will be total no-way with iLO and all that *should be there* signals, at least part of the plan with extracting embedded NIC Ethernet signals from a few blades and routing them into one switch (or pass-thru) module should work. Theoretically. Actually even this interconnect modules could be kinda kinky about powering up without OA. Oh, and one more thing, just to make sure that I understand correctly all the topology. As per my understanding I could have up to all that 16 blades connected to a single switch, but with only ONE NIC. In order to use all TWO embedded NICs I have to use TWO separate switches even if I have one blade. They are just two different networks. Right?


Here's another page with tons of information about the OA, might or might not be useful:
HP BladeSystem Onboard Administrator User Guide
Wow, that's massive. Will be diggin into it also...



Is your proposal about testing minimal bootable config on one of your enclosures still in force?
 

s2600gl

Member
Apr 1, 2016
33
11
8
50
Finland
This "something" should be something like a gigabit switch or router, don't it? I mean ordinary SOHO 100MBit router will not work it out? Also will HP Cisco blade switches do the job also? Like this one: HP Cisco Catalyst | 410916-B21 | WS-CBS3020-HPQ | 432904-001 | 882780367401 | eBay they are cheaper overall and to my understanding could provide basic (and even not basic) switching and all negotiating functions. Another words if everything will go out smooth with the blades I will be able to connect few blades with this one switch to the internet with just one 100MBit RJ45 cable, right?
Yes, that will fit the bill nicely, 1 Gbit towards the backplane and 10/100/1000 to the outside world. Would work nicely with your
100Mbit gear.

Oh, and one more thing, just to make sure that I understand correctly all the topology. As per my understanding I could have up to all that 16 blades connected to a single switch, but with only ONE NIC. In order to use all TWO embedded NICs I have to use TWO separate switches even if I have one blade. They are just two different networks. Right?
Correct, if you want to use both NICs you will need two pass-through modules. It has possibly to do with redundancy, loosing one module
would not take the entire network down.

Is your proposal about testing minimal bootable config on one of your enclosures still in force?
Sure thing :). This week has been really hectic, but I'll try to make time for a little "labbing" next week ;)
 
  • Like
Reactions: tim.yoshi

tim.yoshi

Member
Jun 25, 2017
40
0
6
36
Kiev, Ukraine
Yes, that will fit the bill nicely, 1 Gbit towards the backplane and 10/100/1000 to the outside world. Would work nicely with your
100Mbit gear.
Correct, if you want to use both NICs you will need two pass-through modules. It has possibly to do with redundancy, loosing one module
would not take the entire network down.
Awesome! Thanks. Ordered one Cisco switch to play with.

Sure thing :). This week has been really hectic, but I'll try to make time for a little "labbing" next week ;)
Eagerly waiting :) Especially interested if it will boot without onboard administrator module. As far as I see it now, it should be the root of all the evill :) I expect blade's onboard iLO to be functional as it likely acts as a one side communication from in-blade iLO to in-enclosure administrator module. So hence it is possble that in-blade iLO even don't know at all nothing about fans, PSUs, and other external hardware... If this is true - this is perfect :)
 

voxadam

Member
Apr 21, 2016
107
14
18
Portland, Oregon
I had a thought, there's at least one mezzanine card that's largely passive, the MXM mezzanine adapter board (HP p/n 583496-001). Since MXM is an open standard the pinout is widely available.[1] So, with the aid of a multimeter, a decent pair of probes, and a substantial amount of persistence it should be fairly simple to determine the mapping from the mezzanine to the MXM slot and thus the PCIe signals.

An MXM card has a lot more pins than a PCIe card because it has to accommodate signals for VGA, HDMI, DisplayPort, LVDS, TV, S/PDIF, and a few other things. This means that a lot of the pins on the MXM connector will likely be pulled low, pulled high, unterminated, or tied to ground (likely everything past about pin 157).

Anyway, I just thought I'd share my idea in hopes someone might find it useful.

One last question, what't your goal with this project b778? While looking for a high resolution photo of the HP's mezzanine connector I ran across a two year old forum post and realized you've been working on this to some degree for quiet some time.

[1] MXM Version 2.1A Graphics Module Thermal Electromechanical Specification, Table 7. Connector Pinout, page 44-50
 

tim.yoshi

Member
Jun 25, 2017
40
0
6
36
Kiev, Ukraine
I had a thought, there's at least one mezzanine card that's largely passive, the MXM mezzanine adapter board (HP p/n 583496-001). Since MXM is an open standard the pinout is widely available.[1] So, with the aid of a multimeter, a decent pair of probes, and a substantial amount of persistence it should be fairly simple to determine the mapping from the mezzanine to the MXM slot and thus the PCIe signals.

An MXM card has a lot more pins than a PCIe card because it has to accommodate signals for VGA, HDMI, DisplayPort, LVDS, TV, S/PDIF, and a few other things. This means that a lot of the pins on the MXM connector will likely be pulled low, pulled high, unterminated, or tied to ground (likely everything past about pin 157).

Anyway, I just thought I'd share my idea in hopes someone might find it useful.

One last question, what't your goal with this project b778? While looking for a high resolution photo of the HP's mezzanine connector I ran across a two year old forum post and realized you've been working on this to some degree for quiet some time.

[1] MXM Version 2.1A Graphics Module Thermal Electromechanical Specification, Table 7. Connector Pinout, page 44-50
thanks for the info on Mezz to MXM - I've definitely think it's possible but the main thing here is to have above-mentioned mezz to MXM plate. One could potentially try to do it even without it, i think it's possible. But The question is how to differentiate rx\tx pair numbers?

And one more think - you are likely confusing me with some other user :)
 

s2600gl

Member
Apr 1, 2016
33
11
8
50
Finland
A few tests and some initial results:

Test 1 - 1 Blade, 1 PSU, 1 Gbe pass-through module - no OA module, no Fans
- psu turns on and supplies 12V to the chassis, psu led green
- blade power led amber -> standby
- no signs of life in the pass-through module or Insight display (was to be expected w/o OA)
- any attempt to power on the blade results in blade health led blinking red -> critical condition and psu shutting down (led going from green to amber -> power supply failure)

Test 2 - 1 Blade, 1 PSU, 1 Gbe pass-through, 4 fans (minimum config) - no OA
- psu turns on - > green led
- blade power led amber -> standby
- all 4 fans run at full blast
- no signs of life in pass-through or Insight
- any attempt to power on blade results in blade health led blinking red -> critical condition
- psu stays on -> led green

Test 3 - 1 Blade, 1 PSU, 1 Gbe pass-through, OA module installed - no fans
- psu turns on - > green led
- blade power led amber -> standby
- Insight display turns on showing fan subsystem failure, wants at least 4 fans installed
- Blade health led blinks red even without trying to power it on - > critical condition

Test 4 - 1 Blade, 1 PSU, 1 Gbe pass-through, OA module and 4 fans installed
- psu turns on - > green led
- blade power led amber -> standby
- Insight display turns on showing all green -> happy with the situation
- Blade powers on (led amber -> green)
- Both link and network activity on the pass-through module

So it looks like you have to bite the bullet and get the OA, unfortunately....
 
  • Like
Reactions: tim.yoshi