Mezzanine and Backplane Connectors

Discussion in 'DIY Server and Workstation Builds' started by b778, May 20, 2016.

  1. b778

    b778 New Member

    Joined:
    May 19, 2016
    Messages:
    3
    Likes Received:
    0
    Hi all
    I'm new to the site, so forgive me if I have missed finding previous discussion on this already.
    I have a couple of HP BL460c (Gen1, I think) that I want to experiment with and I would like to work (in)directly with the mezzanine connectors and backplane port. So far all of my searches have come up empty.
    Does anyone have information on pinouts or can you point me in the right direction? Alternatively, do you have thoughts on mezzanine to PCIe?
    Thank you.
    Bernard
     

    Attached Files:

    #1
  2. Sleyk

    Sleyk Active Member

    Joined:
    Mar 25, 2016
    Messages:
    671
    Likes Received:
    142
    Hmm.. Haven't worked with Mezzanine cards. Will see if I can search for any info for you.

    In the meantime...

    Welcome to STH my friend! Pleasure to have you here. Hope you get all the info you need!
     
    #2
  3. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    2,221
    Likes Received:
    305
    I guess the search is somewhat fruitless since the chassis is not really very expensive and those blades need shared power and cooling so really need to run in the chassis.

    The first bl460c was G5 after that we have G6, G7 (using Xeon 56xx, this is the earliest I would waste any time on), then Gen8 and Gen9 which are e5 v1/v2 and v3/v4 respective)
     
    #3
  4. Sleyk

    Sleyk Active Member

    Joined:
    Mar 25, 2016
    Messages:
    671
    Likes Received:
    142
    Yeah, I was googling for some added info for our friend here, but it appears the Gen1 chassis does seem abit old. I know you wanna work on it and play around with it, so I will keep searching around for anything useful. Will keep you posted if I find anything!
     
    #4
  5. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    2,221
    Likes Received:
    305
    Can use a c3000 if you don't want the 16 node c7000.
     
    #5
  6. b778

    b778 New Member

    Joined:
    May 19, 2016
    Messages:
    3
    Likes Received:
    0
    Thank you for your warm welcome and support, and a big hello from Australia.
    A little back story - this came about as I was searching for parts for my very first project, which is a NAS. I am using an old IBM x346 which I am in the process of converting to SATA, when I came across these for $12 and $16 - so I thought, why not? Even if I can't use them the memory on board will return my investment. Cheap hardware seems hard to come by here and if we see it overseas, shipping almost always makes it inaccessible.
    They fire up with just 12V and a local I/O SUV Diagnostic Cable gives me video out and USB access. Each has 2 Quad Core Xeon 5430 and enough HDD and RAM to run Linux. So that started me thinking about putting a switch between them to make a 16 core high performance cluster. From what I have read so far, the rear Molex connector is basically a bunch of Ethernet connections and possibly PCI (more reading needed) but at the moment, the pinout is indecipherable to me. On the other hand, I read that the mezzanine is a form of PCI, which opens up the possibility of cheap 3rd party cards with common connectors, but at added cost of the conversion from the mezzanine connector to PCIe as well as the third party card.
    The only chassis I have seen so far was a three-hour drive away but I will keep looking and in the meantime see what more I can find. I've no specific purpose for this project at the moment, other than the challenge and learning though doing.
    Once again, thanks for your support and for making me feel welcome.
     
    #6
  7. s2600gl

    s2600gl Member

    Joined:
    Apr 1, 2016
    Messages:
    33
    Likes Received:
    10
    Don't know if you have stumbled upon this document in your searches:

    http://pdfstream.manualsonline.com/d/daf0afed-5379-47a6-bb05-4dfabb13f4b3.pdf

    It's the technology brief of the C3000 enclosure. It won't give you the pin out you're looking for, but it explains (briefly) the inner workings of these blades when they are paired together with the enclosure. The interesting stuff starts on page 13.

    Seems you were quite right about the Ethernet / PCI stuff in the Molex connector. The midplane these blades connect to in the enclosure acts like a giant PCIe bus interconnecting the blades with each other and other modules (network / storage). But instead of only supporting memory-semantic protocols (PCIe, memory addresses in packet headers) this bus also supports network-semantic protocols (Ethernet/FC/Infiniband, network addresses in the packet headers). The interconnect modules dictates how you can mix and mash all together.

    They're looking like mighty interesting beasts those C3000 / C7000. And it kind of tells why the blades tend to be dirt-cheap on their own (they really, really, really want to be paired with the enclosure and the interconnect modules).

    I wish you luck on your journey, it's going to be a steep uphill climb. The information will be scarce (unless you have good friends at HP).
    I have the utmost respect for people who take on the task to reverse-engineer and hack hardware, especially when it's done solely to educate oneself / others.
     
    #7
  8. b778

    b778 New Member

    Joined:
    May 19, 2016
    Messages:
    3
    Likes Received:
    0
    Thanks very much for your kind words & encouragement!
    I hadn't seen that and it is a big help.
    I will keep you posted on my progress.
     
    #8
  9. Sleyk

    Sleyk Active Member

    Joined:
    Mar 25, 2016
    Messages:
    671
    Likes Received:
    142
    Nice find s2600gl! I know it will help our friend out greatly!
     
    #9
  10. tim.yoshi

    tim.yoshi New Member

    Joined:
    Jun 25, 2017
    Messages:
    28
    Likes Received:
    0
    Sorry to resurrect old thread but have anybody had any progress on that? I'm in the kinda same boat. Want dirt cheap standalone server converted from old HP blade. So far I learned that it could be easily started with just 12V power and HP SUV cable which could provide VGA and USB. Also one could use usb network cards easily available everywhere. So basically this thing could easily be adapted to use as a regular PC. But the trick is to make iLO and embedded NIC to work also. As per my understanding one just have to know exact pinout of rear end of this server to make it all work as expected... So any info?
     
    #10
  11. s2600gl

    s2600gl Member

    Joined:
    Apr 1, 2016
    Messages:
    33
    Likes Received:
    10
    Judging by the lack of updates I'd say that @b778 either a) took the three-hour drive to pick up the chassis or b) parted the blade
    and headed on towards new adventures. My money is on option b, but I've been wrong before...

    As for your question, if you're happy with running it as a "regular PC" it's going to be dirt cheap. But if you want to take use of the
    mezzanine features of the blade (nic/fc/ib) the labor and expenses are going to start to build up. The midplane in these chassis (c7000 / c3000)
    is a passive one (i.e. just a bunch of traces on a PCB) so you could theoretically just use wire instead.

    But you'd still need the interconnect modules for the blades to be able to access "the outside world". And most probably also the administrator module to get all the settings done. By the time you have everything glued, taped and hacked together you probably wish that you would have picked up that chassis in the first place...

    As @Evan said, those chassis aren't too pricey anyway and if you're lucky they can even be dirt cheap. Expect to pay a little premium for the
    c3000, as they are smaller, quieter and hogs less power than the c7000. The c7000 OTOH is a 500lb monster and devours anything that you
    throw at it so you might even find someone that will gladly give one to you for free.

    But there's a caveat even if you manage to get the chassis and the blades for free. They are not going to sip power even at idle, they are
    going to gobble it !. I had one of my chassis :)rolleyes:) populated with a single blade and just two psus and that thing drew 250ish watts just
    sitting idle doing nothing (not even an OS running)... Granted, that was a FH Gen 1 blade, and I had all fans running.

    Sorry for the long rant that probably didn't help you one bit (other than perhaps persuade you to get the chassis). Don't get me wrong,
    these things are fun to tinker with and power up now and then (when the juice is cheap), but I would never consider running them 24/7.
     
    #11
    tim.yoshi likes this.
  12. tim.yoshi

    tim.yoshi New Member

    Joined:
    Jun 25, 2017
    Messages:
    28
    Likes Received:
    0
    @s2600gl thanks.
    I'd be happy to pick up one C7000. I saw one on eBay as low as 38 bucks for the full enclosure. Without PSUs however, but with FANs and all other hardware that should be there. But here is the problem - I'm not in US. Shipping to my country this, as you say 500lb monster, could cost as an average old small sedan cost in US, so... no way at all :) I tried to find locally this thing, but no luck - only saw one or two complete machines and no parts at all.
    Noise and power draw is not a problem for me.
    So basically I'd need one midplate from either C7000 or C3000 to make total or partial pinout. I guess it will follow one simple logic, like pin A1 goes to pin A1 into interconnect BAY 1, Pin A2 goes to A1 into BAY2 and so on. I expect it to be easily interpolated into other bays and pins once initial tests with few dozen pins will be done. Firstly I thought that it could be as easy as insert bare copper wires into blades main connector appropriate socket pins and on the other end to crimp it with regular RJ45 connector to connect it's internal NIC to the outer world. Now, after reading all the data available in the internet I doubt it will be so easy. Likely one will need at least Ethernet switch module to allow server use it's internal NICs. All because I believe they are using something like dynamic pins switching, or something like that on electrical signal level. At least they want it to look like this from their brochures and documents. However counting number of pins in the main blade connector and figuring out possible expected combinations quite well correlates. So I still now expect that there is *maybe* a chance to stick bare copper into main connector :)
    Anyways I'm already up for that adventure. One bare motherboard and SUV cable are on their way, right now found very cheap midplane board perfectly striped to the bare minimum - only plain PCB. So no need to pay extra for the shipping. Made a bid and hope it will be successful. The one thing that worries me is iLO. This thing could not accept working without fans signals and with absence of administration panel and control board. While they are not that expensive I doubt it will be economically viable to buy all that equipment for a very single board. However if one will make a cluster of few ghetto blades in da shitty housing of wood with one administrator control board and interconnect modules that could be quite cheap setup... Ok, only time will tell looks like.
     
    #12
  13. s2600gl

    s2600gl Member

    Joined:
    Apr 1, 2016
    Messages:
    33
    Likes Received:
    10
    Aaah, those US prices...:(, no such luck at this side of the pond. But hang in there @tim.yoshi, you'll never know when one of those might just
    fall into your lap. I was on-off hunting for a c7000 chassis for at least 2 years but had the same problem as you, shipping it would have
    doubled it's price (on a good day). So I had pretty much abandoned the idea until earlier this year when I found one less than a hours drive from home.

    Long story short, it turned out that the guy actually had 3 of these lying around and he made me an offer I just couldn't pass...
    To add insult to injury, it wasn't even a week after this I got a message from a friend that a local company was getting rid of their old gear and
    was literally giving it away. And I bet you can never guess what the gear was - yup, a c7000 chassis with ~ half a dozen blades :rolleyes:.

    After a quick inventory, it turned out that one of the blades had 12 x 16Gb DDR3 modules in it => WIN :).

    Hope it pans out for you with the midplane, I'm really curious to see if someone succeeds in making a "ghetto-chassis". If you want, I
    could pull out everything but a blade, a network interconnect and a psu in one of my chassis to try and emulate the situation you would be in.
    Just to see what errors that would generate and would there be a way to circumvent them.
     
    #13
    tim.yoshi likes this.
  14. tim.yoshi

    tim.yoshi New Member

    Joined:
    Jun 25, 2017
    Messages:
    28
    Likes Received:
    0
    @s2600gl wow! That is awesome proposal! Thanks, would be glade about that "virtual" experiment. I think it should be easily to emulate it's original fans tacho signal from just ordinary regular PC-like fans, and split this one signal to all the input tacho pins. Board should be fine with it and should not see the forgery. I know coz I successfully did this on another Dell server :) However administrator board, administrator insight display and iLO is what really bothers me... I would be very thankful if you could find a minimal boot configuration... I mean really minimal :)

    Was lucky enough yesterday to win eBay auction for this bare midplane with dirt cheap price. Now I likely should find and buy appropriate interconnect module. By the way, could you advise? I'm not that familiar with all that blade topology at all. What interconnect module will I need to connect a blade to basic regular 100 Mbit ethernet router. I do not have any sophisticated network infrastructure. However I'd like to have both of embedded NICs active like a regular + reserve fail-safe channel scheme. Also I hope it is possible to set in BIOS to iLO "shared" connection with main NIC? So that I will could use only two Ethernet cables to connect blade to internet and to be able to connect into blade itself.

    Very specific hardware. Very low odds I will find it locally with a reasonable price tag, however I set few alarms for the local searches...
     
    #14
  15. s2600gl

    s2600gl Member

    Joined:
    Apr 1, 2016
    Messages:
    33
    Likes Received:
    10
    I think that the fan signal emulation might be a little trickier than that. The OA module is controlling both power, thermal and interconnect modules through a I2C bus. So to fake the presence of the fan you would need to tap into the I2C bus and generate the replies the OA is expecting.
    I found another document describing the innards of the c7000 chassis:

    https%3A%2F%2Fcommunity.hpe.com%2Fhpeb%2Fattachments%2Fhpeb%2Fbladescategory04%2F4014%2F1%2FTechnologies%2520in%2520the%2520HP%2520BladeSystem%2520c7000%2520Enclosure.pdf

    OTOH it seems that there's no I2c communication between the blades and the OA, and since you are already bypassing the psus they might
    not care that you have no fans. So the question might be, will the interconnect modules work without the OA ?
    That same picture shows that the iLO is separately routed so you could probably not share it over the NICs.

    As for interconnect modules, the HP 1GB Ethernet Pass-through module would be enough. Mind you, though, that module is a fixed speed
    1Gbit meaning it cannot auto negotiate to 100Mbit if that is the speed of your network. You would need something in-between to sort that out.

    Here's another page with tons of information about the OA, might or might not be useful:

    HP BladeSystem Onboard Administrator User Guide
     
    #15
    tim.yoshi likes this.
  16. tim.yoshi

    tim.yoshi New Member

    Joined:
    Jun 25, 2017
    Messages:
    28
    Likes Received:
    0
    Wow, that came unexpectedly. Will look more on it.

    The link was somehow broken, but I found a way to extrract this document. Thanks, will be reading it.


    Well, it all is just my assumptions based on what I've seen on the internet. My blade is in the mail yet somewhere beyond the sea. But what I noticed - they all disable onboard iLO with jumpers config in order to make it boot without warnings. So yeah, that's the question how enabled iLO will look at absent fans, PSU's, enclosure, "sea of sensors" and so on. Gotta pleasure it with some faked signals likely :)




    This "something" should be something like a gigabit switch or router, don't it? I mean ordinary SOHO 100MBit router will not work it out? Also will HP Cisco blade switches do the job also? Like this one: HP Cisco Catalyst | 410916-B21 | WS-CBS3020-HPQ | 432904-001 | 882780367401 | eBay they are cheaper overall and to my understanding could provide basic (and even not basic) switching and all negotiating functions. Another words if everything will go out smooth with the blades I will be able to connect few blades with this one switch to the internet with just one 100MBit RJ45 cable, right?

    Not that good, but OTOH it could be easier to "extract" bare copper pinout for "bare copper" tap-in to iLO from onboard connector to real world. But what worries me is that they are using, as they say in documentation "Serializer/Deserializer" on every physical signal connection to the blade. Reading further about it (SerDes - Wikipedia) I realize that this is a tricky part - if that is true to my current understanding, one could in no way tap into connector. One would need properly configured deserializer part to work properly with signals. So basically what comes from it's mighty connector is not a few direct Ethernet lines, few direct PCI-e lines, etc, but a bunch of "SOMETHING encoded" that should be "decoded" on the other end. They are decreasing number of physical lines needed with that. But this deserialiser should be inside the pass-thru or switch interconnect modules for sure. So theoretically even if it will be total no-way with iLO and all that *should be there* signals, at least part of the plan with extracting embedded NIC Ethernet signals from a few blades and routing them into one switch (or pass-thru) module should work. Theoretically. Actually even this interconnect modules could be kinda kinky about powering up without OA. Oh, and one more thing, just to make sure that I understand correctly all the topology. As per my understanding I could have up to all that 16 blades connected to a single switch, but with only ONE NIC. In order to use all TWO embedded NICs I have to use TWO separate switches even if I have one blade. They are just two different networks. Right?


    Wow, that's massive. Will be diggin into it also...



    Is your proposal about testing minimal bootable config on one of your enclosures still in force?
     
    #16
  17. s2600gl

    s2600gl Member

    Joined:
    Apr 1, 2016
    Messages:
    33
    Likes Received:
    10
    Yes, that will fit the bill nicely, 1 Gbit towards the backplane and 10/100/1000 to the outside world. Would work nicely with your
    100Mbit gear.

    Correct, if you want to use both NICs you will need two pass-through modules. It has possibly to do with redundancy, loosing one module
    would not take the entire network down.

    Sure thing :). This week has been really hectic, but I'll try to make time for a little "labbing" next week ;)
     
    #17
    tim.yoshi likes this.
  18. voxadam

    voxadam Member

    Joined:
    Apr 21, 2016
    Messages:
    105
    Likes Received:
    11
  19. tim.yoshi

    tim.yoshi New Member

    Joined:
    Jun 25, 2017
    Messages:
    28
    Likes Received:
    0
    Awesome! Thanks. Ordered one Cisco switch to play with.

    Eagerly waiting :) Especially interested if it will boot without onboard administrator module. As far as I see it now, it should be the root of all the evill :) I expect blade's onboard iLO to be functional as it likely acts as a one side communication from in-blade iLO to in-enclosure administrator module. So hence it is possble that in-blade iLO even don't know at all nothing about fans, PSUs, and other external hardware... If this is true - this is perfect :)
     
    #19
  20. tim.yoshi

    tim.yoshi New Member

    Joined:
    Jun 25, 2017
    Messages:
    28
    Likes Received:
    0
    #20
Similar Threads: Mezzanine Backplane
Forum Title Date
DIY Server and Workstation Builds Any instructions on replacing backplane for SM 836TQ to 836A? Jul 24, 2017
DIY Server and Workstation Builds JBOD Chassis/Backplane Jul 19, 2017
DIY Server and Workstation Builds question about Supermicro 846A backplane SAS cable lengths... Apr 14, 2017
DIY Server and Workstation Builds Where to purchase good quality hot-swap backplanes Dec 4, 2012

Share This Page