Patrick

HP / HPE Moonshot Unofficial Reference Guide

barmalej

New Member
May 31, 2019
9
1
3
These OEM heatsinks are impossible to find. I talked to sellers on Alibiba, we can order a custom-made ones with heatpipes but the minimum order is 30
 

Scott Laird

Active Member
Aug 30, 2014
292
130
43
One thing I noticed on mine is that the M.2 retaining screws appear to be part of the heat sink retainer assembly. Without the heat sink, there's no way to actually keep M.2s in place. It should be easy enough to draw up a part and 3D print it.
 

adam_kf

New Member
Apr 17, 2017
17
5
3
37
One thing I noticed on mine is that the M.2 retaining screws appear to be part of the heat sink retainer assembly. Without the heat sink, there's no way to actually keep M.2s in place. It should be easy enough to draw up a part and 3D print it.
Yes, the rear bracket is actually a separate HPE SKU. Here's the SKU's as far as I've been able to tell:

  • 867458-001 or 864712-001 - M510 Heatsink Extended
  • 867511-001 - M510 Heatsink Backplate
Note: the backplate is available through HPE spare parts, so the only thing we need is the actual CPU heatsink assembly.
 

Scott Laird

Active Member
Aug 30, 2014
292
130
43
Wow, the pricing on that is kind of wild. A quick scan showed prices between $47 and $130 with no real consensus at all. I need 4, and even $47 is kind of steep for what it is. I mean, it couple probably build my own with an hour or so of measurement and cad work and 3 or 4 print-fit-revise iterations.
 

Scott Laird

Active Member
Aug 30, 2014
292
130
43
I fired up my EL4000 with one M510 installed, and eventually got it working. It wouldn't power on with both PSUs installed but only one plugged in; I'm not sure if that's a feature or a bad PSU. Once I fixed that, I was able to log in to iLO and poke around a bit. I was able to boot the Ubuntu installer using remote media, which was cool, but then it cut me off once it booted because I don't have an advanced ILO license. So presumably I'll have to install via the serial console, which looks straightforward enough. Just ssh to Administrator@<ILO IP>, log in with the same password, and go from there.

It runs around 40C with no heatsink while booting. I don't want to put any load on it until I have heatsinks, but it looks like it's installable for now.
 

Scott Laird

Active Member
Aug 30, 2014
292
130
43
Here's my first pass at a heat sink mount: scottlaird/HPE_M510

That's still *slightly* too thick; the next rev will probably be about 1mm thinner. I'm using one of these right now, and it fits into the slot fine, but it's obviously slightly thicker than it should be.

It's designed to use M3 heat-set inserts to hold the screws from the heat sink. It's specifically intended for a specific model of insert from McMaster-Carr, but anything similar will probably be fine. I've included M.2 mounting points for 2260, 2280, and 22110 where possible, using M2 screws. This rev doesn't include holes for the SATA M.2 port.
 

Attachments

Last edited:
  • Like
Reactions: adam_kf

Scott Laird

Active Member
Aug 30, 2014
292
130
43
I now have 4 M510s running in an EL4000 with a newer rev of my heatsink mount. I had to *really* shorten it in order to fit underneath the second DIMM slot, so it's not using brass thread inserts anymore, as every one that I could find was too thick. So the heat sink screws simply thread into the plastic. Not ideal, but probably the best that can be done for cheap. I'm working on getting a few sample heatsinks, but that'll probably take 2-3 weeks.

I updated the GitHub project that holds the STL for the mount. The README has a few more details, including a link to the Fusion 360 source for the mount.
 
  • Like
Reactions: adam_kf and Patriot

adam_kf

New Member
Apr 17, 2017
17
5
3
37
I'm working on getting a few sample heatsinks, but that'll probably take 2-3 weeks.
Awesome works Scott! I'm going to try to print on my end, see how I go. I picked up one of these and subsequently another 3 more recently. Looking forward to see how you go with the heatsinks as well.

What's the ordering process look like for them?
 

Scott Laird

Active Member
Aug 30, 2014
292
130
43
I found a few places online that do semi-custom heatsinks. Which is to say, they take stock heatsink stock and machine it down to match a spec. I have an automated quote from one of them (low $20s each, quantity 10), along with an estimate of 3-ish weeks. Hopefully I'll hear more next week. It looks like they're going to generate a part number for the heatsink that I'm ordering, so it should be simple for people to order their own. Or, if a bunch of people were interested, then a group buy would drive the price down, but even quantity 5 the price wasn't horrible.
 

adam_kf

New Member
Apr 17, 2017
17
5
3
37
Interesting. I found a few Alibaba sellers last night who look like they do custom jobs with heat pipes and all. Unsure of cost but willing to investigate there as well. I'd love to get my hand on a proper 867458-001 heatsink to reference, but they're impossible to find... anywhere...

Also, you mentioned you were working with your m510's inside an EL4000. This is all _without_ heatsinks? Are you using the 16 core variety? How are the thermals looking?
 

Scott Laird

Active Member
Aug 30, 2014
292
130
43
Oddly, the M510s (16 core) without heatsinks run fine, at around 40C when idle. But they boot fairly slowly, which makes me think that they're hitting thermal throttling limits in between spot checks. They work well enough so I can deal with OS installs and so forth while waiting for my heatsinks to arrive. Or maybe "exist" would be better?

I should probably throw this in here: installing Ubuntu without paying for an advanced iLO license is actually fairly easy. The problem is that the standard iLO stops doing console redirection once the OS has booted, which makes talking to the installer tricky. So here's what I did (once for each M510):
  1. Boot the node with an Ubuntu 20.04 server install USB drive using the iLO console redirection.
  2. In parallel to this, ssh into the iLO, using the same username and password used for the iLO web interface. So 'ssh Administrator@1.2.3.4'. Then type 'vsp' to start the virtual serial port connection. You can watch the POST while you wait.
  3. In the iLO Java console redirection window: Once the node finishes its POST interrupt the grub bootloader so it wouldn't automatically proceed to boot linux. Hitting one of the arrow keys will accomplish this.
  4. Hit 'e' to edit the settings for the Ubuntu install Linux boot option.
  5. Add 'console=ttyS0,115200' to the 'linux' command line. I actually replaced the 'quiet' option so I could verify that the serial port worked, but it probably doesn't matter.
  6. Hit 'control-x' to tell Grub to start booting Linux (there's a prompt for this on the screen).
  7. Now watch the serial/ssh window. You'll see the kernel boot, and then it'll be quiet for a minute while installer messages scroll on the iLO console window. After the installer finishes booting, it'll prompt for input on the serial/ssh window. You should be able to install like normal at this point.
  8. After installing, the default Ubuntu server config includes a serial console, but the VGA console is the default so you won't see all of the boot messages and may not be able to interact with it if it fails to boot correctly. You can edit the grub config to change this. Just follow any of the random web pages that discuss setting up serial consoles for Ubuntu (or really any Linux distribution; they're all going to be the same at this level).
 

Scott Laird

Active Member
Aug 30, 2014
292
130
43
HPE's website says that there is a "critical" iLO firmware upgrade to v2.61 from early 2021. Mine came with v2.54, from Jan 24, 2018. Unfortunately, the link that my iLO provided for HPE firmware updates wasn't helpful.

I was able to download the latest version from here:


Basically, I searched for "moonshot" and filtered to OS-independent software. This produced a 'ilo4_261_moonshot.bin' file that I could upload via the iLO webpage. Seems to work fine. Better than fine, actually: it adds HTML5 remote console support. No more Java!
 

Scott Laird

Active Member
Aug 30, 2014
292
130
43
I have 5 custom heatsinks heading to manufacturing now. Hopefully they'll work on the first pass and I won't need to order a second (or third) batch. Quantity 5, it comes to about $35 each, but for 50+ it's under $10. That seems surprisingly reasonable for a custom heatsink.
 

adam_kf

New Member
Apr 17, 2017
17
5
3
37
exciting! i'm waiting with bated breath!

Fingers crossed for the "hole in one" first batch. I know the feeling... had a few custom engineering builds in the past (cases mostly) that were never "quite right" but nothing a Dremel and a drill couldn't fix :)
 
  • Like
Reactions: Scott Laird

adrian_sa

New Member
Oct 5, 2020
3
2
3
It would be interesting to get a M510 blade working without the chassis. I think that the first small part of the edge connector is for power since it appears that the two bigger connectors have differential pairs, that could be for PCIe and SATA/SAS.
 

Scott Laird

Active Member
Aug 30, 2014
292
130
43
Scott - what is power draw are you getting from the 4 nodes setup you have in EL4000? I am debating about downsizing my current lab using the same setup.
My EL4000's onboard power meter claims around 120W average. That seems high if it's a per-node number, but low for 4 nodes. So I'm not really sure how much power it's drawing.

However, I also have an EL1000 with a single M510 sitting here. The EL1000 is the single-node cousin of the EL4000. This one has 2x 32G DIMMs, 1x 1T Sabrent Rocket 4 M.2 SSD, a 500W AC power supply, and a 2x40G ConnectX-3 card installed. Since it has a single node, it's a lot less ambiguous, and it's also plugged into a metered PDU at the moment. So I have good data on power for it.

With the M510 off, the chassis draws around 21W. Booting Ubuntu 22.04 over the network (via iLO) peaked around 110W, and then dropped to ~82W while installing the first time. It kept getting louder and louder, and I finally realized that I'd left it sitting with the fan intake blocked. Oops. After that, it dropped to ~63W while installing and 53W at the Ubuntu login prompt. So I guess it *could* draw ~20W for the chassis, ~7W for the Mellanox NIC, and ~26W for the M510. That'd make 120W for an EL4000 possible.

FWIW, the EL1000 is kind of neat, although I wish it was small enough to fit 2 onto a 2U shelf. As it is, it's the size of a moderately small desktop PC (7 liters--not tiny, but compact). 2x PCIe x8 slots, 1 PS, 2x 2.5" SATA bays, 2x optional Mini-PCI slots for WiFi + LTE. Unfortunately, all I could find for cheap was the 2x 1G model, but there's a 2x SFP+ model out there somewhere.
 
  • Like
Reactions: Patrick

Scott Laird

Active Member
Aug 30, 2014
292
130
43
A spare EL4000 has arrived, so I threw a single M510 (same config as the last message, but without the extra NIC) and a 500W power supply (not technically supported, but seems to work) into it to compare. It draws around 28W with the node powered off, or slightly more than the EL1000 did. It draws 60W during POST and 52W sitting idle in the Ubuntu installer.

I might move the node from the EL1000 into the EL4000 later today, just to see what that gives me for numbers. Probably ~85W or so total, or a bit higher if I throw CX3s into the EL4000 as well.

So 120W with 4 nodes isn't unreasonable.
 
  • Like
Reactions: Patrick