HP / HPE Moonshot Unofficial Reference Guide
- Thread starter Patrick
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
One thing I noticed on mine is that the M.2 retaining screws appear to be part of the heat sink retainer assembly. Without the heat sink, there's no way to actually keep M.2s in place. It should be easy enough to draw up a part and 3D print it.
Yes, the rear bracket is actually a separate HPE SKU. Here's the SKU's as far as I've been able to tell:
- 867458-001 or 864712-001 - M510 Heatsink Extended
- 867511-001 - M510 Heatsink Backplate
Wow, the pricing on that is kind of wild. A quick scan showed prices between $47 and $130 with no real consensus at all. I need 4, and even $47 is kind of steep for what it is. I mean, it couple probably build my own with an hour or so of measurement and cad work and 3 or 4 print-fit-revise iterations.
I fired up my EL4000 with one M510 installed, and eventually got it working. It wouldn't power on with both PSUs installed but only one plugged in; I'm not sure if that's a feature or a bad PSU. Once I fixed that, I was able to log in to iLO and poke around a bit. I was able to boot the Ubuntu installer using remote media, which was cool, but then it cut me off once it booted because I don't have an advanced ILO license. So presumably I'll have to install via the serial console, which looks straightforward enough. Just ssh to Administrator@<ILO IP>, log in with the same password, and go from there.
It runs around 40C with no heatsink while booting. I don't want to put any load on it until I have heatsinks, but it looks like it's installable for now.
It runs around 40C with no heatsink while booting. I don't want to put any load on it until I have heatsinks, but it looks like it's installable for now.
Here's my first pass at a heat sink mount: scottlaird/HPE_M510
That's still *slightly* too thick; the next rev will probably be about 1mm thinner. I'm using one of these right now, and it fits into the slot fine, but it's obviously slightly thicker than it should be.
It's designed to use M3 heat-set inserts to hold the screws from the heat sink. It's specifically intended for a specific model of insert from McMaster-Carr, but anything similar will probably be fine. I've included M.2 mounting points for 2260, 2280, and 22110 where possible, using M2 screws. This rev doesn't include holes for the SATA M.2 port.
That's still *slightly* too thick; the next rev will probably be about 1mm thinner. I'm using one of these right now, and it fits into the slot fine, but it's obviously slightly thicker than it should be.
It's designed to use M3 heat-set inserts to hold the screws from the heat sink. It's specifically intended for a specific model of insert from McMaster-Carr, but anything similar will probably be fine. I've included M.2 mounting points for 2260, 2280, and 22110 where possible, using M2 screws. This rev doesn't include holes for the SATA M.2 port.
Attachments
Last edited:
I now have 4 M510s running in an EL4000 with a newer rev of my heatsink mount. I had to *really* shorten it in order to fit underneath the second DIMM slot, so it's not using brass thread inserts anymore, as every one that I could find was too thick. So the heat sink screws simply thread into the plastic. Not ideal, but probably the best that can be done for cheap. I'm working on getting a few sample heatsinks, but that'll probably take 2-3 weeks.
I updated the GitHub project that holds the STL for the mount. The README has a few more details, including a link to the Fusion 360 source for the mount.
I updated the GitHub project that holds the STL for the mount. The README has a few more details, including a link to the Fusion 360 source for the mount.
Awesome works Scott! I'm going to try to print on my end, see how I go. I picked up one of these and subsequently another 3 more recently. Looking forward to see how you go with the heatsinks as well.
What's the ordering process look like for them?
I found a few places online that do semi-custom heatsinks. Which is to say, they take stock heatsink stock and machine it down to match a spec. I have an automated quote from one of them (low $20s each, quantity 10), along with an estimate of 3-ish weeks. Hopefully I'll hear more next week. It looks like they're going to generate a part number for the heatsink that I'm ordering, so it should be simple for people to order their own. Or, if a bunch of people were interested, then a group buy would drive the price down, but even quantity 5 the price wasn't horrible.
Interesting. I found a few Alibaba sellers last night who look like they do custom jobs with heat pipes and all. Unsure of cost but willing to investigate there as well. I'd love to get my hand on a proper 867458-001 heatsink to reference, but they're impossible to find... anywhere...
Also, you mentioned you were working with your m510's inside an EL4000. This is all _without_ heatsinks? Are you using the 16 core variety? How are the thermals looking?
Also, you mentioned you were working with your m510's inside an EL4000. This is all _without_ heatsinks? Are you using the 16 core variety? How are the thermals looking?
Oddly, the M510s (16 core) without heatsinks run fine, at around 40C when idle. But they boot fairly slowly, which makes me think that they're hitting thermal throttling limits in between spot checks. They work well enough so I can deal with OS installs and so forth while waiting for my heatsinks to arrive. Or maybe "exist" would be better?
I should probably throw this in here: installing Ubuntu without paying for an advanced iLO license is actually fairly easy. The problem is that the standard iLO stops doing console redirection once the OS has booted, which makes talking to the installer tricky. So here's what I did (once for each M510):
I should probably throw this in here: installing Ubuntu without paying for an advanced iLO license is actually fairly easy. The problem is that the standard iLO stops doing console redirection once the OS has booted, which makes talking to the installer tricky. So here's what I did (once for each M510):
- Boot the node with an Ubuntu 20.04 server install USB drive using the iLO console redirection.
- In parallel to this, ssh into the iLO, using the same username and password used for the iLO web interface. So 'ssh Administrator@1.2.3.4'. Then type 'vsp' to start the virtual serial port connection. You can watch the POST while you wait.
- In the iLO Java console redirection window: Once the node finishes its POST interrupt the grub bootloader so it wouldn't automatically proceed to boot linux. Hitting one of the arrow keys will accomplish this.
- Hit 'e' to edit the settings for the Ubuntu install Linux boot option.
- Add 'console=ttyS0,115200' to the 'linux' command line. I actually replaced the 'quiet' option so I could verify that the serial port worked, but it probably doesn't matter.
- Hit 'control-x' to tell Grub to start booting Linux (there's a prompt for this on the screen).
- Now watch the serial/ssh window. You'll see the kernel boot, and then it'll be quiet for a minute while installer messages scroll on the iLO console window. After the installer finishes booting, it'll prompt for input on the serial/ssh window. You should be able to install like normal at this point.
- After installing, the default Ubuntu server config includes a serial console, but the VGA console is the default so you won't see all of the boot messages and may not be able to interact with it if it fails to boot correctly. You can edit the grub config to change this. Just follow any of the random web pages that discuss setting up serial consoles for Ubuntu (or really any Linux distribution; they're all going to be the same at this level).
HPE's website says that there is a "critical" iLO firmware upgrade to v2.61 from early 2021. Mine came with v2.54, from Jan 24, 2018. Unfortunately, the link that my iLO provided for HPE firmware updates wasn't helpful.
I was able to download the latest version from here:
support.hpe.com
Basically, I searched for "moonshot" and filtered to OS-independent software. This produced a 'ilo4_261_moonshot.bin' file that I could upload via the iLO webpage. Seems to work fine. Better than fine, actually: it adds HTML5 remote console support. No more Java!
I was able to download the latest version from here:
HPE Support Center
Basically, I searched for "moonshot" and filtered to OS-independent software. This produced a 'ilo4_261_moonshot.bin' file that I could upload via the iLO webpage. Seems to work fine. Better than fine, actually: it adds HTML5 remote console support. No more Java!
I have 5 custom heatsinks heading to manufacturing now. Hopefully they'll work on the first pass and I won't need to order a second (or third) batch. Quantity 5, it comes to about $35 each, but for 50+ it's under $10. That seems surprisingly reasonable for a custom heatsink.
exciting! i'm waiting with bated breath!
Fingers crossed for the "hole in one" first batch. I know the feeling... had a few custom engineering builds in the past (cases mostly) that were never "quite right" but nothing a Dremel and a drill couldn't fix
Fingers crossed for the "hole in one" first batch. I know the feeling... had a few custom engineering builds in the past (cases mostly) that were never "quite right" but nothing a Dremel and a drill couldn't fix
Scott - what is power draw are you getting from the 4 nodes setup you have in EL4000? I am debating about downsizing my current lab using the same setup.
My EL4000's onboard power meter claims around 120W average. That seems high if it's a per-node number, but low for 4 nodes. So I'm not really sure how much power it's drawing.
However, I also have an EL1000 with a single M510 sitting here. The EL1000 is the single-node cousin of the EL4000. This one has 2x 32G DIMMs, 1x 1T Sabrent Rocket 4 M.2 SSD, a 500W AC power supply, and a 2x40G ConnectX-3 card installed. Since it has a single node, it's a lot less ambiguous, and it's also plugged into a metered PDU at the moment. So I have good data on power for it.
With the M510 off, the chassis draws around 21W. Booting Ubuntu 22.04 over the network (via iLO) peaked around 110W, and then dropped to ~82W while installing the first time. It kept getting louder and louder, and I finally realized that I'd left it sitting with the fan intake blocked. Oops. After that, it dropped to ~63W while installing and 53W at the Ubuntu login prompt. So I guess it *could* draw ~20W for the chassis, ~7W for the Mellanox NIC, and ~26W for the M510. That'd make 120W for an EL4000 possible.
FWIW, the EL1000 is kind of neat, although I wish it was small enough to fit 2 onto a 2U shelf. As it is, it's the size of a moderately small desktop PC (7 liters--not tiny, but compact). 2x PCIe x8 slots, 1 PS, 2x 2.5" SATA bays, 2x optional Mini-PCI slots for WiFi + LTE. Unfortunately, all I could find for cheap was the 2x 1G model, but there's a 2x SFP+ model out there somewhere.
A spare EL4000 has arrived, so I threw a single M510 (same config as the last message, but without the extra NIC) and a 500W power supply (not technically supported, but seems to work) into it to compare. It draws around 28W with the node powered off, or slightly more than the EL1000 did. It draws 60W during POST and 52W sitting idle in the Ubuntu installer.
I might move the node from the EL1000 into the EL4000 later today, just to see what that gives me for numbers. Probably ~85W or so total, or a bit higher if I throw CX3s into the EL4000 as well.
So 120W with 4 nodes isn't unreasonable.
I might move the node from the EL1000 into the EL4000 later today, just to see what that gives me for numbers. Probably ~85W or so total, or a bit higher if I throw CX3s into the EL4000 as well.
So 120W with 4 nodes isn't unreasonable.