A rainy day project - Warning PIC Heavy!!

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
Ok, so as some of you may know, I don't always take the obvious or easy route to solving problems. Sometimes this is out of necessity, and other times I do this just for some fun when I'm bored. Either way, here goes with another one of my crazy meanders down the rabbit hole...

Imagine for a moment that you do a lot of work with older DELL servers. You need to perform some updates, bios etc and if you are anything like myself, you dread the thought of even breathing in the same room as it. You just never know what may break in the process, such is the nature of DELL updates :)

This brings me to the eternal issue of the failed iDRAC/ Lifecycle controller on these things, sticking two fingers up at you after an update. Sure, it all seems to go well, all the updates complete without error or warning and you have no reason to suspect that you should have stayed in bed that day. Upon a reboot you are greeted with a massive long boot time, howling fans which are quickly followed by this lovely image on screen:




You think to yourself, it's OK, I'll power cycle and drain off the flea power, nope. OK I'll redirect the com port, nope. OK I'll try the web interface, nada. A bit more investigation reveals that it doesn't even pull an IP address, sh1t, another $40 wasted. In fact, when you get to this point, there's no way to get back in to it, in order to fix it. If it won't talk to the server, you can't update it or roll it back. Basically your fancy remote access device has just turned into a "will never access it again device". After exhausting every possible option, you will likely be told by the "DELL Experts", that you have either goosed the iDRAC cards, if you're lucky and it has those, or the mainboard or both and will need replacements. How's that for spoiling your Monday morning coffee...

What I'm going to be dealing with here, specifically, is the iDRAC6 modules. These are not built onto the mainboard, they are pluggable modules. Here's what the express module looks like when the plastic carrier is removed:






With such grandiose names like "iDRAC6 Express" and "iDRAC6 Enterprise" one would be led to believe that the modules themselves had small Microcontroller's, ASIC's, or some such devices in them, performing some complicated task. However, in the case of the iDRAC6 enterprise module, it's nothing more than a fancy Broadcom nic card with an SD adapter built on top. The irony is not lost on me. The iDRAC6 express module, much to my amazement, is simply a 2GB Hynix eMMC NAND chip, wrapped up in a bit of plastic. I'm sure it's that damned plastic bit that costs so much money. It's either that or the quality printing on the labels :)

Now I don't know about you, but I have amassed a fair few of these damned things over the years, none of which have worked since that fateful moment, you know, the one where you wish you hadn't told your friend you would quickly perform the updates for him while there doing something else. To add insult, these things sell for anywhere between $25 and $90 each, depending on where you buy them from, so to have a box of them sitting useless is frustrating at the least. For me, this time was the last straw. I was determined to return them to service somehow and figured, if I can do it, so can anyone else reading this. I refuse to spend any more money fixing bad updates on these things, unless I really have no other choice :)

I won't bore you with the technical details, but if you care to dig deeper, look for the Jedec MMC and SD standards. The SD Card standard is actually a subset of the MMC standard, in fact, all SD Cards begin communication as if they were actually MMC cards. They then negotiate appropriate features etc and switch to SD mode. This little feature is really quite handy, as you'll see shortly, because it allows you to probe and prod an eMMC memory device directly, just as if it were an SD memory device, all with a minimal interface. This is exactly what I'm about to embark on here :)

I strongly suggest that if anyone follows in my footsteps to recover their own duff modules, that you use a USB Multi-Card Reader to connect to your Laptop/ PC, rather than using the direct SD slot. If, gawd forbid, you get anything wrong or miss spotting a short circuit etc, you'll be spared the embarrassment of telling your wife that you now have both a duff Laptop AND a box of non-functional iDRAC modules :D

First off, I need to "take one for the team" so to speak. I need to do this so I can locate suitable connection points on the iDRAC6 module in order to hook up my SD port. To this end, I need to pop the eMMC chip off the board. Don't worry, you'll not have to do this, it's a one time deal. When I get around to doing any more BGA rework, I'll probably just reball the chip and stick it back on the iDRAC6 module, just for a spare, but for the time being, this is what it looks like without it's clothes on:








Having now located suitable connection points on the module for my SD port, I can move on to building an adapter to let me actually connect up to the Laptop. Before I do though, let me just mention what you'll need if you want to build this yourself:

You'll need a Micro-SD card adapter, the cheaper the better, as the cheap ones are usually only glued together in a couple of spots.

You will also need some thin hook up wire, preferably different colours, so you don't get confused as to which wire is which. Any wire will do, as long as it's thin enough that it doesn't create a hump in the SD adapter casing. I robbed mine from an old Cisco console cable, which had a bad RJ plug on it. I needed to cut it off and re-terminate it anyway, so losing a few inches off the end for this is no biggie.

Also you'll need some basic hand tools and a soldering iron with a fine tip and better eyesight than I have to solder it all together!

Lastly you will need either a working iDRAC6 express module to copy from, or a suitable image file to dump back into your duff ones.

A word on wire type, length and connection to the module etc. Keep the wires really short, certainly less than 50mm end to end and hook them up to the module exactly where I have marked on the photo. I have chosen those points carefully to minimise cable reflections and impedance. If your wires are too long, or you use ribbon cable, instead of individual wires, there's a chance you will have unreliable communication, which is a baaaad thing :)

Here's where you need to connect up to on the iDRAC6 express module. Pay close attention to where each wire goes and watch for shorts/ solder splashes when you connect to them. I have colour coded them the same way as I wired my SD adapter to make it easier to follow:




So lets build the adapter:

Carefully crack open the Micro-SD adapter and remove the pin insert intact. Snip off the little crimped/ springy ends of the pins that would normally contact the micro card, just to make them sit nice and flat. Also snip off the three unused pins so you are not tempted to stick a wire on them by mistake.

Once you are satisfied that the wires are soldered on well and not shorting together, place the pin insert back in the carrier, laying the wires straight and flat, then pop the cover on with a few drops of super glue to hold the thing together. Place a line of glue over where the wires will contact the casing to strain relieve them, and be careful you don't glue your fingers on!

Only use super glue, anything else like hot melt etc, might introduce enough stray capacitance to ruin your efforts. Also, a possible gotcha with using super glue, is that it can run onto the pins, so inspect them after the glue dries and scrape them clean again if needed, or you'll be wondering why nothing happens when you plug it in.

If you look closely at the next couple of photos, you'll see exactly what goes where and how it should kind of look when your done with it:










If all has gone well so far, you should be able to fit the adapter easily into an SD slot, without having to put excessive pressure on it:




Next up, carefully hook up your wires, watching for shorts, to your working iDRAC6 express module, or skip ahead to the bad one(s), if you have a suitable image file already on hand. You will likely need to preheat the power and ground points with your soldering iron tip for a minute or so, before attempting to place the wires on, because they are attached to large copper planes internally in the module, which will soak up all the heat. A little patience is key here. Do not tug on the wires to see if they are soldered, you might pull up the pads and ruin the board. Here's what it should sort of look like when you're done:






Ok, so that's it for the hardware. Plug your express module into the card adapter, fire up your favourite flavour of Linux on your Laptop/ PC and plug your adapter into the USB port. Fingers crossed all went well and using fdisk you can now see your express module's partitions listed. Assuming that you do, use dd to image the whole disk, all 2GB of it and store the image somewhere safe. Now remove the card adapter from the USB port, pull your good express module and swap over the wires onto your bad one(s). You can then repeat this procedure as necessary, but this time use dd to write the duff express modules with the image file you just captured from the good one. Both operations will take a little time, because you are only using a minimal interface (single bit mode) to the device, but the iDRAC recovery really is as simple as that, assuming the eMMC on the module is actually working of course :)

Here's what mine looked like:




After you write your image back to the bad express module, remove the card reader and the express module. Desolder the wires and give it a quick clean with something like isopropyl to remove any accidental solder splashes and flux residue. Drop it back into your server and hold your breath...better yet, leave the room :)

Hopefully, all being well, you'll be greeted with the happy message and be able to, once again, access the iDRAC config menu:






As you can see, my test server is all over the map in terms of updates, but a quick run of the latest repo iso and bob's yer auntie, we're back in business:







If like me, you have a few of these things kicking about, that are not playing nice and are tired of wasting money on them and you have the time (fortunately I did this week), then this is a fun way to kill a couple of hours. It took me longer to actually document this, than it did to do it. The fact that I have recovered a pile of money in parts that were otherwise unusable, is a bonus in my eyes. I really don't know why I never got around to doing this before now :)

Obviously in any kind of production environment, one would simply order replacement parts, so I'm not suggesting that anyone consider this a goto fix, well, not unless you're really in a bind, but for the little guy on the cost conscious budget, this really is a money and time saver. The good thing is that pretty much all of the time and effort is taken up by building the SD adapter and capturing a working image. Once you have that step safely tucked away, you can turn a duff module round in 10 minutes whenever you need to in the future.

Anyway, I hope you enjoyed the read and managed to get something out of it...


Edited due to my being blissfully unaware that TinyPic went extinct, taking all my photos with it :oops:

Just an additional thought, deviating slightly here from the above instructions. Run the latest repo iso and bring your server up to date first using your good/ donor iDRAC6 module, then pull the now updated module, hook it up to the adapter and take an image of that with dd, that way you will have the latest good version to dump back into your confused ones. I didn't do that on my first pass through when I wrote this up, as I only had 1 old, but working, donor module and I didn't want to risk borking it before I had a working solution for recovery.
 
Last edited:

saivert

Member
Nov 2, 2015
138
18
18
40
Norway
I assume the next step is to build an adapter that fits the actual socket on the iDRAC card?
Looks to be some kind of Molex PMC Mezzanine connector. One should be able to source this.
You could design a PCB with the mating connector that ends in SD card like gold fingers. Could do for a small production run.
 

ttabbal

Active Member
Mar 10, 2016
743
207
43
47
Probably more trouble than it's worth to spin a PCB for the 5 wires. You would have to be flashing a LOT of them to make it worth the bother. :)
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
Yea, the way I would do it, if I were inclined to be fixing these things in batches of 50 or more daily, would be to accurately place pogo pins on a PCB, in exactly the right locations, and then sit the modules in an actual programming cradle. The connectors that DELL used are not designed for large numbers of insertions and would become unreliable very quickly. All that said, production programming of these was not my intention here, I'm just an old git that is sick to death of being rumped up the a*se for these every time I run a damned DELL update :)

I'm open to do some investigation work on some DELL mainboards that have failed and that also have the iDRAC onboard if anyone cares to ship some in my direction that are out of warranty etc, I'm fairly confident that those will be recoverable also :)
 

TuxDude

Well-Known Member
Sep 17, 2011
616
338
63
Well if you want to go the cheap route, I can print you a cradle with pretty good precision to hold the drac card and pogo pins, and just solder wires to the pins instead of having a PCB. I would just need accurate dimensions.
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
I would doubt that there would be sufficient interest in recovering these modules, unless motivated by a pile of them sitting on your desk, the servers that use them are getting on a bit now. I'm game to dream something up for it, if folks here would find it useful, but I would think most simply replace them. In saying that, having a wee programming cradle that is cheap to put together, turns this idea into a field service tool :)
 

Dww0311

Member
May 19, 2017
49
7
8
57
is there any chance you still have the photos available or could advise how to set up the connections? I have a ton of these which I can not use and would like to try your recovery method. Any help greatly appreciated!
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
is there any chance you still have the photos available or could advise how to set up the connections? I have a ton of these which I can not use and would like to try your recovery method. Any help greatly appreciated!

Thanks for bringing the photo issue to my attention, should be good now :)
 
  • Like
Reactions: T_Minus

Dww0311

Member
May 19, 2017
49
7
8
57
Thanks for bringing the photo issue to my attention, should be good now :)
thank you for reposting it. Super helpful. I successfully constructed the adapter, but try as I might I can not get anything to stick to the pads on the pcb. Isopropyl’ed it, using good quality NC 63/37 and I took the heat as high as I dare (300°c) without frying the board entirely. Any suggestions / helpful tips?
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
Soldering onto the little via's using a small needle tip iron should really be a non issue, as there's practically no copper there to sink the heat away. If you are having problems soldering even those, either the iron is not hot enough, or the tip contact is poor and you are not making good enough thermal contact. If there's conformal coating on the board, scrape the pads clean. Also sometimes just tinning the pad/ via first, helps with thermal contact as it increases the footprint as well as the mass, also the application of a little flux goes a long way. The pads for the VCC and VSS do take a lot of heat, as they are effectively full copper layers. I find that to help things along with large ground planes it's always best to pre-heat the general area first, so that you reach temperature easily and quickly with the smaller iron tip. a few minutes with a good hairdryer or hot air wand is all that's needed. Patience is also key, you can hold the iron tip on for a fairly long time, just don't slide it around on the board in case you soften the pad bond :)
 
  • Like
Reactions: Dww0311

Dww0311

Member
May 19, 2017
49
7
8
57
Soldering onto the little via's using a small needle tip iron should really be a non issue, as there's practically no copper there to sink the heat away. If you are having problems soldering even those, either the iron is not hot enough, or the tip contact is poor and you are not making good enough thermal contact. If there's conformal coating on the board, scrape the pads clean. Also sometimes just tinning the pad/ via first, helps with thermal contact as it increases the footprint as well as the mass, also the application of a little flux goes a long way. The pads for the VCC and VSS do take a lot of heat, as they are effectively full copper layers. I find that to help things along with large ground planes it's always best to pre-heat the general area first, so that you reach temperature easily and quickly with the smaller iron tip. a few minutes with a good hairdryer or hot air wand is all that's needed. Patience is also key, you can hold the iron tip on for a fairly long time, just don't slide it around on the board in case you soften the pad bond :)
I gave it several good tries, but no joy. It just would not bond. I ended up buying a 0PPH2J commercially - those seem to be the latest ones that were released and tend to have new enough microcode on them that they'll work. Seriously ball dropping by Dell with that one.
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
Damn, that's rough, especially after doing the hard bit building the adapter. If you have a few of them laying around, making it worthwhile for you, I would be quite happy to recover them for you, if you want to pay the postage each way :)
 

ketiljo

New Member
Sep 7, 2022
16
7
3
Thanks for the work on this! As I only have one non-working module, would anyone be able to post an image please?
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
Just so folks understand, a recovered "good" image file, from a working iDRAC6 board is likely copyright to Dell, or at least in that grey area where what happens behind closed doors for personal use...Anyway, I won't openly post a file for this, for that reason and that I have no wish to get anyone in trouble or bring unwanted attention to the forum owners, but if you are really stuck, pm me directly and I'll do what I can to help :)

Wow! Has it been THAT LONG since I posted the original article?
 
  • Like
Reactions: Borromini

ketiljo

New Member
Sep 7, 2022
16
7
3
Internet and elephants, never forgets :)

A fair point about the copyright of the image.

Another interesting find with this server is that you can hook up a USB to UART cable to the iDrac UART pins and get access to the Linux terminal (BusyBox). Here you can use racadm to pull the firmimg.d6 over TFTP and restore the image from what I can tell.
I have two of these express cards and none of them works. The first only have 1 GB of eMMC out of some strange reason. I can pull the image over TFTP and is starts to copy it but obviously fails because of the size. The other seems to have a bad chip as it complains about block read errors. But what's interesting is that you can use fdisk to delete the partitions on the eMMC and at the next reboot, all partitions are restored. So I believe the startup script will do the partitioning of the card if it doesn't find any partitions.
I've ordered a couple of 2 GB chips and I'll replace the faulty one to see it I can restore it over TFTP. On a 12 gen servers this works. I have a R720 a well where I replaced the eMMC with a new and empty one and was able to restore the iDrac from firmimg.d7 from a SD card.
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
Perhaps, as with anything else, things become more reliable per iteration/ cycle. That said, any software update comes with risk attached and Dell churn out that many things, that software is not generally given enough attention imo. Read what's in front of you in the documentation. Often with hardware, there is a specific order of applying multiple updates, getting it wrong leaves you wide open to malfunction. Make sure you have stable power before doing anything and do nothing to intererfere with it during the update process. Whatever else you do, ensure that any updates you do deploy, have been carefully compared against a valid hash from Dell, in order to avoid applying a corrupted image, but don't just check the files you download, check them again from the launch media. This may seem obvious, but you would be surprised how many folks just download an update and let 'er rip without testing it first, and I'm my own worst enemy for being in a rush and grabbing an old USB stick. If it's not the number one cause of borking something, it definitely ranks high up the list :)
 

CyklonDX

Well-Known Member
Nov 8, 2022
784
255
63
Is the IDRAC7 less prone to getting fuucked up?
haha, no.

Certain upgrades break it if they aren't sequentially upgraded in minors. (even worse to fix too, as they are on the motherboard - so they cannot be replaced anymore.)
 

ionotto

New Member
Sep 14, 2023
1
0
1
Well, I tried it and did the same... and it works! I can see my idrac6 express as a SDcard of 1GB capacity.
...but I dont have another idrac6 express module in order to extract an image.

(...) Anyway, I won't openly post a file for this, for that reason and that I have no wish to get anyone in trouble or bring unwanted attention to the forum owners, but if you are really stuck, pm me directly and I'll do what I can to help :) (...)
I was about to ask you for that image. But im wondering if that will help me, cause your module has 2GB capacity. Twice as mine XD

So close c': nice thread anyways! Thanks! I ordered another idrac6 express card in aliexpress, so ill take note of this to create a working image for the next time.

Sidenote: in one photo, the 3rd test point next to CMD is marked "DAT0" but its actually the 2nd test point (another photo xD)