Ok, so as some of you may know, I don't always take the obvious or easy route to solving problems. Sometimes this is out of necessity, and other times I do this just for some fun when I'm bored. Either way, here goes with another one of my crazy meanders down the rabbit hole...
Imagine for a moment that you do a lot of work with older DELL servers. You need to perform some updates, bios etc and if you are anything like myself, you dread the thought of even breathing in the same room as it. You just never know what may break in the process, such is the nature of DELL updates
This brings me to the eternal issue of the failed iDRAC/ Lifecycle controller on these things, sticking two fingers up at you after an update. Sure, it all seems to go well, all the updates complete without error or warning and you have no reason to suspect that you should have stayed in bed that day. Upon a reboot you are greeted with a massive long boot time, howling fans which are quickly followed by this lovely image on screen:
You think to yourself, it's OK, I'll power cycle and drain off the flea power, nope. OK I'll redirect the com port, nope. OK I'll try the web interface, nada. A bit more investigation reveals that it doesn't even pull an IP address, sh1t, another $40 wasted. In fact, when you get to this point, there's no way to get back in to it, in order to fix it. If it won't talk to the server, you can't update it or roll it back. Basically your fancy remote access device has just turned into a "will never access it again device". After exhausting every possible option, you will likely be told by the "DELL Experts", that you have either goosed the iDRAC cards, if you're lucky and it has those, or the mainboard or both and will need replacements. How's that for spoiling your Monday morning coffee...
What I'm going to be dealing with here, specifically, is the iDRAC6 modules. These are not built onto the mainboard, they are pluggable modules. Here's what the express module looks like when the plastic carrier is removed:
With such grandiose names like "iDRAC6 Express" and "iDRAC6 Enterprise" one would be led to believe that the modules themselves had small Microcontroller's, ASIC's, or some such devices in them, performing some complicated task. However, in the case of the iDRAC6 enterprise module, it's nothing more than a fancy Broadcom nic card with an SD adapter built on top. The irony is not lost on me. The iDRAC6 express module, much to my amazement, is simply a 2GB Hynix eMMC NAND chip, wrapped up in a bit of plastic. I'm sure it's that damned plastic bit that costs so much money. It's either that or the quality printing on the labels
Now I don't know about you, but I have amassed a fair few of these damned things over the years, none of which have worked since that fateful moment, you know, the one where you wish you hadn't told your friend you would quickly perform the updates for him while there doing something else. To add insult, these things sell for anywhere between $25 and $90 each, depending on where you buy them from, so to have a box of them sitting useless is frustrating at the least. For me, this time was the last straw. I was determined to return them to service somehow and figured, if I can do it, so can anyone else reading this. I refuse to spend any more money fixing bad updates on these things, unless I really have no other choice
I won't bore you with the technical details, but if you care to dig deeper, look for the Jedec MMC and SD standards. The SD Card standard is actually a subset of the MMC standard, in fact, all SD Cards begin communication as if they were actually MMC cards. They then negotiate appropriate features etc and switch to SD mode. This little feature is really quite handy, as you'll see shortly, because it allows you to probe and prod an eMMC memory device directly, just as if it were an SD memory device, all with a minimal interface. This is exactly what I'm about to embark on here
I strongly suggest that if anyone follows in my footsteps to recover their own duff modules, that you use a USB Multi-Card Reader to connect to your Laptop/ PC, rather than using the direct SD slot. If, gawd forbid, you get anything wrong or miss spotting a short circuit etc, you'll be spared the embarrassment of telling your wife that you now have both a duff Laptop AND a box of non-functional iDRAC modules
First off, I need to "take one for the team" so to speak. I need to do this so I can locate suitable connection points on the iDRAC6 module in order to hook up my SD port. To this end, I need to pop the eMMC chip off the board. Don't worry, you'll not have to do this, it's a one time deal. When I get around to doing any more BGA rework, I'll probably just reball the chip and stick it back on the iDRAC6 module, just for a spare, but for the time being, this is what it looks like without it's clothes on:
Having now located suitable connection points on the module for my SD port, I can move on to building an adapter to let me actually connect up to the Laptop. Before I do though, let me just mention what you'll need if you want to build this yourself:
You'll need a Micro-SD card adapter, the cheaper the better, as the cheap ones are usually only glued together in a couple of spots.
You will also need some thin hook up wire, preferably different colours, so you don't get confused as to which wire is which. Any wire will do, as long as it's thin enough that it doesn't create a hump in the SD adapter casing. I robbed mine from an old Cisco console cable, which had a bad RJ plug on it. I needed to cut it off and re-terminate it anyway, so losing a few inches off the end for this is no biggie.
Also you'll need some basic hand tools and a soldering iron with a fine tip and better eyesight than I have to solder it all together!
Lastly you will need either a working iDRAC6 express module to copy from, or a suitable image file to dump back into your duff ones.
A word on wire type, length and connection to the module etc. Keep the wires really short, certainly less than 50mm end to end and hook them up to the module exactly where I have marked on the photo. I have chosen those points carefully to minimise cable reflections and impedance. If your wires are too long, or you use ribbon cable, instead of individual wires, there's a chance you will have unreliable communication, which is a baaaad thing
Here's where you need to connect up to on the iDRAC6 express module. Pay close attention to where each wire goes and watch for shorts/ solder splashes when you connect to them. I have colour coded them the same way as I wired my SD adapter to make it easier to follow:
So lets build the adapter:
Carefully crack open the Micro-SD adapter and remove the pin insert intact. Snip off the little crimped/ springy ends of the pins that would normally contact the micro card, just to make them sit nice and flat. Also snip off the three unused pins so you are not tempted to stick a wire on them by mistake.
Once you are satisfied that the wires are soldered on well and not shorting together, place the pin insert back in the carrier, laying the wires straight and flat, then pop the cover on with a few drops of super glue to hold the thing together. Place a line of glue over where the wires will contact the casing to strain relieve them, and be careful you don't glue your fingers on!
Only use super glue, anything else like hot melt etc, might introduce enough stray capacitance to ruin your efforts. Also, a possible gotcha with using super glue, is that it can run onto the pins, so inspect them after the glue dries and scrape them clean again if needed, or you'll be wondering why nothing happens when you plug it in.
If you look closely at the next couple of photos, you'll see exactly what goes where and how it should kind of look when your done with it:
If all has gone well so far, you should be able to fit the adapter easily into an SD slot, without having to put excessive pressure on it:
Next up, carefully hook up your wires, watching for shorts, to your working iDRAC6 express module, or skip ahead to the bad one(s), if you have a suitable image file already on hand. You will likely need to preheat the power and ground points with your soldering iron tip for a minute or so, before attempting to place the wires on, because they are attached to large copper planes internally in the module, which will soak up all the heat. A little patience is key here. Do not tug on the wires to see if they are soldered, you might pull up the pads and ruin the board. Here's what it should sort of look like when you're done:
Ok, so that's it for the hardware. Plug your express module into the card adapter, fire up your favourite flavour of Linux on your Laptop/ PC and plug your adapter into the USB port. Fingers crossed all went well and using fdisk you can now see your express module's partitions listed. Assuming that you do, use dd to image the whole disk, all 2GB of it and store the image somewhere safe. Now remove the card adapter from the USB port, pull your good express module and swap over the wires onto your bad one(s). You can then repeat this procedure as necessary, but this time use dd to write the duff express modules with the image file you just captured from the good one. Both operations will take a little time, because you are only using a minimal interface (single bit mode) to the device, but the iDRAC recovery really is as simple as that, assuming the eMMC on the module is actually working of course
Here's what mine looked like:
After you write your image back to the bad express module, remove the card reader and the express module. Desolder the wires and give it a quick clean with something like isopropyl to remove any accidental solder splashes and flux residue. Drop it back into your server and hold your breath...better yet, leave the room
Hopefully, all being well, you'll be greeted with the happy message and be able to, once again, access the iDRAC config menu:
As you can see, my test server is all over the map in terms of updates, but a quick run of the latest repo iso and bob's yer auntie, we're back in business:
If like me, you have a few of these things kicking about, that are not playing nice and are tired of wasting money on them and you have the time (fortunately I did this week), then this is a fun way to kill a couple of hours. It took me longer to actually document this, than it did to do it. The fact that I have recovered a pile of money in parts that were otherwise unusable, is a bonus in my eyes. I really don't know why I never got around to doing this before now
Obviously in any kind of production environment, one would simply order replacement parts, so I'm not suggesting that anyone consider this a goto fix, well, not unless you're really in a bind, but for the little guy on the cost conscious budget, this really is a money and time saver. The good thing is that pretty much all of the time and effort is taken up by building the SD adapter and capturing a working image. Once you have that step safely tucked away, you can turn a duff module round in 10 minutes whenever you need to in the future.
Anyway, I hope you enjoyed the read and managed to get something out of it...
Edited due to my being blissfully unaware that TinyPic went extinct, taking all my photos with it
Just an additional thought, deviating slightly here from the above instructions. Run the latest repo iso and bring your server up to date first using your good/ donor iDRAC6 module, then pull the now updated module, hook it up to the adapter and take an image of that with dd, that way you will have the latest good version to dump back into your confused ones. I didn't do that on my first pass through when I wrote this up, as I only had 1 old, but working, donor module and I didn't want to risk borking it before I had a working solution for recovery.
Imagine for a moment that you do a lot of work with older DELL servers. You need to perform some updates, bios etc and if you are anything like myself, you dread the thought of even breathing in the same room as it. You just never know what may break in the process, such is the nature of DELL updates
This brings me to the eternal issue of the failed iDRAC/ Lifecycle controller on these things, sticking two fingers up at you after an update. Sure, it all seems to go well, all the updates complete without error or warning and you have no reason to suspect that you should have stayed in bed that day. Upon a reboot you are greeted with a massive long boot time, howling fans which are quickly followed by this lovely image on screen:
You think to yourself, it's OK, I'll power cycle and drain off the flea power, nope. OK I'll redirect the com port, nope. OK I'll try the web interface, nada. A bit more investigation reveals that it doesn't even pull an IP address, sh1t, another $40 wasted. In fact, when you get to this point, there's no way to get back in to it, in order to fix it. If it won't talk to the server, you can't update it or roll it back. Basically your fancy remote access device has just turned into a "will never access it again device". After exhausting every possible option, you will likely be told by the "DELL Experts", that you have either goosed the iDRAC cards, if you're lucky and it has those, or the mainboard or both and will need replacements. How's that for spoiling your Monday morning coffee...
What I'm going to be dealing with here, specifically, is the iDRAC6 modules. These are not built onto the mainboard, they are pluggable modules. Here's what the express module looks like when the plastic carrier is removed:
With such grandiose names like "iDRAC6 Express" and "iDRAC6 Enterprise" one would be led to believe that the modules themselves had small Microcontroller's, ASIC's, or some such devices in them, performing some complicated task. However, in the case of the iDRAC6 enterprise module, it's nothing more than a fancy Broadcom nic card with an SD adapter built on top. The irony is not lost on me. The iDRAC6 express module, much to my amazement, is simply a 2GB Hynix eMMC NAND chip, wrapped up in a bit of plastic. I'm sure it's that damned plastic bit that costs so much money. It's either that or the quality printing on the labels
Now I don't know about you, but I have amassed a fair few of these damned things over the years, none of which have worked since that fateful moment, you know, the one where you wish you hadn't told your friend you would quickly perform the updates for him while there doing something else. To add insult, these things sell for anywhere between $25 and $90 each, depending on where you buy them from, so to have a box of them sitting useless is frustrating at the least. For me, this time was the last straw. I was determined to return them to service somehow and figured, if I can do it, so can anyone else reading this. I refuse to spend any more money fixing bad updates on these things, unless I really have no other choice
I won't bore you with the technical details, but if you care to dig deeper, look for the Jedec MMC and SD standards. The SD Card standard is actually a subset of the MMC standard, in fact, all SD Cards begin communication as if they were actually MMC cards. They then negotiate appropriate features etc and switch to SD mode. This little feature is really quite handy, as you'll see shortly, because it allows you to probe and prod an eMMC memory device directly, just as if it were an SD memory device, all with a minimal interface. This is exactly what I'm about to embark on here
I strongly suggest that if anyone follows in my footsteps to recover their own duff modules, that you use a USB Multi-Card Reader to connect to your Laptop/ PC, rather than using the direct SD slot. If, gawd forbid, you get anything wrong or miss spotting a short circuit etc, you'll be spared the embarrassment of telling your wife that you now have both a duff Laptop AND a box of non-functional iDRAC modules
First off, I need to "take one for the team" so to speak. I need to do this so I can locate suitable connection points on the iDRAC6 module in order to hook up my SD port. To this end, I need to pop the eMMC chip off the board. Don't worry, you'll not have to do this, it's a one time deal. When I get around to doing any more BGA rework, I'll probably just reball the chip and stick it back on the iDRAC6 module, just for a spare, but for the time being, this is what it looks like without it's clothes on:
Having now located suitable connection points on the module for my SD port, I can move on to building an adapter to let me actually connect up to the Laptop. Before I do though, let me just mention what you'll need if you want to build this yourself:
You'll need a Micro-SD card adapter, the cheaper the better, as the cheap ones are usually only glued together in a couple of spots.
You will also need some thin hook up wire, preferably different colours, so you don't get confused as to which wire is which. Any wire will do, as long as it's thin enough that it doesn't create a hump in the SD adapter casing. I robbed mine from an old Cisco console cable, which had a bad RJ plug on it. I needed to cut it off and re-terminate it anyway, so losing a few inches off the end for this is no biggie.
Also you'll need some basic hand tools and a soldering iron with a fine tip and better eyesight than I have to solder it all together!
Lastly you will need either a working iDRAC6 express module to copy from, or a suitable image file to dump back into your duff ones.
A word on wire type, length and connection to the module etc. Keep the wires really short, certainly less than 50mm end to end and hook them up to the module exactly where I have marked on the photo. I have chosen those points carefully to minimise cable reflections and impedance. If your wires are too long, or you use ribbon cable, instead of individual wires, there's a chance you will have unreliable communication, which is a baaaad thing
Here's where you need to connect up to on the iDRAC6 express module. Pay close attention to where each wire goes and watch for shorts/ solder splashes when you connect to them. I have colour coded them the same way as I wired my SD adapter to make it easier to follow:
So lets build the adapter:
Carefully crack open the Micro-SD adapter and remove the pin insert intact. Snip off the little crimped/ springy ends of the pins that would normally contact the micro card, just to make them sit nice and flat. Also snip off the three unused pins so you are not tempted to stick a wire on them by mistake.
Once you are satisfied that the wires are soldered on well and not shorting together, place the pin insert back in the carrier, laying the wires straight and flat, then pop the cover on with a few drops of super glue to hold the thing together. Place a line of glue over where the wires will contact the casing to strain relieve them, and be careful you don't glue your fingers on!
Only use super glue, anything else like hot melt etc, might introduce enough stray capacitance to ruin your efforts. Also, a possible gotcha with using super glue, is that it can run onto the pins, so inspect them after the glue dries and scrape them clean again if needed, or you'll be wondering why nothing happens when you plug it in.
If you look closely at the next couple of photos, you'll see exactly what goes where and how it should kind of look when your done with it:
If all has gone well so far, you should be able to fit the adapter easily into an SD slot, without having to put excessive pressure on it:
Next up, carefully hook up your wires, watching for shorts, to your working iDRAC6 express module, or skip ahead to the bad one(s), if you have a suitable image file already on hand. You will likely need to preheat the power and ground points with your soldering iron tip for a minute or so, before attempting to place the wires on, because they are attached to large copper planes internally in the module, which will soak up all the heat. A little patience is key here. Do not tug on the wires to see if they are soldered, you might pull up the pads and ruin the board. Here's what it should sort of look like when you're done:
Ok, so that's it for the hardware. Plug your express module into the card adapter, fire up your favourite flavour of Linux on your Laptop/ PC and plug your adapter into the USB port. Fingers crossed all went well and using fdisk you can now see your express module's partitions listed. Assuming that you do, use dd to image the whole disk, all 2GB of it and store the image somewhere safe. Now remove the card adapter from the USB port, pull your good express module and swap over the wires onto your bad one(s). You can then repeat this procedure as necessary, but this time use dd to write the duff express modules with the image file you just captured from the good one. Both operations will take a little time, because you are only using a minimal interface (single bit mode) to the device, but the iDRAC recovery really is as simple as that, assuming the eMMC on the module is actually working of course
Here's what mine looked like:
After you write your image back to the bad express module, remove the card reader and the express module. Desolder the wires and give it a quick clean with something like isopropyl to remove any accidental solder splashes and flux residue. Drop it back into your server and hold your breath...better yet, leave the room
Hopefully, all being well, you'll be greeted with the happy message and be able to, once again, access the iDRAC config menu:
As you can see, my test server is all over the map in terms of updates, but a quick run of the latest repo iso and bob's yer auntie, we're back in business:
If like me, you have a few of these things kicking about, that are not playing nice and are tired of wasting money on them and you have the time (fortunately I did this week), then this is a fun way to kill a couple of hours. It took me longer to actually document this, than it did to do it. The fact that I have recovered a pile of money in parts that were otherwise unusable, is a bonus in my eyes. I really don't know why I never got around to doing this before now
Obviously in any kind of production environment, one would simply order replacement parts, so I'm not suggesting that anyone consider this a goto fix, well, not unless you're really in a bind, but for the little guy on the cost conscious budget, this really is a money and time saver. The good thing is that pretty much all of the time and effort is taken up by building the SD adapter and capturing a working image. Once you have that step safely tucked away, you can turn a duff module round in 10 minutes whenever you need to in the future.
Anyway, I hope you enjoyed the read and managed to get something out of it...
Edited due to my being blissfully unaware that TinyPic went extinct, taking all my photos with it
Just an additional thought, deviating slightly here from the above instructions. Run the latest repo iso and bring your server up to date first using your good/ donor iDRAC6 module, then pull the now updated module, hook it up to the adapter and take an image of that with dd, that way you will have the latest good version to dump back into your confused ones. I didn't do that on my first pass through when I wrote this up, as I only had 1 old, but working, donor module and I didn't want to risk borking it before I had a working solution for recovery.
Last edited:
