When you image a partition with dd

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Fritz

Well-Known Member
Apr 6, 2015
3,372
1,375
113
69
How do you know it's actually happening?

I'm trying to image my old Linux Mint partition using dd and all I see if a blinking square cursor. No progress bar or any other indication that it's actually creating the image. I can't access the partition being imaged so maybe it is happening?
 

Markess

Well-Known Member
May 19, 2018
1,146
761
113
Northern California
If you don’t include one of the display status options in your command, you don’t see much while DD runs.

To get more detail on what’s happening, try including either “status=progress” or “pv” excluding quotation marks, in your command. Use only one though, not both. If you’ve done that already and still only getting blinking cursor, please post the command line you used. Maybe something else is an issue.

Cheers!

EDIT: looks like PV isn’t installed by default in Mint. So if you don’t want to install it, maybe try status=progress first.
 
Last edited:

Fritz

Well-Known Member
Apr 6, 2015
3,372
1,375
113
69
Looks like it trashed the destination drive. :(

EDIT: Did a repair of the file system and managed to recover the drive, sort of. :(

Seems that imaging a partition means something completely different under Linux. What's left on the drive is the partial image and all the original date is gone. I thought it would just create a file with the image in it, not destroy the drive and then clone the source partition to it.

Live and learn. :(
 
Last edited:
  • Sad
Reactions: Markess

Markess

Well-Known Member
May 19, 2018
1,146
761
113
Northern California
That sucks mightily.

DD is a superpowerful command that can do lots of stuff, but unfortunately has zero guardrails and the manpage is pretty complex at first look.

It CAN make an ISO image of a source drive and write it to your destination, leaving the existing files on the destination alone. Which I think is what you wanted?

But it can also clone a source drive onto a destination, making an exact copy of the source and deleting anything already on the destination. Which it seems like what happened?

For home use, where I'm not doing batch or scheduled operations, I tend to use a graphical archive manager to create/write single images whenever possible. For pretty much everything that I don't do regularly, I try to use the graphical tool if one is available. No matter how "skilled" I think I am with the terminal, I've still managed to wipe my data, or mess up my configuration, a time or two...or ten.
 
  • Like
Reactions: Fritz

Fritz

Well-Known Member
Apr 6, 2015
3,372
1,375
113
69
I'm guilty of not doing enough reading before diving in. Also should have searched for a gui tool. :(

I just built a new daily driver and wanted to keep a copy of the old install just in case. I was being overly cautious because I did a full backup before I started. Anyway, nothing lost but time.

I'm using a Supermicro X9SRH-F motherboard in the new computer. I was pleasantly surprised when it arrived with a E5 2697v2 installed. Thought about it for a minute and decided yea, this will work. Takes a bit longer to boot than a desktop board but since it runs all day that's no big deal.
 
  • Like
Reactions: Markess

Markess

Well-Known Member
May 19, 2018
1,146
761
113
Northern California
E5 is definitely the gift that keeps on giving. Not as expensive as scalable but still relatively powerful. Lots of PCIe lanes for stuff, and is well behaved with virtualization if you want to run a second OS in a VM, even with hardware passthrough. Plus, a ton of DDR3 RAM is pretty cheap. Enjoy!
 
  • Like
Reactions: Fritz

Fritz

Well-Known Member
Apr 6, 2015
3,372
1,375
113
69
Yep, definitely a lot of bang for the buck. I only put 64GB of ram in it. Half that would probably suffice, more would be wasted. It's in a Fractal Design R5 case so whisper quiet, even with an extra 120mm fan installed.
 
  • Like
Reactions: Markess

Stephan

Well-Known Member
Apr 21, 2017
920
698
93
Germany
Fritz, you can't use dd for this job. Was the partition mounted? Doubly bad...

Here is how to do it properly: Using a recent gnu-tar, to preserve also ACLs and extended attributes.

If not much is running, especially no SQL, you can try backing up a running root filesystem:

Bash:
#!/bin/sh

if [ "$(id -u 2>/dev/null)" -ne 0 ]; then
        echo "Need to be root, exiting..." >&2
        exit 1
fi

N="ionice -c3 nice -n19 chrt -i 0"

$N tar \
        --exclude='/swapfile' \
        --exclude='/var/swap/0' \
        --exclude='/home/fritz/Media/*' \
        --exclude='/home/fritz/.cache' \
        --use-compress-program="/usr/bin/lbzip2" \
        --create \
        --preserve-permissions \
        --one-file-system \
        --sparse \
        --numeric-owner \
        --acls \
        --xattrs \
        --xattrs-include='*' \
        --warning=no-file-ignored \
        --totals \
        --file=- \
        /boot / | ssh fritz@192.168.1.50 "cat > mint-backup.tar.bz2"
As you can see in this example, swap files in certain locations should also be excluded. Mostly for space reasons. Can be recreated later.

I like lbzip2 because performance is really good, uses all cores, compression ratio also good, and because literally everyone has bzip2 to decompress.

You obviously want to preserve permissions, skip any special filesystems so only do the root and /boot (usually a FAT16 partition with UEFI binaries to let BIOS boot). We want to keep sparse files with holes sparse. Numeric owner because the restoring Linux usually has no clue what usernames mean which may be present in the backup file. So we use numeric ids. Keep ACLs and extended attributes. Otherwise ping, which needs root for the raw network socket, will only be a normal non-suid binary. Then normal users will no longer be able to ping so keep that. Print totals. Pipe everything to a remote machine using SSH, delivering the backup into the home directory.

To restore, use:
Bash:
tar --xattrs --xattrs-include='*' --acls -xvjpf mint-backup.tar.bz2 -C /mnt/newrootdrive
If you have a new disk already in place and mounted, you could also use rsync with appropriate flags:

rsync -haxHAXDS8 --inplace --progress / /mnt/newroot
(mkfs and mount /boot...)
rsync -haxHAXDS8 --inplace --progress /boot/ /mnt/newroot/boot
Be mindful of the trailing "/" for the source. Otherwise /boot will be /boot/boot/...

Creating a new GPT EFI target disk is possible like so:
  • fdisk /dev/sdX
    • g
    • n 1 2048 +1024M
    • t 1 1 (EFI-System)
    • n + 3x RETURN
    • w
  • Create file-systems
    • mkfs.vfat -F16 -n EFI /dev/sdX1
    • mkfs.ext4 -m 1 -O metadata_csum,64bit /dev/sdX2
      • Caveat: metadata_csum requires CRC modules in the kernel or in ramdisk. I like errors=panic and ext4 checksums at least for the metadata. On Arch, I have therefore the line MODULES=(crc32c_generic crc32c-intel ...) in /etc/mkinitcpio.conf.
  • Mount everything in proper order
    • mount /dev/sdX2 /mnt
    • mkdir /mnt/boot
    • mount /dev/sdX1 /mnt/boot
These days except for laptops which I want it for encrypted hibernation, I use systemd-swap with zRAM instead of swap partition or swap file. YMMV. For laptops you dd to fill a file, can't use sparse files. For just swap, I use "fallocate -l 8G /swapfile" followed by "chmod 0600 /swapfile" and "mkswap /swapfile" together with fstab line "/swapfile none swap sw 0 0".
 
Last edited:
  • Like
Reactions: Fritz

Fritz

Well-Known Member
Apr 6, 2015
3,372
1,375
113
69
Thanks Stephan, all duly noted for the next time I need it. Imaging and cloning are much easier on the Windows side of the fence. Maybe someday Linux will catch up. Until then I'll slog through the forest as best I can. One thing I learned this time is that "image" means something different in Linux. In windows a partition is a file, In Linux apparently it's a clone. Starting out I never intended to destroy the data already on the destination drive. There was plenty of empty space to hold everything in the source partition. Biggest thing I learned is that dd is as dumb as I am and won't warn you if you're about to screw up. Next time I'll search for a gui solution.
 

Stephan

Well-Known Member
Apr 21, 2017
920
698
93
Germany
While I never used it for Linux, maybe try out Drive Snapshot - Disk Image Backup for Windows NT/2000/XP/2003/X64 by booting a recent Windows ISO and start it from recovery option command prompt and a USB thumb drive. Has a GUI. Claims it can do ext4.

What was the aim, transfer that Mint installation to a new drive?

Yes, dd is dangerous. Recent dd from coreutils 8.24+ also has progress display. Harmless test: dd if=/dev/urandom of=/dev/null status=progress
 

Markess

Well-Known Member
May 19, 2018
1,146
761
113
Northern California
Imaging and cloning are much easier on the Windows side of the fence. Maybe someday Linux will catch up.
I bet some of that is just because you're more used to Windows. Plus, if you compare using a graphical program for Windows with using linux from a terminal, Windows wins hands down. If you compare doing it Windows from C:\ instead, I bet they stack up pretty equally ...i.e. both ways are tuff if you aren't used to them.

Here's my recommendation for doing it in Linux without having a PhD., and using only the tools you already have. This assumes you still have the Mint live USB that you used to install Mint on your new build:

1. Boot from the live USB..Using the live USB avoids any potential isssues with trying to image the disk the OS is on while you're using it. Linux is generally pretty picky about doing stuff like that to the disk the OS is on while the OS is running. So easiest to boot from the Live USB.
2. Open "Disks" from the menu. I'm not sure what sub-menu its in with Mint (in Ubuntu Mate its in "Preferences", but in Mint it may be "Accessories"?)
3. Identify the disk you recently installed Mint on (or Mint and Windows if you did both) that you want to create an image of. Click on the disk in the left hand pane.
4. Click the "more" icon"(the 3 stacked elipses) and select "Create Disk Image".
5. Name the image, and pick where to save it. If you're saving it to a second hard drive in your system or external drive, you may want to check ahead of time that its mounted in the live session. You can check that, and mount the destination drive if necessary, right there in Disks. The Mint live USB will often auto-mount all the disks it finds, but best to check.
6. Click "Start Creating" or "Start", or etc. If you've got a sizable install....it could take a while, but you'll get a progress bar.

Pictures here are just for reference. It may look a bit different for you as I'm on Ubuntu Mate, but its the same underlying program:
Screenshot at 2022-11-07 15-19-39.png

Screenshot at 2022-11-07 15-15-41.png

Not overly hard, eh? Sometimes I think that there'd be more people adopting desktop Linux if more tutorials referred to available graphical tools. After all, there's always a lot of tutorials on how to do things in Linux from the terminal, while almost nobody publishes tutorials on how to accomplish things in Windows from the command prompt.
 
Last edited:
  • Like
Reactions: Fritz

Stephan

Well-Known Member
Apr 21, 2017
920
698
93
Germany
Nota Bene: I also use command line on Windows...

@echo off
cd /d %~dp0
powercfg /s 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c
md Scratch
dism /Capture-Image /ScratchDir:%~dp0Scratch /ImageFile:intel-nuc7i5dnke-win10-pro-2004-20200910.wim /CaptureDir:C:\ /Name:Win10 /Compress:fast /Verify

;)
 
  • Haha
Reactions: Markess

Fritz

Well-Known Member
Apr 6, 2015
3,372
1,375
113
69
I bet some of that is just because you're more used to Windows. Plus, if you compare using a graphical program for Windows with using linux from a terminal, Windows wins hands down. If you compare doing it Windows from C:\ instead, I bet they stack up pretty equally ...i.e. both ways are tuff if you aren't used to them.

Here's my recommendation for doing it in Linux without having a PhD., and using only the tools you already have. This assumes you still have the Mint live USB that you used to install Mint on your new build:

1. Boot from the live USB..Using the live USB avoids any potential isssues with trying to image the disk the OS is on while you're using it. Linux is generally pretty picky about doing stuff like that to the disk the OS is on while the OS is running. So easiest to boot from the Live USB.
2. Open "Disks" from the menu. I'm not sure what sub-menu its in with Mint (in Ubuntu Mate its in "Preferences", but in Mint it may be "Accessories"?)
3. Identify the disk you recently installed Mint on (or Mint and Windows if you did both) that you want to create an image of. Click on the disk in the left hand pane.
4. Click the "more" icon"(the 3 stacked elipses) and select "Create Disk Image".
5. Name the image, and pick where to save it. If you're saving it to a second hard drive in your system or external drive, you may want to check ahead of time that its mounted in the live session. You can check that, and mount the destination drive if necessary, right there in Disks. The Mint live USB will often auto-mount all the disks it finds, but best to check.
6. Click "Start Creating" or "Start", or etc. If you've got a sizable install....it could take a while, but you'll get a progress bar.

Pictures here are just for reference. It may look a bit different for you as I'm on Ubuntu Mate, but its the same underlying program:
View attachment 25212

View attachment 25213

Not overly hard, eh? Sometimes I think that there'd be more people adopting desktop Linux if more tutorials referred to available graphical tools. After all, there's always a lot of tutorials on how to do things in Linux from the terminal, while almost nobody publishes tutorials on how to accomplish things in Windows from the command prompt.
I did not know that. Yes, it's the same in Mint. I would have definitely used it if I knew it was there. Funny how my Googling didn't turn up the first reference to it.

One great thing about Linux is that if you trash your install you don't have to get permission from someone in Redmond to reinstall it and no activation or license key to keep up with.
 
  • Like
Reactions: Markess

Sean Ho

seanho.com
Nov 19, 2019
768
352
63
Vancouver, BC
seanho.com
Good learning experience; glad your backups are working properly! dd can certainly copy the contents of a block device (e.g., unmounted partition) to a file on a mounted file system, e.g.

Code:
dd if=/dev/sda1 of=/mnt/otherdisk/partition.img
but as Stephan noted you usually wouldn't want to do this; it'd be a 1:1 copy of the partition including unused space, uncompressed. tar and actual backup software like restic, borg, etc. would copy the contents of the filesystem, rather than the entire block device.
 
  • Like
Reactions: Markess

Markess

Well-Known Member
May 19, 2018
1,146
761
113
Northern California
Funny how my Googling didn't turn up the first reference to it.
There's a core group of programs that run via terminal sessions and are common among pretty much all linux distributions. DD is one of them, so you will see a lot of tutorials on how to do things with DD. Its not always the best program for a task, but the chances are really good that it will be in your linux install.

But, any mature Linux Desktop Environment (like Mint) will have its own suite of GUI based tools, and there are many other GUI based programs available to do most everything else you'd want to do. You can always use a terminal session for managing software firewall rules, network configuration, printers, displays, sound and all kinds of other stuff. But, there's GUI tools available for each as well. Some are just GUI wrappers for the terminal based programs, but they make them easier to use and give you some common sense guardrails.

So, if you just want to get something done and don't want to have to learn the ins and outs of a terminal based program, you can always go GUI.

Before someone mentions it, I will say that there's a lot of situations where you want the terminal over a GUI. If you're an IT guy and need to do repetitive operations or make the identical configuration change in 1000 systems, you'll be very glad for scripts and the terminal.
 

nabsltd

Active Member
Jan 26, 2022
339
207
43
I'm using a Supermicro X9SRH-F motherboard in the new computer. I was pleasantly surprised when it arrived with a E5 2697v2 installed. Thought about it for a minute and decided yea, this will work. Takes a bit longer to boot than a desktop board but since it runs all day that's no big deal.
My daily driver for the past 6 years has been an X9-SRA. I'm just about to retire it for an X11SRA, since anything newer from Intel has fewer than 24 PCIe lanes in the form of slots. I run a GPU and a LSI RAID card, so that's 24 lanes right there. I could get by with a board that does x16/x0 or x8/x8 in two slots, but even those are hard to find these days. Most of the LGA1700 boards are x16/x4 and maybe some PCH slots. There are a few with more, but those are very expensive, and without digging into manuals, it's hard to tell what happens when you populate every slot.
 
  • Like
Reactions: Markess and Fritz

Stephan

Well-Known Member
Apr 21, 2017
920
698
93
Germany
Funny how my Googling didn't turn up the first reference to it.
Not funny, fact. Can't tell you how many times I've been googling an SMD chip marking, flash chip, looked for some USB hub chipset or similar and Google only came up with a bunch of unhelpful dung. Turned to Bing, not much better. Image search worse on Bing. Want to know who I turn to for tech stuff since 12-18 months? Yandex. Yep, the Russians. "Finds everything..." well not quite but a 100 times better than what Google seems to be doing. I don't know what is up with Google but major mojo is gone. Previously it found stuff and most helpful hit was in 1st half of 1st page. Now all it wants to do is sell me new pants and shrimp farm equipment. Also helpful so let me mention it is archive.org. For pages that are gone in 2022. This is why I can only recommend to run your own ZFS uhh, shrimp farm. Bitrot prevention included. I am an avid downloader of stuff I find useful or insightful. Tomorrow the file or YT video could be gone forever. Next time recommend to try Yandex.
 
  • Like
Reactions: Fritz

Fritz

Well-Known Member
Apr 6, 2015
3,372
1,375
113
69
Not funny, fact. Can't tell you how many times I've been googling an SMD chip marking, flash chip, looked for some USB hub chipset or similar and Google only came up with a bunch of unhelpful dung. Turned to Bing, not much better. Image search worse on Bing. Want to know who I turn to for tech stuff since 12-18 months? Yandex. Yep, the Russians. "Finds everything..." well not quite but a 100 times better than what Google seems to be doing. I don't know what is up with Google but major mojo is gone. Previously it found stuff and most helpful hit was in 1st half of 1st page. Now all it wants to do is sell me new pants and shrimp farm equipment. Also helpful so let me mention it is archive.org. For pages that are gone in 2022. This is why I can only recommend to run your own ZFS uhh, shrimp farm. Bitrot prevention included. I am an avid downloader of stuff I find useful or insightful. Tomorrow the file or YT video could be gone forever. Next time recommend to try Yandex.
Thanks. I'll try Yandex. I agree about Google. Used to be you could search a Supermicro MB model number and the first link was to the relevant page on SM site. Not anymore.
 

Markess

Well-Known Member
May 19, 2018
1,146
761
113
Northern California
Thanks. I'll try Yandex. I agree about Google. Used to be you could search a Supermicro MB model number and the first link was to the relevant page on SM site. Not anymore.
I won't disagree about Google! Especially when searching products, which in the last few months has gotten almost as bad as Amazon. You have to page through a bunch of ad supported/algorithm selected non-applicable, stuff to find the first hit that includes what you want.

But, on the original topic of getting something done in Linux Mint, I suggest hitting the Mint forums directly and entering your search term in the search bar there: Linux Mint Forums - Index page

For programs that need to run in the terminal (i.e. DD and etc.), if you don't find a good answer on the Mint forums, try the Arch Wiki ArchWiki or Arch Forums Arch Linux Forums and enter your search term there. Its true that architecture-wise, Arch is nothing like Mint, but the behavior of terminal based programs like DD is similar enough in Arch that often the Arch response will work...and the Arch forums are VERY VERY extensive on the topic of getting things done via the terminal.

Mint is based on Ubuntu, so you can also try that forum Ubuntu Forums or Ask Ubuntu Newest Questions

When it comes to info from each of these Forums, it seems to me a Google search is more likely to turn up where to buy Mint scented air fresheners than steps on how to refresh a frozen screen in Linux Mint (I'm kind of proud of that witicism :p ). Searching on more than one individual forum if required is more effort, but in my opinion is 99% more likely to produce useful results.

Just my 2 cents.
 
  • Haha
Reactions: Fritz

Sean Ho

seanho.com
Nov 19, 2019
768
352
63
Vancouver, BC
seanho.com
I've been increasingly using DDG recently, but even Google has no issues looking up manuals for SM boards or other server parts: x11sra-f manual filetype:pdf .

For Linux CLI: good old man (or Linux man pages or man7.org). I find the SNR to be a bit low on the Ubuntu forums; sometimes a bit of the blind leading the blind. ArchWiki is fantastic, even if not using Arch.

I empathise that SMD chips can be hard to look up due to non-specific part numbering. Digikey, Mouser, et al are helpful but still struggle with this.

For ZFS: TrueNAS forums, ZoL mailing list, or even posts right here by @ gea and other ZFS wizards.