Question, about dd and/or qemu, for disk cloning

zapper · 2023-09-12 18:51:15

Anyone know how to copy an entire hard drive onto a qcow2 image but with only the space being used as being counted? This would truly be helpful to know.

Would save me a lot of irritations lol.

I need it to be done this way btw, other solutions likely would be more messy or have too many dependencies that can't be supported. This has to work for both Hyperbola and Devuan.

~~Like if an image has 500GB of space as the maximum being possible to use but since only 100 is used, only 100 is stored on the disk.~~

~~qemu-img create -f qcow2 your-img 500GB does something similar more or less.~~

~~Edit: Well as far as creating the image and then you install, etc...~~

~~But I have files I want to keep on current drive... so... yeah.~~

Forget what I wrote there, I have a better explanation:

Like say 200GB is used on hdd, but 450 is available on hdd.

I want only the 200GB to be counted but expansion be possible at the same time.

That is what I actually desire in a qemu or dd solution.

I want to be able to do either this way.

I tried a qemu method, but it was already at 9GB and had only completed 2% of the job.

This was what I did:
qemu-img convert -c -O qcow2 /dev/sdXn backup.qcow2

It would have copied the whole disk including the unused space. :S

Last edited by zapper (2023-09-13 19:33:20)

delgado · 2023-09-12 20:07:22

https://unix.stackexchange.com/question … -browse-re

shows some examples using e2image, also for a qcow2- image. Not sure whether this works for your case too.

zapper · 2023-09-13 05:30:00

delgado wrote:

https://unix.stackexchange.com/question … -browse-re
shows some examples using e2image, also for a qcow2- image. Not sure whether this works for your case too.

Sadly no, but I did find something on that page, that might work.

I will just have to see though...

boughtonp · 2023-09-13 16:11:49

zapper wrote:

Anyone know how to copy an entire hard drive onto a qcow2 image but with only the space being used as being counted?

No, but I have spent a significant amount of time trying various solutions that didn't work, before eventually giving up and converting each partition to its own file, with qemu-image convert.

Last edited by boughtonp (2023-09-13 16:13:11)

zapper · 2023-09-13 19:29:13

boughtonp wrote:

zapper wrote:
Anyone know how to copy an entire hard drive onto a qcow2 image but with only the space being used as being counted?
No, but I have spent a significant amount of time trying various solutions that didn't work, before eventually giving up and converting each partition to its own file, with qemu-image convert.

Hmm... that method you mentioned didn't work for me, for some reason.

You know what I mean though, right?

Like say 200GB is used but 450 is available on hdd.

I want only the 200GB to be counted but expansion be possible at the same time.

I am guessing you understand what I am saying, but felt I should clarify anyways. Especially for other people.

I might add to the first post...

boughtonp · 2023-09-13 22:18:25

Yeah, like if you had this situation:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   20G   30G  40% /
/dev/sda2       450G   80G  270G  18% /home

You'd want a single 100GB sparse file, which still retains the 500GB capacity.

ralph.ronnquist · 2023-09-13 23:07:47

@zapper, I think you can use the zerofree program for ext[234] filesystems to get all unused blocks of their partitions to be zeroed. Perhaps similar tools exist for other filesystems, if that's what you have.

Then according to its man page, qemu-img has the -S size argument whereby it translates contiguous runs of zeros as holes.

And dd can do something similar with the conv=sparse argument, though only into a sparse raw image file.

boughtonp · 2023-09-14 12:09:17

Heh, I don't recall encountering zerofree before - I wonder if that might have been the missing step that stopped some of my attempts from working.

delgado · 2023-09-14 13:32:41

The point of zerofree (or just doing cat /dev/zero>zero.bin; rm zero.bin) is to reduce the size of a packed image file created with
dd or cat, e.g. cat /dev/sdXn | gzip >sdXn.img.gz.

When creating a partition image file as sparse file, these sectors are omitted anyway.

zapper · 2023-09-15 22:45:17

so zerofree would work for what I want? Is that correct?

I guess I should try this then?

zapper · 2023-09-15 22:48:10

boughtonp wrote:

Yeah, like if you had this situation:
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   20G   30G  40% /
/dev/sda2       450G   80G  270G  18% /home
You'd want a single 100GB sparse file, which still retains the 500GB capacity.

Correct, but actually, I may have changed my mind slightly, is there any easy solution in devuan included in doing this? Even that might be helpful. Preferably one that isn't mega slow.

2 hours is fine, anything longer would be painful.

500GB disk btw.

alphalpha · 2023-09-15 23:14:21

you can copy the whole disc and then shrink the qcow

ralph.ronnquist · 2023-09-15 23:25:33

You can use dd to create a 500G "empty" sparse file:

dd if=/dev/zero of=/mysparsefile bs=1G count=0 seek=500

Though after that you still have the problem of identifying which blocks on /dev/sda are in use so as to copy those. If you have run zerofree, i.e. made all unused blocks be zeroed, then another dd would do the job, e.g.

dd if=/dev/sda2 of=/mysparsefile conv=sparse,notrunc

Obviously mysparsefile needs to be somewhere with enough space but not on /dev/sda (as it then will fill up the 500G by copying its own blocks)

zapper · 2023-09-16 04:46:52

alphalpha wrote:

you can copy the whole disc and then shrink the qcow

I have tried that, but for some reason, errors occurred for me and so I gave it up.

Maybe I am missing something?

fest3er · 2023-10-16 16:54:51

I haven't tried this, but it ought to work: https://gist.github.com/shamil/62935d9b456a6f9877b5.

In short:

modprobe nbd
Create the QCOW2 image same (virtual) size as the HD.
Connect to the QCOW2 (see link above).
Mount the HD's filesystems.
On each of these FSen, dd if=/dev/zero of=null.zero bs=1024k; sync; rm null.zero to use up all free blocks, fill them with nulls, then erase the file. Neither perfect nor complete, but it'll zero most free blocks. You will have to <CTRL/C> dd if it doesn't quit when it exhausts free space. (This'll work on all FSen, not just EXT[234].) Also be aware that some FSen may still have limits on the maximum file size; you might have to create several nullX.zero files.
Unmount the HD's filesystems.
As root, dd --sparse the raw HD device to the QCOW2 NBD device.
Disconnect from the QCOW2 (see link above)
rmmod nbd

Given the price and size of storage these days (and sufficient funds), it might be better to buy a large SSD or NVME and copy the HD onto a corner of it. If, of course, the above doesn't work well.

Whichever way you use, it will take quite some time. 500GB at 1.5 Gb/s ain't fast. And you're doing it twice (once to zero free blocks, and again to copy the drive).

zapper · 2023-10-17 20:35:04

I use ivy bridge fest3er, so it might be more like 300mb per second depending on the drive lol.

The officially official Devuan Forum!

#1 2023-09-12 18:51:15

Question, about dd and/or qemu, for disk cloning

#2 2023-09-12 20:07:22

Re: Question, about dd and/or qemu, for disk cloning

#3 2023-09-13 05:30:00

Re: Question, about dd and/or qemu, for disk cloning

#4 2023-09-13 16:11:49

Re: Question, about dd and/or qemu, for disk cloning

#5 2023-09-13 19:29:13

Re: Question, about dd and/or qemu, for disk cloning

#6 2023-09-13 22:18:25

Re: Question, about dd and/or qemu, for disk cloning

#7 2023-09-13 23:07:47

Re: Question, about dd and/or qemu, for disk cloning

#8 2023-09-14 12:09:17

Re: Question, about dd and/or qemu, for disk cloning

#9 2023-09-14 13:32:41

Re: Question, about dd and/or qemu, for disk cloning

#10 2023-09-15 22:45:17

Re: Question, about dd and/or qemu, for disk cloning

#11 2023-09-15 22:48:10

Re: Question, about dd and/or qemu, for disk cloning

#12 2023-09-15 23:14:21

Re: Question, about dd and/or qemu, for disk cloning

#13 2023-09-15 23:25:33

Re: Question, about dd and/or qemu, for disk cloning

#14 2023-09-16 04:46:52

Re: Question, about dd and/or qemu, for disk cloning

#15 2023-10-16 16:54:51

Re: Question, about dd and/or qemu, for disk cloning

#16 2023-10-17 20:35:04

Re: Question, about dd and/or qemu, for disk cloning

Board footer