The officially official Devuan Forum!

You are not logged in.

#1 2023-09-12 18:51:15

zapper
Member
Registered: 2017-05-29
Posts: 972  

Question, about dd and/or qemu, for disk cloning

Anyone know how to copy an entire hard drive onto a qcow2 image but with only the space being used as being counted? This would truly be helpful to know.

Would save me a lot of irritations lol.

I need it to be done this way btw, other solutions likely would be more messy or have too many dependencies that can't be supported. This has to work for both Hyperbola and Devuan.

Like if an image has 500GB of space as the maximum being possible to use but since only 100 is used, only 100 is stored on the disk.

qemu-img create -f qcow2 your-img 500GB does something similar more or less. wink

Edit: Well as far as creating the image and then you install, etc...

But I have files I want to keep on current drive... so... yeah.

Forget what I wrote there, I have a better explanation:


Like say  200GB is used on hdd, but 450 is available on hdd.

I want only the 200GB to be counted but expansion be possible at the same time.

That is what I actually desire in a qemu or dd solution.

I want to be able to do either this way.

I tried a qemu method, but it was already at 9GB and had only completed 2% of the job.

This was what I did:
qemu-img convert -c -O qcow2 /dev/sdXn backup.qcow2

It would have copied the whole disk including the unused space. :S

Last edited by zapper (2023-09-13 19:33:20)


Freedom is never more than one generation away from extinction. Feelings are not facts
If you wish to be humbled, try to exalt yourself long term  If you wish to be exalted, try to humble yourself long term
Favourite operating systems: Hyperbola Devuan OpenBSD
Peace Be With us All!

Offline

#2 2023-09-12 20:07:22

delgado
Member
Registered: 2022-07-14
Posts: 216  

Re: Question, about dd and/or qemu, for disk cloning

https://unix.stackexchange.com/question … -browse-re

shows some examples using e2image, also for a qcow2- image. Not sure whether this works for your case too.

Offline

#3 2023-09-13 05:30:00

zapper
Member
Registered: 2017-05-29
Posts: 972  

Re: Question, about dd and/or qemu, for disk cloning

delgado wrote:

https://unix.stackexchange.com/question … -browse-re

shows some examples using e2image, also for a qcow2- image. Not sure whether this works for your case too.

Sadly no, but I did find something on that page, that might work.

I will just have to see though...


Freedom is never more than one generation away from extinction. Feelings are not facts
If you wish to be humbled, try to exalt yourself long term  If you wish to be exalted, try to humble yourself long term
Favourite operating systems: Hyperbola Devuan OpenBSD
Peace Be With us All!

Offline

#4 2023-09-13 16:11:49

boughtonp
Member
From: UK
Registered: 2023-01-19
Posts: 212  
Website

Re: Question, about dd and/or qemu, for disk cloning

zapper wrote:

Anyone know how to copy an entire hard drive onto a qcow2 image but with only the space being used as being counted?

No, but I have spent a significant amount of time trying various solutions that didn't work, before eventually giving up and converting each partition to its own file, with qemu-image convert.

Last edited by boughtonp (2023-09-13 16:13:11)


3.1415P265E589T932E846R64338

Offline

#5 2023-09-13 19:29:13

zapper
Member
Registered: 2017-05-29
Posts: 972  

Re: Question, about dd and/or qemu, for disk cloning

boughtonp wrote:
zapper wrote:

Anyone know how to copy an entire hard drive onto a qcow2 image but with only the space being used as being counted?

No, but I have spent a significant amount of time trying various solutions that didn't work, before eventually giving up and converting each partition to its own file, with qemu-image convert.

Hmm... that method you mentioned didn't work for me, for some reason.

You know what I mean though, right?

Like say 200GB is used but 450 is available on hdd.

I want only the 200GB to be counted but expansion be possible at the same time.

I am guessing you understand what I am saying, but felt I should clarify anyways. Especially for other people.

I might add to the first post...


Freedom is never more than one generation away from extinction. Feelings are not facts
If you wish to be humbled, try to exalt yourself long term  If you wish to be exalted, try to humble yourself long term
Favourite operating systems: Hyperbola Devuan OpenBSD
Peace Be With us All!

Offline

#6 2023-09-13 22:18:25

boughtonp
Member
From: UK
Registered: 2023-01-19
Posts: 212  
Website

Re: Question, about dd and/or qemu, for disk cloning

Yeah, like if you had this situation:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   20G   30G  40% /
/dev/sda2       450G   80G  270G  18% /home

You'd want a single 100GB sparse file, which still retains the 500GB capacity.


3.1415P265E589T932E846R64338

Offline

#7 2023-09-13 23:07:47

ralph.ronnquist
Administrator
From: Battery Point, Tasmania, AUS
Registered: 2016-11-30
Posts: 1,268  

Re: Question, about dd and/or qemu, for disk cloning

@zapper, I think you can use the zerofree program for ext[234] filesystems to get all unused blocks of their partitions to be zeroed. Perhaps similar tools exist for other filesystems, if that's what you have.

Then according to its man page,  qemu-img has the -S size argument whereby it translates contiguous runs of zeros as holes.

And dd can do something similar with the conv=sparse argument, though only into a sparse raw image file.

Offline

#8 2023-09-14 12:09:17

boughtonp
Member
From: UK
Registered: 2023-01-19
Posts: 212  
Website

Re: Question, about dd and/or qemu, for disk cloning

Heh, I don't recall encountering zerofree before - I wonder if that might have been the missing step that stopped some of my attempts from working.


3.1415P265E589T932E846R64338

Offline

#9 2023-09-14 13:32:41

delgado
Member
Registered: 2022-07-14
Posts: 216  

Re: Question, about dd and/or qemu, for disk cloning

The point of zerofree (or just doing cat /dev/zero>zero.bin; rm zero.bin) is to reduce the size of a packed image file created with
dd or cat, e.g. cat /dev/sdXn | gzip >sdXn.img.gz.

When creating a partition image file as sparse file, these sectors are omitted anyway.

Offline

#10 2023-09-15 22:45:17

zapper
Member
Registered: 2017-05-29
Posts: 972  

Re: Question, about dd and/or qemu, for disk cloning

so zerofree would work for what I want? Is that correct?

I guess I should try this then?


Freedom is never more than one generation away from extinction. Feelings are not facts
If you wish to be humbled, try to exalt yourself long term  If you wish to be exalted, try to humble yourself long term
Favourite operating systems: Hyperbola Devuan OpenBSD
Peace Be With us All!

Offline

#11 2023-09-15 22:48:10

zapper
Member
Registered: 2017-05-29
Posts: 972  

Re: Question, about dd and/or qemu, for disk cloning

boughtonp wrote:

Yeah, like if you had this situation:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   20G   30G  40% /
/dev/sda2       450G   80G  270G  18% /home

You'd want a single 100GB sparse file, which still retains the 500GB capacity.

Correct, but actually, I may have changed my mind slightly, is there any easy solution in devuan included in doing this? Even that might be helpful. Preferably one that isn't mega slow.

2 hours is fine, anything longer would be painful.

500GB disk btw.


Freedom is never more than one generation away from extinction. Feelings are not facts
If you wish to be humbled, try to exalt yourself long term  If you wish to be exalted, try to humble yourself long term
Favourite operating systems: Hyperbola Devuan OpenBSD
Peace Be With us All!

Offline

#12 2023-09-15 23:14:21

alphalpha
Member
From: Germany
Registered: 2018-01-23
Posts: 148  

Re: Question, about dd and/or qemu, for disk cloning

you can copy the whole disc and then shrink the qcow

Offline

#13 2023-09-15 23:25:33

ralph.ronnquist
Administrator
From: Battery Point, Tasmania, AUS
Registered: 2016-11-30
Posts: 1,268  

Re: Question, about dd and/or qemu, for disk cloning

You can use dd to create a 500G "empty" sparse file:

dd if=/dev/zero of=/mysparsefile bs=1G count=0 seek=500

Though after that you still have the problem of identifying which blocks on /dev/sda are in use so as to copy those. If you have run zerofree, i.e. made all unused blocks be zeroed, then another dd would do the job, e.g.

dd if=/dev/sda2 of=/mysparsefile conv=sparse,notrunc

Obviously mysparsefile needs to be somewhere with enough space but not on /dev/sda (as it then will fill up the 500G by copying its own blocks)

Offline

#14 2023-09-16 04:46:52

zapper
Member
Registered: 2017-05-29
Posts: 972  

Re: Question, about dd and/or qemu, for disk cloning

alphalpha wrote:

you can copy the whole disc and then shrink the qcow

I have tried that, but for some reason, errors occurred for me and so I gave it up.

Maybe I am missing something?


Freedom is never more than one generation away from extinction. Feelings are not facts
If you wish to be humbled, try to exalt yourself long term  If you wish to be exalted, try to humble yourself long term
Favourite operating systems: Hyperbola Devuan OpenBSD
Peace Be With us All!

Offline

#15 2023-10-16 16:54:51

fest3er
Member
Registered: 2022-07-06
Posts: 8  

Re: Question, about dd and/or qemu, for disk cloning

I haven't tried this, but it ought to work: https://gist.github.com/shamil/62935d9b456a6f9877b5.

In short:

  1. modprobe nbd

  2. Create the QCOW2 image same (virtual) size as the HD.

  3. Connect to the QCOW2 (see link above).

  4. Mount the HD's filesystems.

  5. On each of these FSen, dd if=/dev/zero of=null.zero bs=1024k; sync; rm null.zero to use up all free blocks, fill them with nulls, then erase the file. Neither perfect nor complete, but it'll zero most free blocks. You will have to <CTRL/C> dd if it doesn't quit when it exhausts free space. (This'll work on all FSen, not just EXT[234].) Also be aware that some FSen may still have limits on the maximum file size; you might have to create several nullX.zero files.

  6. Unmount the HD's filesystems.

  7. As root, dd --sparse the raw HD device to the QCOW2 NBD device.

  8. Disconnect from the QCOW2 (see link above)

  9. rmmod nbd

Given the price and size of storage these days (and sufficient funds), it might be better to buy a large SSD or NVME and copy the HD onto a corner of it. If, of course, the above doesn't work well.

Whichever way you use, it will take quite some time. 500GB at 1.5 Gb/s ain't fast. And you're doing it twice (once to zero free blocks, and again to copy the drive).

Offline

#16 2023-10-17 20:35:04

zapper
Member
Registered: 2017-05-29
Posts: 972  

Re: Question, about dd and/or qemu, for disk cloning

I use ivy bridge fest3er, so it might be more like 300mb per second depending on the drive lol.


Freedom is never more than one generation away from extinction. Feelings are not facts
If you wish to be humbled, try to exalt yourself long term  If you wish to be exalted, try to humble yourself long term
Favourite operating systems: Hyperbola Devuan OpenBSD
Peace Be With us All!

Offline

Board footer