Refracta-SnapShot - Question

grafiksinc · 2020-08-16 05:58:41

Hello Forum- before I go into detail about SnapShot. I wanted to make sure I am in the right spot.
Is this the correct place to pose a question about this package?

Thank you in advance

fsmithred · 2020-08-16 13:16:14

Sometimes questions about refracta tools appear in the Derivatives section, sometimes in DIY and sometimes in Installation. You have my attention, so right here will also work.

Here's the documentation for it:
https://refracta.org/docs/readme.refractasnapshot.txt
https://refracta.org/docs/refractasnapshot.conf.txt
https://refracta.org/docs/snapshot_exclude.list.txt

grafiksinc · 2020-08-18 11:59:37

@fsmithred – thanks! For the help.
Let me preface this as using snapshot with the refracta-installer everything works as expected.

Version of SnapShot being used:
refractasnapshot-base_10.2.10_all.deb
refractasnapshot-gui_10.2.10_all.deb

Operating System environments tested:
Ubuntu 20.04
Debian 10 (Buster)
Both running in Virtual Box

The problem below happens on both distros with the exact same error log

Problem:
I want to take a “snapshot” of my build, however I want to use “calamares” as the installer.
I also want to use the “calamares-settings-debian (10.0.20-1+deb10u3)“ package for my installer setup.
That package is located here: https://packages.debian.org/stable/cala … ngs-debian
Now…. these settings install on both distros listed above and run with no problem, and the settings work just fine with calamares.

***NOTE: Before setting up calamares with the “calamares-settings-debian” package I ran snapshot and it completed successfully using the “01 Create a snapshot” option***

After setting up calamares and then applying the “calamares-settings-debian” package snapshot does Not complete. It errors out with the following error:

+ [[ -n '' ]]
+ '[' yes = yes ']'
+ clean_initrd
+ extract_initrd
+ mkdir /tmp/extracted
+ pushd /tmp/extracted
++ egrep -o 'gzip compressed|XZ compressed|cpio archive'
++ file -L /boot/initrd.img
+ COMPRESSION='cpio archive'
+ '[' 'cpio archive' = 'gzip compressed' ']'
+ '[' 'cpio archive' = 'XZ compressed' ']'
+ '[' 'cpio archive' = 'cpio archive' ']'
+ echo 'Archive is cpio archive...'
+ cpio -i
62 blocks
+ cpio -i
+ zcat

gzip: stdin: not in gzip format
cpio: premature end of archive
+ exit_code=2
+ '[' 2 -ne 0 ']'
+ cpio -i
cpio: kernel/x86/microcode/AuthenticAMD.bin not created: newer or same age version exists
62 blocks
+ cpio -i
+ xzcat
xzcat: (stdin): File format not recognized
cpio: premature end of archive
+ exit_code=2
+ '[' 2 -ne 0 ']'
+ echo 'Decompression error'
+ exit 1

Looking through the snapshot script I have been able to find where it is interrupted, that is in this snippet:

extract_initrd () {

	mkdir /tmp/extracted
	pushd /tmp/extracted

	COMPRESSION=$(file -L "$initrd_image" | egrep -o 'gzip compressed|XZ compressed|cpio archive')

	if [ "$COMPRESSION" = "gzip compressed" ]; then
		echo "Archive is gzip compressed..."
		zcat "$initrd_image" | cpio -i

	elif [ "$COMPRESSION" = "XZ compressed" ]; then
		echo "Archive is XZ compressed..."
		xzcat "$initrd_image" | cpio -d -i -m

	elif [ "$COMPRESSION" = "cpio archive" ]; then
		echo "Archive is cpio archive..."
		(cpio -i ; zcat | cpio -i) < "$initrd_image"
		exit_code="$?"
		if [ "$exit_code" -ne 0 ] ; then
			(cpio -i ; xzcat | cpio -i) < "$initrd_image"
			exit_code="$?"
			if [ "$exit_code" -ne 0 ] ; then
				echo "Decompression error" && exit 1
			else
				COMPRESSION="XZ compressed"
			fi
		else
			COMPRESSION="gzip compressed"
		fi
		
		echo "COMPRESSION is $COMPRESSION"

	else
		echo "Decompession error..." && exit 1
	fi

	popd

	echo "Initrd is extracted"

}

Now…..I know what is causing this issue but I am not sure how to resolve it.
Back to the “calamares-settings-debian” package:
https://packages.debian.org/stable/cala … ngs-debian
one of the dependencies is cryptsetup a dummy package that installs cryptsetup-initramfs and cryptsetup-run that is causing snapshot to stop,

So I uninstalled cryptsetup and ran snapshot and it completed no problem, but now the calamares installation in the iso, does not complete because cryptsetup is not found.
Of course I want to use cryptsetup, because of LUKS for encryption, for the sake of installation I want to keep it around,

Which brings be to my question, the error that gets thrown is a decompression error is there an additional command I can use to get past decompression because it seems to be using gzip, XZ, or cpio.
Or am I all wrong about how I am using cryptsetup?

Other reference links:
https://packages.debian.org/buster/cryptsetup
https://packages.debian.org/buster/cryptsetup-initramfs
https://packages.debian.org/buster/cryptsetup-run

Last edited by grafiksinc (2020-08-18 12:03:18)

HevyDevy · 2020-08-18 13:31:06

for the xzcat: (stdin): File format not recognized error maybe you are missing the package xz-utils ?

fsmithred · 2020-08-18 13:38:12

You should be able to make a snapshot with cryptsetup installed. I do it all the time.

What does calamares-settings-debian do? Does it make changes to initrd.img?

I haven't tried making a snapshot with calamares installed. I can take a look at this tonight.

fsmithred · 2020-08-18 18:16:16

I am unable to reproduce the problem. I tried with and without calamares-settings-debian and refractasnapshot ran both times. I also tried with initrd_crypt=yes in refractasnapshot.conf. Are you using a custom kernel and initrd? Also tried with intel-microcode and then amd64-microcode installed.

Please post the output of this command.

file -L /initrd.img

grafiksinc · 2020-08-19 13:03:36

@HevyDevy - that solved my problem on the Debian 10 --- thank you very very much! I did start off of the net install could have been a flook...;)

@fsmithred - Thanks to HevyDevy - the Debian is good to go....... now the Ubuntu is the stickler..... It does have xz-utils and gzip, that is not the issue...
and this command file -L /initrd.img returns
Ubuntu - Screen

hmmm... so doing boot a directory comparison Ubuntu on the left and Debian on the right

you can see the file structure is of course different -

Just for the sake of concept I did run file -L /initrd.img on Debian and it did return the following:
/initrd.img: ASCII cpio archive (SVR4 with no CRC)

Being that the structure on Ubuntu is different it is worth mentioning the kernel and initrd setting for the .conf file for snapshot in etc had to be set like this:

kernel_image ="/boot/vmlinuz"
initrd_image ="/boot/initrd.img"

Where as on debian this is all it needed:

kernel_image ="/vmlinuz"
initrd_image ="/initrd.img"

As I mentioned on Ubuntu everything works fine until I install cryptsetup.

Another thing... I did I try with initrd_crypt=yes in the .conf file.... and removed cryptsetup that allows Snapshot to complete successfully... but the calamaeres settings does not like that, because its looking for that specific cryptsetup file structure.

I know this is long but I really appreciate the advice..:)

HevyDevy · 2020-08-19 15:49:25

I dont profess to know what is going on with the ubuntu side of things but how does ubuntu generate its hooks when building/updating the initramfs?

For instance, in devuan/debian the conf hook is located as per below..

sudo nano /etc/cryptsetup-initramfs/conf-hook

Last edited by HevyDevy (2020-08-19 16:03:40)

fsmithred · 2020-08-19 16:00:55

Then do this...

file -L /boot/initrd.img

or run 'file' on whatever actual file you use for initramfs.

As I mentioned on Ubuntu everything works fine until I install cryptsetup.
Another thing... I did I try with initrd_crypt=yes in the .conf file.... and removed cryptsetup that allows Snapshot to complete successfully... but the calamaeres settings does not like that, because its looking for that specific cryptsetup file structure.

Setting initrd_crypt=yes without cryptsetup installed won't put cryptsetup in the initrramfs like it's meant to do.

What error messages are you getting and when?
Have you tried running snapshot with cryptsetup installed but calamares-settings removed?
If refractasnapshot is failing, I'd like to see the error log. If you're using the cli version, please run it in debug mode (sudo refractasnapshot -d). It'll be too big to post here. You can email it or paste it at paste.debian.net

grafiksinc · 2020-08-20 10:43:50

@fsmithred - here is the result of file -L /boot/initrd.img

Also I did as you suggested I ran snapshot with JUST cryptsetup installed and NOT calamares the result is the same.
Here is the terminal output from debug mode.

Do you want the whole log, file? if so I emailed it to.

@HevyDevy - Yep it is in the same location -

fsmithred · 2020-08-20 13:37:27

From the log:

gzip: stdin: not in gzip format
cpio: premature end of archive

xzcat: (stdin): File format not recognized
cpio: premature end of archive

It's not gzip and it's not xz compression, so it must be something else. If so, you can change the compression to gzip or xz and run 'update-initramfs -u', or else the script needs to be fixed to accommodate other types of compression.

Run this (or read the file, it's not long) to see what compression is used.

grep "COMPRESS=" /etc/initramfs-tools/initramfs.conf

HevyDevy · 2020-08-20 14:27:36

^ sorry to correct you fsmithred but missing directory path there with grep command i think.

grep "COMPRESS=" /etc/initramfs-tools/initramfs.conf

is ubuntu using zstd for compression maybe?

Last edited by HevyDevy (2020-08-20 14:34:34)

fsmithred · 2020-08-20 17:34:54

Fixed the path, thanks.

According to the comments in the config file, the choices are:

# COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz ]

The other way to find out which it is would be to remove the amd64-microcode package and then run 'file -L /boot/initrd.img' again.

Edit: Or maybe not. When I install amd64-microcode, checking the initrd with file tells me it's gzip compresses, same as if I did not install that microcode. But installing the intel-microcode does change it to ASCII cpio archive. And the commands to extract it work correctly.

grafiksinc · 2020-08-21 12:21:04

All.....Sorry for the late reply, I am in Japan....

Looks like it is lz4

where as my Debian is gzip...

Last edited by grafiksinc (2020-08-21 12:22:38)

HevyDevy · 2020-08-21 13:43:25

As fsmithred mentioned it looks like the script (refractasnapshot) needs to accommodate lz4 or perhaps an easier way would be to change ubuntus /etc/initramfs-tools/initramfs.conf compression method (gzip instead of lz4 then update initramfs) and see how that goes with another snapshot and calamares testing.

Wait for fsmithred to confirm though, im just curious on this subject as i have something planned.

Last edited by HevyDevy (2020-08-21 13:48:20)

fsmithred · 2020-08-21 13:45:53

Edit: Yup. ^^^

So, you can either change the compression to a supported type as I mentioned above, or you can manually unpack the initrd, remove crypttab, repack it and replace /home/work/iso/live/initrd.img with the modified copy, then re-run refractasnapshot and just select the option to re-run xorriso. You must have set save_work=yes in the config file for this to work.

If you keep the lz4 and do the manual re-pack, your users may run into the same problem if they try to make a snapshot on an encrypted system.

grafiksinc · 2020-08-22 00:00:04

Thank you so much! for the advice I think I will
Try changing the compression to gzip. and give it ago...

If that does not work then I will update the script to include lz4 compression.
I will update this post, with the results. and what I did when it is all said and done .;)

Last edited by grafiksinc (2020-08-22 02:38:54)

grafiksinc · 2020-08-22 12:16:29

@HevyDevy and @fsmithred

Here are my results,
**I have not tried the manual method @fsmithred mentioned yet.

I did change the compression in the initramfs.conf to gzip and updated the initramfs no issues there...
Ran Snapshot and it ...came back with the same Decompression error.

Now....for myself I know exactly where the problem area is in the snapshot script. That is here which is what you guys have already mentioned.

	elif [ "$COMPRESSION" = "cpio archive" ]; then
		echo "Archive is cpio archive..."
		(cpio -i ; zcat | cpio -i) < "$initrd_image"
		exit_code="$?"
		if [ "$exit_code" -ne 0 ] ; then
			(cpio -i ; xzcat | cpio -i) < "$initrd_image"
			exit_code="$?"
			if [ "$exit_code" -ne 0 ] ; then
				echo "Decompression error" && exit 1

Before I try the manual method I wanted to ask the following:
Ubuntu talks about using unmkinitramfs to extract an initramfs image.... because they are using lz4
The manpage is here: https://manpages.debian.org/buster/init … .8.en.html
Starting with 20.04...... lz4 is the standard used by Ubuntu to compress an initramfs image.

My question is......is unmkinitramfs the correct tool that I should try to add to the snapshot script?
or.... am I going in the wrong direction

Last edited by grafiksinc (2020-08-22 12:20:01)

fsmithred · 2020-08-22 14:07:54

You can try using mkinitramfs. I've never used it, but it looks like it would be easier than figuring out the right decompression commands. But I'd really like to know why it doesn't work with your rebuilt initrd. Did you verify that it's really in gzip format by running 'file' on it?

fsmithred · 2020-08-22 14:42:06

I got an ubuntu iso (20.04.1-desktop)
Booted it in qemu and used unmkinitramfs to unpack it into an empty directory.
Re-packed it as gzip with find . -print0 | cpio -0 -H newc -o | gzip -c > ../initrd.img-custom
I then unpacked the custom initrd to verify that the amd64 microcode was still there. It was.

Note 1: I chose gzip because the command was easier to type than xz, and I did not attempt to figure out the correct lz4 command. I just wanted to make sure the microcode didn't get lost.

Note 2: Since it was a live session, there were only dead symlinks in /boot, so I used /cdrom/casper/initrd for the extraction.

grafiksinc · 2020-09-18 12:03:21

Hello guys --- I wanted to update this post just to have closure...
Again thank you so much for advice and support....

Long story short I could not get around the compression problem with Ubuntu....
So, with Ubuntu what we did was take the "Snapshot without" crypt-setup, and then later I used a tool called "Cubic" to update the ISO.
https://launchpad.net/cubic
And that was problem solved.

But....I am glad I came here, because the work you guys have done and the freedom you offer speaks volumes.
I think a lot of folks just don't understand just how much systemd touches so many things...

Who knows maybe I can convince my community, and team to be open to a Devuan base
I hear OpenRc may be the init to use though....is that true I am really ignorant on that subject..?

Again.... thank you!

samhain · 2023-12-15 20:51:28

I just ran into the very same problem. Changing line 333 in refractasnapshot from:

(cpio -i ; zcat | cpio -i) < "$initrd_image"

to this:

cpio -i < "$initrd_image"

succeeds.

I don't know why that 'zcat' is there when initrd is uncompressed.

fsmithred · 2023-12-16 10:42:18

The extra zcat was needed at the time it was added because the microcode put an extra layer on the initrd.img. I guess that got fixed and snapshot needs to be changed again. That section of code also needs to support the new compression, zstd.

Thanks for posting this.

siva · 2023-12-24 19:11:25

samhain wrote:

I don't know why that 'zcat' is there when initrd is uncompressed.

Which kernel/OS are you running?

https://linuxconfig.org/how-to-uncompre … t-on-linux

Last edited by siva (2023-12-24 19:29:27)

oui · 2024-02-07 09:13:19

installed through import of the 6 packages from sourceforge in an old Ubuntu, the ubuntu apt version miss a file live-config and declares that the installation is now wrong. The live-config system man page explains that is so, as the installed system is not a live system. how to repare the system (the 2 refracta package from sourceforge were installed, but ubuntu's apt can't remove / puge them.

live-boot-initramfs-tools_20221008~fsr1_all.deb
refractainstaller-base_9.6.6_all.deb
refractasnapshot-base_10.2.12_all.deb
f@ubuntu:~$ sudo apt install ./*deb
[sudo] password for f:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'live-boot' instead of './live-boot_20221008~fsr1_all.deb'
Note, selecting 'live-boot-initramfs-tools' instead of './live-boot-initramfs-tools_20221008~fsr1_all.deb'
Note, selecting 'refractainstaller-base' instead of './refractainstaller-base_9.6.6_all.deb'
Note, selecting 'refractasnapshot-base' instead of './refractasnapshot-base_10.2.12_all.deb'
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
refractasnapshot-base : Depends: live-config (>= 2.0.15) but it is not installable
Recommends: live-config-sysvinit but it is not installable or
live-config-systemd but it is not installable or
live-config-upstart but it is not installable or
live-config-backend but it is not installable
E: Unable to correct problems, you have held broken packages.

Last edited by oui (2024-02-07 09:33:56)

The officially official Devuan Forum!

#1 2020-08-16 05:58:41

Refracta-SnapShot - Question

#2 2020-08-16 13:16:14

Re: Refracta-SnapShot - Question

#3 2020-08-18 11:59:37

Re: Refracta-SnapShot - Question

#4 2020-08-18 13:31:06

Re: Refracta-SnapShot - Question

#5 2020-08-18 13:38:12

Re: Refracta-SnapShot - Question

#6 2020-08-18 18:16:16

Re: Refracta-SnapShot - Question

#7 2020-08-19 13:03:36

Re: Refracta-SnapShot - Question

#8 2020-08-19 15:49:25

Re: Refracta-SnapShot - Question

#9 2020-08-19 16:00:55

Re: Refracta-SnapShot - Question

#10 2020-08-20 10:43:50

Re: Refracta-SnapShot - Question

#11 2020-08-20 13:37:27

Re: Refracta-SnapShot - Question

#12 2020-08-20 14:27:36

Re: Refracta-SnapShot - Question

#13 2020-08-20 17:34:54

Re: Refracta-SnapShot - Question

#14 2020-08-21 12:21:04

Re: Refracta-SnapShot - Question

#15 2020-08-21 13:43:25

Re: Refracta-SnapShot - Question

#16 2020-08-21 13:45:53

Re: Refracta-SnapShot - Question

#17 2020-08-22 00:00:04

Re: Refracta-SnapShot - Question

#18 2020-08-22 12:16:29

Re: Refracta-SnapShot - Question

#19 2020-08-22 14:07:54

Re: Refracta-SnapShot - Question

#20 2020-08-22 14:42:06

Re: Refracta-SnapShot - Question

#21 2020-09-18 12:03:21

Re: Refracta-SnapShot - Question

#22 2023-12-15 20:51:28

Re: Refracta-SnapShot - Question

#23 2023-12-16 10:42:18

Re: Refracta-SnapShot - Question

#24 2023-12-24 19:11:25

Re: Refracta-SnapShot - Question

#25 2024-02-07 09:13:19

Re: Refracta-SnapShot - Question

Board footer