The officially official Devuan Forum!

You are not logged in.

#1 2020-08-16 05:58:41

grafiksinc
Member
Registered: 2020-08-16
Posts: 8  

Refracta-SnapShot - Question

Hello Forum- before I go into detail about SnapShot. I wanted to make sure I am in the right spot.
Is this the correct place to pose a question about this package?

Thank you  in advance

Offline

#2 2020-08-16 13:16:14

fsmithred
Administrator
Registered: 2016-11-25
Posts: 1,645  

Re: Refracta-SnapShot - Question

Sometimes questions about refracta tools appear in the Derivatives section, sometimes in DIY and sometimes in Installation. You have my attention, so right here will also work.

Here's the documentation for it:
https://refracta.org/docs/readme.refractasnapshot.txt
https://refracta.org/docs/refractasnapshot.conf.txt
https://refracta.org/docs/snapshot_exclude.list.txt

Offline

#3 2020-08-18 11:59:37

grafiksinc
Member
Registered: 2020-08-16
Posts: 8  

Re: Refracta-SnapShot - Question

@fsmithred – thanks! For the help.
Let me preface this as using snapshot with the refracta-installer  everything works as expected.

Version of SnapShot being used:
refractasnapshot-base_10.2.10_all.deb
refractasnapshot-gui_10.2.10_all.deb

Operating System environments tested:

Ubuntu 20.04
Debian 10 (Buster)
Both running in Virtual Box

The problem below happens on both distros with the  exact same error log

Problem:
I want to take a “snapshot” of my build, however I want to use “calamares” as the installer.
I also want to use the  “calamares-settings-debian (10.0.20-1+deb10u3)“ package for my installer setup.
That package is located here: https://packages.debian.org/stable/cala … ngs-debian
Now…. these settings install on both distros listed above and run with no problem, and the settings work just fine with calamares.

***NOTE:  Before setting up  calamares with the “calamares-settings-debian” package I ran snapshot  and it completed successfully using the “01 Create a snapshot” option***

After setting up calamares and then applying the “calamares-settings-debian” package snapshot does Not complete.  It errors out with the following error:


+ [[ -n '' ]]
+ '[' yes = yes ']'
+ clean_initrd
+ extract_initrd
+ mkdir /tmp/extracted
+ pushd /tmp/extracted
++ egrep -o 'gzip compressed|XZ compressed|cpio archive'
++ file -L /boot/initrd.img
+ COMPRESSION='cpio archive'
+ '[' 'cpio archive' = 'gzip compressed' ']'
+ '[' 'cpio archive' = 'XZ compressed' ']'
+ '[' 'cpio archive' = 'cpio archive' ']'
+ echo 'Archive is cpio archive...'
+ cpio -i
62 blocks
+ cpio -i
+ zcat

gzip: stdin: not in gzip format
cpio: premature end of archive
+ exit_code=2
+ '[' 2 -ne 0 ']'
+ cpio -i
cpio: kernel/x86/microcode/AuthenticAMD.bin not created: newer or same age version exists
62 blocks
+ cpio -i
+ xzcat
xzcat: (stdin): File format not recognized
cpio: premature end of archive
+ exit_code=2
+ '[' 2 -ne 0 ']'
+ echo 'Decompression error'
+ exit 1

Looking through the snapshot script I have been able to find where it is interrupted, that is in this snippet:

extract_initrd () {

	mkdir /tmp/extracted
	pushd /tmp/extracted

	COMPRESSION=$(file -L "$initrd_image" | egrep -o 'gzip compressed|XZ compressed|cpio archive')

	if [ "$COMPRESSION" = "gzip compressed" ]; then
		echo "Archive is gzip compressed..."
		zcat "$initrd_image" | cpio -i

	elif [ "$COMPRESSION" = "XZ compressed" ]; then
		echo "Archive is XZ compressed..."
		xzcat "$initrd_image" | cpio -d -i -m

	elif [ "$COMPRESSION" = "cpio archive" ]; then
		echo "Archive is cpio archive..."
		(cpio -i ; zcat | cpio -i) < "$initrd_image"
		exit_code="$?"
		if [ "$exit_code" -ne 0 ] ; then
			(cpio -i ; xzcat | cpio -i) < "$initrd_image"
			exit_code="$?"
			if [ "$exit_code" -ne 0 ] ; then
				echo "Decompression error" && exit 1
			else
				COMPRESSION="XZ compressed"
			fi
		else
			COMPRESSION="gzip compressed"
		fi
		
		echo "COMPRESSION is $COMPRESSION"

	else
		echo "Decompession error..." && exit 1
	fi

	popd

	echo "Initrd is extracted"

}

Now…..I know what is causing this issue but I am not sure how to resolve it.
Back to the “calamares-settings-debian” package:
https://packages.debian.org/stable/cala … ngs-debian
one of the dependencies is  cryptsetup  a dummy package that installs cryptsetup-initramfs and cryptsetup-run that is causing snapshot to stop,

So I uninstalled cryptsetup and ran snapshot and it completed no problem, but now the calamares installation in the iso, does not complete because cryptsetup is not found.
Of course I want to use cryptsetup, because of LUKS for encryption, for the sake of installation I want to keep it around,

Which brings be  to my question,  the error that gets thrown is a decompression error  is there an additional command I can use to get past decompression because it seems to be using gzip, XZ, or cpio.
Or am I all wrong about how I am using cryptsetup?

Other reference links:
https://packages.debian.org/buster/cryptsetup
https://packages.debian.org/buster/cryptsetup-initramfs
https://packages.debian.org/buster/cryptsetup-run

Last edited by grafiksinc (2020-08-18 12:03:18)

Offline

#4 2020-08-18 13:31:06

HevyDevy
Member
Registered: 2019-09-06
Posts: 358  

Re: Refracta-SnapShot - Question

for the xzcat: (stdin): File format not recognized error maybe you are missing the package xz-utils ?

Offline

#5 2020-08-18 13:38:12

fsmithred
Administrator
Registered: 2016-11-25
Posts: 1,645  

Re: Refracta-SnapShot - Question

You should be able to make a snapshot with cryptsetup installed. I do it all the time.

What does calamares-settings-debian do? Does it make changes to initrd.img?

I haven't tried making a snapshot with calamares installed. I can take a look at this tonight.

Offline

#6 2020-08-18 18:16:16

fsmithred
Administrator
Registered: 2016-11-25
Posts: 1,645  

Re: Refracta-SnapShot - Question

I am unable to reproduce the problem. I tried with and without calamares-settings-debian and refractasnapshot ran both times. I also tried with initrd_crypt=yes in refractasnapshot.conf. Are you using a custom kernel and initrd? Also tried with intel-microcode and then amd64-microcode installed.

Please post the output of this command.

file -L /initrd.img

Offline

#7 2020-08-19 13:03:36

grafiksinc
Member
Registered: 2020-08-16
Posts: 8  

Re: Refracta-SnapShot - Question

@HevyDevy - that solved my problem on the Debian 10 --- thank you very very much! I  did start off of the net install could have been a flook...;)

@fsmithred - Thanks to HevyDevy - the Debian is good to go....... now the Ubuntu is the stickler..... It does have xz-utils and gzip, that is not the issue...
and this command file -L /initrd.img returns
Ubuntu - Screen
g4SqUhHl.png

hmmm... so doing boot a directory comparison Ubuntu on the left and Debian on the right
eyVxDBhl.png

you can see the file structure is of course different -

Just for the sake of concept I did run file -L /initrd.img on Debian and it did return the following:
/initrd.img: ASCII cpio archive (SVR4 with no CRC)

Being that the structure on Ubuntu is different it is worth mentioning the kernel and initrd setting for the .conf file for snapshot  in etc had to be set like this:

kernel_image ="/boot/vmlinuz"
initrd_image ="/boot/initrd.img"


Where as on debian this is all it needed:

kernel_image ="/vmlinuz"
initrd_image ="/initrd.img"

As I mentioned on Ubuntu everything works fine until I install cryptsetup.

Another thing... I did I try with initrd_crypt=yes in the .conf file.... and removed cryptsetup that allows Snapshot to complete successfully... but the calamaeres settings  does not like that, because its looking for that specific cryptsetup file structure.

I know this is long but I really appreciate the advice..:)

Offline

#8 2020-08-19 15:49:25

HevyDevy
Member
Registered: 2019-09-06
Posts: 358  

Re: Refracta-SnapShot - Question

I dont profess to know what is going on with the ubuntu side of things but how does ubuntu generate its hooks when building/updating the initramfs?

For instance, in devuan/debian the conf hook is located as per below..

sudo nano /etc/cryptsetup-initramfs/conf-hook

Last edited by HevyDevy (2020-08-19 16:03:40)

Offline

#9 2020-08-19 16:00:55

fsmithred
Administrator
Registered: 2016-11-25
Posts: 1,645  

Re: Refracta-SnapShot - Question

Then do this...

file -L /boot/initrd.img

or run 'file' on whatever actual file you use for initramfs.

As I mentioned on Ubuntu everything works fine until I install cryptsetup.

Another thing... I did I try with initrd_crypt=yes in the .conf file.... and removed cryptsetup that allows Snapshot to complete successfully... but the calamaeres settings  does not like that, because its looking for that specific cryptsetup file structure.

Setting initrd_crypt=yes without cryptsetup installed won't put cryptsetup in the initrramfs like it's meant to do.

What error messages are you getting and when?
Have you tried running snapshot with cryptsetup installed but calamares-settings removed?
If refractasnapshot is failing, I'd like to see the error log. If you're using the cli version, please run it in debug mode (sudo refractasnapshot -d). It'll be too big to post here. You can email it or paste it at paste.debian.net

Offline

#10 2020-08-20 10:43:50

grafiksinc
Member
Registered: 2020-08-16
Posts: 8  

Re: Refracta-SnapShot - Question

@fsmithred - here is the result of file -L /boot/initrd.img
  fRXWnY2l.png

Also I did as you suggested I ran snapshot  with JUST cryptsetup installed and NOT calamares the result is the same.
Here is the terminal output from debug mode.
  cpGSiucl.png

Do you want the whole log, file? if so I emailed it to.

@HevyDevy - Yep it is in the same location -
CvOuVngl.png

Offline

#11 2020-08-20 13:37:27

fsmithred
Administrator
Registered: 2016-11-25
Posts: 1,645  

Re: Refracta-SnapShot - Question

From the log:

gzip: stdin: not in gzip format
cpio: premature end of archive

xzcat: (stdin): File format not recognized
cpio: premature end of archive

It's not gzip and it's not xz compression, so it must be something else. If so, you can change the compression to gzip or xz and run 'update-initramfs -u', or else the script needs to be fixed to accommodate other types of compression.

Run this (or read the file, it's not long) to see what compression is used.

grep "COMPRESS=" /etc/initramfs-tools/initramfs.conf

Offline

#12 2020-08-20 14:27:36

HevyDevy
Member
Registered: 2019-09-06
Posts: 358  

Re: Refracta-SnapShot - Question

^ sorry to correct you fsmithred but missing directory path there with grep command i think.

grep "COMPRESS=" /etc/initramfs-tools/initramfs.conf

is ubuntu using zstd for compression maybe?

Last edited by HevyDevy (2020-08-20 14:34:34)

Offline

#13 2020-08-20 17:34:54

fsmithred
Administrator
Registered: 2016-11-25
Posts: 1,645  

Re: Refracta-SnapShot - Question

Fixed the path, thanks.

According to the comments in the config file, the choices are:

# COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz ]

The other way to find out which it is would be to remove the amd64-microcode package and then run 'file -L /boot/initrd.img' again.

Edit: Or maybe not. When I install amd64-microcode, checking the initrd with file tells me it's gzip compresses, same as if I did not install that microcode. But installing the intel-microcode does change it to ASCII cpio archive. And the commands to extract it work correctly.

Offline

#14 2020-08-21 12:21:04

grafiksinc
Member
Registered: 2020-08-16
Posts: 8  

Re: Refracta-SnapShot - Question

All.....Sorry for the late reply, I am in Japan....

Looks like it is lz4

xtNu70il.png


where as my Debian is gzip...

Last edited by grafiksinc (2020-08-21 12:22:38)

Offline

#15 2020-08-21 13:43:25

HevyDevy
Member
Registered: 2019-09-06
Posts: 358  

Re: Refracta-SnapShot - Question

As fsmithred mentioned it looks like the script (refractasnapshot) needs to accommodate lz4 or perhaps an easier way would be to change ubuntus /etc/initramfs-tools/initramfs.conf compression method (gzip instead of lz4 then update initramfs) and see how that goes with another snapshot and calamares testing.

Wait for fsmithred to confirm though, im just curious on this subject as i have something planned.

Last edited by HevyDevy (2020-08-21 13:48:20)

Offline

#16 2020-08-21 13:45:53

fsmithred
Administrator
Registered: 2016-11-25
Posts: 1,645  

Re: Refracta-SnapShot - Question

Edit: Yup. ^^^

So, you can either change the compression to a supported type as I mentioned above, or you can manually unpack the initrd, remove crypttab, repack it and replace /home/work/iso/live/initrd.img with the modified copy, then re-run refractasnapshot and just select the option to re-run xorriso. You must have set save_work=yes in the config file for this to work.

If you keep the lz4 and do the manual re-pack, your users may run into the same problem if they try to make a snapshot on an encrypted system.

Offline

#17 2020-08-22 00:00:04

grafiksinc
Member
Registered: 2020-08-16
Posts: 8  

Re: Refracta-SnapShot - Question

Thank you so much! for the advice I think I will
Try changing the compression to gzip. and give it ago...

If that does not work then I will  update the script to include lz4 compression.
I will update this post, with the results. and what I did when it is all said and done .;)

Last edited by grafiksinc (2020-08-22 02:38:54)

Offline

#18 2020-08-22 12:16:29

grafiksinc
Member
Registered: 2020-08-16
Posts: 8  

Re: Refracta-SnapShot - Question

@HevyDevy and @fsmithred

Here are my results,
**I have not tried the manual method @fsmithred mentioned yet.

I did change the compression in the initramfs.conf to gzip and updated the initramfs no issues there...
Ran Snapshot and it ...came back with the same Decompression error.

Now....for myself I know exactly where the  problem area is in the snapshot script. That is here which is what you guys have already mentioned.

	elif [ "$COMPRESSION" = "cpio archive" ]; then
		echo "Archive is cpio archive..."
		(cpio -i ; zcat | cpio -i) < "$initrd_image"
		exit_code="$?"
		if [ "$exit_code" -ne 0 ] ; then
			(cpio -i ; xzcat | cpio -i) < "$initrd_image"
			exit_code="$?"
			if [ "$exit_code" -ne 0 ] ; then
				echo "Decompression error" && exit 1

Before I try the  manual method I wanted to ask the following:
Ubuntu talks about using unmkinitramfs to extract an  initramfs image.... because they are using lz4
The manpage is here: https://manpages.debian.org/buster/init … .8.en.html
Starting with 20.04...... lz4 is the standard used by Ubuntu to compress an initramfs image.

My question is......is unmkinitramfs the correct tool that I should try to add to the snapshot script?
or.... am I going in the wrong direction

Last edited by grafiksinc (2020-08-22 12:20:01)

Offline

#19 2020-08-22 14:07:54

fsmithred
Administrator
Registered: 2016-11-25
Posts: 1,645  

Re: Refracta-SnapShot - Question

You can try using mkinitramfs. I've never used it, but it looks like it would be easier than figuring out the right decompression commands. But I'd really like to know why it doesn't work with your rebuilt initrd. Did you verify that it's really in gzip format by running 'file' on it?

Offline

#20 2020-08-22 14:42:06

fsmithred
Administrator
Registered: 2016-11-25
Posts: 1,645  

Re: Refracta-SnapShot - Question

I got an ubuntu iso (20.04.1-desktop)
Booted it in qemu and used unmkinitramfs to unpack it into an empty directory.
Re-packed it as gzip with find . -print0 | cpio -0 -H newc -o | gzip -c > ../initrd.img-custom
I then unpacked the custom initrd to verify that the amd64 microcode was still there. It was.

Note 1: I chose gzip because the command was easier to type than xz, and I did not attempt to figure out the correct lz4 command. I just wanted to make sure the microcode didn't get lost.

Note 2: Since it was a live session, there were only dead symlinks in /boot, so I used /cdrom/casper/initrd for the extraction.

Offline

#21 2020-09-18 12:03:21

grafiksinc
Member
Registered: 2020-08-16
Posts: 8  

Re: Refracta-SnapShot - Question

Hello guys --- I wanted to update this post just to have closure...
Again thank you so much for advice and support....

Long story short I could not get around the compression problem with Ubuntu....
So, with Ubuntu what we did was take the "Snapshot without" crypt-setup, and then later  I used  a tool called "Cubic" to update the ISO.
https://launchpad.net/cubic
And that was problem solved.

But....I am glad I came here, because the work you guys have done and the  freedom you offer speaks volumes. 
I think a lot of folks just don't understand just how much systemd touches so many things... 

Who knows maybe I can convince my community, and team to be open to a Devuan base wink 
I hear OpenRc may be the init to use though....is that true I am really ignorant on that subject..?

Again.... thank you!

Offline

Board footer