Testing for beowulf 3.1 point release

fsmithred · 2020-12-04 14:14:05

Message copied from dng mailing list:

The Devuan Developers are preparing for a beowulf point release (3.1). It would be very helpful to have the new versions of packages tested more widely. If you feel able to help with that and are prepared to debug any breakage with us, please
- add beowulf-proposed-updates to apt sources.list:
deb http://deb.devuan.org/devuan beowulf-proposed-updates main
Run:
apt update
apt upgrade
beowulf-proposed-updates contains packages built from the following updated source versions.
base-files | 10.3+devuan3.5
choose-init | 0.3
cryptsetup-modified-functions | 19.09.02+devuan1
dbus | 1.12.20-0+deb10u1+devuan1
eudev | 3.2.9-8~beowulf1
init-system-helpers | 1.56+nmu1+devuan3
lightdm | 1.26.0-4+devuan1
refractainstaller-base | 9.5.6
refractainstaller-gui | 9.5.6
tomcat9 | 9.0.31-1~deb10u2+devuan1
udev | 1:3.2.9+devuan4
Depending on your setup, you will get some, but not necessarily all of those.
If you encounter problems, please use reportbug to send a report to Devuan BTS.

Note: If you do it wrong like I did, you may also get some updates from debian package we do not fork. (See later post for explanation.) One example is Thunderbird. I wish I'd kept that one back. - fsmithred

fsmithred · 2020-12-04 16:04:05

I made some preliminary desktop-live isos with the updates. Please test.

https://get.refracta.org/files/experime … p-live.iso
https://get.refracta.org/files/experime … p-live.iso

sha256sums:

bf477a3c0dc27866509407fde670d8fa0b903296effef7f67c2222a48fd9eb2a  devuan_beowulf_3.1.0_2020-11-30_amd64_desktop-live.iso
6b5a47918d983850500829d008ea0b96daba8b2fae45f26456b28a4fedba926f  devuan_beowulf_3.1.0_2020-11-30_i386_desktop-live.iso

fsmithred · 2020-12-04 19:41:36

The instructions say to use
deb http://deb.devuan.org/devuan beowulf-proposed-updates main

but I used
deb http://deb.devuan.org/merged beowulf-proposed-updates main

which explains why I got more than our own forked packages.

dev-1-dash-1 · 2020-12-05 18:54:36

Testing the amd64 live-iso.

Burned to physical dvd.

Udev seems fine, all modules loaded on first try (with or without "toram").
Network cards and sound working - the system even detected which output I had plugged and set default output accordingly. This is even more than I expected - usually some tweaking is needed on my setup to select the correct output. Very good.
Switching betweet manual/dhcp, as well as lan/wi-fi using wicd - all ok.
Cryptsetup, lvm work ok.
Added backports (with contrib and non-free sections), built zfs-dkms modules, imported the pools - all ok.
Used "dkms mkbmdeb" to pack the zfs modules into a deb, rebooted and installed the deb to avoid recompile of zfs modules - all ok.

Module count, as indicated by "lsmod | wc -l" is the same w/ or wo/ "toram" parameter.

Problem:
Trying to load the iso from hd through grub entry failed:
"Warning: can not mount /dev/sda3".
That's an ntfs partition where I've placed the iso, and, as I suspected, the ntfs modules are missing from /live/boot.img.
Deliberate policy or accidental omission? I hope the latter.
I used to be able to boot debian/devuan/ubuntu and many others from ntfs partitions like this.
When placing the iso to an ext4 partition, the boot proceeds normally.

Minor issues:
1)Incosistent formatting wrt tabs/spaces in "/etc/init.d/eudev" between start) and other) sections. Has no run-time effect.
2)Usage of "start-stop-daemon" to start eudev is not neccessary. It doesn't do any damage in this configuration, but it's still not needed. Udev will be perfectly fine when started as "udevd --daemon" - the way it was in jessie/ascii, unless I am mistaken.
3)acpi-fakekeyd: not neccessary anymore since eudev startup seems to be in order. May be used as a fail-safe for live-cd, but maybe consider adding it to exclude list, if possible, for the actual refracta-install process. Some people have complained that system installed from live-image uses a bit more resources - it's not much, and does no harm, but acpi-fakekey may be one of the reasons.

Other than that, live-image seems very good.

Keep Devuan rock-solid.

fsmithred · 2020-12-06 15:02:52

Thanks for the detailed report. I don't know what decisions went into using start-stop-daemon. There were a few iterations of the patch, and they all seemed to work.

NTFS: If something is missing, I have no idea what it is. I did not remove any packages and ntfs-3g is installed in the live system. Do you maybe need to load a grub module before loading the kernel?

acpi-fakekey is still there because I didn't think to remove it. I will do so in the rebuild.

dev-1-dash-1 · 2020-12-06 15:43:52

fsmithred wrote:

I don't know what decisions went into using start-stop-daemon. There were a few iterations of the patch, and they all seemed to work.

As it stands now, udev seems to work correctly, so it's not a big deal.
The only "problem" here may be a few microseconds and a few kb of ram temporarily used when starting the "start-stop-daemon" binary itself. I was only pointing this out because I believe any unneccessary "extra layer", even one as simple as start-stop-daemon, should be justified, and, as you know it had already caused some problems in the past.

Again, right now everything seems to work correctly wrt udev, so no problems here.

NTFS: If something is missing, I have no idea what it is. I did not remove any packages and ntfs-3g is installed in the live system. Do you maybe need to load a grub module before loading the kernel?

This is not a grub problem - grub loads the kernel and the initrd into memory from "within" the iso when the iso itself is on an ntfs partition.
The problem appears later, I think, when initrd is parsing the "fromiso=/dev/sda3:/path/to.iso" kernel command line parameter. It tries to mount the iso but fails, because it can not mount ntfs filesystem.
Edit:The full grub entry looks like this (has been working since forever with debian,devuan, ubuntu with some changes):

menuentry 'Devuan Ascii 2.1 Live cd' {
        set root="hd0,3"
        set isofile="/devuan_ascii_2.1_amd64_desktop-live.iso"
        set gfxpayload=keep
        loopback loop ($root)$isofile
        linux   (loop)/live/vmlinuz boot=live username=devuan fromiso=/dev/sda3$isofile
        initrd  (loop)/live/initrd.img
    }

This seems to be an initrd problem, it might be necessary to regenerate it and include the ntfs modules. Here's why I think so:
1) Open/unpack the "devuan_beowulf_3.1.0_2020-11-30_amd64_desktop-live.iso" file.
2) Within it: go to "/live" directory.
3) There is a file "initrd.img"
4) Open/unpack this file (on ascii, use "unmkinitramfs" command)
5) Go to "/lib/modules/4.19.0-13-amd64/kernel/fs" within it.
6) Note there are directories like btrfs, ext4, fat, xfs, but there is no "ntfs" directory
7) Now, inspect the "devuan_ascii_2.1_amd64_desktop-live.iso", for example, and see that "ntfs" directory is there. (obviously the path is a bit different due to different kernel version, "/lib/modules/4.9.0-11-amd64/kernel/fs". That one loads the iso from ntfs partition just fine.

Also, when unpacking the initrd from ascii 2.1 live iso when using "unmkinitramfs", we get two directories - "early" and "main", "early" contains processor microcode, "main" contains the initrd tree itself. That means there are basically two (2) archives in the initrd itself.

Shouldn't the "beowulf-3.1" initrd also include processor microcode?
Note: I am using the ascii "unmkinitramfs" script to unpack a "beowulf" initrd, that may be a source of the problem.

Please doublecheck.

acpi-fakekey is still there because I didn't think to remove it. I will do so in the rebuild.

Again, it doesn't cause problems, it's simply not needed if udev has done all the work wrt detecting mouse/kbd/sound etc. As I've said, udev seems to be working correctly.
However acpi-fakekey may be needed on the live image itself, if perhaps, something fails on an unusual configuration. Having an extra layer of reliability is not a bad idea for livecd.
But perhaps you can exclude it from the system which will be installed to hdd itself.

Last edited by dev-1-dash-1 (2020-12-06 15:53:45)

fsmithred · 2020-12-06 18:22:10

I can confirm that the ntfs kernel modules are not in the beowulf iso's initramfs, and that they are in the ascii iso's initramfs.

3.1
# lsinitramfs mnt/live/initrd.img |grep ntfs
scripts/local-bottom/ntfs_3g
scripts/local-premount/ntfs_3g
usr/bin/ntfs-3g
usr/lib/x86_64-linux-gnu/libntfs-3g.so.883
usr/lib/x86_64-linux-gnu/libntfs-3g.so.883.0.0
usr/sbin/mount.ntfs
usr/sbin/mount.ntfs-3g

2.1
# lsinitramfs mnt/live/initrd.img |grep ntfs
bin/ntfs-3g
sbin/mount.ntfs-3g
sbin/mount.ntfs
scripts/local-premount/ntfs_3g
scripts/local-bottom/ntfs_3g
lib/x86_64-linux-gnu/libntfs-3g.so.871
lib/x86_64-linux-gnu/libntfs-3g.so.871.0.0
lib/modules/4.9.0-11-amd64/kernel/fs/ntfs
lib/modules/4.9.0-11-amd64/kernel/fs/ntfs/ntfs.ko

Where the hell did ntfs.ko go? On my installed beowulf system, it's missing. Same ntfs packages are installed as in my ascii, which does have ntfs.ko. In ascii, apt-file shows that it comes with the kernel. In beowulf, apt-file doesn't find ntfs.ko.

dpkg -l |grep ntfs
ii  libntfs-3g883                           1:2017.3.23AR.3-3                   amd64        read/write NTFS driver for FUSE (runtime library)
ii  ntfs-3g                                 1:2017.3.23AR.3-3                   amd64        read/write NTFS driver for FUSE

# find /lib/modules/4.19.0-13-amd64/ -name "*ntfs*"
#

# locate ntfs.ko
#

dev-1-dash-1 · 2020-12-06 18:55:16

fsmithred wrote:

Where the hell did ntfs.ko go? On my installed beowulf system, it's missing. Same ntfs packages are installed as in my ascii, which does have ntfs.ko.

Thanks for paying attention and replying, fsmithred. That's why I (and I'm sure many others) use devuan - because of the sensible attitude towards resolving actual issues.

However, in this case:
FALSE ALARM!
I've just tested the debian-live-10.7.0-mate-amd64.iso directrly from debian.

It behaves exactly the same. No ntfs.ko in initramfs. "Warning: unable to mount /dev/sda3" when trying to access iso on ntfs partition.

Therefore: it's a Debian change, and Devuan simply mirrored it, and there's no direct obligation for Devuan team to do anything about it since it isn't connected to *ystemd.

In ascii, apt-file shows that it comes with the kernel. In beowulf, apt-file doesn't find ntfs.ko.

I can see the same results for ascii. I don't have a beowulf system right now near me.

If we were to try:
The simplest way to include ntfs.ko into the initramfs would probably be modifying "/etc/initramfs-tools/modules", but first making sure that the modules themselves are installed on the base system of course. I don't know why they aren't there on beowulf.

Can you tell me how big a partition I would need to install a minimal beowulf 3.1 system from the live image? If I have a partition big enough I can maybe try and find an easy solution.

In the meantime - this isn't a Devuan problem, so it's a non-blocking issue if you are planning to release 3.1.
As I described, the beowulf 3.0 problems with eudev are not present in 3.1, so progress has already been made.

Update:
Debian has dropped the support of ntfs.ko: (see: 29 June 2019: GParted Live 1.0.0-3 Stable Release)
https://gparted.org/news.php?item=224Release

And here people seem to be solving the same problem:
http://gparted-forum.surf4.info/viewtopic.php?pid=35358

Last edited by dev-1-dash-1 (2020-12-06 19:00:32)

fsmithred · 2020-12-06 19:33:03

I think this post has the magic sauce: https://lists.debian.org/debian-live/20 … 00038.html

live-boot's function 'is_supported_fs' should add one statement,'if test "$fstype" = "ntfs" -a -e /sbin/mount.ntfs -a -e /bin/ntfs-3g;then return 0;fi '.
ntfs-3g's command is /bin/ntfs-3g,and /sbin/mount.ntfs (also /sbin/mount.ntfs-3g) is a soft link to /bin/ntfs-3g.when user installed ntfs-3g,it's /var/lib/dpkg/ntfs-3g.postinst will auto remake initrd.img (call update-initramfs),include live-boot'script in new initrd.img,but,when live-boot not change it's code of function 'is_supported_fs',new initrd.img will boot failed on ntfs.All live-boot need to do is add one statement to it.

That function, is_supported_fs is in /lib/live/boot/9990-misc-helpers.sh
But I'm not sure where that test goes. Maybe early in the function to avoid some of the other tests.

dev-1-dash-1 · 2020-12-07 13:00:10

fsmithred wrote:

I think this post has the magic sauce: https://lists.debian.org/debian-live/20 … 00038.html
That function, is_supported_fs is in /lib/live/boot/9990-misc-helpers.sh

That seems close to the target.
If I have some time and energy in the next few days, maybe I'll try looking into it more. It might be just a few lines of code needed to fix it.

For the moment, we can conclude that it's another feature that has been working for decades but has been cancelled by Debian.
Thus it doesn't block release of beowulf 3.1.

fsmithred · 2020-12-07 13:31:00

If you try patching 9990-misc-helpers.sh please let me know how it goes. That file already gets patched for refracta2usb, and I could add the ntfs support next time I touch that package.

dev-1-dash-1 · 2020-12-07 13:31:53

I've tested the gparted-live-1.0.0-3 live image, which is being discussed in the forum post I've linked at message #8 in this thread.
The url for the actual live image is:
https://sourceforge.net/projects/gparte … o/download

I was able to boot it to desktop from an ntfs partition.

I looked into the initrd, /lib/live/boot directory and ran a diff command between beowulf's files at the same location in the initrd.
There are a number of differences, I'm not posting full diff because most of them don't seem related to ntfs.
However, here's the diff between the 9990-misc-helpers.sh file:

82c82
<   elif echo "${sysfs_path}" | grep -q '^/block/vd[a-z]$'
---
>   elif echo "${sysfs_path}" | grep -q '^/block/[hsv]d[a-z]$'
474a475,482 
>   # For ntfs, since user space program ntfs-3g will be used. Check ntfs-3g instead of kernel module.
>   if [ "${fstype}" = "ntfs" ]; then
>       if type ntfs-3g >/dev/null 2>&1; then
>           return 0
>       else
>           return 1
>       fi
>   fi
1222c1230
<       if [ "$(cat ${sysblock}/removable)" = "1" ]
---
>       if [ "$(cat ${sysblock}/removable 2>/dev/null)" = "1" ]
1270c1278
<       if [ "$(cat ${sysblock}/removable)" = "0" ]
---
>       if [ "$(cat ${sysblock}/removable 2>/dev/null)" = "0" ]

The second difference looks almost exactly like the post you linked to in message #9 of this thread.

dev-1-dash-1 · 2020-12-07 13:36:34

fsmithred wrote:

If you try patching 9990-misc-helpers.sh please let me know how it goes. That file already gets patched for refracta2usb, and I could add the ntfs support next time I touch that package.

I've tried patching it myself, but I am unable to boot with a modified initrd which I've reconstructed by myself.

Maybe you can suggest something?

I unpack the beowulf-live initrd with

unmkinitramfs initrd.img /path/to/extracted/initrd-tree

After I've applied some changes, how do I compress it in exactly the same format?
I've tried the following:

find . | cpio -H newc -o > ../initramfs.cpio
cat initramfs.cpio | gzip > initramfs.igz

But boot failed immediately and dropped me to the shell even though I've only added some debug prints.

Do I have to insert the initrd into the iso image itself, or can I simply modify grub entry to load the custom initrd and it should work?

fsmithred · 2020-12-07 13:48:48

Quick answer on compressing initrd. Look at extract_initrd() rebuild_initrd() starting at lines 316 and 385.
https://git.devuan.org/devuan/refractas … tasnapshot
There's extra code if the cpu microcode is installed.

If you're doing this on a usb, you can mount the usb, copy the modified initrd to the /live folder and edit the boot menu. This won't work if you used dd to put the isohybrid image on the usb. It needs to be a real first partition.

dev-1-dash-1 · 2020-12-07 21:31:22

Got it to work by applying the same changes gparted-live people did.

Patching the file

9990-misc-helpers.sh

with the following patch

474a475,481
>   if [ "${fstype}" = "ntfs" ]; then
>       if type ntfs-3g >/dev/null 2>&1; then
>           return 0
>       else
>           return 1
>       fi
>   fi

seems to be enough.

Pro-tip: for a quick test you can avoid rebuilding the initrd by booting the iso with "break=mount" command line parameter, and applying the patch right in the shell.

Andre4freedom · 2020-12-13 12:07:47

Hello all, hello fsmithred,
late, but finally done, I tested the 2 procedures as you asked for in your original post.

1. - Updating a perfectly working Beowulf (Devian3) machine (EFI) worked very well. A few packages were updated, and I'm glad to see that the Devuan bugreport #438 has been dealt with. Great. (The issue with the lightdm greeter's greyed activity menus)

2. - I have downloaded the installer iso and tested it on another machine: (refracta installer devuan_beowulf_3.1.0_2020-11-30_amd64_desktop-live.iso)
The iso and the installer worked, but: A: I had troubles getting the grub-install working. I have resolved it by doing it manually in a maintenance boot-session.
This was on a BIOS/MBR machine.
Then, when installing lightdm, the old bug #438 re-appeared. (I'm quite confident the definitive installer isos -devuan 3.1- will have that fixed too.)

I hope this information to be useful. It comes from a user's perspective.
I'm very happy to work with Devuan Linux.
Greetings, Andre4freedom

fsmithred · 2020-12-13 17:27:23

Andre,

Thanks for testing and reporting. What trouble did you have with grub? I've installed with and without a separate /boot partition. Other than installing the bootloader twice, it works correctly for me. Did you look at the error log? It should be in the user's home.

FYI: What we call "installer isos" refers to the isos with the debian(devuan)-installer. Not the live isos. Although either can be used to install devuan, as you know.

Bug#438 was fixed. You got the old lightdm because the fix is still in beowulf-proposed-updates, which is not in sources.list by default. We're working on another bug in lightdm, and it will get moved to the main beowulf repo when that one is fixed. Check this before you install the newer version:
https://bugs.devuan.org/cgi/bugreport.cgi?bug=529

I've been using that newer version since the package was put in the repo, and I didn't notice any problem until I tried to connect with vnc.

fsmithred

Andre4freedom · 2020-12-14 09:48:27

fsmithred,
thank you for your answer.
Again, I only had the problem with the live-iso (refracta installer) the first time.
The machine is a Dell Optiplex GX780, a BIOS/MBR machine.
A Windows 7 is installed on it, and a lot of free space for Linux is availble.

Layout /dev/sda: (Disklabel DOS )
/dev/sda1 100M ntfs System reserved
/dev/sda2 100G ntfs Windows7

I booted the mentioned live-iso and performed the installation.

Layout /dev/sda: (Disklabel DOS)
/dev/sda1 100M ntfs System reserved
/dev/sda2 100G ntfs Windows7
/dev/sda3 1G    ext3 /boot
/dev/sda4 extended
/dev/sda5 40G    ext4 /
/dev/sda6 80G    ext4 /home
/dev/sda7 8G    swap swap

After the installation
I installed the grub-package as instructed and tried to do the "sudo grub-install /dev/sda".
This resulted in an error message the device /dev/sda not being available.

I then rebooted the system and found myself in Windows again.
So I rebooted from a standard Devuan3.0 Beowulf installer-iso to the maintenance mode.

From there I could perform the
grub-install /dev/sda
and
update-grub
and everything was fine.

Now, that the disk had a GRUB bootblock, I repeated the live-iso refracta installer again, including the dpkg -i grub-pc...., grub-install and update-grub and it was a success.
Unfortunately, the log of the first, failed attempt was lost due to the second installation.

I guess had I installed the first time on a wiped disk with no Windows using the live-iso I would have been successful right away.

I suggest not to invest a lot of time into that issue, since the true installer always works well and the installer on the live-isos were repeatedly slightly troublesome.

Hopefully that information is not too messy.... thank you for your thoughts. Andre

The officially official Devuan Forum!

#1 2020-12-04 14:14:05

Testing for beowulf 3.1 point release

#2 2020-12-04 16:04:05

Re: Testing for beowulf 3.1 point release

#3 2020-12-04 19:41:36

Re: Testing for beowulf 3.1 point release

#4 2020-12-05 18:54:36

Re: Testing for beowulf 3.1 point release

#5 2020-12-06 15:02:52

Re: Testing for beowulf 3.1 point release

#6 2020-12-06 15:43:52

Re: Testing for beowulf 3.1 point release

#7 2020-12-06 18:22:10

Re: Testing for beowulf 3.1 point release

#8 2020-12-06 18:55:16

Re: Testing for beowulf 3.1 point release

#9 2020-12-06 19:33:03

Re: Testing for beowulf 3.1 point release

#10 2020-12-07 13:00:10

Re: Testing for beowulf 3.1 point release

#11 2020-12-07 13:31:00

Re: Testing for beowulf 3.1 point release

#12 2020-12-07 13:31:53

Re: Testing for beowulf 3.1 point release

#13 2020-12-07 13:36:34

Re: Testing for beowulf 3.1 point release

#14 2020-12-07 13:48:48

Re: Testing for beowulf 3.1 point release

#15 2020-12-07 21:31:22

Re: Testing for beowulf 3.1 point release

#16 2020-12-13 12:07:47

Re: Testing for beowulf 3.1 point release

#17 2020-12-13 17:27:23

Re: Testing for beowulf 3.1 point release

#18 2020-12-14 09:48:27

Re: Testing for beowulf 3.1 point release

Board footer