The officially official Devuan Forum!

mirrortokyo

Member

Registered: 2021-04-08

Posts: 54  

Re: merged /usr

It's helpful to know what is different in your dmesg output compared with booting before usrmerge.

I had enough "fun" with incrementally adding symbolic links from old locations within /bin /lib /sbin to where the new locations in /usr/bin /usr/lib and /usr/sbin as individula packages changed the location of their files and finally installing the usrmerge package after I had manually done all the work.

Unfortunately when there are a lot of changes in one hit, it is difficult to identify the specific changes/errors that broke things.

TheLancashireman

Member

Registered: 2019-07-01

Posts: 13  

Re: merged /usr

I can't get the dmesg output for the last known good case, but I can extract the information from syslog. It'll take a while though.

In the cases where the kernel doesn't find the root fs there's no dmesg and no log, of course.

I just tried transplanting the kernel files (config-, initrd.img-, System.map- and vmlinuz-6.5.0-4-amd) from my laptop. Also ceres, but not updated for a while. I never had 6.5.0-4 on my desktop so I wouldn't expect it to work properly (missing modules) but in fact it works almost as well as the setup with the forced modules in the initrd. With that in mind, I can switch back to a boot set from an earlier kernel (probably 6.1.xxx). I still have those kernels installed, but all the initrd files in /boot have been polluted by mkinitramfs. Fortunately I made a backup before I started fiddling :-)

One more "interesting" fact: my work laptop runs debian sid/unstable. A couple of updates ago it appeared to stop booting at the point where the text display switches from 80x25 to high-res. In fact, it didn't stop booting - it was the console that didn't work. An older kernel worked fine, so I uninstalled the generic kernel package and nailed the kernel version at 6.1.27-1. Now that I look at the fs more closely, it has been usrmerged - /bin, /sbin etc. are symlinks. So maybe my trouble there was an early indication of usrmerge trouble. I'm leaving the company at the end of March, so I won't do any more updates in case they break something.

Last edited by TheLancashireman (2024-02-26 15:27:51)

TheLancashireman

Member

Registered: 2019-07-01

Posts: 13  

Re: merged /usr

First inkling of a problem:  https://bugs.devuan.org/cgi/bugreport.cgi?bug=828

That was before usrmerge was forced. Some scripts had hard-coded paths to /bin and /sbin, but the programs they use moved to /usr/bin and /usr/sbin

But the problem of hard-coded paths should go away after usrmerge, shouldn't it?

Last edited by TheLancashireman (2024-02-26 16:47:42)

golinux

Administrator

Registered: 2016-11-25

Posts: 3,316  

Re: merged /usr

That depends on how and when the merge is activated as well as the non-standard choices made by the user prior to the merge. Please see this usrmerge announcement posted a few days ago.

---

Get answers - Help Devuan - Report a bug - Search packages

TheLancashireman

Member

Registered: 2019-07-01

Posts: 13  

Re: merged /usr

The non-standard choices that I made earlier were:
* adding modprobe, lsmod & co as symlinks to kmod
* editing some shellscripts to add /usr/bin and /usr/sbin to the hard coded paths at the top

Anyway, FWIW the logs are here: https://experimental.thelancashireman.org/devuan-logs/
There's a description file (info.txt)

TheLancashireman

Member

Registered: 2019-07-01

Posts: 13  

Re: merged /usr

I've found a solution or a workaround: downgraded to eudev_3.2.12-4+deb11u1_amd64.deb and libeudev1_3.2.12-4+deb11u1_amd64.deb
Removed my hacked list of modules from /etc/initramfs-tools/modules and ran update-initramfs -u

Now my system boots kernel 6.6.15 correctly. Keyboard and mouse both working. At some point I should tidy up /boot again and regenerate all the initrd.img files

So it might not be related to usrmerge after all.

What now? Should I report a bug against eudev?

Last edited by TheLancashireman (2024-02-26 20:44:28)

mirrortokyo

Member

Registered: 2021-04-08

Posts: 54  

Re: merged /usr

Unpacking dpkg (1.22.5) over (1.22.4) ...
Setting up dpkg (1.22.5) ...
dpkg: warning: This system uses merged-usr-via-aliased-dirs, going behind dpkg's
dpkg: warning: back, breaking its core assumptions. This can cause silent file
dpkg: warning: overwrites and disappearances, and its general tools misbehavior.
dpkg: warning: See <https://wiki.debian.org/Teams/Dpkg/FAQ#broken-usrmerge>.

But the self-contradiction continues:

Q: Does dpkg support merged-/usr-via-aliased-dirs?
A: No. This approach is considered broken by design and breaks many common expectations.

In dpkg the expected breakage includes:

failing to notice file conflicts with the subsequent silent file overwrites by f.ex. dpkg, dpkg-divert and update-alternatives,

files disappearing during package upgrades or diversion installation,
failing to activate triggers on pathnames,
failing to find pathnames on dpkg-query -S searches,

failing to install packages shipping the same filename in real and aliased directories (with rather cryptic errors), f.ex. to be able to install otherwise dependency satisfiable packages needed by old software,
completely messing up the filesystem by simply using dpkg-deb -x or tar -x.

If you have a system that has been installed recently (since Debian buster) or switched via the usrmerge hack, you might want to consider using the dpkg-fsys-usrunmess program (but beware that it should not be used in systemd's emergency mode) or reinstalling. For further information see Teams/Dpkg/MergedUsr.

Debian officially only supports merged-/usr-via-aliased-dirs systems. Converting to an unmerged-/usr setup might break the system in unexpected ways in the future, including data loss or failure to boot.

boughtonp

Member

From: UK

Registered: 2023-01-19

Posts: 212  

Website

Re: merged /usr

See //sources.debian.org/src/dpkg/1.22.5/debian/dpkg.postinst/

Presumably the Devuan vendor value is not "debian", and someone needs to do what Kali have done and submit a patch.

Last edited by boughtonp (2024-02-27 16:21:49)

---

3.1415P265E589T932E846R64338

TheLancashireman

Member

Registered: 2019-07-01

Posts: 13  

Re: merged /usr

For what it's worth - just ran apt-update and noticed new versions of eudev and libeudev1 incoming. After upgrading them (and only them) PC boots with no problems.
Then did a full upgrade. Now cryptsetup is causing trouble, but that's a different matter.

I got the warning from dpkg too, and a few warnings about dependency problems during the upgrade.

Last edited by TheLancashireman (2024-02-28 18:00:23)

soren

Member

Registered: 2023-04-30

Posts: 142  

Re: merged /usr

Has anyone got an idea how to fix this?
I tried upgrading from stable(daedalus) to unstable(ceres) on a spare partition and seems it has failed spectacularly due to usrmerge.

The system was rsync'ed to a new partition and then modified.

So /etc/apt/sources.list looks like below.

deb http://deb.devuan.org/merged ceres main contrib non-free non-free-firmware

The update / upgrade was going fine until i did apt-get upgrade

Error messages as follows....

# apt-get upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libc-dev-bin : Depends: libc6 (> 2.38) but 2.36-9+deb12u3 is installed
                Recommends: manpages-dev but it is not installed
 libc6-dev : Depends: libc6 (= 2.38-12.1) but 2.36-9+deb12u3 is installed
 locales : Depends: libc-bin (> 2.38) but 2.36-9+deb12u3 is installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

apt --fix-broken install gives below error

The following packages will be upgraded:
  libc-bin libc6
2 upgraded, 0 newly installed, 0 to remove and 828 not upgraded.
5 not fully installed or removed.
Need to get 0 B/3,381 kB of archives.
After this operation, 289 kB disk space will be freed.
Do you want to continue? [Y/n] Y

Y

Reading changelogs... Done
Preconfiguring packages ...
(Reading database ... 90625 files and directories currently installed.)
Preparing to unpack .../libc6_2.38-12.1_amd64.deb ...

The current installation does not have a merged-/usr layout.
This is unsupported and unpacking libc6 would break the system.
Refusing to unpack. Please install the usrmerge package and try again.

dpkg: error processing archive /var/cache/apt/archives/libc6_2.38-12.1_amd64.deb (--unpack):
 new libc6:amd64 package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/libc6_2.38-12.1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Its just stuck in a loop now, im thinking i may have needed to install usrmerge first??

EDIT to add. I am upgrading the devuan partition from an artix linux system, so im chrooting into the devuan system via artix linux.

EDIT 2, i cant install usrmerge either if anyone is wondering, it throws similar errors...

/# apt install usrmerge
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libc-dev-bin : Depends: libc6 (> 2.38) but 2.36-9+deb12u3 is to be installed
                Recommends: manpages-dev but it is not going to be installed
 libc6-dev : Depends: libc6 (= 2.38-12.1) but 2.36-9+deb12u3 is to be installed
 locales : Depends: libc-bin (> 2.38) but 2.36-9+deb12u3 is to be installed
 usrmerge : Depends: libfile-find-rule-perl but it is not going to be installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

Last edited by soren (2024-06-07 08:06:07)

ralph.ronnquist

Administrator

From: Battery Point, Tasmania, AUS

Registered: 2016-11-30

Posts: 1,251  

Re: merged /usr

Yes, that's what it says, isn't it?

The current installation does not have a merged-/usr layout.
This is unsupported and unpacking libc6 would break the system.
Refusing to unpack. Please install the usrmerge package and try again.

soren

Member

Registered: 2023-04-30

Posts: 142  

Re: merged /usr

So usrmerge needs to be installed first before a big upgrade like this? Noted.

soren

Member

Registered: 2023-04-30

Posts: 142  

Re: merged /usr

Thats fine, but i cant install usrmerge, i cant install anything now, apt/dpkg is broken.
EDIT: this was a reply to someone but they deleted the post.

Last edited by soren (2024-06-07 08:15:03)

soren

Member

Registered: 2023-04-30

Posts: 142  

Re: merged /usr

Okay so i removed the filesystem and started again, this time installing usrmerge first. I get this error now??
Note that the system is stable(daedalus) no upgrade has occurred yet.

FATAL ERROR:
Both /lib/x86_64-linux-gnu/libsystemd.so.0 and /usr/lib/x86_64-linux-gnu/libsystemd.so.0 exist.

You can try correcting the errors reported and running again
/usr/lib/usrmerge/convert-usrmerge until it will complete without errors.
Do not install or update other Debian packages until the program
has been run successfully.

E: usrmerge failed.
dpkg: error processing package usrmerge (--configure):
 installed usrmerge package post-installation script subprocess returned error exit status 1
Processing triggers for man-db (2.11.2-2) ...
Errors were encountered while processing:
 usrmerge

soren

Member

Registered: 2023-04-30

Posts: 142  

Re: merged /usr

Ok so moving both /lib/x86_64-linux-gnu/libsystemd.so.0 and /usr/lib/x86_64-linux-gnu/libsystemd.so.0 out of the way successfully converted the system, i hope.

apt install usrmerge
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
usrmerge is already the newest version (37~deb12u1).
0 upgraded, 0 newly installed, 0 to remove and 68 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] 
Setting up usrmerge (37~deb12u1) ...
The system has been successfully converted.

soren

Member

Registered: 2023-04-30

Posts: 142  

Re: merged /usr

Upgrade from daedalus to ceres went fine after installing usrmerge first. Im not sure what i was thinking, i thought maybe apt might throw a warning to install usrmerge first before upgrading, but probably too soon as stable to unstable not covered. Sorry for the noise.

delgado

Member

Registered: 2022-07-14

Posts: 212  

Re: merged /usr

Thats fine, but i cant install usrmerge, i cant install anything now, apt/dpkg is broken.
EDIT: this was a reply to someone but they deleted the post.

Sorry for the inconvenience. The post's online time was less than 5 minutes. Tight timing!
It was the 3rd post in less than three minutes (after ralph and soren), saying that you need usrmeger for ceres.
BTW: In such cases, It is possible install a downloaded package via dpkg -i /path_to/some-package.deb manually.

Anyway. Good to hear that you made it!

chris2be8

Member

Registered: 2018-08-11

Posts: 307  

Re: merged /usr

Re post 89 by soren (at 09:32:57) my first response would be:
ls -li /lib/x86_64-linux-gnu/libsystemd.so.0 /usr/lib/x86_64-linux-gnu/libsystemd.so.0
and see if one was a symlink to the other or if they are hard links (same inode numbers). If different files but the same size I'd then try diff /lib/x86_64-linux-gnu/libsystemd.so.0 /usr/lib/x86_64-linux-gnu/libsystemd.so.0 to see if one is a copy of the other.

If they are the same inode numbers I'd check if /lib and /usr/lib are symlinks (ls -lid /lib/x86_64-linux-gnu/ /usr/lib/x86_64-linux-gnu/ then ls -lid /lib/ /usr/lib/). That would give me a better idea what state things are in.

mirrortokyo

Member

Registered: 2021-04-08

Posts: 54  

Re: merged /usr

Hmm, upgrade

[UPGRADE] base-files:amd64 13.3devuan1 -> 13.4devuan1

failed because I had symlinks from /bin -> /usr/bin and /sbin -> /usr/sbin still present.

However, a lot of scripts in /etc/init.d still have /bin and /sbin hard-coded, and removing those symbolic links causes a boot failure.

I've put package base-files on hold for now.

Last edited by mirrortokyo (2024-08-02 05:16:57)

mirrortokyo

Member

Registered: 2021-04-08

Posts: 54  

Re: merged /usr

What I needed to have in / was:

bin -> usr/bin
lib -> usr/lib
lib64 -> usr/lib64
sbin -> usr/sbin

After that, the base-files upgrade worked.

PS, on my i386 installation I had to use /usr/lib/klibc/bin/ln if the default ln program was unavailable while changing symbolic links.

Last edited by mirrortokyo (2024-08-04 14:37:35)

stultumanto

Member

Registered: 2023-12-12

Posts: 68  

Re: merged /usr

There are also a couple of directories for 32-bit libraries that become links in a default merged system. Here is a full listing from a system I created by running debootstrap with the --merged-usr option:

$ find / -maxdepth 1 -type l -xtype d -printf '%p -> %l\n'
/libx32 -> usr/libx32
/sbin -> usr/sbin
/bin -> usr/bin
/lib32 -> usr/lib32
/lib64 -> usr/lib64
/lib -> usr/lib

fanderal

Member

Registered: 2017-01-14

Posts: 80  

Re: merged /usr

I'm on Ceres and had to install usrmerge a month or so ago. No problems. Thanks to @GlennW and @Altoid, finding any symlinks usrmerge missed can be corrected by hand.

# symlinks -csrv / | grep dangling