OpenSSH fails to start with the latest update in Devuan 6 Excalibur

vazhnov · 2024-07-07 12:02:22

Hi!

I have a Proxmox host with KVM virtual machine with Devuan GNU/Linux 6 (excalibur/ceres) + OpenRC init.

Yesterday I've installed updates in the virtual machine guest and after reboot, SSHD couldn't start:

/var/log/syslog:

2024-07-07T01:00:42.824732+02:00 devuan6-kube01 /etc/init.d/ssh[1822]: start-stop-daemon: pam error: Authentication failure
2024-07-07T01:00:42.830420+02:00 devuan6-kube01 /etc/init.d/ssh[1821]: start-stop-daemon: failed to start `/usr/sbin/sshd'

Updates, from /var/log/apt/history.log:

Start-Date: 2024-07-07  00:59:42
Commandline: apt-get -V --no-install-recommends dist-upgrade
Requested-By: alexey (1000)
Upgrade: base-passwd:amd64 (3.6.3, 3.6.4), libeinfo1:amd64 (0.54-2, 0.54.2-1), gping:amd64 (1.16.1-1, 1.16.1-2), debian-keyring:amd64 (2024.03.24, 2024.06.24), librc1t64:amd64 (0.54-2, 0.54.2-1), openrc:amd64 (0.54-2, 0.54.2-1), ruby-xmlrpc:amd64 (0.3.2-2, 0.3.3-2)
End-Date: 2024-07-07  00:59:56

When I login through KVM console and start the service manually with sudo service ssh start, it starts.

I will investigate this. Just wanted to notify users to not install the latest updates if they have similar configuration.

vazhnov · 2024-07-07 12:19:06

Also I found that after the upgrade, Devuan fails to stop some services at shutdown:

2024-07-07T01:00:07.694393+02:00 devuan6-kube01 shutdown[2776]: shutting down for system reboot
2024-07-07T01:00:07.802340+02:00 devuan6-kube01 init: Switching to runlevel: 6
2024-07-07T01:00:07.803922+02:00 devuan6-kube01 kernel: Not activating Mandatory Access Control as /sbin/tomoyo-init does not exist.
2024-07-07T01:00:07.810263+02:00 devuan6-kube01 init: No inittab.d directory found
2024-07-07T01:00:07.916679+02:00 devuan6-kube01 start-stop-daemon: Will stop /usr/sbin/sshd
2024-07-07T01:00:07.916702+02:00 devuan6-kube01 start-stop-daemon: Will stop PID 1833
2024-07-07T01:00:07.917112+02:00 devuan6-kube01 start-stop-daemon: Sending signal 15 to PID 1833
2024-07-07T01:00:07.950597+02:00 devuan6-kube01 /etc/init.d/rsync[2806]: ERROR: rsync failed to stop
2024-07-07T01:00:08.032347+02:00 devuan6-kube01 start-stop-daemon: Will stop PID 1765
2024-07-07T01:00:08.032929+02:00 devuan6-kube01 start-stop-daemon: Sending signal 15 to PID 1765
2024-07-07T01:00:08.216425+02:00 devuan6-kube01 avahi-daemon[1643]: Got SIGTERM, quitting.
2024-07-07T01:00:08.217919+02:00 devuan6-kube01 avahi-daemon[1643]: Leaving mDNS multicast group on interface eth0.IPv6 with address 2a01:112f:xxx:yyy:zzz:xxx:yyy:zzz.
2024-07-07T01:00:08.218882+02:00 devuan6-kube01 avahi-daemon[1643]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.168.155.137.
2024-07-07T01:00:08.219253+02:00 devuan6-kube01 avahi-daemon[1643]: Leaving mDNS multicast group on interface lo.IPv6 with address ::1.
2024-07-07T01:00:08.219426+02:00 devuan6-kube01 avahi-daemon[1643]: Leaving mDNS multicast group on interface lo.IPv4 with address 127.0.0.1.
2024-07-07T01:00:08.236026+02:00 devuan6-kube01 avahi-daemon[1643]: avahi-daemon 0.8 exiting.
2024-07-07T01:00:08.364213+02:00 devuan6-kube01 start-stop-daemon: Will stop PID 1625
2024-07-07T01:00:08.364234+02:00 devuan6-kube01 start-stop-daemon: Will stop processes owned by UID 100
2024-07-07T01:00:08.364783+02:00 devuan6-kube01 start-stop-daemon: Sending signal 15 to PID 1625
2024-07-07T01:00:08.387275+02:00 devuan6-kube01 /etc/init.d/dbus[2928]: ERROR: dbus failed to stop

Last edited by vazhnov (2024-07-07 12:19:31)

fsmithred · 2024-07-08 13:35:28

I'm not seeing this problem before or after upgrade today of excalibur amd64 on hardware with sysvinit. SSH login from remote works, rsync and dbus stop/start normally as do other services.

vazhnov · 2024-07-08 22:34:16

fsmithred, thanks. This is OpenRC-related issue, probably connected with an upgrade openrc:amd64 (0.54-2, 0.54.2-1).

I've added rc_verbose=yes into /etc/rc.conf and now I see in /var/log/boot:

Tue Jul  9 00:07:18 2024: Starting system message bus: dbus * start-stop-daemon: fopen `/var/run/dbus/pid': No such file or directory
Tue Jul  9 00:07:18 2024:  * Detaching to start `/usr/bin/dbus-daemon' ...
Tue Jul  9 00:07:18 2024: .
Tue Jul  9 00:07:18 2024:  * Executing: /usr/lib/rc/sh/openrc-run.sh /usr/lib/rc/sh/openrc-run.sh /etc/init.d/avahi-daemon start
Tue Jul  9 00:07:18 2024: Starting Avahi mDNS/DNS-SD Daemon: avahi-daemon.
Tue Jul  9 00:07:18 2024:  * Executing: /usr/lib/rc/sh/openrc-run.sh /usr/lib/rc/sh/openrc-run.sh /etc/init.d/bootlogs start
Tue Jul  9 00:07:19 2024:  * Executing: /usr/lib/rc/sh/openrc-run.sh /usr/lib/rc/sh/openrc-run.sh /etc/init.d/console-setup.sh start
Tue Jul  9 00:07:19 2024: Setting up console font and keymap...done.
Tue Jul  9 00:07:19 2024:  * Executing: /usr/lib/rc/sh/openrc-run.sh /usr/lib/rc/sh/openrc-run.sh /etc/init.d/cron start
Tue Jul  9 00:07:19 2024: Starting periodic command scheduler: cron.
Tue Jul  9 00:07:19 2024:  * Executing: /usr/lib/rc/sh/openrc-run.sh /usr/lib/rc/sh/openrc-run.sh /etc/init.d/earlyoom start
Tue Jul  9 00:07:19 2024:  * Executing: /usr/lib/rc/sh/openrc-run.sh /usr/lib/rc/sh/openrc-run.sh /etc/init.d/elogind start
Tue Jul  9 00:07:19 2024: Starting session management daemon: elogind * start-stop-daemon: fopen `/run/elogind.pid': No such file or directory
Tue Jul  9 00:07:19 2024:  * Detaching to start `/usr/libexec/elogind' ...
Tue Jul  9 00:07:19 2024: .
Tue Jul  9 00:07:19 2024:  * Executing: /usr/lib/rc/sh/openrc-run.sh /usr/lib/rc/sh/openrc-run.sh /etc/init.d/sudo start
Tue Jul  9 00:07:19 2024:  * Executing: /usr/lib/rc/sh/openrc-run.sh /usr/lib/rc/sh/openrc-run.sh /etc/init.d/rmnologin start
Tue Jul  9 00:07:19 2024:  * Executing: /usr/lib/rc/sh/openrc-run.sh /usr/lib/rc/sh/openrc-run.sh /etc/init.d/rsync start
Tue Jul  9 00:07:19 2024:  * Executing: /usr/lib/rc/sh/openrc-run.sh /usr/lib/rc/sh/openrc-run.sh /etc/init.d/ssh start
Tue Jul  9 00:07:19 2024: Starting OpenBSD Secure Shell server: sshd * start-stop-daemon: fopen `/run/sshd.pid': No such file or directory
Tue Jul  9 00:07:19 2024:  * Detaching to start `/usr/sbin/sshd' ...
Tue Jul  9 00:07:19 2024:  *   start-stop-daemon: pam error: Authentication failure
Tue Jul  9 00:07:20 2024:  *   start-stop-daemon: failed to start `/usr/sbin/sshd'
Tue Jul  9 00:07:20 2024:  failed!
Tue Jul  9 00:07:20 2024:  * ERROR: rc.local needs service(s) 0, 6
Tue Jul  9 00:07:20 2024:  * Executing: /usr/lib/rc/sh/openrc-run.sh /usr/lib/rc/sh/openrc-run.sh /etc/init.d/stop-bootlogd start

I don't have custom records in /etc/fstab to mount /run…

Analysing git diff debian/0.54-2 debian/0.54.2-1 in https://salsa.debian.org/debian/openrc (web diff).

vazhnov · 2024-07-10 22:03:51

Currently there are two versions of start-stop-daemon binaries from two packages:

 % dpkg -S start-stop-daemon|sort                      
dpkg: /usr/sbin/start-stop-daemon
dpkg: /usr/share/man/de/man8/start-stop-daemon.8.gz
dpkg: /usr/share/man/fr/man8/start-stop-daemon.8.gz
dpkg: /usr/share/man/man8/start-stop-daemon.8.gz
dpkg: /usr/share/man/nl/man8/start-stop-daemon.8.gz
dpkg: /usr/share/man/pt/man8/start-stop-daemon.8.gz
dpkg: /usr/share/man/sv/man8/start-stop-daemon.8.gz
openrc: /usr/lib/rc/sbin/start-stop-daemon
openrc: /usr/lib/rc/sh/start-stop-daemon.sh

They are different:

 % file /usr/sbin/start-stop-daemon /usr/lib/rc/sbin/start-stop-daemon
/usr/sbin/start-stop-daemon:        ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=fb0de15cbe08c53aa308fddb2925afe3423ed34e, for GNU/Linux 3.2.0, stripped
/usr/lib/rc/sbin/start-stop-daemon: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6b69feea4068615a048ce10c763efb5869f798f8, for GNU/Linux 3.2.0, stripped

As an experiment, I decided to make OpenRC version of start-stop-daemon as the main one:

sudo cp -pv /usr/sbin/start-stop-daemon /usr/sbin/start-stop-daemon.dpkg
sudo cp -pv /usr/lib/rc/sbin/start-stop-daemon /usr/sbin/start-stop-daemon

After this change, I see no difference in Devuan shutdown/start (the same errors and SSHD can't start), but this time I couldn't start SSHD manually with sudo /etc/init.d/ssh start, I received the same error "start-stop-daemon: pam error: Authentication failure". Before the binaries swap, I was able to start the SSHD manually.

Then I returned binaries back and made second experiment:

sudo cp -pv /usr/sbin/start-stop-daemon.dpkg /usr/lib/rc/sbin/start-stop-daemon

And now my system starts and stops normally, SSHD starts automatically!

vazhnov · 2024-07-21 22:51:57

Checked with clean minimal Devuan 5 Daedalus installation: the same issue.

put openrc package on hold,
installed usrmerge,

upgraded to Devuan 6 Excalibur,

 % dpkg -l | { head -n4; grep -E -i 'sysvinit|sysv-rc|openrc|runit|sinit|66-devuan'; }
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                Version                  Architecture Description
ii  openrc                              0.45.2-2+deb12u1         amd64        dependency based service manager (runlevel change mechanism)
ii  runit-helper                        2.16.3                   all          dh-runit implementation detail
rc  sysv-rc                             3.06-4devuan3            all          System-V-like runlevel change mechanism
ii  sysvinit-core                       3.09-2devuan1            amd64        System-V-like init
ii  sysvinit-utils                      3.09-2devuan1            amd64        System-V-like utilities

 % sudo tail /var/log/boot
Mon Jul 22 02:29:13 2024: bound to 10.0.2.15 -- renewal in 35829 seconds.
Mon Jul 22 02:29:13 2024: done.
Mon Jul 22 02:29:13 2024: Cleaning up temporary files....
Mon Jul 22 02:29:13 2024:  * Mounting cgroup filesystem ...                                                                                                                         [ ok ]
INIT: Entering runlevel: 2
Mon Jul 22 02:29:14 2024: Setting up console font and keymap...done.
Mon Jul 22 02:29:14 2024: Starting enhanced syslogd: rsyslogd.
Mon Jul 22 02:29:14 2024: Starting periodic command scheduler: cron.
Mon Jul 22 02:29:14 2024: Starting system message bus: dbus.
Mon Jul 22 02:29:14 2024: Starting session management daemon: elogind.
Mon Jul 22 02:29:14 2024: Starting OpenBSD Secure Shell server: sshd.
Mon Jul 22 02:29:14 2024:  * ERROR: rc.local needs service(s) 0, 6

restarted the OS,

then upgraded the openrc package only:

alexey@devuan6-aqemu1 ~ % sudo apt-get -V --no-install-recommends dist-upgrade                               
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be REMOVED:
   librc1 (0.45.2-2+deb12u1)
The following NEW packages will be installed:
   librc1t64 (0.54.2-1)
The following packages will be upgraded:
   openrc (0.45.2-2+deb12u1 => 0.54.2-1)
1 upgraded, 1 newly installed, 1 to remove and 0 not upgraded.
Need to get 217 kB of archives.
After this operation, 82.9 kB of additional disk space will be used.
…
 * Caching service dependencies ...
Service `rc.local' needs non existent service `0'
Service `rc.local' needs non existent service `6'
Error: udev is the name of a real and virtual service.                                                                                                                                      [ ok ]
…

and restarted the OS. OpenSSH couldn't start, I had to start it manually from console:

alexey@devuan6-aqemu1 ~ % sudo tail /var/log/boot
INIT: Entering runlevel: 2
Mon Jul 22 02:39:27 2024: Setting up console font and keymap...done.
Mon Jul 22 02:39:27 2024: Starting enhanced syslogd: rsyslogd.
Mon Jul 22 02:39:28 2024: Starting periodic command scheduler: cron.
Mon Jul 22 02:39:28 2024: Starting system message bus: dbus.
Mon Jul 22 02:39:28 2024: Starting session management daemon: elogind.
Mon Jul 22 02:39:28 2024: Starting OpenBSD Secure Shell server: sshd * start-stop-daemon: pam error: Authentication failure
Mon Jul 22 02:39:28 2024:  * start-stop-daemon: failed to start `/usr/sbin/sshd'
Mon Jul 22 02:39:28 2024:  failed!
Mon Jul 22 02:39:28 2024:  * ERROR: rc.local needs service(s) 0, 6

And also workaround with copying /usr/sbin/start-stop-daemon into /usr/lib/rc/sbin/start-stop-daemon solves the issue.

Last edited by vazhnov (2024-07-21 22:54:51)

vazhnov · 2024-08-21 18:58:31

Where should I file a ticket — in bugs.devuan.org or in bugs.debian.org?

Lorenzo · 2024-08-21 20:59:30

Hi!

Is this is a side effects of the fix for #1038789 ? (not sure)
https://bugs.debian.org/cgi-bin/bugrepo … %231038789

Where should I file a ticket — in bugs.devuan.org or in bugs.debian.org?

the openrc package is not forked in Devuan, so please report directly to Debian!

Cheers,
Lorenzo

The officially official Devuan Forum!

#1 2024-07-07 12:02:22

OpenSSH fails to start with the latest update in Devuan 6 Excalibur

#2 2024-07-07 12:19:06

Re: OpenSSH fails to start with the latest update in Devuan 6 Excalibur

#3 2024-07-08 13:35:28

Re: OpenSSH fails to start with the latest update in Devuan 6 Excalibur

#4 2024-07-08 22:34:16

Re: OpenSSH fails to start with the latest update in Devuan 6 Excalibur

#5 2024-07-10 22:03:51

Re: OpenSSH fails to start with the latest update in Devuan 6 Excalibur

#6 2024-07-21 22:51:57

Re: OpenSSH fails to start with the latest update in Devuan 6 Excalibur

#7 2024-08-21 18:58:31

Re: OpenSSH fails to start with the latest update in Devuan 6 Excalibur

#8 2024-08-21 20:59:30

Re: OpenSSH fails to start with the latest update in Devuan 6 Excalibur

Board footer