system with complicated mdadm/cryptsetup does not boot

markus23 · 2025-08-21 05:23:33

Dear Devuan Community,

probably I should have reached out earlier, now the situation is a bit complicated. Let me start with the most recent problem.

recent problem

Today my Devuan System (12.11 with Kernel 5.10.0-32-amd64) did not boot anymore, it got stuck with the message:

Starting early crypto disks..md-crypt (starting)...Please unlock disk md-crypt:
[  28.84] udevd[4238]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
...
[ 209.84] udevd[4236]: timeout '/sbin/lvm vgchange -aay --autoactivation event lvm-raid'
[ 210.25] udevd[602]: worker[4236] /devices/virtual/block/dm-10 timeout; kill it
[ 210.25] udevd[602]: seq 3283 '/devices/virtual/block/dm-10' killed
[ 210.25] udevd[602]: worker[4236] failed while handling '/devices/virtual/block/dm-10'

Ctrl+Print rebooted, leading to the same error again.
I tried 3 times.

/etc/crypttab is:

# <target name> <source device>         <key file>      <options>
#/dev/md0 
md-crypt UUID=...-3d90f2cea4dc none luks

/etc/mdadm/mdadm.conf is (shortened):

HOMEHOST <system>
ARRAY /dev/md/0  metadata=1.2 UUID=...:42d0672f name=markusbyte:0
# This configuration was auto-generated on Thu, 14 May 2020 10:29:04 +0200 by mkcon

I was able to boot via rescue mode and by manually executing

/sbin/lvm vgchange -aay --autoactivation event lvm-raid

and then

mount -a

all data was there and everything worked.

Before it completely stopped booting, it seemed that bootup was slower and slower every time, with many udevd errors.

earlier problem

You probably ask: why is this system running such an old kernel?
This dates back to another problem I already faced when installing Devuan on that machine, still present when trying to boot with a version 6.1 Linux kernel. When booting with 6.1 I consistently get following errors (also already before the recent problem turned up):

mdadm: No devices listed in conf file were found.
... [many times repated]
mdadm: No devices listed in conf file were found.
mdadm: error opening /dev/md?*: No such file or directory
Gaving up waiting for suspend/resume device

questions

When shutting down there are also several errors of crypt devices.

Reinstallation of devuan would be quite a challenge, as I have many (partly self-created) packages.

How can I get my system again booting and shutting down normally, ideally with the recent kernels, without errors?

Help is very much appreciated.

more information and logs

root is also on lvm (on sda) but separated from the big lvm-raid (all other disks) which seems to cause the bootup problems.

lsblk says:

sda                                     8:0    0   1,8T  0 disk  
└─sda1                                  8:1    0   1,8T  0 part  
  ├─lvm--ssd-swap                     253:0    0    40G  0 lvm   [SWAP]
  ├─lvm--ssd-root-real                253:2    0   200G  0 lvm   
  │ │                                                            /
...
sdb                                     8:16   0   7,3T  0 disk  
└─md0                                   9:0    0  29,1T  0 raid6 
  ├─md-crypt_unformatted              253:10   0  29,1T  0 crypt 
  │ ├─lvm--raid-home                  253:11   0   260G  0 lvm   /home
...
sdg                                     8:96   0   7,3T  0 disk  
└─md0                                   9:0    0  29,1T  0 raid6 
  ├─md-crypt_unformatted              253:10   0  29,1T  0 crypt 
  │ ├─lvm--raid-home                  253:11   0   260G  0 lvm   /home
...

/proc/mdstat contains:

md0 : active raid6 sdd[7] sdc[8] sde[9] sdf[6] sdb[0] sdg[5]
      31255576576 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/6] [UUUUUU]
      bitmap: 0/59 pages [0KB], 65536KB chunk

mdadm -D /dev/md0 says:

/dev/md0:
           Version : 1.2
     Creation Time : Fri Mar  6 22:04:49 2020
        Raid Level : raid6
        Array Size : 31255576576 (29.11 TiB 32.01 TB)
     Used Dev Size : 7813894144 (7.28 TiB 8.00 TB)
      Raid Devices : 6
     Total Devices : 6
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Thu Aug 21 06:57:59 2025
             State : clean 
    Active Devices : 6
   Working Devices : 6
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : markusbyte:0  (local to host markusbyte)
              UUID : ...:42d0672f
            Events : 1073251

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       8       8       32        1      active sync   /dev/sdc
       7       8       48        2      active sync   /dev/sdd
       5       8       96        3      active sync   /dev/sdg
       6       8       80        4      active sync   /dev/sdf
       9       8       64        5      active sync   /dev/sde

vgdisplay says:

  --- Volume group ---
  VG Name               lvm-raid
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  47
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                16
  Open LV               13
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <29,11 TiB
  PE Size               4,00 MiB
  Total PE              7630751
  Alloc PE / Size       6488373 / 24,75 TiB
  Free  PE / Size       1142378 / <4,36 TiB
  VG UUID               ...-LlfW-hjm3zb
   
  --- Volume group ---
  VG Name               lvm-ssd
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  61
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <1,82 TiB
  PE Size               4,00 MiB
  Total PE              476931
  Alloc PE / Size       343643 / 1,31 TiB
  Free  PE / Size       133288 / <520,66 GiB
  VG UUID               ...-I148YJ

uname -a (with the only kernel that actually boots, but from today only via rescue mode)

Linux markusbyte 5.10.0-32-amd64 #1 SMP Debian 5.10.223-1 (2024-08-10) x86_64 GNU/Linux

sudo lvdisplay lvm-raid contains many devices including:

  --- Logical volume ---
  LV Path                /dev/lvm-raid/p
  LV Name                p
  VG Name                lvm-raid
  LV UUID                ...-ZkXWUR
  LV Write Access        read/write
  LV Creation host, time markusbyte, 2020-... +0100
  LV Status              available
  # open                 0
  LV Size                80,00 GiB
  Current LE             20480
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:19

which is again a encrypted device within the encrypted volume, usually not mounted.

udev rules I only have about usb and net, so it should be unrelated?

SUBSYSTEM=="usb", SYSFS{idVendor}=="...", SYSFS{idProduct}=="...", ACTION=="add", GROUP="audio", MODE="0664"
SUBSYSTEM=="usb", ATTR{idVendor}=="...", MODE="0666", GROUP="plugdev
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="...", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

dmesg excerpt:

[    0.000000] microcode: microcode updated early to revision 0x28, date = 2019-11-12
[    0.000000] Linux version 5.10.0-32-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.223-1 (2024-08-10)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.10.0-32-amd64 root=/dev/mapper/lvm--ssd-root ro single
...
[    0.024315] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.10.0-32-amd64 root=/dev/mapper/lvm--ssd-root ro single
...
[    1.335601] ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.SAT0.SPT5._GTF.DSSP], AE_NOT_FOUND (20200925/psargs-330)
[    1.335669] ACPI Error: Aborting method \_SB.PCI0.SAT0.SPT5._GTF due to previous error (AE_NOT_FOUND) (20200925/psparse-529)
[    1.336530] ata3.00: ATA-10: TOSHIBA HDWF180, GX0B, max UDMA/100
[    1.336587] ata3.00: 15628053168 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    1.336731] ata6.00: ATA-10: TOSHIBA HDWN180, GX2M, max UDMA/100
[    1.336807] ata6.00: 15628053168 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    1.338075] ata1.00: supports DRM functions and may not be fully accessible
[    1.338180] ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.SAT0.SPT5._GTF.DSSP], AE_NOT_FOUND (20200925/psargs-330)
[    1.338280] ACPI Error: Aborting method \_SB.PCI0.SAT0.SPT5._GTF due to previous error (AE_NOT_FOUND) (20200925/psparse-529)
[    1.338477] ata3.00: configured for UDMA/100
[    1.338785] ata1.00: disabling queued TRIM support
[    1.338796] ata1.00: ATA-11: Samsung SSD 870 QVO 2TB, SVQ02B6Q, max UDMA/133
[    1.338884] ata1.00: 3907029168 sectors, multi 1: LBA48 NCQ (depth 32), AA
[    1.339456] ata6.00: configured for UDMA/100
[    1.340355] ata2.00: ATA-9: WDC WD82PURZ-85TEUY0, 82.00A82, max UDMA/133
[    1.340434] ata2.00: 15628053168 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    1.340958] ata1.00: supports DRM functions and may not be fully accessible
[    1.341466] ata1.00: disabling queued TRIM support
[    1.343167] ata1.00: configured for UDMA/133
[    1.343415] scsi 0:0:0:0: Direct-Access     ATA      Samsung SSD 870  2B6Q PQ: 0 ANSI: 5
[    1.349259] ata2.00: configured for UDMA/133
[    1.349623] scsi 1:0:0:0: Direct-Access     ATA      WDC WD82PURZ-85T 0A82 PQ: 0 ANSI: 5
[    1.350382] scsi 2:0:0:0: Direct-Access     ATA      TOSHIBA HDWF180  GX0B PQ: 0 ANSI: 5
[    1.373015] ata4.00: ATA-10: ST8000DM004-2CX188, 0001, max UDMA/133
[    1.373089] ata4.00: 15628053168 sectors, multi 16: LBA48 NCQ (depth 32), AA
...
[    1.515135] scsi 6:0:0:0: Direct-Access     ATA      WDC WD80EFAX-68L 0A83 PQ: 0 ANSI: 
...
[    1.991816] ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    1.995229] ata8.00: ATA-9: WDC WD80EZZX-11CSGA0, 83.H0A03, max UDMA/133
[    1.995302] ata8.00: 15628053168 sectors, multi 0: LBA48 NCQ (depth 32), AA
[    2.001054] ata8.00: configured for UDMA/133
...
[    2.080299] sd 1:0:0:0: [sdb] Attached SCSI disk
[    2.084207] sd 6:0:0:0: [sdf] Attached SCSI disk
[    2.092075] sd 2:0:0:0: [sdc] Attached SCSI disk
[    2.092084] sd 5:0:0:0: [sde] Attached SCSI disk
...
[    2.964920] md/raid:md0: device sdg operational as raid disk 3
[    2.965556] md/raid:md0: raid level 6 active with 6 out of 6 devices, algorithm 2
[    3.017539] md0: detected capacity change from 0 to 32005710413824
[    3.023856]  md0: AHDI p1 p2 p3
...
[    9.415062] udevd[601]: starting version 3.2.12
[    9.424943] udevd[601]: unknown key 'SYSFS{idVendor}' in /etc/udev/rules.d/41-odvr.rules:1
[    9.425007] udevd[601]: invalid rule '/etc/udev/rules.d/41-odvr.rules:1'
[    9.453655] udevd[602]: starting eudev-3.2.12
[    9.456412] udevd[602]: unknown key 'SYSFS{idVendor}' in /etc/udev/rules.d/41-odvr.rules:1
[    9.456476] udevd[602]: invalid rule '/etc/udev/rules.d/41-odvr.rules:1'
[    9.458943] udevd[607]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[    9.459200] udevd[610]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[    9.459492] udevd[612]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[    9.459704] udevd[614]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[    9.459783] udevd[616]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
...
[   10.020743] uvcvideo: Failed to set UVC probe control : -32 (exp. 26).
...
[   29.263665]  md0: AHDI p1 p2 p3
[   29.264815] udevd[4228]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[   29.265029] udevd[4230]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[   29.265187] udevd[4231]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[   29.265438] udevd[4232]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[   29.275859] udevd[4238]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[   29.334635] udevd[4242]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[   29.356814] udevd[4243]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[   29.396869] udevd[4246]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[   29.396941] udevd[4248]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[   29.397091] udevd[4250]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[   90.395837] udevd[602]: worker [4225] /devices/virtual/block/dm-10 is taking a long time
[  210.365044] udevd[4225]: timeout '/sbin/lvm vgchange -aay --autoactivation event lvm-raid'
[  210.365140] udevd[4225]: slow: '/sbin/lvm vgchange -aay --autoactivation event lvm-raid' [4244]
[  210.507867] udevd[602]: worker [4225] /devices/virtual/block/dm-10 timeout; kill it
[  210.507991] udevd[602]: seq 3259 '/devices/virtual/block/dm-10' killed
[  210.508669] udevd[602]: worker [4225] terminated by signal 9 (Killed)
[  210.508835] udevd[602]: worker [4225] failed while handling '/devices/virtual/block/dm-10'
[  216.758729] udevd[4386]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  217.010675] udevd[4388]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  217.104395] udevd[4390]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
... (below probably occurred while I was manually executing mount -a in the rescue mode):
[ 1470.716659] EXT4-fs (dm-11): mounted filesystem with ordered data mode. Opts: (null)
[ 1475.975692] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
[ 1484.983625] EXT4-fs (dm-14): mounted filesystem with ordered data mode. Opts: (null)
[ 1485.118698] EXT4-fs (dm-15): mounted filesystem with ordered data mode. Opts: (null)
[ 1485.231346] EXT4-fs (dm-16): mounted filesystem with ordered data mode. Opts: (null)
[ 1485.341633] EXT4-fs (dm-12): mounted filesystem with ordered data mode. Opts: (null)
[ 1485.544592] EXT4-fs (dm-26): mounted filesystem with ordered data mode. Opts: (null)
[ 1485.720898] EXT4-fs (dm-17): mounted filesystem with ordered data mode. Opts: (null)
[ 1486.017107] EXT4-fs (dm-21): mounted filesystem with ordered data mode. Opts: (null)
[ 1486.173414] EXT4-fs (dm-20): mounted filesystem with ordered data mode. Opts: (null)
[ 1486.362840] EXT4-fs (dm-24): mounted filesystem with ordered data mode. Opts: (null)
[ 1486.508462] EXT4-fs (dm-13): mounted filesystem with ordered data mode. Opts: (null)
[ 1486.634011] EXT4-fs (dm-18): mounted filesystem with ordered data mode. Opts: (null)
[ 1486.757678] SGI XFS with ACLs, security attributes, realtime, quota, no debug enabled
[ 1486.759493] udevd[4497]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[ 1486.809792] XFS (dm-23): Mounting V5 Filesystem
[ 1488.390585] XFS (dm-23): Ending clean mount
[ 1488.455285] xfs filesystem being mounted at /home/data-xfs supports timestamps until 2038 (0x7fffffff)
...
[ 1530.166601] udevd[4536]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[ 1530.273388] udevd[4949]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[ 1530.290600] udevd[5005]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[ 1530.297367] udevd[5021]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[ 1530.298652] udevd[5022]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[ 1530.316103] udevd[5043]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[ 1530.319615] udevd[5061]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[ 1530.321359] udevd[5065]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[ 1530.321402] udevd[5066]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[ 1530.326623] audit: type=1400 audit(1755746174.019:2): apparmor="DENIED" operation="change_onexec" info="label not found" error=-2 profile="unconfined" name="system_tor" pid=5069 comm="aa-exec"
[ 1530.335599] udevd[5075]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
...

Last edited by markus23 (2025-08-21 05:27:45)

g4sra · 2025-08-21 08:14:41

For now...you are being presented with the prompt

Please unlock disk md-crypt:

Ignore all further udev errors and enter nothing but and only the passphrase

Start by dealing with

apparmor="DENIED" operation="change_onexec"

temporarily suppress apparmor while you error trace.

Then fix

invalid rule '/etc/udev/rules.d/41-odvr.rules:1'

before rebooting.

NB: I never set up my systems to automount crypted mdraid (\home in your case) because of the nightmare when things go wrong.
My preference for sysvinit makes it trivial to boot runlevel 2 and providing that is error free automatically trigger runlevel 3 which then mounts the md also starting any daemons dependant on it.
When an error does occur the system still boots and root can login (over the network if headless) in runlevel 2 and repair the raid.

markus23 · 2025-08-21 09:45:34

Dear g4sra,

thank you for the quick reply!

Of course I entered the correct passphrase. This is actually a prerequisite of the error showing up. Unlike Debian, Devuan doesn't seem to timeout as long the passphrase is not entered. This is a great feature, as it does not force me to wait in front of the computer and immediately enter the passphrase.

temporarily suppress apparmor while you error trace.

I don't understand the sentence. Can you please elaborate? Is

apt remove apparmor

okay?

invalid rule

Is commenting out the rule okay?

My preference for sysvinit makes it trivial to boot runlevel 2 and providing that is error free automatically trigger runlevel 3 which then mounts the md also starting any daemons dependant on it.

Good idea, it would have made my system ending up in runlevel 2 instead of failing completely?

Do you have your setup somewhere described? But probably I first should fix this problem described above before making other changes?

Thanks again.

Last edited by markus23 (2025-08-21 09:47:33)

g4sra · 2025-08-21 10:05:05

Assuming you are using sysvinit...

Just (sudo if not root) for now to prevent apparmor startup at boot

sudo update-rc.d apparmor disable

Once your system is booting properly, turn it back on

sudo update-rc.d apparmor enable

The directory '/etc/udev/rules.d/' is for custom rules created by the administrator (not from an installed package).
I have no idea what that rule is for, if short, post its contents to this thread.

But probably I first should fix this problem

Always fix what is broken first before making 'enhancements'.

markus23 · 2025-08-21 11:29:26

Thank you for the clarifications! Unfortunately, it did not help, both kernels still don't boot with same/similar error messages.

What I did?

1. I removed all my custom udev rules except the /etc/udev/rules.d/70-persistent-net.rules

2. disabled apparmor

> sudo update-rc.d apparmor disable
insserv: warning: current start runlevel(s) (empty) of script `apparmor' overrides LSB defaults (S).
insserv: warning: current stop runlevel(s) (S) of script `apparmor' overrides LSB defaults (empty).

What happened?

1. With the 6.1 kernel I still get

mdadm: No devices listed in conf file were found.

2. With the 5.10 kernel, I got the very similar error message than before (manually typed from a photo, so beware of typos, better read dmesg below):

Starting early crypto disks..md-crypt (starting)...Please unlock disk md-crypt:
md0: AHDI p1 p2 P3
[  28.84] udevd[4238]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
... (again several times)
[ 232] udevd[601] worker [4236] /devices/virtual/block/dm-10 is taking a long time
[ 351] udevd[4236]: timeout '/sbin/lvm vgchange -aay --autoactivation event lvm-raid'
[ 351] udevd[4236] slow:  '/sbin/lvm vgchange -aay --autoactivation event lvm-raid'
[ 351] udevd[602]: worker[4236] /devices/virtual/block/dm-10 timeout; kill it
[ 351] udevd[602]: seq 3283 '/devices/virtual/block/dm-10' killed
[ 252] udevd[602]: worker[4236] terminated by signal 9 (killed)
[ 252] udevd[602]: worker[4236] failed while handling '/devices/virtual/block/dm-10'
[  28.84] udevd[4238]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
... (again several times)

dmesg says now (without typos and correct times/pids):

...
[    2.976547] md/raid:md0: device sdd operational as raid disk 2
[    2.976601] md/raid:md0: device sdf operational as raid disk 4
[    2.976655] md/raid:md0: device sdb operational as raid disk 0
[    2.977312] md/raid:md0: raid level 6 active with 6 out of 6 devices, algorithm 2
[    3.030750] md0: detected capacity change from 0 to 32005710413824
[    3.037153]  md0: AHDI p1 p2 p3
[    9.114539] PM: Image not found (code -22)
[    9.117030] PM: hibernation: Marking nosave pages: [mem 0x00000000-0x00000fff]
[    9.117032] PM: hibernation: Marking nosave pages: [mem 0x0009d000-0x000fffff]
[    9.117034] PM: hibernation: Marking nosave pages: [mem 0xcc2d6000-0xcc2dcfff]
[    9.117035] PM: hibernation: Marking nosave pages: [mem 0xcc725000-0xccd35fff]
[    9.117056] PM: hibernation: Marking nosave pages: [mem 0xde366000-0xde3f4fff]
[    9.117059] PM: hibernation: Marking nosave pages: [mem 0xde441000-0xdeffefff]
[    9.117098] PM: hibernation: Marking nosave pages: [mem 0xdf000000-0xffffffff]
[    9.117480] PM: hibernation: Basic memory bitmaps created
[    9.117814] PM: hibernation: Basic memory bitmaps freed
[    9.199066] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[    9.238095] Not activating Mandatory Access Control as /sbin/tomoyo-init does not exist.
[    9.433415] udevd[600]: starting version 3.2.12
[    9.470542] udevd[601]: starting eudev-3.2.12
[    9.475879] udevd[606]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[    9.476197] udevd[609]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[    9.476472] udevd[610]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[    9.476593] udevd[612]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[    9.476688] udevd[614]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[    9.477141] udevd[618]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[    9.477234] udevd[619]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[    9.477434] udevd[622]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[    9.477474] udevd[624]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[    9.509732] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input2
...
[   12.628315] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  170.680562]  md0: AHDI p1 p2 p3
[  170.681867] udevd[4887]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  170.682098] udevd[4889]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  170.682251] udevd[4890]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  170.682466] udevd[4891]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  170.692189] udevd[4897]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  170.761191] udevd[4901]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  170.763189] udevd[4902]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  170.873657] udevd[4905]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  170.873720] udevd[4907]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  170.873883] udevd[4909]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  232.018315] udevd[601]: worker [4885] /devices/virtual/block/dm-10 is taking a long time
[  351.691902] udevd[4885]: timeout '/sbin/lvm vgchange -aay --autoactivation event lvm-raid'
[  351.691998] udevd[4885]: slow: '/sbin/lvm vgchange -aay --autoactivation event lvm-raid' [4903]
[  352.123899] udevd[601]: worker [4885] /devices/virtual/block/dm-10 timeout; kill it
[  352.123983] udevd[601]: seq 3259 '/devices/virtual/block/dm-10' killed
[  352.124480] udevd[601]: worker [4885] terminated by signal 9 (Killed)
[  352.124549] udevd[601]: worker [4885] failed while handling '/devices/virtual/block/dm-10'
[  372.779844] udevd[5048]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  373.033539] udevd[5050]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  373.106437] udevd[5055]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  421.779937] EXT4-fs (dm-11): mounted filesystem with ordered data mode. Opts: (null)
[  421.816705] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
[  422.005089] EXT4-fs (dm-14): mounted filesystem with ordered data mode. Opts: (null)
[  422.142728] EXT4-fs (dm-15): mounted filesystem with ordered data mode. Opts: (null)
[  422.266537] EXT4-fs (dm-16): mounted filesystem with ordered data mode. Opts: (null)
[  422.375852] EXT4-fs (dm-12): mounted filesystem with ordered data mode. Opts: (null)
[  422.566794] EXT4-fs (dm-26): mounted filesystem with ordered data mode. Opts: (null)
[  422.752475] EXT4-fs (dm-17): mounted filesystem with ordered data mode. Opts: (null)
[  423.072050] EXT4-fs (dm-21): mounted filesystem with ordered data mode. Opts: (null)
[  423.205209] EXT4-fs (dm-20): mounted filesystem with ordered data mode. Opts: (null)
[  423.418098] EXT4-fs (dm-24): mounted filesystem with ordered data mode. Opts: (null)
[  423.592117] EXT4-fs (dm-13): mounted filesystem with ordered data mode. Opts: (null)
[  423.681453] EXT4-fs (dm-18): mounted filesystem with ordered data mode. Opts: (null)
[  423.813930] SGI XFS with ACLs, security attributes, realtime, quota, no debug enabled
[  423.816044] udevd[5101]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  423.862621] XFS (dm-23): Mounting V5 Filesystem
[  425.567980] XFS (dm-23): Ending clean mount
[  425.669955] xfs filesystem being mounted at /home/data-xfs supports timestamps until 2038 (0x7fffffff)
[  430.627981] Not activating Mandatory Access Control as /sbin/tomoyo-init does not exist.
[  433.748583] udevd[5139]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  433.852460] udevd[5524]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  433.871622] udevd[5576]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  433.879495] udevd[5591]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  433.881231] udevd[5598]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  433.893024] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[  433.895254] udevd[5623]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  433.897923] udevd[5641]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  433.898479] udevd[5645]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  433.898737] udevd[5646]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  433.906813] audit: type=1400 audit(1755774538.281:2): apparmor="DENIED" operation="change_onexec" info="label not found" error=-2 profile="unconfined" name="system_tor" pid=5660 comm="aa-exec"
[  433.909421] udevd[5672]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
[  434.839927] audit: type=1400 audit(1755774539.217:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="docker-default" pid=5837 comm="apparmor_parser"

Workaround?

In case someone else has this problem, too, here the full sequence of how to boot again:

0. boot into rescue mode
1. type in password and wait until timeout
2. press "Print"/"Druck" and type root password
3. lvm vgchange -aay --autoactivation event lvm-raid
4. mount -a
5. /sbin/resolvconf --enable-updates
6. init 2

Further suggestions are very much welcomed.

Last edited by markus23 (2025-08-21 11:30:15)

g4sra · 2025-08-21 14:16:19

A very quick quickie, I have just had my ear bent by the boss this morning for being here when I should be working on other things.

That all looks very fishy to me, what is the history of this system ?
Was it a fresh Devuan install or did you migrate from something else ?

I havn't seen your /etc/fstab so this is an assumption...
Appending 'nofail' as an option to the mount line for '/home' may permit the boot to complete without having to reboot into rescue

The historic errors shutting down your crypt device are most probably because of 'still active' lvm on it.
The fix is to call 'lvchange -an' before cryptsetup trying to close it, possibly in /etc/init.d/crypsetup.

I have many (partly self-created) packages.

First, have you kept these up to date ?
Second, Why was this necessary ?

Anybody else feel free to jump in, my time is up.

markus23 · 2025-08-21 16:08:54

Thank you so much for your time!

That all looks very fishy to me, what is the history of this system ?
Was it a fresh Devuan install or did you migrate from something else ?

IIRC it was a Debian 10, 11, 12 and I followed the migration guide to Devuan, probably https://www.devuan.org/os/documentation … o-daedalus

Fstab is boring, it only refers to (many) LABELs:

LABEL=root / ext4 defaults,errors=remount-ro 0 1
LABEL=swap none swap sw 0 0
LABEL=home /home ext4 rw,suid,dev,exec,auto,nouser 0 2
...

The fix is to call 'lvchange -an' before cryptsetup trying to close it, possibly in /etc/init.d/crypsetup.

Do you mean /etc/init.d/cryptdisks or am I missing an important file here?

First, have you kept these up to date ?
Second, Why was this necessary ?
Anybody else feel free to jump in, my time is up.

Probably you are right and I should reinstall. The packages mostly contain self-made scripts. Will try to reinstall on another partition on the weekend if no other ideas come in.

Thanks again.

g4sra · 2025-08-21 17:12:12

it was a Debian 10, 11, 12

Right! I believe your main issue is (esp the HAL stuff) there is hangover files from systemd.
Examine your system carefully for SystemD packages that have not been purged.
I have never had to do this so someone else may be able to advise you a good way to do so.

LABEL=home /home ext4 rw,suid,dev,exec,auto,nouser,nofail 0 2

Adding 'nofail' *may* enable booting without recovery, it depends on your LVM configuration which I ha
ve not seen.
Either way it wont harm (other than /home not getting mounted on an error)

Do you mean /etc/init.d/cryptdisks

My mistake, yes.
I searched my personal historic docs and found that I actually modified the shell library that /etc/init.d/cryptdisks pulls in

Check /lib/cryptsetup/cryptdisks-functions and the do_stop() function

--- cryptdisks-functions.orig   2023-10-10 14:51:52.654685939 +0100
+++ cryptdisks-functions        2023-10-10 14:55:10.184788099 +0100
@@ -191,6 +191,19 @@
     devno_rootfs="$(get_mnt_devno /)" || devno_rootfs=""
     devno_usr="$(get_mnt_devno /usr)" || devno_usr=""

+    if [ "$INITSTATE" = "remaining" ]; then
+       if [ -x /sbin/lvm ]; then
+          vgs="$(/sbin/lvm vgscan | sed -n '/"/s/^.*"\([^'\'']*\)".*$/\1/p')"
+          if [ -n "${vgs}" ]; then
+              log_action_cont_msg "\nDeactivating volume groups:"
+              for vg in ${vgs}; do
+                  log_action_cont_msg " \"${vg}\""
+                  /sbin/lvm vgchange -a n ${vg} >/dev/null 2>&1
+              done
+          fi
+       fi
+    fi
+
     crypttab_foreach_entry _do_stop_callback
     log_action_end_msg 0
 }

That was how I dealt with the issue of Debian not supporting LVM on LUKS in 2023, I do not know if that is still required today.
Updating cryptsetup in the future will obviously clobber any changes you make.

Last edited by g4sra (2025-08-21 17:15:35)

fsmithred · 2025-08-21 19:12:40

Installing cryptsetup-modified-functions might do the same thing as your edited version. I didn't need this in daedalus, but the author of the patch (devujan) did need it. I don't know if he still needs it in excalibur or if it needs to be changed.
https://dev1galaxy.org/viewtopic.php?pid=41064#p41064
https://git.devuan.org/devuan/cryptsetu … -functions

markus23 · 2025-08-22 14:28:31

Thank you, there is little progress now, at shutdown there are no errors anymore due to cryptsetup-modified-functions.

I also got rid of most errors from bootup (removing left-over files of hal, policy-kit, systemd) but bootup is still broken with the same error. Now dmesg looks just like this:

[   28.411279]  md0: AHDI p1 p2 p3
[   89.728840] udevd[601]: worker [2937] /devices/virtual/block/dm-10 is taking a long time
[  209.512068] udevd[2937]: timeout '/sbin/lvm vgchange -aay --autoactivation event lvm-raid'
[  209.512213] udevd[2937]: slow: '/sbin/lvm vgchange -aay --autoactivation event lvm-raid' [2946]
[  209.843673] udevd[601]: worker [2937] /devices/virtual/block/dm-10 timeout; kill it
[  209.843812] udevd[601]: seq 3259 '/devices/virtual/block/dm-10' killed
[  209.844486] udevd[601]: worker [2937] terminated by signal 9 (Killed)
[  209.844584] udevd[601]: worker [2937] failed while handling '/devices/virtual/block/dm-10'

What are my best bets?

Roll back to an old snapshot and see if it is working there?

Trying to upgrade to excalibur?

Fresh install of excalibur?

Something else?

Is it the file /usr/lib/udev/rules.d/69-lvm.rules which causes the problem for me? Should I simply duplicate the line 89 so that vgchange is executed twice?

Last edited by markus23 (2025-08-22 16:42:58)

markus23 · 2025-08-24 11:45:26

Update:

Old snapshots and backups did not help, the error persists, so probably there is something different in the hardware or the encrypted filesystem (and not that an update in initrd caused the issue)

I also tried upgrading to excalibur but it did not help. I found that in the rescue mode of the 6.1 and 6.12 kernels, there is no mdadm error but there is also the timeout error.

With the excalibur live-usb-stick I tried to manually encrypt /dev/md0 and found it hangs on the first attempt. With later invocation of --test-passphrase however it should be fast enough within 30sec (a few seconds with the first slot, last slot even with mistyping below 30 seconds).

I found some post with a similar issue that says using mdev instead of udev in the initramfs fixed the problem. Is this a good idea? Is someone else using mdev in the initramfs? Are there tutorials how to change it?

Last edited by markus23 (2025-08-24 11:49:51)

markus23 · 2025-08-25 15:56:10

Update:

I tried the solution as described in https://dev1galaxy.org/viewtopic.php?pid=57578#p57578 but it didn't work, I still get the "mdadm: No devices listed in conf file were found." error, even if I comment out my devices completely from /etc/mdadm/mdadm.conf and run update-initramfs -u -k all and verified the mdadm.conf file it looked like:

AUTO -all
ARRAY <ignore> UUID=00000000:00000000:00000000:00000000
HOMEHOST <system>
MAILADDR mail@markus-raab.org
#ARRAY /dev/md/0  metadata=1.2 UUID=...:42d0672f name=markusbyte:0

But there is little progress, I saw that my workaround works with 6.1 as well, so at least I have a more recent kernel now. My current workaround is:

1. boot 6.1 kernel rescue mode
2. type in password and wait until timeout
3. press "Print"/"Druck" and type root password
4. lvm vgchange -aay --autoactivation event lvm-raid
5. mount -a
6. /sbin/resolvconf --enable-updates
7. init 2

The officially official Devuan Forum!

#1 2025-08-21 05:23:33