You are not logged in.
After installing new kernel, the on booting system is stuck for 2-3 minutes at "waiting for /dev to be fully populated... udevd[664]: speci.. known ". But, the kernel-5.6 boots after the delay.
I installed linux-image-5.6.0-0.bpo.2-amd64 in Beowulf. It removed crda and wireless-regdb packages when new kernel is installed. I have to install a newer version of crda package along with wireless-regdb from backports repo. I checked /var/log/messages for clue
# apt policy wireless-regdb
wireless-regdb:
Installed: 2019.06.03-1~bpo10+1
# apt policy crda
crda:
Installed: 4.14+git20191112.9856751-1~bpo10+1
This is the only lines I can find:
udevd[121]: starting version 3.2.7
[ 0.810367] random: udevd: uninitialized urandom read (16 bytes read)
[ 0.810408] random: udevd: uninitialized urandom read (16 bytes read)
[ 0.811452] udevd[122]: starting eudev-3.2.7
[ 0.821120] random: udevd: uninitialized urandom read (16 bytes read)
UPDATE: Resolved it finally. The problem package is "modemmanger". This package installs new rules to /lib/udev/rules.d/ which interferes.
Another observation: I installed recently with debian-installer (devuan net-iso). Completed without installing kernel. Chrooted to the Devuan Beowulf install and installed linux-image-5.6.0-0.bpo.2-amd64. It removed crda and wireless-regdb packages. Gnome is working fine. I think modemmanager may be put into banned list or edit the script to fix the problems.
Last edited by Debuser2018 (2020-07-10 11:28:16)
Offline
System boots fine with kernel 4.19. But, with 5.6 (backport), udev is acting. I have a usb wireless adapter and usb bluetooth adapter in the desktop system. Both are working fine with 5.6 kernel also. But, seems like error and udev delaying due to these usb devices?
udev waits for 3 minutes approximately and times out. When, I shutdown the system, can see this error mssage:
Currently running processes (pstree):
init-*-rc--openrc---openrc-run---sendsigs---pstree
*-udevd---3*[udevd]
Killing all processes... : worker [xyzab] failed while handling '/devices/pci0000:00/0000:00:1c.3/0000:09:00:0/usb4/4-0'
udevd worker [xxxx] failed while handling '/devices/pci0000:00/0000:00:1c.3/0000:09:00:0/usb3/3-0:1-0'
Offline
Almost 3 minutes of "waiting for /dev/ to be populated" before timing out. Cannot find the reason for udev worker time out with kernel 5.6. No issues with 4.19 kernel. Should I file a bug report (if beowulf-backports is supported). I have even eudev & co backported from ceres and installed in case if it's a udev issue.
Here is the result of "diff config-4.19.0-9-amd64 config-5.6.0-0.bpo.2-amd64"
https://paste.debian.net/1153509/
The following packages are locally rebuilt from ceres sources in the hope that the udevd issue may get fixed. It did not.
:~# apt list --installed |grep ,local
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
crda/now 4.14+git20191112.9856751-1~bpo10+1 amd64 [installed,local]
eudev/now 3.2.9-7~bpo10+2 amd64 [installed,local]
init-system-helpers/now 1.57+devuan1~bpo10+1 all [installed,local]
init/now 1.57+devuan1~bpo10+1 amd64 [installed,local]
libeinfo1/now 0.42-1~bpo10+1 amd64 [installed,local]
libeudev1/now 3.2.9-7~bpo10+2 amd64 [installed,local]
librc1/now 0.42-1~bpo10+1 amd64 [installed,local]
openrc/now 0.42-1~bpo10+1 amd64 [installed,local]
refind/now 0.12.0-1 amd64 [installed,local]
wireless-regdb/now 2020.04.29-1~bpo10+1 all [installed,local]
Last edited by Debuser2018 (2020-06-25 23:13:42)
Offline
I tried to reproduce your problem and failed. Maybe it's because I'm testing in qemu instead of hardware, maybe it's specific to your hardware, and maybe it's specific to the version of openrc that you're using. I can't find that one. I'm using 0.40.3-1 in beowulf with the current 5.6 backports kernel.
Where did you get your version of openrc? (0.42-1~bpo10+1)
Offline
Guess its system specific. Got two machines here that work fine with Beowulf and kernel linux-image-5.6.0-0.bpo.2-amd64. Though I am using the openrc version that comes with Beowulf.
rolfie
Offline
I tried to reproduce your problem and failed. Maybe it's because I'm testing in qemu instead of hardware, maybe it's specific to your hardware, and maybe it's specific to the version of openrc that you're using. I can't find that one. I'm using 0.40.3-1 in beowulf with the current 5.6 backports kernel.
Where did you get your version of openrc? (0.42-1~bpo10+1)
Hi, the problem is reproducible with openrc-0.40.3-1.
I have openRC rebuilt from ceres along with eudev, crda, wireless-regdb (needed for 5.6 kernel to work) also from the ceres. I did this, in the hope that udev worker timing out issue may be fixed in latest openrc, eudev packages. I captured the log while shutting down:
lspci and lsusb
https://paste.debian.net/1153837/
Offline
Guess its system specific. Got two machines here that work fine with Beowulf and kernel linux-image-5.6.0-0.bpo.2-amd64. Though I am using the openrc version that comes with Beowulf.
rolfie
Stock kernel 4.19 boots just fine. But, with linux-image-5.6.0-0.bpo.2-amd64, udev worker timing out. I have bootlogd installed. But, nothing much useful in the log. I tried booting with "quiet" line removed. Still, nothing much. If I run "udevadm settle", it takes few minutes and no error messages. While shutting down, as the photo shows, "connection refused" messages shown.
Should I check /etc/init.d/ for this udev issue?
These are the doubtful messages from /var/log/kern.log shows. Python3 segfaults, rfkill disabled.. :~
Jun 26 03:51:08 devuan kernel: [ 0.814444] Run /init as init process
Jun 26 03:51:08 devuan kernel: [ 0.814894] with arguments:
Jun 26 03:51:08 devuan kernel: [ 0.814894] /init
Jun 26 03:51:08 devuan kernel: [ 0.814895] with environment:
Jun 26 03:51:08 devuan kernel: [ 0.814895] HOME=/
Jun 26 03:51:08 devuan kernel: [ 0.814896] TERM=linux
Jun 26 03:51:08 devuan kernel: [ 0.834931] random: udevd: uninitialized urandom read (16 bytes read)
Jun 26 03:51:08 devuan kernel: [ 0.835718] random: udevd: uninitialized urandom read (16 bytes read)
Jun 26 03:51:08 devuan kernel: [ 0.847931] random: udevd: uninitialized urandom read (16 bytes read)
Jun 26 03:52:29 devuan kernel: [ 212.280879] rfkill: input handler disabled
Jun 26 04:07:06 devuan kernel: [ 1089.877525] show_signal_msg: 10 callbacks suppressed
Jun 26 04:07:06 devuan kernel: [ 1089.877530] python3[11624]: segfault at 18 ip 00007f220ee90af0 sp 00007ffe6ee2f6e8 error 4 in libgtk-3.so.0.2404.1[7f220ec4a000+352000]
Jun 26 04:07:06 devuan kernel: [ 1089.877543] Code: dc ff 48 8b 44 24 08 64 48 33 04 25 28 00 00 00 75 0f 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 e8 64 e1 db ff 0f 1f 40 00 <48> 8b 47 18 48 8b 40 10 c3 0f 1f 80 00 00 00 00 53 48 8b 5f 18 48
Offline
I tried booting with rw init=/bin/bash and started services manually. /etc/rcS.d/S02eudev takes a minute to finish with "done. (time out)" message shown.
Offline
@fsmithred: I did a fresh install of Devuan Beowulf using netinstall iso (Debian installer) and reusing my /home directory, which is a separate slice. Everything was working fine until I installed Gnome. Before, "devmapper" line was appearing after udevd. After Gnome3, it is stuck at udevd.. worker "timing out". The python segfault and other error messages thankfully not observed.
Offline
This is just a wild guess, but maybe you are being affected by this bug:
https://bugs.devuan.org/cgi/bugreport.cgi?bug=483
Simple test is to add sleep 1 to /etc/init.d/eudev as shown in the diff.
Offline
This is just a wild guess, but maybe you are being affected by this bug:
https://bugs.devuan.org/cgi/bugreport.cgi?bug=483Simple test is to add sleep 1 to /etc/init.d/eudev as shown in the diff.
Yes I tried (added sleep 1 after sleep 15 fi). It did not work.
linux-image-4.19.0-9-amd64 boots fine. Problem is with the latest kernel from backports.
With 5.6.0-0.bpo.2-amd64, crda and wireless-regdb are removed. I have to get the new versions from ceres backported to eliminate warning messages about missing regulatory.db files.
I have uploaded logs in the hope that something helpful can be found:
kern.log: https://pastebin.ubuntu.com/p/XbzSB98Qxp/
/var/log/messages: https://pastebin.ubuntu.com/p/9fTpV3nsy2/
/var/log/syslog: https://pastebin.ubuntu.com/p/S39wPxjS3j/
Offline
I'm seeing the same configs for crda and regdb in 4.19.0-9-amd64, 5.6.0-0.bpo.2 and 5.6.0-1 in ceres.
CONFIG_CFG80211_CRDA_SUPPORT=y
CONFIG_CFG80211_REQUIRE_SIGNED_REGDB=y
CONFIG_CFG80211_USE_KERNEL_REGDB_KEYS=y
Offline
But, if I have to install linux-image-5.6.0-0.bpo.2-amd64, both crda and wireless-regdb needs to be removed. When I installed crda & wireless-regdb from ceres (backported), the regulatory.db error message disappears.
Apart from crda & wireless-regdb, rest of the system is lean and stock. I'm sure that something breaks udev with kernel-5.x when you install gnome3. I'm ready to dig it. But, totally clueless now.
~$ apt policy wireless-regdb
wireless-regdb:
Installed: 2020.04.29-1~bpo10+1
Candidate: 2020.04.29-1~bpo10+1
Version table:
*** 2020.04.29-1~bpo10+1 100
100 /var/lib/dpkg/status
2019.06.03-1~bpo10+1 100
100 http://deb.devuan.org/merged beowulf-backports/main amd64 Packages
2016.06.10-1 500
500 http://deb.devuan.org/merged beowulf/main amd64 Packages
:~$ apt policy crda
crda:
Installed: 4.14+git20191112.9856751-1~bpo10+2
Candidate: 4.14+git20191112.9856751-1~bpo10+2
Version table:
*** 4.14+git20191112.9856751-1~bpo10+2 100
100 /var/lib/dpkg/status
3.18-1 500
500 http://deb.devuan.org/merged beowulf/main amd64 Packages
:~$ diff /boot/config-4.19.0-9-amd64 /boot/config-5.6.0-0.bpo.2-amd64 |tee diff-kernels.txt
https://paste.debian.net/1154234/
Offline
Now, there is not much anything about why the 3 minutes delay after udevd starts. Except for the time out message. I checked the /var/log/messages and saw only this:
Jun 28 23:52:19 dev1box kernel: [ 0.850092] random: udevd: uninitialized urandom read (16 bytes read)
Jun 28 23:52:19 dev1box kernel: [ 0.850661] random: udevd: uninitialized urandom read (16 bytes read)
Jun 28 23:52:19 dev1box kernel: [ 0.862455] random: udevd: uninitialized urandom read (16 bytes read)
Offline
Is this a re-emergence in the new kernel of this reported bug in Debian Buster, which has never seemed to be an issue in Beowulf?
5.1.5. Daemons fail to start or system appears to hang during boot
Due to systemd needing entropy during boot and the kernel treating such calls as blocking when available entropy is low, the system may hang for minutes to hours until the randomness subsystem is sufficiently initialized (random: crng init done). For amd64 systems supporting the RDRAND instruction this issue is avoided by the Debian kernel using this instruction by default (CONFIG_RANDOM_TRUST_CPU).Non-amd64 systems and some types of virtual machines need to provide a different source of entropy to continue fast booting. haveged has been chosen for this within the Debian Installer project and may be a valid option if hardware entropy is not available on the system. On virtual machines consider forwarding entropy from the host to the VMs via virtio_rng.
If you read this after upgrading a remote system to buster, ping the system on the network continuously as this adds entropy to the randomness pool and the system will eventually be reachable by ssh again.
See the wiki https://wiki.debian.org/BoottimeEntropyStarvation and DLange's overview of the issue https://daniel-lange.com/archives/152-hello-buster.html for other options.
source: https://www.debian.org/releases/buster/ … starvation
Offline
In both 4.19.0-9-amd64 and 5.6.0-0.bpo.2-amd64 kernels, searching config showed: CONFIG_RANDOM_TRUST_CPU=y
Earlier, I tried recent eudev and elogind, openRC from ceres in the hope to resolve the problem. Nothing much to try.
May be unrelated. But, I earlier saw udevd worker time out with two usb adapters - bluetooth and wireless adapters. (Now no warning or error messages seen, but only "time out"). Of this, bluetooth adapter is not working despite the device detected hci0 and shows as up running. WiFi adapter works fine.
Bus 004 Device 004: ID 0a12:0001 Cambridge Silicon Radio, Ltd Bluetooth Dongle (HCI mode)
Bus 004 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 005: ID 046d:c077 Logitech, Inc. M105 Optical Mouse
Bus 003 Device 004: ID 413c:2107 Dell Computer Corp.
Bus 003 Device 003: ID 148f:7601 Ralink Technology, Corp. MT7601U Wireless Adapter
Bus 003 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
lsusb -v results for 0a12:0001 and 148f:7601 :
https://paste.debian.net/1154312/
Offline
What else to try? Should I try updating eudev, elogind, openrc etc from unstable repo?
Offline
Install haveged.
Offline
Install haveged.
Installed already. But, udev waiting & time out continues. What is least helpful with this fresh Beowulf install is, I cannot find any logs about what is actually failing or "hangs" while booting or in any log files. In the previous installation (from ascii to beowulf), when I boot with linux-5.6.x, I can see some usb id's related error messages can be seen. (I guess a bluetooth adapter and wireless adapter).
Right now, I saw some message while shutting down. I think it's complaining about usb device and udevd worker. (will confirm)
@fsmithred: Where do I start with? I tried starting init scripts manually and even eudev started fine. If I run "udevadm settle" it takes few minutes.
Offline
One way to get more information is to edit /etc/udev/udev.conf and set udev_log="debug" and make sure it's uncommented.
If you want to try with the system that I'm using for testing, I can upload a live-iso later today.
Offline
One way to get more information is to edit /etc/udev/udev.conf and set udev_log="debug" and make sure it's uncommented.
If you want to try with the system that I'm using for testing, I can upload a live-iso later today.
I will try. Since I installed with debian-installer the base system. I observed that the problem cropped up after installing gnome. I will try removing whole of Gnome and related dependencies and will update.
Offline
udev_log="debug" did not do anything. I tried purging all of Gnome packages, xorg etc to get bare minimum system and tried booting with linux-image-5.6. But, the same problem happens - 3 minutes delay .
@fsmithred: Please upload your version of live-iso.
Offline
@fsmithred: At last, I got some debug messages when I opened virtual terminal (tty2,3 etc). These messages are running through the virtual terminals. Something to do with usb rules?
[ 2321.518059] udevd[657]: seq 23488 queued, 'change' 'usb'
[ 2321.518117] udevd[698]: created db file '/run/udev/data/+usb:2-0:1.0' for '/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb2/2-0:1.0'
[ 2348.631334] elogind-daemon[2318]: VT changed to 2
[ 2348.631410] elogind-daemon[2318]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/login1/session/_32 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=427 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 2348.632510] elogind-daemon[2318]: card0: Found udev node /dev/dri/card0 for seat seat0
[ 2348.632579] elogind-daemon[2318]: hwC0D0: Found udev node /dev/snd/hwC0D0 for seat seat0
[ 2348.632643] elogind-daemon[2318]: hwC0D3: Found udev node /dev/snd/hwC0D3 for seat seat0
[ 2348.632712] elogind-daemon[2318]: pcmC0D0c: Found udev node /dev/snd/pcmC0D0c for seat seat0
[ 2348.632779] elogind-daemon[2318]: pcmC0D0p: Found udev node /dev/snd/pcmC0D0p for seat seat0
[ 2348.632849] elogind-daemon[2318]: pcmC0D1p: Found udev node /dev/snd/pcmC0D1p for seat seat0
[ 2348.632914] elogind-daemon[2318]: pcmC0D2c: Found udev node /dev/snd/pcmC0D2c for seat seat0
[ 2348.632981] elogind-daemon[2318]: pcmC0D3p: Found udev node /dev/snd/pcmC0D3p for seat seat0
[ 2352.418157] udevd[657]: seq 23774 queued, 'change' 'usb'
[ 2352.418252] udevd[679]: created db file '/run/udev/data/+usb:4-0:1.0' for '/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb4/4-0:1.0'
[ 2352.427829] udevd[657]: seq 23775 queued, 'change' 'usb'
[ 2352.428010] udevd[679]: passed 531 byte device to netlink monitor 0x55b69c38d1d0
[ 2352.432565] udevd[679]: seq 12989 processed
[ 2352.433966] udevd[657]: passed 426 byte device to netlink monitor 0x55b69c377ab0
[ 2352.433981] udevd[679]: seq 12991 running
[ 2352.437758] udevd[679]: IMPORT builtin 'hwdb' /lib/udev/rules.d/50-udev-default.rules:14
[ 2383.546370] udevd[657]: seq 24062 queued, 'change' 'usb'
[ 2383.546466] udevd[679]: created db file '/run/udev/data/+usb:4-0:1.0' for '/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb4/4-0:1.0'
[ 2383.550878] udevd[657]: seq 24063 queued, 'change' 'usb'
[ 2383.552016] udevd[679]: passed 531 byte device to netlink monitor 0x55b69c38d1d0
[ 2383.559139] udevd[679]: seq 13134 processed
[ 2383.560619] udevd[657]: passed 426 byte device to netlink monitor 0x55b69c377ab0
[ 2383.560713] udevd[679]: seq 13137 running
[ 2383.563513] udevd[679]: IMPORT builtin 'hwdb' /lib/udev/rules.d/50-udev-default.rules:14
[ 2383.762379] udevd[657]: seq 24064 queued, 'change' 'usb'
[ 2383.762502] udevd[698]: created db file '/run/udev/data/+usb:2-0:1.0' for '/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb2/2-0:1.0'
[ 2388.730404] udevd[657]: seq 24110 queued, 'change' 'usb'
[ 2388.730544] udevd[679]: created db file '/run/udev/data/+usb:4-0:1.0' for '/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb4/4-0:1.0'
[ 2388.737062] udevd[657]: seq 24111 queued, 'change' 'usb'
[ 2388.738964] udevd[679]: passed 531 byte device to netlink monitor 0x55b69c38d1d0
[ 2388.743008] udevd[679]: seq 13159 processed
[ 2388.744628] udevd[657]: passed 426 byte device to netlink monitor 0x55b69c377ab0
[ 2388.744653] udevd[679]: seq 13160 running
[ 2388.749236] udevd[679]: IMPORT builtin 'hwdb' /lib/udev/rules.d/50-udev-default.rules:14
[ 2388.950428] udevd[657]: seq 24112 queued, 'change' 'usb'
[ 2388.950529] udevd[698]: created db file '/run/udev/data/+usb:2-0:1.0' for '/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb2/2-0:1.0'
[ 2392.019663] elogind-daemon[2318]: Inhibitor Debian-gdm (GNOME needs to lock the screen) pid=15173 uid=109 mode=delay stopped.
[ 2392.022544] elogind-daemon[2318]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/login1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=536 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 2392.029489] elogind-daemon[2318]: Inhibitor Debian-gdm (GNOME handling keypresses) pid=15171 uid=109 mode=block stopped.
[ 2392.031972] elogind-daemon[2318]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/login1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=537 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 2392.037393] elogind-daemon[2318]: Electing new display for user Debian-gdm
[ 2392.040120] elogind-daemon[2318]: Ignoring session c3
[ 2392.042933] elogind-daemon[2318]: Removed session c3.
[ 2392.045238] elogind-daemon[2318]: Sent message type=signal sender=n/a destination=:1.85 path=/org/freedesktop/login1/session/c3 interface=org.freedesktop.login1.Session member=PauseDevice cookie=538 reply_cookie=0 signature=uus error-name=n/a error-message=n/a
[ 2392.055886] elogind-daemon[2318]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/login1 interface=org.freedesktop.login1.Manager member=SessionRemoved cookie=539 reply_cookie=0 signature=so error-name=n/a error-message=n/a
[ 2392.060328] elogind-daemon[2318]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/login1/user/_109 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=540 reply_cookie=0 signature=sa{sv}as
error-name=n/a error-message=n/a
[ 2393.918382] udevd[657]: seq 24158 queued, 'change' 'usb'
[ 2393.918452] udevd[679]: created db file '/run/udev/data/+usb:4-0:1.0' for '/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb4/4-0:1.0'
[ 2393.926095] udevd[657]: seq 24159 queued, 'change' 'usb'
[ 2393.930347] udevd[679]: passed 531 byte device to netlink monitor 0x55b69c38d1d0
[ 2393.934214] udevd[679]: seq 13183 processed
[ 2393.936634] udevd[657]: passed 426 byte device to netlink monitor 0x55b69c377ab0
[ 2393.936665] udevd[679]: seq 13184 running
[ 2393.942043] udevd[679]: IMPORT builtin 'hwdb' /lib/udev/rules.d/50-udev-default.rules:14
[ 2394.134441] udevd[657]: seq 24160 queued, 'change' 'usb'
[ 2394.134499] udevd[698]: created db file '/run/udev/data/+usb:2-0:1.0' for '/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/usb2/2-0:1.0'
/lib/udev/rules.d/50-udev-default.rules (line number 14):
ENV{MODALIAS}!="", IMPORT{builtin}="hwdb --subsystem=$env{SUBSYSTEM}"
Last edited by Debuser2018 (2020-07-01 09:43:57)
Offline
https://get.refracta.org/files/experime … 0_1456.iso
sha256sum:
dc97d884f860b12791f40fbe4d1a4b04a1a2e2cbe4e937dd4a64d6ed598998be refracta-test-oblx_5.6bpo_openrc-20200630_1456.iso
Offline
I installed that iso in a qemu VM in uefi mode. I can't get it to boot at all without booting from the iso and using grub commands to boot the virtual hard disk. That's a qemu issue, I'm pretty sure. Anyway, it boots like that, and when I install gnome, it still boots that way, and it's pretty fast.
If you try installing from that iso, user the cli installer, not the graphical one in the apps menu. I ran into gtk problems with that. Just run 'refractainstaller' as root.
Offline