You are missing the point.

Device Tree files
There are various Device Tree blob files, which have the extension .dtb. These contain the hardware definitions of the various models of Raspberry Pi, and are used on boot to set up the kernel according to which Pi model is detected.

https://www.raspberrypi.org/documentati … _folder.md

The keyword here is: Detected.

I'm taking a look at the full history to check whether there is additional information of value.  I'm unsure which of us missed what, but it seems we're on opposite sides of a canyon.  We've got different goals and our priorities are very different.

You desire fully operational overlays and the advantages those give.  Notably you can use the full Raspberry PI ecosystem of hardware.

I meanwhile desire greater control over the boot process and the advantages that gives.  This allows for loading rather interesting OS setups, notably it can load Linux on Xen.

I don't know whether we're diametrically opposed or not.  What I can say is our goals are wildly different, thus we're going to make very different choices and push things in different directions.

My current thinking is we really need a minimum baseline of compatible device-trees.  Enough for storage and console I/O to function.  Once you have that what we really need is a program which can disassemble the device-tree passed in by the RP4's bootloader and replace the passed-in overlays with overlays associated with the current kernel.  Then turn this into a new device-tree and have the running kernel load this reinterpreted device-tree.

I'm unsure how complicated this is.  I recall seeing a kernel option to replace the passed-in device-tree with one built into the kernel, but I'm unsure whether the possibility of replacing the device-tree during initial ramdisk phase was offered.  I certainly wouldn't want to add this to U-Boot or GRUB, this is too Raspberry PI specific and a user-mode utility in initial ramdisk seems the right sort of place.

Device Tree files
There are various Device Tree blob files, which have the extension .dtb. These contain the hardware definitions of the various models of Raspberry Pi, and are used on boot to set up the kernel according to which Pi model is detected.

https://www.raspberrypi.org/documentati … _folder.md

The keyword here is: Detected.

The DTB file should not be defined, as the bootloader detects which version of the pi and loads it accordingly. You can however define it inside the /boot/config.txt file, but that is usually only done if it's not in its default location or someone is using a custom DTB. Either way, this is done during the normal boot process of the Pi and has nothing to do with u-boot ==> {boot.scr,extlinux,grub}

So the problem still becomes, how does each kernel know which DTB to load if it is not definable when using u-boot and only takes place after the fact? I believe you had mentioned before that it may be/is possible for grub to copy the correct dtb into place during the boot process and if so I assume this would be doable. But then we still have the problem of Kernel packaging, which at this time isn't suitable.

You are more than welcome to work out the logistical elements of this and contribute if you wanna make this happen. You can find me on irc @ libera: #devuan-arm or #arm-img-builder

On other SoCs the device tree blob/binary needs to be loaded with the kernel in Extlinux and is defined by using

fdtdir /location

and

fdt /location/file

but on the PI this can't been done. Not sure why Grub would be able to handle this, but ok? I'll read over ur post further when I have some time.

Why wouldn't GRUB be able to handle this?  Loading a device-tree like this is simply loading a file and placing it at an appropriate address address in an appropriate structure in memory.

The setup I've described is accomplishing this in two pieces.  "flash-kernel" copies the kernel device-tree into /boot/dtb-${kernel}.  The GRUB patch modifies /etc/grub.d/10_linux which in turn adds lines to the resultant grub.cfg file, the crucial line is "devicetree /boot/dtb-${kernel}" (where ${kernel} is the kernel version string).

I've been given the impression GRUB's "devicetree" command exists in 2.02.  Yet without the linked to patch (or an equivalent) the invocation of "devicetree" doesn't appear in grub.cfg and the device-tree never gets loaded.

fdtdir /location

and

fdt /location/file

but on the PI this can't been done. Not sure why Grub would be able to handle this, but ok? I'll read over ur post further when I have some time.

So if I'm reading that correctly it depends highly on kernel packaging, yes? I believe I've already went over this and suggested that the current packaging isn't adequate. It simply isn't suitable for multi kernel boot and I never set it up to do so.

No, it simply needs appropriate scripts to be loaded on the machine.  Noticed how many on target machine scripts x86 kernel packages rely upon?  I will agree the packaging of kernels is suboptimal, but it is mostly good enough (notably on ARM Debian kernel packages shouldn't depend upon "flash-kernel", but upon "linux-kernel-installer" which is provided by "flash-kernel").

That doesn't make sense.  `extlinux` is for loading a Linux kernel from an ext2/3/4 or btrfs filesystem.  The true earliest stage was the binary blob loaded by the Video Core portion of the chip, next stage is something which might be based off of syslinux which loads the kernel off of vfat (I recall reading the RP4 may skip the Video Core blob off of vfat).

I've been reporting experiments in places where others were reporting things.  I had hopes SuSE's approach would take over due to shear superiority (again, GRUB has much better boot functionality).  Since I need to give the outline here too...

Install the packages "u-boot-rpi", "raspi-firmware", "grub-efi-arm64", and "flash-kernel" (yuck, need a cut-down package "html5-kernel" which simply includes the scripts for copying device-tree files into appropriate placed, ideally using `cp -l`).

Setup the boot vfat filesystem.  Mount this as /boot/efi and then do the following:

cp /usr/lib/raspi-firmware/* /boot/efi

cp /usr/lib/u-boot/rpi_arm64/u-boot.bin /boot/efi/u-boot64.bin

cp /usr/lib/u-boot/rpi_3/u-boot.bin /boot/efi/u-boot3.bin
cp /usr/lib/u-boot/rpi_4/u-boot.bin /boot/efi/u-boot4.bin

grub-install --bootloader-id=BOOT
cp /boot/efi/EFI/BOOT/grubaa64.efi /boot/efi/EFI/BOOT/bootaa64.efi

echo bootaa64 > /boot/efi/startup.nsh

Now, I'm unsure whether `cp /boot/dtbs/$(uname -r)/broadcom/bcm2*-rpi*.dtb /boot/efi` is required; or perhaps as you've suggested U-Boot can make due with its built-in device-tree.  I've also run into a mention of GRUB merely being able to overwrite an incoming device-tree and not being able to install one from scratch.  You'll also want to adjust /etc/grub.d/10_linux* in line with the patch from here (the original patch is out of date).

These directions are not 100%, but they're most of the way there.  Biggest issue is getting the versions of everything lined up.

I'll extend an olive branch. How would one define which DTB to use for each kernel, if the DTB is undefinable?

This edit came in after I was already typing.  What do you mean?

Unfortunately the functionality is usually provided by the `flash-kernel` package, but if `flash-kernel` is installed it will copy the appropriate DTB from /usr/lib/<kernel-version> to /boot/dtb-<kernel-version>.  The GRUB wishlist bug 52939 includes a patch for /etc/grub.d/10_linux to cause GRUB to load the device-tree from /boot/dtb-<kernel-version>.  The patch was made for an older version of GRUB thus needs manual application and also needs to modify /etc/grub.d/10_linux_xen.

If you've had it working for 6 months why are you keeping it a secret and being a jerk about it here? Why not just contribute?

Lets say we use u-boot. The DTB file can't be defined in extlinux, as the next stage of the bootloader will get confused and at worst not boot at all. If it did boot by chance, it would be all out of whack and certain onboard things wouldn't work. How do I know this? I've experimented with it

Okay.  So what?  You're using `extlinux` which cannot load device-trees.  GRUB can and I already pointed to someone who had a patch for their scripts which functioned (updates to the scripts mean the patch no longer applies, but it isn't difficult to update by hand).

I already had it functioning more than 6 months ago.  Polish is needed, but the boot sequence U-Boot => GRUB is workable.

Lets say we use u-boot. The DTB file can't be defined in extlinux, as the next stage of the bootloader will get confused and at worst not boot at all. If it did boot by chance, it would be all out of whack and certain onboard things wouldn't work. How do I know this? I've experimented with it

Your hate of the device-tree doesn't change how the bootloader works. In all honesty it would probs be easier to get a multi OS boot before a multi kernel boot on the same medium.

As for the device-tree, its essential. There are no if and or buts about it. This is not your Dads x86 computer Not all DTB files are gonna support the next kernel exactly the same and there isn't much any of us can do about it and unfortunately defining that DTB using extlinux and or grub is gonna create a headache in the boot process.

Of course, I'm just going on my own testing. I welcome you to play with it and see if you can get further.

Edit:
I'll extend an olive branch. How would one define which DTB to use for each kernel, if the DTB is undefinable?

I'm not ignoring it, I just don't care I find it at this time to be a bit fruitless, hence why I suggested if you wanna put in the work to make it happen. Have at it!

Numerous people have demonstrated it isn't much work, and has distinct benefits.  Most of the work is getting people claiming it is too much work to get out of the way.

I think you just demonstrated why I'm highly underwhelmed by device-trees.  Their implementation is awful.

No, it isn't.  I'm far from expert in these pieces, but I've gotten things functioning without much work.  A very little bit of scripting might be needed, the main work is keeping the pieces synchronized.  Mostly needing to keep a combination of U-Boot, GRUB and device-tree which are compatible together.

Have I mentioned device-trees are garbage?  (this could merely be an issue of lack of standards, whereas UEFI was an organization behind it keeping things mostly sane)

c0rnelius wrote:

ehem wrote:

You use U-Boot's UEFI support to have it load GRUB (SuSE uses this approach).  While wasteful due to having an extra bootloader stage, you gain GRUB's massively better functionality.

You aren't taking under account kernel packaging and the problems therein. Not to mention overlays and whatnots are gonna be different between kernel revisions, so now that would also need to be taken under consideration. Grub would also need to be scripted to know what to look for, which presents even more headaches.

Even other SBCs that use grub aren't trying to boot more than one kernel, as there is a lot of things to account for. But hey, if you feel like looking into it and investing the time and effort to make it work, go for it.

There was some discussion on Debian bug #824954.  Some of the necessary scripting can be found as GRUB wishlist #52939.

This though is only needed for covering the most extreme case, where GRUB needs to load a kernel-specific device-tree.  This isn't needed if you're switching between kernel patch levels, which is adequate if you're switching between 5.10.0-1 and 5.10.0-2 (which would be security update versions).  This is also overkill if you're simply rebuilding Devuan's kernel source with custom configuration.

You're also yet again ignoring the other highly important case.  It is possible, but quite difficult to boot Xen directly from U-Boot.  Whereas, GRUB will boot Xen out of the box.

In other news I find device-trees rather underwhelming.  Most UEFI implementations will boot Linux kernels from 3.0 to 5.10 with the exact same implementation.  The same implementation will also boot *BSD, or other closed OSes from Redmond.  Yet a device-tree implementation is only likely to work with one kernel version of one OS.

I'm not ignoring it, I just don't care I find it at this time to be a bit fruitless, hence why I suggested if you wanna put in the work to make it happen. Have at it!

You can simply review the latest changes in mainline to find what ur saying about kernel revisions isn't accurate at this time: https://git.kernel.org/pub/scm/linux/ke … 10.52&dt=2

I'm not against getting this working, I'm simply saying that the effort needed to get that done would be very time consuming.

Cheers!

ehem wrote:

c0rnelius wrote:

As for Grub, I've played around with it on other SBCs, but am not 100% sure how to set that up correctly on a Pi.

You use U-Boot's UEFI support to have it load GRUB (SuSE uses this approach).  While wasteful due to having an extra bootloader stage, you gain GRUB's massively better functionality.

You aren't taking under account kernel packaging and the problems therein. Not to mention overlays and whatnots are gonna be different between kernel revisions, so now that would also need to be taken under consideration. Grub would also need to be scripted to know what to look for, which presents even more headaches.

Even other SBCs that use grub aren't trying to boot more than one kernel, as there is a lot of things to account for. But hey, if you feel like looking into it and investing the time and effort to make it work, go for it.

There was some discussion on Debian bug #824954.  Some of the necessary scripting can be found as GRUB wishlist #52939.

This though is only needed for covering the most extreme case, where GRUB needs to load a kernel-specific device-tree.  This isn't needed if you're switching between kernel patch levels, which is adequate if you're switching between 5.10.0-1 and 5.10.0-2 (which would be security update versions).  This is also overkill if you're simply rebuilding Devuan's kernel source with custom configuration.

You're also yet again ignoring the other highly important issue.  It is possible, but quite difficult to boot Xen directly from U-Boot.  Whereas, GRUB will boot Xen out of the box.

In other news I find device-trees rather underwhelming.  Most UEFI implementations will boot Linux kernels from 3.0 to 5.10 with the exact same implementation.  The same implementation will also boot *BSD, or other closed OSes from Redmond.  Yet a device-tree implementation is only likely to work with one kernel version of one OS.

c0rnelius wrote:

ehem wrote:

Is there any hope Raspberry PI 4B images with U-Boot => GRUB will show up?

I like being able to easily boot previous kernels when the latest fails (modified kernels do fail and sometimes security update kernels have introduced severe reversions).  Also the Raspberry PI 4B has hardware suitable for running Xen and recent versions of GRUB (2.02 has the module, but only >=2.04 has an appropriate /etc/grub.d/20_linux_xen) will load Xen/ARM64 out of the box.

This would be easier to accomplish using u-boot and extlinux, with extlinux you can add an optional menu. One problem though, would be the current kernel packaging, as currently it purges the old kernel, dtbs, initrd, overlays and headers during install.

That doesn't seem easier except for the first time setup.  Maintenance is harder since it means spinning everything by hand, whereas GRUB has everything scripted out of the box.

More importantly, you missed the second part.  I don't believe extlinux can load Xen (two kernels plus an initial ramdisk).

c0rnelius wrote:

As for Grub, I've played around with it on other SBCs, but am not 100% sure how to set that up correctly on a Pi.

You use U-Boot's UEFI support to have it load GRUB (SuSE uses this approach).  While wasteful due to having an extra bootloader stage, you gain GRUB's massively better functionality.

You aren't taking under account kernel packaging and the problems therein. Not to mention overlays and whatnots are gonna be different between kernel revisions, so now that would also need to be taken under consideration. Grub would also need to be scripted to know what to look for, which presents even more headaches.

Even other SBCs that use grub aren't trying to boot more than one kernel, as there is a lot of things to account for. But hey, if you feel like looking into it and investing the time and effort to make it work, go for it.

ehem wrote:

Is there any hope Raspberry PI 4B images with U-Boot => GRUB will show up?

I like being able to easily boot previous kernels when the latest fails (modified kernels do fail and sometimes security update kernels have introduced severe reversions).  Also the Raspberry PI 4B has hardware suitable for running Xen and recent versions of GRUB (2.02 has the module, but only >=2.04 has an appropriate /etc/grub.d/20_linux_xen) will load Xen/ARM64 out of the box.

This would be easier to accomplish using u-boot and extlinux, with extlinux you can add an optional menu. One problem though, would be the current kernel packaging, as currently it purges the old kernel, dtbs, initrd, overlays and headers during install.

That doesn't seem easier except for the first time setup.  Maintenance is harder since it means spinning everything by hand, whereas GRUB has everything scripted out of the box.

More importantly, you missed the second part.  I don't believe extlinux can load Xen (two kernels plus an initial ramdisk).

You use U-Boot's UEFI support to have it load GRUB (SuSE uses this approach).  While wasteful due to having an extra bootloader stage, you gain GRUB's massively better functionality.