Perhaps it assumes an initial type FAT boot partition, which wouldn't be unusual. That would at least have boot.scr for booting, and usually that will also contain kernel and initrd. Or for example, if your u-boot supports the sysboot command you may use that for an extlinux boot from a differt, type ext partition, which then would contain the extlinux.conf configuration file as well as kernel, FDT and initrd.

(EDIT: added mentioning of FDT = Flat Device Tree, aka DTB = Device Tree Blob)

Ideally you get a serial cable for the box so you can see u-boot logging.

But my suggestion is that you first set up your box with an initial FAT partition for boot.scr, The command to use in boot.cmd would be like this

sysboot mmc 0:2 ext4 0x7ffff000 /boot/extlinux.conf

where mmc is the drive type (for sdcard), 0:2 identifies the boot sdcard, partition 2, ext4 indicates the partition type, and it's then followed by the pathname of the configuration file.
You'll then need to mkimage that into boot.scr.

EDIT: The argument 0x7ffff000 is the load address for the extlinux.conf file, and you must choose a RAM address within your actual RAM. High up in your RAM is good.

The second partition can be a single partition for everything, i.e. boot and root filesystem. Note that I've used ext4 rather than ext3, but that's only because I'm incredibly modern Though you might have better luck telling sysboot it's an ext2 even if it's an ext4, or perhaps use ext3... whichever.

EDIT: actually, you want rootfs to be encrypted, so then you'll need 3 partitions...

What I did:

I used file from linux-u-boot-orangepi3b to update bootloader,factory one is from 2017.

I made a Devuan system with the help of debootstrap using the default variant, version ceres, and I added DietPi mirrors (because they have a kernel that fits my board). I installed cryptsetup, the kernel, Wi‑Fi-related packages, added firmware blobs to /lib/firmware, and compiled booster for the initramfs tool.

I wrote boot.cmd and compiled it to boot.scr, and also wrote extlinux.conf (because I had seen one of the Manjaro ARM images boot on my board; it didn't have the syslinux package but also no boot.scr).

The bootloader still doesn't recognize the partition; there was no blinking LED. Debian system from DietPi has no problem booting, I wish DietPi images had a LUKS option.

I also tried converting the fully LUKSv2 partition to LUKSv1 to see if this was a problem, since the first article I saw was https://u-boot.org/blog/unlocking-disks … rint=print ; but I later found https://u-boot.org/blog/unlocking-moder … 2-support/ when digging through their blog, so this was not a problem by itself.