Note:
This setup will give you a working xserver without the need for a displaymanager or a desktop enviroment.

1. List all available video drivers in the repos with a short description

apt-cache search xserver-xorg-video-.* | pager

2. Install the one(s) you need

apt-get install xserver-xorg-video-Your_GPU_Vendor_Here

3. Install the minimum packages to get a working and useable xserver

apt-get install x11-xserver-utils xfonts-100dpi xfonts-75dpi xfonts-base xfonts-scalable xinit xorg xserver-xorg-input-kbd xserver-xorg-input-libinput xserver-xorg-input-mouse

4. Install a window manager (I use evilwm, but you can use what ever you want)

apt-get install evilwm

5. Install a terminal emulator (I use rxvt-unicode, but you can use what ever you want)

apt-get install rxvt-unicode

6. Install a hotkey utility to start programs with keybindings
I use keylaunch since ages.
Sadly the developer has dropped it, but the deb-files for it are still available here:
https://packages.debian.org/bullseye/keylaunch
It still works on Devuan 5 Daedalus and on Devuan 6 Excalibur.
It is very small, it only dependencies are libc6 and libx11-6 (so packages every system has normally) and has only one configuration file in text format, wich is pretty easy to understand and to edit.

Download the deb-file you need and install it with:

dpkg --install keylaunch_1.3.9+b2_amd64.deb

7. Create the configuration file for keylaunch in the home directory of the user who will run the xsession

touch ~.keylaunchrc

8. Edit the configuration file for keylaunch
Put this at the top of the file content, it explains the whole file:

# Format:
# key=...KeyName:Command
#
# ... No modifier
# *.. Shift
# .*. Ctrl
# ..* Alt

And then some actual binding, here for example to start the rxvt-unicode terminal emulator:

key=*.*t:rxvt-unicode

This means, when you in a running xsession, you can press and hold down Shift and Alt and then press t and this will spawn a terminal.
Of course you can use the hotkey combination you like the most, this is just an example.

9. Create a xinit file in the home directory of the user who will run the xsession

touch ~/.xinitrc

10. Put the starting command for the windowmanager in that file, so that the windowmanager will start right after the xsession has started

evilwm

This is an example, if you use a different windowmanager than evilwm, use the start command of that instead.

11. Reboot your computer

reboot

You need to do this for the drivers, so you could have done it before, but to do it after setting all things up is easier.

12. Start the xsession with the prefered user (so the user who has the .xinitrc and .keylaunchrc files in the home directory).

startx

13. End the xsession by killing the windowmanager

pidof evilwm | xargs kill