@UnixRocks, @steve_v
Then tigerVNC and x2go it is for the next tries.

@nixer
Thanks for the detailed description, Xephyr sounds very interesting. I need time to get used to the concept, currently the nested / recursive X-server creates a knot in my head. As you mentioned, specifying the correct DISPLAY sounds like challange.

All three suggestions can be used with ssh, which is great.

PS:
Yesterday, while trying to theme lightdm ... .
The arch wiki mentioned a test mode lightdm --test-mode --debug, which works using xephyr. lightdm.conf has a VNC section and there is the installble lightdm-remote-session-x2go debian package. First I thought these components were addressing the server-side function, but the x2go module seems to be client related. Anyway.

ssh, vnc, rdesktop is basically all I know.

There are a lot of possibilities here and I will throw out the one example that I use the most:  Xephyr, Xming, and SSH and the technology is called Nested Xsessions.

Xephyr is a linux program which runs nested X sessions on the client (local) system.  Xming is the windows program that displays the remote system on a local windows computer.  SSH is the tunnel to transport the xsession from remote to local.  You might have to also use Putty with windows and Xming.  At the time, I had never heard of "nested Xsessions" before so this was all new to me.  I prefer it over VNC, X11vnc, etc. as you do not have to have an Xsession running on the remote system.  It only has to be installed, and you start the remote xsession from the local machine and it displays within the Xephyr window on the local desktop.

There is a lot written about this on the web, so check if you think it may fit your needs.  The many tutorials written on the web explains it better than I can.  The hardest part for me to get it working initially was getting the Display number right.  Also, you might have to make a few configuration changes on the server involving ssh xforwarding, so root access would be required there.

For my needs, I use it to access a remote server that is running in headless mode which has a lightweight xserver installed on it (Xfce).  It is not perfect but it is the best that I have used to access a remote server which does not have a X session running.  I have two aliases configured in my local client bash profile so it is easy and quick to access the remote server.

My goal is to get JWM on the remote server instead of Xfce, since it is uses less resources.  This is a work in progress.

Lab: TigerVNC Server, FreeBSD 12.3-RELEASE-p2
Home: TigerVNC Viewer, Devuan GNU/Linux 4 (chimaera)

I'm asking for suggestions for remote desktop access (the server component). What are you using?

ssh, vnc, rdesktop is basically all I know. "X-application with display on a different X-host" can be an option. Maybe there are other protocols commonly used. Something straight and simple would be nice.

The connection ability from anydesk / teamviewer to pass a router is not needed.
File transfer would be nice, but also not needed.
It may be a virtual session or mirror of the current display.
If there is a real killer application, it could be wayland too.
Sorry, but I need a client for windows too (work; can not change that).

xrdp: Currently used. The keymap issue is only one thing. Usage feels not very responsive, even in the same network. Moving a browser window ... raises +20 years old memories.
vnc: occasionally used, but good memories. Small and simple to setup. What would it be? tigervnc, tightvnc, x11vnc, ... ?

Thanks and regards.