Gems of knowledge into my sea of ignorance .. I hope the OP gets to mark this as solved soon.

Sorry, i hate coming across as a smartass Anyways, i also hope there will be a satisfying solution.

There are probably many holes in my understanding, but isn't it so, that the server-vpn program already interacts on eth0 for handling its clients; so it's response packets are already outbound on eth0 without any routing?

Well, not saying i am total expert here either. I've done a bit of weird stuff like per user routing and so on but it's not like i deal with this on a regular basis. Still to my best knowledge the kernel doesn't care where a packet was received when sending a reply. It just goes by what the routing table says. The reply packets disappearing from eth0 as soon as the tunY default route is up imo underlines this. Also when you look into reverse path filtering you'll see that it deals with exactly such cases (seems it's not used by default in Devuan - Ubuntu for example does use it by default i think - otherwise OP would have to disable it to not have the kernel outright drop the incoming packets).

mmm ... all 192.168.x.0/24 (ethX) traffic should go out without ado due to the network route.

Sure but if his internal VPN server gets a connection from the internet that won't be from 192.168.x.0/24 or any kind of local address but a random WAN IP (which OP doesn't know beforehand so no chance to set a conventional route for it) and when the server attempts to answer the only route matching the destination IP will be the default route pointing at tunY.

It's a bit of a weird setup to actually have a need for routing based on source port (well, at least that's how i'd single out traffic originating from his internal VPN server) rather than destination IP but in this case i don't see how it could work otherwise. Well, i guess he might get away with source based routing also but in that case all traffic originating at eth0 (or rather it's IP) would get routed through eth0 (his normal internet connection). That might spare him the tagging of individual packets but it's not exactly (as i understand it) what he is asking for and imo it wouldn't be all that much simpler.

That's a good set of rules for doing no filtering at all, yes

If you don't mind, rather dump the output if iptables-save, just to confirm all the tables. Basically, you shouldn't need any iptables rules, except the masquerading of the outbound traffic on tunY. That one is necessary to allow packets with original source from IP 10.8.x.0/24 (i.e., your server-vpn clients) or 192.168.y.0/24 (your wlanX neighbors), as well as allowing packets from 192.168.x.0/24 (your ethX neighbors) to be forwarded and masqueraded through tunY.

Agreed. He'll likely need a couple of additional rules to route selected internet traffic (communication with his internal VPN server) over eth0 though. At least that's the best idea i have to actually have replies (those packet's will have a non local destination) to packet's that reach his VPN server ever eth0 exit there again while having a default route that points towards tunY

If you don't mind, rather dump the output if iptables-save, just to confirm all the tables. Basically, you shouldn't need any iptables rules, except the masquerading of the outbound traffic on tunY. That one is necessary to allow packets with original source from IP 10.8.x.0/24 (i.e., your server-vpn clients) or 192.168.y.0/24 (your wlanX neighbors), as well as allowing packets from 192.168.x.0/24 (your ethX neighbors) to be forwarded and masqueraded through tunY.

On the other hand, with route-nopull disabled, I saw the packets trying to reach ethX, but as you said, no replies.  And I've gotta be totally honest with you, but I have no idea what that means.

Well, it's not 100% sure but it pretty much implies that the routing "works". Sadly not "works" as in does what you but rather does what it says though. I am quite certain the missing replies aren't really missing but just exit on the wrong interface as the default route that's in place when you remove route-nopull tells them. I guess you could see them when point tcpdump to tunY rather than eth0. That's basically what i was aiming at with the custom routing table but it seems that didn't work for some reason. I am quite sure that would be the way to go though.

could you post outputs of the following commands

ip route show table isp

ip rule show

iptables -vnL

iptables -t mangle -vnL

before and after running

iptables -A PREROUTING -i eth0 -t mangle -p udp --sport [YOUR-INTERNAL-VPN-SERVER-PORT] -j MARK --set-mark 1
ip rule add fwmark 1 table isp
ip route add default via 192.168.x.1 dev eth0 table isp

I know this is going to be a lot of output but i am trying to wrap my head around where it could go wrong and it's nothing i can easily test locally. Also your internal OpenVPN server uses UDP and when looking at tcpdump it's reply packets used [YOUR-INTERNAL-VPN-SERVER-PORT] as source port, right? Otherwise the iptables rule that should mark the packets would be wrong.

What would be the best way to check?

Remove the rule

iptables -A FORWARD -j REJECT

to see if it works without it.

Same behavior.  Here's my current config:

# iptables -L --line-numbers
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
2    ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         
1    ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
2    ACCEPT     all  --  anywhere             anywhere            
3    ACCEPT     all  --  anywhere             anywhere            
4    ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
5    ACCEPT     all  --  anywhere             anywhere            
6    ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    ACCEPT     udp  --  anywhere             anywhere             udp spt:domain
2    ACCEPT     udp  --  anywhere             anywhere             udp spt:domain

What would be the best way to check?

Remove the rule

iptables -A FORWARD -j REJECT

to see if it works without it.

Then add it back, preceded by rules that allow forwarding between tunX and tunY.

Well, as long as you have to default route on tunY VPN config should be fine. What i'd do at this point is fire up tcpdump to debug:

tcpdump -ni eth0

Do you see the packets from your client connecting to your internal VPN server? If yes do also see the replies going from the server to the client? If you see the packets from the client but don't see any replies can you see replies when running

tcpdump -ni tunY

Edit: You can add

port [YOUR-INTERNAL-VPN-SERVER-PORT]

to the tcpdump commandline to filter traffic. Might be a bit hard to spot anything otherwise.

Okay, so I have no idea why I never thought to run tcpdump, but you nailed part of the diagnosis, which puts me at ease.  With route-nopull, I can see packets to-and-from my mobile device, but it doesn't use tunY.

On the other hand, with route-nopull disabled, I saw the packets trying to reach ethX, but as you said, no replies.  And I've gotta be totally honest with you, but I have no idea what that means.

Again, in all honesty, I'm not sure.  What would be the best way to check?

tcpdump -ni eth0

Do you see the packets from your client connecting to your internal VPN server? If yes do also see the replies going from the server to the client? If you see the packets from the client but don't see any replies can you see replies when running

tcpdump -ni tunY

Edit: You can add

port [YOUR-INTERNAL-VPN-SERVER-PORT]

to the tcpdump commandline to filter traffic. Might be a bit hard to spot anything otherwise.

Edit2: It not working with nopull (as that should have eliminated the default route toward tunY) seems a bit fishy to me. If it worked and the default route points towards eth0 i don't see how it could stop you from connecting from outside. Are you sure the packets aren't getting blocked by iptables?

echo 200 isp >> /etc/iproute2/rt_tables
iptables -A PREROUTING -i eth0 -t mangle -p udp --sport [YOUR-INTERNAL-VPN-SERVER-PORT] -j MARK --set-mark 1
ip rule add fwmark 1 table isp
ip route add default via 192.168.x.1 dev eth0 table isp

Obviously can't really test it but it should basically mark packets coming from your internal VPN server and add a rule to make them use an entirely different routing table which does nothing but send the packets through your ISP instead of the external VPN. Rationale being: I think the problem isn't really not being able to reach the box from out side (the packets arrive just fine) the packets that the internal VPN server sends back to it's outer client get routed through the external VPN instead of being routed through your ISP as the client expects.

Interesting idea.  Unfortunately, the behaviour is the same.  :\  (I tried it with and without route-nopull; those behaviours haven't changed at all.)  Should this be used in conjunction with some option added to my server/client configs?

echo 200 isp >> /etc/iproute2/rt_tables
iptables -A PREROUTING -i eth0 -t mangle -p udp --sport [YOUR-INTERNAL-VPN-SERVER-PORT] -j MARK --set-mark 1
ip rule add fwmark 1 table isp
ip route add default via 192.168.x.1 dev eth0 table isp

Obviously can't really test it but it should basically mark packets coming from your internal VPN server and add a rule to make them use an entirely different routing table which does nothing but send the packets through your ISP instead of the external VPN. Rationale being: I think the problem isn't really not being able to reach the box from out side (the packets arrive just fine) the packets that the internal VPN server sends back to it's outer client get routed through the external VPN instead of being routed through your ISP as the client expects.