Remote systems will need to allow remote access. Changing the port ssh listens on will cut the amount of noise in the logs but doesn't provide much real security. The one I help administer has root logons disabled and the accounts on it have non-obvious names and strong passwords (we normally log on with SSH keys anyway). So an attacker would need to guess both an ID and it's password.

And we have backups so it's not a disaster if it did get hacked.

But there are always people who apparently feel good at misunderstanding.

I'd suggest using port knocking for all ssh service. There are several port knocking schemes available, and you can easily set up your client .ssh/config to issue port knocking sequences in any scheme prior to making the connection. And at the server(s) you may utilise iptables with ipset to keep the ssh port blocked normally, and then open briefly for an client IP that presents the "right knock".

One class of knock detection would use UDP message(s) with special code(s) to pre-defined ports, where perhaps the server have special iptables rules to detect those. Rules like

-A knock -p tcp -m set --match-set GATE4 src -j SET --add-set GATE4 src --exist --timeout 600
-A knock -m set --match-set GATE4 src -j ACCEPT
-A knock -p udp -m udp --dport 25 -m string --string "I'mTheUrbanSpaceman" --algo bm --from 28 -j SET --add-set GATE4 src --timeout 5
-A knock -p tcp -m tcp --dport 22 -j DROP
-A INPUT -j knock
-A OUTPUT -p tcp -m set --match-set GATE4 src -j SET --add-set GATE4 src --exist --timeout 600

That collection of rules opens port 22 for 5 seconds upon receiving a UDP message with special content on port 25, and then refreshes the hole to 10 minute openings for any input or output packet from/to that same IP.

I'm sure there are packaged schemes available as well, if iptables rule editing feels scary.

the attac was performed through a login as root via ssh. With the correct password. No attepts to guess it could be found.

inside job

Anyway, scanbot.me seems to be pretty well known as a phishing site which usually works through email appendices. Which rises the question if and if yes, how, the door could have been the mail server (older postfix with greylisting, no DKIM/DMARC, but otherwise thoroughly configured) in combination with recent thunderbird.

I am running claws-mail which cannot execute garbage in the emails, but the other users (ex wife and my son) use thunderbird.

I know, ASCII is really outdated, but it was not upgradable due to technical issues - the virtual machine running it limited glibc to version 3.10 which meant that nothing more recent could be run.

What happened:

the attac was performed through a login as root via ssh. With the correct password. No attepts to guess it could be found.

Strange enough, the ssh server had been opened to allow root login. Which i had disallowed from the very first setup of that server. But i just saw that ssh root login is enabled by default in excalibur without a warning. 

To me it looks as if the intruder actually had compromised the system earlier,  learning the password (it was strong 20 years ago, but weak for modern resources).

Anyway:

Apparently this is closely connected to the phishing site scanbot.me showing that that site even intrudes mail servers if possible.

the intruder installed a bot-net client in /tmp/.drone/ hiding in the process list as
"-bash" and "crond"

hidden in

/tmp/.drone/

It was conncted to server 45.58.183.18 through port 6667 and to 51.79.74.212 through port 80. There are more possible controlling servers configured.

the crontab had been modified to restart the client within 5 minutes and to load data from scanbot.me

of course the "authorized_keys" of any user on the system are also compromised.