sektor
Having password access enabled for SSH access to your server is a bad security model. You are asking to be hacked & probably in time you might get hacked.
Even using a non standard port (ie something other than 22) does not enhance security at all, it might stop many junior hackers who do not bother to try other ports, but serious hackers can easily scan for other open ports eg SSH on port 221 or whatever & start their attacks.
While multiple layers of authentication may be an answer, SME server does not have that, & changing templates may introduce other security holes.
At present the use of public/private keys to access SSH is very secure (with password access disabled), without the key it is impossible (in real world practical terms) to access SSH. Maybe with a Cray computer you might hack the key after a few years, but most people do not have that resource available to them or the time, energy & dedication to keep up such a sustained attack on a personal or small business server.
As DanB35 says, & I use it also, you can configure key access on most smartphones using good quality free or at cost SSH apps.
Once you have secure SSH access established you can then use the text based server manager if you really need access to that, or create a SSH tunnel to then allow web browser GUI server manager access via
http://localhost/server-managerIf your work LAN access blocks some of those, then try tethering your smartphone via USB & use the 3G/4G Internet connection to access your home server.
Also re your comments about using VPN, well that is no longer secure. There was comment in these forums some months ago about that, & you can google for upstream advisories.
I think there is a suggestion to remove the current VPN from SME server & replace it with a secure version (refer one of the current contribs).
The only secure SSH connection method is public/private keys. There is a very good Howto (see link at top of forums) & it is fairly straightforward to implement. You can export the key to your smartphone app & carry the key with you on a USB memory stick, & even have the Putty exe on the USB, to use wherever you are in the world.