1. Persistent Reverse SSH Tunnel
One of my recent projects was to build a powerful, but relatively low cost deep learning computer (under $1000) with an older server workstation, a 6-core 3.5ghz Xeon processor, and a GTX 1080TI. Having done that, I didn't want to be chained to my home desk all of the time. However, since the internet at home is shared and the router is not accessible, I was limited in setting up port forwarding. Luckily, there is a relatively straightforward solution utilizing reverse SSH tunneling. The idea is simple. If you have a remote computer that is sitting behind a firewall that can't be accessed directly (or in my case behind an inaccessible router), but that computer can access another (relay) server which in turn can be accessed from the outside, say, by my laptop, then I can create an SSH tunnel from the remote computer to that relay server. With my laptop, I can also connect to that relay server, which will in turn allow me to SSH into the remote computer.
There are a great many use cases, but I'm primarily interested in the freedom it affords in an increasingly mobile world. This could also work as a kind of private VPN, but don't depend on it in places like China where the Great Firewall's machine learning implementation and deeper packet inspection seem to make it harder to establish a stable tunnel.
This guide assumes that you have 1) a remote server/computer you want to access from outside, 2) access to an outside server accessible by your remote server, and 3) a local computer (for less confusion we'll call it a laptop in this guide). This guide is written for linux/unix systems. We will create a persisting tunnel so that if something happens to our internet connection, it will automatically reestablish itself when it can, or it will autostart a tunnel connection upon bootup (we will use this feature later with remote Wake on LAN so we don't waste electricity letting our remote server sit idle). Please note that port numbers below are examples. You should use ports that are available on your systems.
Before we start, make sure you can ssh into your relay server from both your remote computer and your laptop. It is also important to take this time to setup ssh public key authentication into your relay server for security purposes (and of course testing by sshing with the correct credentials). Lastly, you'll need to install the OpenSSH package on all systems. It is a remote SSH tool with traffic encryption and other security protocols. In place of creating cron-jobs to check our tunnel connection and reestablishing them in case of broken connections, we will use a package called AutoSSH to do that. Check off the list below before proceeding:
✓ Setup public key authentication on 1) relay server, 2) remote computer, 3) laptop
✓ Install OpenSSH on all 3 systems with sudo apt-get install openssh (with mac, install homebrew, then try brew install openssh)
✓ Install AutoSSH on the remote system with sudo apt-get install autossh (with mac and homebrew, try brew install autossh)
✓ Note the ports you will use below and make sure that security rules on your systems allow those ports to be accessed
II. Reverse Tunnel on Remote Computer
Lets create a reverse tunnel from our remote computer to the relay server:
ssh -N -f -R 50000:localhost:22 -i /path/to/pub_key relayuser@relayIP
-N : non-interactive flag which says all we want is a tunnel, we wont run any remote commands (saves resources)
-f : our ssh tunnel will stay alive in the background; we don't need to run anything in order to keep it alive
-R : tells our relay server to open a (R)emote entry point/listening port (port 50000 in this example), which will forward anything from that port to localhost:22 (a.k.a the ssh client, i.e. your remote computer)
-i : accompanied usually by the path to your pub_key, this will tell ssh where to look to send credential information to login to your server.
This ssh command from our remote computer tells the relay server to open up a remote port, port 50000 (a listening port) and send whatever attaches to it back to localhost, via port 22 (our remote computer).
III. Connect to Remote Computer from Laptop
From our laptop, lets establish a connection to our relay server, and then when that is done, we can create a final connection that will allow us to ssh directly to our remote server:
ssh -N -f -L 40000:localhost:50000 -i /path/to/pub_key relayuser@relayIP
ssh -p 40000 remoteuser@localhost
-L : this is the usual local forwarding flag that tells our computer that anything attaching to port 40000 on our laptop will be forwarded to localhost port 50000 on our relay server.
-p : specify the port to connect via
By doing a local forwarding, anything that attaches to port 40000 on our laptop will be forwarded to our relay server's port 50000. Remember from earlier that we created a reverse tunnel which opened a listening port on the relay server. Anything hitting port 50000 on the relay server will be sent to our remote server! Hence, in the last step, we just ssh specifying port 40000 on our laptop, and we use the login name of the remote user we want to access. Done!
IV. Automating and Simplifying
Before proceeding, if you have tested the commands above and it worked for you, you will want to close those processes so you can test the next section. Search for them with:
ps -ef | grep ssh
Note the process ID (PID) of the processes that you spawned by invoking ssh. Terminate them with:
sudo kill PID
Now we will make use of AutoSSH (documentation: https://linux.die.net/man/1/autossh) to take care of broken connections for us. First lets setup the ssh config file so we can call autossh to create our tunnel with much more ease. On your remote computer, we will modify the ssh config file:
Host connect_RTUNNEL Hostname relayIP User relayuser IdentityFile /path/to/pub_key RemoteForward 50000 localhost:22 ServerAliveCountMax 3 ServerAliveInterval 30
We have now created a profile called connect_RTUNNEL. In place of an exceedingly long and hard to remember command, "autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -L 40000:localhost:50000 -i /path/to/pub_key relayuser@relayIP" we will simply input into the terminal:
autossh -M 0 -N connect_RTUNNEL
-M : This option is for monitoring port to keep connections alive, which we will will turn off with 0. Documentation suggests the next two flags to be superior practice.
-ServerAliveInterval : seconds between packets inspecting whether or not the server is alive
-ServerAlivecountMax : how many times it fails before declaring the connection dead (multiply this with ServerAliveInterval for approximate time between tunnel reconnection attempts)
On your laptop, you will now create an ssh config profile for your localforward tunnel, and an alias to ssh to your remote server:
Host connect_TUNNEL Hostname relayIP User relayuser IdentityFile /path/to/pub_key LocalForward 40000 localhost:50000 Host connect_remote Hostname localhost User remoteuser Port 40000
The first profile is called connect_TUNNEL, which is just a profile for our local forwarded tunnel. The second profile called connect_remote simply connects us to the remote server via port 40000. Lets invoke those profiles (you can choose to use autossh or not depending on your needs):
ssh -N -f connect_TUNNEL ssh connect_remote
If everything works, go ahead and kill these processes and we will proceed to the last section. Wouldn't it be wonderful if upon reboot, our remote computer automatically created the reverse tunnel connection and is ready on demand? It would be very useful if we wanted to turn the remote computer on/off, remotely! To do that, we will ask systemd (d stands for daemon, it is the first process that spawns all other processes thus it has PID 1). It is a simple matter of creating a new .service file and loading it:
sudo nano /etc/systemd/system/remote_tunnel.service
[Unit] Description=AutoSSH reverse tunnel to relay server After=network.target [Service] User=remoteuser Environment="AUTOSSH_GATETIME=0" ExecStart=/usr/bin/autossh -M 0 -v -N connect_TUNNEL [Install] WantedBy=multi-user.target
Make sure that you specify the "User" field as the sudo user that created this service, otherwise autossh wont be able to access the pub_key in the ~/.ssh directory of that user. Notice the added -v flag to help debug if there are any problems. Finally we will load the configuration:
sudo systemctl daemon-reload sudo systemctl start remote_tunnel.service sudo systemctl enable remote_tunnel.service sudo systemctl status remote_tunnel.service
You can use the last command to check on the status of your service, especially if you hit a snag somewhere. We are now fully automated on the remote side!