SSH fundamentals, CSSH and Fabric.

The idea behind this post is to inform readers what is possible with SSH, and how to use it. If you have anything to correct or add, then feel free to add a comment. I hope you will enjoy reading the post.

The level of complexity will increase a bit as you scroll.

Table of Contents

• What is SSH
• How to install SSH
• How to connect to a SSH server from a SSH client
• Basic SSH server security
• How to copy files over SSH from one server to another server
• How to use SSH tunneling
• Fabric - streamline SSH with Python
• Cluster SSH

What is SSH

SSH (secure shell) is used to remote connect to another computer (with a SSH daemon/server). SSH is usually used through the command-line (terminal) of your POSIX-system, but can also be used through Windows with PuTTY or even better, Cygwin (http://www.cygwin.com/). Later it has also been made possible to SSH to machines from your web browser.

Examples of web-based SSH clients:
GateOne: https://github.com/liftoff/GateOne
Shell In A Box: https://code.google.com/p/shellinabox/
Chrome extension: https://chrome.google.com/webstore/detail/secure-shell/pnhechapfaindjhompbnflcldabbghjo

SSH's default port is 22.

How to install SSH

The most common SSH software is OpenSSH (or OpenBSD Secure Shell), which you can install with one of the following lines.

For SSH client only:

$ sudo apt-get install openssh-client

For SSH server only:

$ sudo apt-get install openssh-server

For the whole OpenSSH package:

$ sudo apt-get install openssh

With these packages you should be ready to go.

How to connect to a SSH server from a SSH client

Basics

Simply login in to a newly installed SSH server like this:
$ ssh user@remote-ip-address

This will usually be with the root user (eg. root@109.202.159.44).

This will prompt you for password, and afterwards log you in to the given server/computer.

If the port is not default (22) use the -p flag, eg. if port is 42:
$ ssh user@remote-ip-address -p 42

If you do not want to login and stay logged in, but only want to fire off one command use:
$ ssh user@remote-ip-address [command]

Keys

Some servers need you to have your public key in the remote servers authorized_keys file. Create your SSH keys with this line:
$ ssh-keygen -t rsa

This is a tool which comes with the OpenSSH package. This will generate your SSH public key, which you can find at ~/.ssh/id_rsa.pub.

Install this by coyping it into the remote hosts authorized_keys file, or do it with ease with the following line:
$ ssh-copy-id user@remote-ip-address

This will prompt you for password, and afterwards copy your public key directly into the authorized_keys file in the remote server.

Your public key registered in the server will save you for all the hassle with typing password everytime you need to remote login to the server aswell, and it is much more safer than password protection for the server itself.

You can think of the public key as a key in your hand, and the authorized_keys-file in the server as a lock which decide if you have access to open it or not.

~/.ssh/config

You can make a list of saved SSH connection in the config-file of your ~/.ssh directory. If it is not there already, create the config file: $ touch ~/.ssh/config

Here you can save your connections, and login to them more easily. Edit the config-file with your prefered editor. The config files could eg. look like this:

 Host web01
     HostName 12.34.56.1
     User root
     Port 42

 Host web02
     HostName 12.34.56.2
     User root
     Port 42

With this you can be able to login just like this, instead of remembering all the ip-adresses:
$ ssh web01

Basic SSH server security

Basics

With a newly installed SSH server (and maybe a completely new server in general) it is very important to do a few things which will have big impact on security. This is a few things you can do to secure your SSH a bit - we do not want any evil crackers to get access!

User

If you have not made a new user yet, and only have root-access, start with making a new user:
$ adduser your-username

Then give the user all rights:
$ visudo

Copy the line with the root-user, and paste it with your own username.

root    ALL=(ALL:ALL) ALL
your-username    ALL=(ALL:ALL) ALL

SSH server config (/etc/ssh/sshd_config)

Open your sshd_config file, to do a few configurations, with your prefered editor. In my case Vim.

$ vim /etc/ssh/sshd_config

Insert or edit a few lines, so they matches:

  #No root-login
  PermitRootLogin no 

  #Allow your new user to have access
  AllowUsers your-username 

  #Change SSH server port to another than the default (22) - optional
  Port 42

  #Make sure you use Protocol 2 (SSH2), usually default
  Protocol 2 

You do not have to change your SSH port, and some people says it is wrong to do this because of the complications it creates in a big server farm, but no matter what it is a proved quick fix for security if you of some reason do not want to base your servers access on public keys alone, although this is recommended.

Optionally you can restrict password authentication on the server, so it is only possible to login if your public key is in the SSH servers authorized_keys file (we created earlier). Make sure this is the case, if you choose to use this. You can restrict this with this line in the sshd_config file:

PasswordAuthentication no

Restart your SSH to make the new configurations active:
$ sudo service ssh restart

Fail2ban

Read about how to prevent SSH intrusion with Fail2ban here in my previous post.

How to copy files over SSH from one server to another server

There is a lot of ways to achieve this. You can eg. do this directly with SSH, with SSH + Rsync, or with SCP. I really like SCP, because it keeps things simple and it is default on most POSIX-systems.

SCP

SCP uses SSH1, and is just a tool to copy files between computers/hosts. You can use flag -1 to force SCP to use SSH2 (protocol 2, which is more secure).

You can download files and upload files with ease with SCP.

To upload files use it like this:
$ scp -r /upload/from/this/path user@remote-ip-address:/upload/to/this/path

To download files, do the opposite, and use it like this:
$ scp -r user@remote-ip-address:/download/from/this/path /download/to/this/path

If you only have to transfer one file, the -r flag is not neccessary, but if you have to transfer a directory, you have to use it. It transfer recursively entire directories with the -r flag.

A little sweet trick from bsdadmin, from Reddit:
"Lets say you have box A, box B, and box C. You need a file from box A on box C. However, they have no access to each other. Box B has access to both. How would you copy the file? It's easy! On Box B, do this:
scp -3 user@A:/file user@C:/file"

See more in the man-page:
$ man scp

SSH

I will only make one simple example with SSH alone.

This could eg. be done with a simple txt file, directly from SSH, like this:
$ ssh user@remote-ip-address "cat file-on-server.txt" > local-file.txt

This basically just open a SSH instance on a server, print out what is in a file with "cat", close the connection, and redirects standard output (stdout) to the local file local-file.txt ([output from server] > [file output is redirected to]).

SSH + Rsync

The transfer of files from one server to another can also be achieved with Rsync together with SSH. Rsync can be used in many ways tho, and is also a great tool for local use.

This would be the line to use, to download a whole directory:
$ rsync -avz -e ssh user@remote-ip-address:/download/from/this/path /download/to/this/path

Flag descriptions from $ man rsync:

-a, --archive - archive mode
-z, --compress - compress file data during the transfer
-v, --verbose - increase verbosity
-e, --rsh=COMMAND - specify the remote shell to use

If you just need to download one file, it would be ok only to specify the remote shell (-e) and not use the archive (-a) / compress (-z) or the increase verbosity (-v, optional) flags.
$ rsync -e ssh user@remote-ip-address:/download/from/this/path /download/to/this/path

To upload a file from your machine to a remote machine you should the opposite and switch the remote-server and the local path, so the local path is first and the remote path after. Like in SCP. Should look like this:
$ rsync -avz ssh /upload/from/this/path user@remote-ip-address:/upload/to/this/path

See more in the man-page:
$ man rsync

How to use SSH tunneling

You can do a lot with SSH. Also port tunneling. This is a handy tool to secure access with various programs which depends on network.

This is often used with databases, and first time i used port tunneling with SSH was with a MySQL database.

So i will use this example. There was a MySQL database which was only open for clients local on that machine, and i needed to have access to it to work with the database on my local machine. This is was a problem.

Therefore, i set up a SSH port tunnel to this server which pointed to the port where MySQL was running, the default MySQL port 3306.

Then i tried to connect to the MySQL database in my local project again, and this time successfully. My own machine thought it was connecting to a local MySQL database, but it was actually the remote-machines MySQL database through the tunnel.

I did this with the following line:
$ ssh -L 3306:localhost:3306 user@remote-ip-address

First we start of with calling the ssh-command, then we tell it to use the -L flag which is the flag used for port forwarding. Then we tell it which port it should listen to locally through the tunnel (3306:localhost:3306), then we tell it to use the remote-host localhost (3306:localhost:3306) and then finally we tell which port it needs to use from the remote host (3306:localhost:3306).

So a summary is this: local-port:remote-host:remote-port.

Then we finish the line of with username and remote-ip-address for the server you want to use the tunnel with (local-port:remote-host:remote-port user@remote-ip-address).

It is easy to see the effect if you test with port 80 (default http port), and have a server serving some web-content somewhere.

Run this on you local machine:
$ ssh -L 80:localhost:80 user@remote-ip-address-with-web-server

This would require root permissions, since the port is privileged (port < 1024).

Then go to "http://localhost" in your browser, and you can see the web content served from the server, like it was local on your computer, although it is actually served from the server you just started a tunnel with.

I actually like SSH man-page on the -L flag:
"-L [bindaddress:]port:host:hostport Specifies that the given port on the local (client) host is to be forwarded to the given host and port on the remote side. This works by allocating a socket to listen to port on the local side, optionally bound to the specified bindaddress. Whenever a connection is made to this port, the connection is forwarded over the secure channel, and a connection is made to host port hostport from the remote machine."

Fabric - streamline SSH with Python

Basics

Fabric is a library and a command-line tool which it makes it really simple to execute commands locally and remotely on your servers with Python. I think it is perfect to collect data about various servers and to make a set of commands you want executed on those.

Installation

Install Fabric with pip:
$ sudo pip install fabric
Or with apt-get:
$ sudo apt-get install python-fabric

Introduction to functionality

I think Fabric's simplicity makes communication between servers through SSH really easy, and i love the the logic behind the functions.

Some of the basic functions:
local() - execute command locally
run() - execute command remotely
cd() - execute command from a specific dir
sudo() - execute command remotely with sudo
put() - similar to SCP or Rsync. Upload files.
get() - similar to SCP or Rsync. Download files.
prompt() - Prompt for a value to use in script.

Fabric as a CLI

With Fabric as a CLI you should create the so called fabfile. Create a file called fabfile.py in your current directory, with something similar to:

#!/usr/bin/env python
from fabric.api import *

def uptime():
  local('uptime')
  run('uptime')

def helloworld():
  print('Hello World - everything works in Fabric')

You can then call it from the command-line with the command $ fab

Try to write $ fab -l, this will return:

Available commands:

    helloworld
    uptime

Then you can use $ fab helloworld for example which will print out "Hello World - everything works in Fabric". You will fast experience that Fabric is a very flexibile, simple and strong module.

The uptime command would run the bash command "uptime" locally on your machine (the local() command), and on the machines you have assigned to the fabfile or the command. You can assign machines with the -H flag or in the fabfile. I like to have them collected in the fabfile:

#!/usr/bin/env python
from fabric.api import *

env.hosts = ['remote-ip-address-1', 'remote-ip-address-2', 'remote-ip-address-3', 'remote-ip-address-4']
env.user = 'user'

def uptime():
  local('uptime')
  run('uptime')

def helloworld():
  print('Hello World - everything works in Fabric')

If you called $ fab uptime now it would run $ uptime on all four remote machines + your local machine.

It is also possible to wrtite env.password = '', to specify a password, but you should never write a plain password in your files. Then rather stick with a public key in the servers authorized_keys - then no need for the password hassle.

Read more in the Fabric documentation - there is a lot more examples there with complex use.

Fabric is also rather popular for deployment. I like the Fabric example from Flask's (web framework) documentation:
http://flask.pocoo.org/docs/patterns/fabric/

Fabric as a library

To use Fabric as a library, things are not much different. It is mostly a question about structuring it in another way.

You actually just need to call the functions you define in your scripts within your document instead of calling them from the command-line with the $ fab-command.

Here i made very simple backup script:

from fabric.api import put, env

class Backup():

    def __init__(self, local_dir, remote_dir):

        env.users = ['root']
        env.hosts = ['remote-ip-address']

        self.local_dir = local_dir
        self.remote_dir = remote_dir

        print 'Starting backup - '

        self.upload()

    def upload(self):

        try:
            put(self.local_dir, self.remote_dir)
        finally:
            print 'Backup made'

  Backup('/local/path', '/remote/path')

I will add more examples with the most used Fabric functions soon.

Cluster SSH

Basics

With ClusterSSH you can login at multiple servers at one time and write the same commands for all the machines. This is VERY handy if you need to do the same configuration at several machine. Especially if you know it is just a one-timer. Then you do not have to make a Puppet script or similar.

ClusterSSH is supported by most operating systems. I have used it for both OS X and Debian-based Linux distros.

Download

Download link:
http://sourceforge.net/projects/clusterssh/

Example

You can just use it with its built-in command "cssh", eg. like this:
$ cssh user@web{0,1,2,3,4,5,6,7}

This will open 8 windows with SSH-instances (web0, web1, web2, web3, etc.), and a command line where you write commands for all the instances at once.

So if you for example want to update/upgrade packages on 20 machines, you can open them in CSSH with one command (the one i described above), and write $ sudo apt-get update && apt-get upgrade in the CSSH command-line, and it will type it and run the command for all the instances.

Screenshot