Encrypted streaming backup

I have a development server to backup. For additional safety I wanted to use a FTP host to store another copy of its contents. This was the third backup alternative, so there was no need for anything incremental - this was a last chance backup if everything else failed.

But there was an issue - the server had a fast but quite small hard drive. Around 30GB total, over 60% in use. This meant, that a simple tar of / and sending it to FTP was not going to work.

The solution I found was using tar in combination with ncftpput and mcrypt for encryption:

# overview of command, see script below for full use
tar cjf - ... | mcrypt | ncftpput ...

First, tar cjf - ... archived the required directories with bzip2 compression using stdout as output. Then mcrypt encrypts it (using environment for its configuration). Finally ncftpput -c is used for the upload.

Full backup script

(download)

#!/usr/bin/env bash

# Streams an encrypted backup to an FTP server
# The backup replaces the previous one, but only after it is successfully sent
# See https://chanibal.pl/notes/encrypted-streaming-backup/ for up to date version

# Configuration start
export MCRYPT_ALGO="xtea"
export MCRYPT_MODE="ecb"
export MCRYPT_KEY="your-very-secret-password"
FTP_USER="your-ftp-user@your-ftp-host.tld"
FTP_PASS="your-ftp-password"
FTP_HOST="your-ftp-host"
FTP_FILE="your-backup-name.tar.bz2.mcrypt"
FTP_USE_TEMPORARY=false
# Configuration end

set -eo pipefail

if [[ "$MCRYPT_KEY" = "your-very-secret-password" ]]; then
  echo "Error: you should properly configure the script before using it."
  exit 2
fi

case "${1:-}" in
  "backup")
    if [[ ! $# -eq 1 ]]; then
      echo "Backup does not support positional params"
      exit 1
    fi

    USE_TEMPORARY_FLAG=""
    if [ "$FTP_USE_TEMPORARY" == true ]; then
      USE_TEMPORARY_FLAG="-S .tmp"
    fi

    echo "Backup started $(date -Is)"
    ncftpls -al -u "$FTP_USER" -p "$FTP_PASS" "ftp://${FTP_HOST}/${FTP_FILE}" \
      | awk '1 { printf ( "Previous backup size: %.3fGB\n", $5 / 1073741824 ) }'
    tar cjf - /etc /srv /home | mcrypt \
    | ncftpput -c $USE_TEMPORARY_FLAG -u "$FTP_USER" \
                -p "$FTP_PASS" "$FTP_HOST" "$FTP_FILE"
    ncftpls -al -u "$FTP_USER" -p "$FTP_PASS" "ftp://${FTP_HOST}/${FTP_FILE}" \
      | awk '1 { printf ( "New backup size: %.3fGB\n", $5 / 1073741824 ) }'
    echo "Backup ended $(date -Is)"
    ;;

  "restore")
    if [[ ! $# -eq 3 ]]; then
      echo "Restore requires two arguments"
      exit 1
    fi

    restore_file="$2"
    restore_destination="$3"
    
    mcrypt -d <"$restore_file" \
      | (mkdir -p "$restore_destination"; cd "$restore_destination"; tar xjf -)
    ;;

  "--help"|"-h"|"help"|*)
    echo "Streaming backup, usage:"
    echo "./backup-stream.sh backup"
    echo "./backup-stream.sh restore <restore_file> <restore_destination>"
    exit
    ;;
esac

Example output on error

Backup started 2020-10-16T02:00:01+02:00
Previous backup size: 17.422GB
tar: Removing leading `/' from member names
Remote write failed after 4613111808 bytes had been sent: Connection reset by peer.
Signal 13 caught. Exiting.
tar: -: Wrote only 4096 of 10240 bytes
tar: Child returned status 141
tar: Error is not recoverable: exiting now

Changelog

2018-12-01 First use of streaming backup
2019-04-15 Original note
2021-05-25 Hosting threw us out, lol
2021-05-28 Move to Dropbox
2022-01-08 Cleanup and publish