Koozali.org formerly Contribs.org

Using rclone for making back-ups

Offline holck

  • ****
  • 272
Re: Using rclone for making back-ups
« Reply #15 on: December 14, 2018, 10:19:42 AM »
I have changed the shell script, in order to avoid moving the remote directories. The script makes a remote copy of selected directories to Amazon S3, in the directory recent.  When a file is changed in recent, the previous version is moved to one of a rotating set of remote directories. If you have a set of 7 rotating directories, and you run the script daily, you will be able to go a week back in history and find your missing file there. If you just want the most recent version of your files or directories, you can find them in recent.

The first time the script is run, it will most likely take a few hours. After this, it will be quite fast. 10 minutes for my server.

I have a small problem finding the right names for variables etc. The script is, of course, a backup-script. But in rclone-terms, what I do is a cloning with a "backup-dir", containing the previous versions of changed files.

I will be thankful if someone here can help me improve the script, and maybe turn it into a contrib.


The script now looks like this:
Code: [Select]
#!/bin/bash

SERVER=$(hostname -s)
REMOTE="amazonS3:${SERVER}"                          # Name of the remote service 
MYDIR="/home/e-smith/files/users/holck/dev/rclone/"  # Home-directory of this script 
FILTER="filter.txt"                                  # Filter, naming local directories to in- and exclude in backup 
BACKUPLIST="bak.list"                                # Local file with names of a rotating set of remote directories to use for older file-versions
VERBOSE=""                                           # Add more v's for more verbosity: -v = info, -vv = debug 
LOG="/tmp/rclone_log.txt"                            # Temporary log file 

cd $MYDIR

# Rotate the BACKUPLIST-file, deleting the top entry and adding a new entry at the bottom 
OLDEST=$(head -1 $BACKUPLIST | sed 's/ .*//')        # The oldest, extra backup-version
COUNT=$(cat $BACKUPLIST | wc -l)                     # Number of files in list                                                   
((COUNT=$COUNT-1))
tail -$COUNT $BACKUPLIST > $BACKUPLIST.tmp
echo $OLDEST $(date) >> $BACKUPLIST.tmp              # Add a new line to the bottom, including today's date   

# Purge oldest remote version
date
echo "Purging oldest version ($OLDEST)..."
nice rclone purge ${REMOTE}/${OLDEST} ${VERBOSE} &>> ${LOG}
echo "...done"

# Perform actual backup
date
echo "Performing backup to ${REMOTE}..."
nice rclone sync / ${REMOTE}/recent --backup-dir ${REMOTE}/${OLDEST} --filter-from ${FILTER} ${VERBOSE} --skip-links --update --use-server-modtime &
echo "...done"
date
 
mv -f $BACKUPLIST.tmp $BACKUPLIST   # Update the BACKUPLIST
cat ${LOG} >> log.txt

Here is a sample filter:
Code: [Select]
+ /etc/e-smith/templates-custom/**
+ /etc/e-smith/templates-user-custom/**
+ /etc/group/**
+ /etc/gshadow/**
+ /etc/passwd/**
+ /etc/samba/secrets.tdb/**
+ /etc/samba/smbpasswd/**
+ /etc/shadow/**
+ /etc/ssh/**
+ /etc/sudoers/**
+ /home/e-smith/**
+ /opt/search/**
+ /root/**
+ /usr/lib/mailman/aliases/**
+ /usr/share/wordpress/wp-content/**
+ /var/lib/mailman/**
+ /var/log/**
- *

And a sample baklist-file:
Code: [Select]
old3
old2
old1
old7 Wed Dec 12 13:42:40 CET 2018
old6 Thu Dec 13 10:00:01 CET 2018
old5 Thu Dec 13 10:05:01 CET 2018
old4 Fri Dec 14 01:30:02 CET 2018


......

Offline ReetP

  • *
  • 1,875
Re: Using rclone for making back-ups
« Reply #16 on: December 14, 2018, 10:32:15 AM »
I'll try and take a look later.

Making a contrib could be done for the configuration side but I can't see a el6 rpm in a repo - there is a el7 one.

That makes it a bit trickier.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 1,875
Re: Using rclone for making back-ups
« Reply #17 on: December 14, 2018, 11:47:19 AM »
FYI it seems you can download a rpm of rclone direct from their site:

https://downloads.rclone.org/v1.45/rclone-v1.45-linux-amd64.rpm

I'm not sure on the best way forward. We could add the rpm to contribs, though adding the source then requires Go to be added as well to compile it.

Have to have a think about it.

Quote
I have a small problem finding the right names for variables etc. The script is, of course, a backup-script. But in rclone-terms, what I do is a cloning with a "backup-dir", containing the previous versions of changed files.

So can you describe where you are getting stuck?
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation