Koozali.org: home of the SME Server

Using rclone for making back-ups

Offline holck

  • ****
  • 317
  • +1/-0
Re: Using rclone for making back-ups
« Reply #15 on: December 14, 2018, 10:19:42 AM »
I have changed the shell script, in order to avoid moving the remote directories. The script makes a remote copy of selected directories to Amazon S3, in the directory recent.  When a file is changed in recent, the previous version is moved to one of a rotating set of remote directories. If you have a set of 7 rotating directories, and you run the script daily, you will be able to go a week back in history and find your missing file there. If you just want the most recent version of your files or directories, you can find them in recent.

The first time the script is run, it will most likely take a few hours. After this, it will be quite fast. 10 minutes for my server.

I have a small problem finding the right names for variables etc. The script is, of course, a backup-script. But in rclone-terms, what I do is a cloning with a "backup-dir", containing the previous versions of changed files.

I will be thankful if someone here can help me improve the script, and maybe turn it into a contrib.


The script now looks like this:
Code: [Select]
#!/bin/bash

SERVER=$(hostname -s)
REMOTE="amazonS3:${SERVER}"                          # Name of the remote service 
MYDIR="/home/e-smith/files/users/holck/dev/rclone/"  # Home-directory of this script 
FILTER="filter.txt"                                  # Filter, naming local directories to in- and exclude in backup 
BACKUPLIST="bak.list"                                # Local file with names of a rotating set of remote directories to use for older file-versions
VERBOSE=""                                           # Add more v's for more verbosity: -v = info, -vv = debug 
LOG="/tmp/rclone_log.txt"                            # Temporary log file 

cd $MYDIR

# Rotate the BACKUPLIST-file, deleting the top entry and adding a new entry at the bottom 
OLDEST=$(head -1 $BACKUPLIST | sed 's/ .*//')        # The oldest, extra backup-version
COUNT=$(cat $BACKUPLIST | wc -l)                     # Number of files in list                                                   
((COUNT=$COUNT-1))
tail -$COUNT $BACKUPLIST > $BACKUPLIST.tmp
echo $OLDEST $(date) >> $BACKUPLIST.tmp              # Add a new line to the bottom, including today's date   

# Purge oldest remote version
date
echo "Purging oldest version ($OLDEST)..."
nice rclone purge ${REMOTE}/${OLDEST} ${VERBOSE} &>> ${LOG}
echo "...done"

# Perform actual backup
date
echo "Performing backup to ${REMOTE}..."
nice rclone sync / ${REMOTE}/recent --backup-dir ${REMOTE}/${OLDEST} --filter-from ${FILTER} ${VERBOSE} --skip-links --update --use-server-modtime &
echo "...done"
date
 
mv -f $BACKUPLIST.tmp $BACKUPLIST   # Update the BACKUPLIST
cat ${LOG} >> log.txt

Here is a sample filter:
Code: [Select]
+ /etc/e-smith/templates-custom/**
+ /etc/e-smith/templates-user-custom/**
+ /etc/group/**
+ /etc/gshadow/**
+ /etc/passwd/**
+ /etc/samba/secrets.tdb/**
+ /etc/samba/smbpasswd/**
+ /etc/shadow/**
+ /etc/ssh/**
+ /etc/sudoers/**
+ /home/e-smith/**
+ /opt/search/**
+ /root/**
+ /usr/lib/mailman/aliases/**
+ /usr/share/wordpress/wp-content/**
+ /var/lib/mailman/**
+ /var/log/**
- *

And a sample baklist-file:
Code: [Select]
old3
old2
old1
old7 Wed Dec 12 13:42:40 CET 2018
old6 Thu Dec 13 10:00:01 CET 2018
old5 Thu Dec 13 10:05:01 CET 2018
old4 Fri Dec 14 01:30:02 CET 2018


......

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Using rclone for making back-ups
« Reply #16 on: December 14, 2018, 10:32:15 AM »
I'll try and take a look later.

Making a contrib could be done for the configuration side but I can't see a el6 rpm in a repo - there is a el7 one.

That makes it a bit trickier.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Using rclone for making back-ups
« Reply #17 on: December 14, 2018, 11:47:19 AM »
FYI it seems you can download a rpm of rclone direct from their site:

https://downloads.rclone.org/v1.45/rclone-v1.45-linux-amd64.rpm

I'm not sure on the best way forward. We could add the rpm to contribs, though adding the source then requires Go to be added as well to compile it.

Have to have a think about it.

Quote
I have a small problem finding the right names for variables etc. The script is, of course, a backup-script. But in rclone-terms, what I do is a cloning with a "backup-dir", containing the previous versions of changed files.

So can you describe where you are getting stuck?
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline prophetSe7en

  • 3
  • +0/-0
Re: Using rclone for making back-ups
« Reply #18 on: July 23, 2019, 11:51:12 PM »
Hi

This script looks amazing. I do have some questions I hope you can answer

If I understand it correctly I add paths I want to backup to filter.txt and then only those folders will be included in the backup. Is this correct? I tried adding a path to a folder in the filter.txt during a dry-run, and from what I can tell it tries to backup my entire server and not just the path I added to filter.txt

Edit: Found my error. Was missing the
Code: [Select]
- * in the filter file  ;)

Code: [Select]
FILTER="/boot/logs/filter.txt"
Code: [Select]
+ /mnt/user/various/BackupSource/**
« Last Edit: July 23, 2019, 11:58:35 PM by prophetSe7en »

Offline prophetSe7en

  • 3
  • +0/-0
Re: Using rclone for making back-ups
« Reply #19 on: July 25, 2019, 07:14:31 PM »
So I almost got this working. Only issue is that the bak.list dont rotate, so it put every "old backup" into old1 every time.

Code: [Select]
old1
old2
old3
old4
old5
old1 Thu Jul 25 14:50:47 CEST 2019
old1 Thu Jul 25 14:52:15 CEST 2019
old1 Thu Jul 25 14:53:53 CEST 2019
old1 Thu Jul 25 14:55:17 CEST 2019
old1 Thu Jul 25 15:04:05 CEST 2019

Using this code to rotate bak.list

Code: [Select]
OLDEST=$(head -1 $BACKUPLIST | sed 's/ .*//')
COUNT=$(cat $BACKUPLIST | wc -l)                                           
((COUNT=$COUNT-1))
tail -$COUNT $BACKUPLIST > $BACKUPLIST.tmp
echo $OLDEST $(date) >> $BACKUPLIST.tmp 
« Last Edit: July 25, 2019, 07:16:30 PM by prophetSe7en »

Offline holck

  • ****
  • 317
  • +1/-0
Re: Using rclone for making back-ups
« Reply #20 on: July 26, 2019, 08:34:13 AM »
You seem to be missing this line:
Code: [Select]
mv -f $BACKUPLIST.tmp $BACKUPLIST
In the script, I make a new version of bak.list in bak.list.tmp, then at the end I overwrite bak.list with bak.list.tmp.

Thanks for your comments and for testing the script :-)
......

Offline prophetSe7en

  • 3
  • +0/-0
Re: Using rclone for making back-ups
« Reply #21 on: July 26, 2019, 12:48:15 PM »
You seem to be missing this line:
Code: [Select]
mv -f $BACKUPLIST.tmp $BACKUPLIST
In the script, I make a new version of bak.list in bak.list.tmp, then at the end I overwrite bak.list with bak.list.tmp.

Thanks for your comments and for testing the script :-)

No I had that line, thats why it wrote old1 at bottom on each backup. Not sure what I was missing for it to delete line 1 in the script, but it is working now after i started over.

Do you still use the script or have you changed anything to it?

Offline holck

  • ****
  • 317
  • +1/-0
Re: Using rclone for making back-ups
« Reply #22 on: July 27, 2019, 09:31:58 AM »
So far, I haven't changed anything in the script. It runs very smoothly, and a back-up i performed every night. A nice thing for me is that if I accidentally deleted a single file or directory, AWS's web-interface allows be to easily recover it. No need for complex arguments to dar or tar ... And it's good to know, that I will still have back-up available, even if the closet where I keep my server, should get damaged.

I hope you will start using the script and report your experiences.

/Jesper H, Denmark 
......