Koozali.org: home of the SME Server

Potential Contrib? A new system for FULL Backups?

Offline Elliott

  • ***
  • 150
  • +0/-0
Potential Contrib? A new system for FULL Backups?
« on: February 02, 2007, 03:28:12 PM »
Since the Experienced Users Forum is locked down I figure this is the place to ask.

I'm trying to get a good backup routine in place to make sure that my SME mail server is covered in my offsite backup strategy. I manually ran these steps last night but of course I can't chance a restore routine on the production machine. I'll test it on a test machine once one comes free.

In the meantime, can anyone with a head for this stuff give me a general yay or nay on the process that I've chosen?

I'm using a combination of tar/bzip to compress and backup the system and then rsync to push it to my backup server. The code is below:

Code: [Select]
BACKUP
------
tar cvpjf backup.tar.bz2 --exclude=/proc --exclude=/lost+found \
--exclude=/backup.tar.bz2 --exclude=/mnt --exclude=/sys \
--exclude=/dev /

RSYNC to NETDUMP
----------------
rsync -a backup.tar.bz2 test@10.11.12.3::sme/mail/

RSYNC from NETDUMP
----------------
rsync -a test@10.11.12.3::sme/mail/backup.tar.bz2 /

RESTORE
-------
tar xvpfj backup.tar.bz2 -C


I'm thinking of adding the backup and rsync out options to my crontab as soon as I figure out how to automate the password phase. The tar/bzip command took about 3 hours to complete last night on a mail system with about 15 users and the zipped file was about 1.8GB

I'm trying this routine as I don't have a tape solution and the built-in backup is already reporting that the system is too large. Also, none of the contribs (that aren't in a testing phase) seem to be able to do what I want which is a full system backup that I can then push out to my rsync server. I'm also thinking that I might just rsync the entire system to a local /backup directory and then try tar/bzipping that across a CIFS/Samba share to the backup server.

My main concerns are getting all of the files necessary to restore the system on the same hardware and this obviously includes keeping permissions. So can anyone verify that this backup could be restored after a clean install of SME was performed to bring all of my user accounts and their email back to the last backup time?

Thanks.
-E
Elliott

Offline Elliott

  • ***
  • 150
  • +0/-0
Potential Contrib? A new system for FULL Backups?
« Reply #1 on: February 02, 2007, 03:31:02 PM »
Actually I guess it makes more sense to rsync the files straight to the rsync server so that I'm sure about the benefits of rsync getting used. I'm thinking I'll follow the basic guidelines outlined in the well known Automated Snapshot-style Backups with Linux and Rsync webpage.

So can anyone point out what directories I can safely ignore from my backups if I am wanting to be able to restore to the same hardware? What I mean is that I've got several identically configured HP rack servers so what directories can I safely ignore (e.g. /dev) and still be able to setup SME on another machine and rsync my files back to it and reboot to get the users and email back?

I guess I can do the whole thing but if it's not necessary and can save space I think I should exclude some directories. And since none of the contribs I've reviewed seem to handle this task as thouroughly perhaps someone can point me in the direction of how to create a contrib? I don't know if I have the technical aptitude to do such a thing but I'm willing to try.

-E
Elliott

Offline byte

  • *
  • 2,183
  • +2/-0
Potential Contrib? A new system for FULL Backups?
« Reply #2 on: February 02, 2007, 08:18:22 PM »
Moving this topic to the SME 7.x contribs forum, it is more appropriate there. Thanks!
--[byte]--

Have you filled in a Bug Report over @ http://bugs.contribs.org ? Please don't wait to be told this way you help us to help you/others - Thanks!

Offline Elliott

  • ***
  • 150
  • +0/-0
Potential Contrib? A new system for FULL Backups?
« Reply #3 on: February 03, 2007, 06:11:01 AM »
And after stumbling onto a few more threads I'm seeing that getting a clean dump of the mysql database is going to be necessary but I don't see how to automate that spelled out clearly anywhere.

Normally I'd try to install webmin to help with some of the automation tasks but I'm thinking that that wouldn't make much sense here with all of the template hooplah.

I get the feeling that I'm digging myself in deeper....  :lol:
Elliott

Offline raem

  • *
  • 3,972
  • +4/-0
Potential Contrib? A new system for FULL Backups?
« Reply #4 on: February 03, 2007, 03:25:49 PM »
Elliott

This will dump the mysql dbs
signal-event pre-backup

You should examine the existing backup & restore scripts in sme7 to see how a backup is done and what files are included & what happens during a restore.
You should also install and have a play with the backup2 & backup dar contribs to see what they do.

I'd suggest tar'ing the folders/files mentioned in one of the posts below plus the mysql dump and then resync that file to your external server.

See these also
http://forums.contribs.org/index.php?topic=35429.0

http://forums.contribs.org/index.php?topic=34143.0

http://forums.contribs.org/index.php?topic=34187.0

http://forums.contribs.org/index.php?topic=33989.0

http://forums.contribs.org/index.php?topic=33844.0

http://forums.contribs.org/index.php?topic=33762.0
...

Offline pfloor

  • ****
  • 889
  • +1/-0
Potential Contrib? A new system for FULL Backups?
« Reply #5 on: February 04, 2007, 07:26:58 AM »
Quote from: "RayMitchell"
Elliott

This will dump the mysql dbs
signal-event pre-backup

And after your backup is complete don't forget to run:
signal-event post-backup

You might also consider rdiff-backup.  Gordon Rowell has put this into a couple of easy to use RPM's.  I use it daily to perform local (to USB disks) and remote backups.  Works very well and does incremintal backups and saves previous versions of every file (6 weeks by default).

However: CAUTION this is beta so use at your own risk!

http://mirror.contribs.org/smeserver/releases/7/smedev/i386/RPMS/smeserver-rdiff-backup-0.0.1-19.el4.sme.noarch.rpm

http://mirror.contribs.org/smeserver/releases/7/smedev/i386/RPMS/rdiff-backup-1.1.7-2.el4.sme.i386.rpm

Edit: You should use yum to install.

# yum --enablerepo=smedev install *rdiff-backup


And some notes to go with it:

# sudo /sbin/e-smith/config show rdiff-backup
# rdiff-backup=service
    EnableLocal=yes
    EnableRemote=no
    Hour=2
    LocalPath=/media/usbdisk/rdiff-backup
    Minute=8
    RemoteHost=timtam
    RemoteIdentity=/root/.ssh/id_rsa_rdiff-backup
    RemotePath=/var/backup/rdiff-backup
    RemotePort=22
    RemoteUser=rdiff-backup
    RemoveOlderThan=6W
    status=enabled

In the configuration above, my local backup is enabled, and backs up to
/media/usbdisk1/rdiff-backup/$sysconfig{SystemID}/

The remote backup is currently disabled, but backups up to a host called
'timtam' using the SSH key shown above, with target path of
/var/backup/rdiff-backup/$sysconfig{SystemID}/

The remote backup is done as user 'rdiff-backup' remotely (root locally).

To enable LOCAL backups:

- mount the media you want to use for backups, create the directory on the
mounted device for the backup (this must be <target directory>/rdiff-backup)
and then make sure /media/usbdisk/rdiff-backup is owned by rdiff-backup.

# mount /media/usbdisk (adjust mount point according to your system setup).
# mkdir /media/usbdisk/rdiff-backup
# chown rdif-backup:rdiff-backup /media/usbdisk/rdiff-backup


Note: Currently /media/xxxx devices are not automounted by default.
You must mount the intended media after each bootup or the backup won't work.
This mounting process may be automated in the future

- Set your configuration:

# config setprop rdiff-backup LocalPath <whatever>/rdiff-backup
# config setprop rdiff-backup EnableRemote no
# config setprop rdiff-backup EnableLocal yes
# config setprop rdiff-backup status enabled

# expand-template /etc/crontab

To enable REMOTE backups:

- install smeserver-rdiff-backup on both ends
- This will create a 'rdiff-backup' user

On the machine to be backed up:

# config setprop rdiff-backup RemoteHost <whatever>
# config setprop rdiff-backup RemotePath <whatever>/rdiff-backup
# config setprop rdiff-backup EnableRemote yes
# config setprop rdiff-backup EnableLocal no
# config setprop rdiff-backup status enabled

# expand-template /etc/crontab

# ssh-keygen -t rsa -f /root/.ssh/id_rsa_rdiff-backup -N ''
# expand-template /root/.ssh/config

On the backup server:

- Create OR Append /root/.ssh/id_rsa_rdiff-backup.pub to
~rdiff-backup/.ssh/authorized_keys2

Note: Permissions should also be 0400 rdiff-backup:rdiff-backup
Note2: If both ends are trusted then you can put the ssh in root's .ssh directory and run the remote as root.  This will mantain proper file ownership.  Otherwise all the backup files are saved as the user rdiff-backup.

# usermod -p '*' rdiff-backup          # This enables SSH only

Note: You can enable both Local and Remote backups at the same time.  rdiff-backup
will perform the local backup first then the remote.

Note: Always use a subdirectory 'rdiff-backup'. If the backup disk
isn't mounted, this saves you from filling the parent filesystem.
In life, you must either "Push, Pull or Get out of the way!"

duncan

Potential Contrib? A new system for FULL Backups?
« Reply #6 on: February 04, 2007, 08:34:48 AM »
Just out of curiosity - why not just use the DAR package.

It`s integrated into the server-manager, you can choose a number of backup destinations and any directory you wish. It will also handle compression and restoration.

EDIT - Ok - offsite backups. I did an rsnapshot contrib a while back that is pretty good for automated rsync backups.

Offline Elliott

  • ***
  • 150
  • +0/-0
Potential Contrib? A new system for FULL Backups?
« Reply #7 on: February 05, 2007, 07:25:42 PM »
Quote from: "duncan"
I did an rsnapshot contrib a while back that is pretty good for automated rsync backups.


Where might I find this?
Elliott

Offline Elliott

  • ***
  • 150
  • +0/-0
Potential Contrib? A new system for FULL Backups?
« Reply #8 on: February 05, 2007, 07:31:26 PM »
Quote from: "pfloor"
You might also consider rdiff-backup.  


The problem here is that my rsync server is a proprietary box that simply accepts rsync connections. I can't really install any software on it. I can set up users and rsync to it from SME (tested) but hacking it isn't in the cards for me right now as I have only one and no budget for another one to toy with.

Quote from: "pfloor"
Quote from: "RayMitchell"

This will dump the mysql dbs
signal-event pre-backup

And after your backup is complete don't forget to run:
signal-event post-backup



So if I simply include the dump and delete commands in the appropriate places in my script I should be able to bypass the calls to pre- and post- backup, correct?

I'm still looking into what exactly gets backed up in the other contribs so I know what to include and exclude. Actually can anyone point me to the scripts for the built-in backup? I thought they'd be in the events directory tree but none of the names other than conf-backup jumped out at me and there's nothing worthwhile in that tree.

Thanks,
E
Elliott

duncan

Potential Contrib? A new system for FULL Backups?
« Reply #9 on: February 05, 2007, 10:31:43 PM »

Offline pfloor

  • ****
  • 889
  • +1/-0
Potential Contrib? A new system for FULL Backups?
« Reply #10 on: February 05, 2007, 10:37:54 PM »
By default these are backet up:

/home/e-smith
/etc/e-smith/templates-custom
/etc/e-smith/templates-user-custom
/etc/ssh
/root
/etc/sudoers
/etc/passwd
/etc/shadow
/etc/group
/etc/gshadow
/etc/samba/secrets.tdb
/etc/samba/smbpasswd
/etc/smbpasswd
In life, you must either "Push, Pull or Get out of the way!"

Offline Elliott

  • ***
  • 150
  • +0/-0
Potential Contrib? A new system for FULL Backups?
« Reply #11 on: February 05, 2007, 10:42:38 PM »
Quote from: "pfloor"
By default these are backet up:

/home/e-smith
/etc/e-smith/templates-custom
/etc/e-smith/templates-user-custom
/etc/ssh
/root
/etc/sudoers
/etc/passwd
/etc/shadow
/etc/group
/etc/gshadow
/etc/samba/secrets.tdb
/etc/samba/smbpasswd
/etc/smbpasswd


Thank you, thank you, thank you! So are these all that are required (along with the mysql tables that would need to be dumped as listed below) to give me the ability to restore?

In plainer (:D ) English, if I pull the appropriate code from the singnal-event pre/post-backup scripts and then back the directories you listed along with the dumped mysql tables, should I be set for a full restore of my system?

I obviously will test this I'm just wondering if anyone can tell me right off if I'd be missing something?

Thanks!
-E
Elliott

Offline pfloor

  • ****
  • 889
  • +1/-0
Potential Contrib? A new system for FULL Backups?
« Reply #12 on: February 05, 2007, 10:50:29 PM »
Just put:

signal-event pre-backup
<your backup routine>
signal-event post-backup

in your script and the events will do what you need.
In life, you must either "Push, Pull or Get out of the way!"

Offline raem

  • *
  • 3,972
  • +4/-0
Potential Contrib? A new system for FULL Backups?
« Reply #13 on: February 06, 2007, 07:36:53 AM »
Elliott

> So are these all that are required (along with the mysql tables that would need to be dumped....)

You might also include
/opt
some contribs install stuff there.
This also needs to be added to the backup dar jobs too.
...

Offline Elliott

  • ***
  • 150
  • +0/-0
Potential Contrib? A new system for FULL Backups?
« Reply #14 on: February 08, 2007, 05:53:50 PM »
Should these errors worry me?

Code: [Select]
/bin/tar: Removing leading `/' from member names
/bin/tar: Removing leading `/' from hard link targets
Elliott