Koozali.org: home of the SME Server

[UPDATE 7.2] e-smith-backup with dar

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #45 on: September 06, 2007, 06:15:25 PM »
jpl

I wonder if you could advise on the following situation please, ie how best to resolve it.

I successfully did a full backup of approx 95GB of data (87 file parts) to a removable 120Gb drive on a workstation. I then did two more daily incremental backups OK (4 parts & 1 part).
For the next three nights the backup has terminated with exit code 11, and messages saying that user mail folders were changed during reading.
WARNING! File modified while reading it for backup

The incomplete backup files (4 parts) are left in the tmp folder on the backup drive each night, and these get overwritten the next night, only to be left there again when the backup terminates again the following night.

Reading dar literature this appears to be correct behaviour, as the file being read is in an unknown state, but it certainly does not achieve the objective of completing the backup.

There are users email accounts that are automatically receiving & sending messages, so it's possible that email folders could change at any time.
At this rate I don't know when the system will complete an incremental backup again.

Any advice/comments/suggestion, thanks.

Emailed reports below:
No terminal found for user interaction. All questions will be assumed a negative answer (less destructive choice), which most of the time will abort the program.
WARNING! File modified while reading it for backup: /home/e-smith/files/users/xxxxxx/home/email/xxx yyy - Inbox.dbx


 --------------------------------------------
 9422 inode(s) saved
 with 75 hard link(s) recorded
 1 inode(s) changed at the moment of the backup
 628563 inode(s) not saved (no inode/file change)
 0 inode(s) failed to save (filesystem error)
 254 inode(s) ignored (excluded by filters)
 2445 inode(s) recorded as deleted from reference backup
 --------------------------------------------
 Total number of inode considered: 640685
 --------------------------------------------
Dar error during backup : 11
Backup terminated: backup failed - status: 7424

==================================
DAILY BACKUP TO WORKSTATION REPORT
==================================
Backup started at Thu Sep  6 23:55:14 2007
Backup of mysql databases has been done.
Mounting backup shared directory stationxx/backup
Backup temp directory /mnt/smb/tmp_dir/server.xxxxxxxx.xxx.xx is mounted and is writable
Using set number 0 of 1
Attempt to incremental backup number 3 of 7
Backup base file name is inc-003-20070906
Making backup on temporary dir...
using a backup session timeout of : 17970 seconds
*** No backup allowed or error during backup ***
Dar error during backup : 11


Thanks
...

Offline Ptah

  • 3
  • +0/-0
  • so long and thanks for all the disk space...
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #46 on: September 07, 2007, 11:04:02 AM »
Geez, excuse me as I am very new to SME server.

I get what you saying about it not being in the server manager. Without having someone laugh at me now but how do you access the existing Backup or restore panel. As far as my knowledge stretch(erm..) there is two management gui's. One from the ssh-client with login as admin and the other is the webfront server management page. Am I missing something here??

 :?
...light travels faster than sound. That's why some people appear bright until you hear them speak.
:)

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #47 on: September 07, 2007, 04:05:43 PM »
Ptah

I think you really need to read all of the manuals from start to finish, linked at the top of the forum as your knowledge and understanding seems weak.

I'm referring to what you call the "webfront server management page", which is commonly (and always as far as I know) called server manager, with a URL https://www.yourdomain.com/server-manager
and which by default is only accessible from your local network unless you enable remote access & enter your remote host IP.

Open server manager and click on the heading for Backup or restore, then select one of the options from the pull down menu (which should now have extra options if you have installed the correct e-smith-backup rpm).

With that knowledge please go and read the earlier posts and the wiki entry again, perhaps they will make more sense to you now.
...

Offline jester

  • *
  • 496
  • +1/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #48 on: September 07, 2007, 05:41:26 PM »
JPL,
an actionscript to mount and unmount alone will not do the trick. If there is no tape in the drive (or in case of an usbdisk not being present) the mount will fail, so then the backup has to be canceled and a warning mail be sent that there was no tape in the drive. I don't think this can and i wouldn't know how this can be accomplished by an action-script.

I know the Backup2 contrib had a mount/unmount option (and if i remember correctly even an eject option) and it worked for my REV-drive. If it still works on SME7.2 this might be the better option for usb-REV-drives/usb-disks (although it is not maintained anymore) for now.

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #49 on: September 09, 2007, 09:51:36 PM »
Hi jpl

You  wrote to another poster:

I appreciate your contribution as it is exactly what I am looking for.  However I also find that it is not quite ready for use for backups to USB. ...


Hi Jim,

Thanks for your comments. I agree my job needs improvement for usb disk backup. Here are some further details.

When usb type backup is used, no mount or umount is made. Dar backup files are written on the directory specified in the server-manager panel.
So if you specify /media/usbdisk in your panel and you connect an usk disk whitout mounting it, backup is made on your server's local /media/usbdisk directory.

Present script cares about a real mount for smbfs, cifs, and nfs, but not for usb.

Quote
I think that the /mnt/smb directory is used for the temporary files
No. Even temporary files are on the backup system.

So, what could be done ?

We cannot trust about the fact that the usb mount point is named /media/usbdisk. It can be another name. The admin must know this name and specify it in the server-manager.
But I think the problem of the real mount can be solved by adding a test of /proc/mounts, to verify ther is a real mount (not only for usb) in the backup script.
About the possibilty to do the mount, I think we should take in account the two situations : admin keeps his usb disk always mounted, or admin wants the usb disk is mounted before each backup, and unmounted after each backup.

If we want to provide the two possibilities, we have two solutions :
- keep things unchanged and resquire the admin to add an action which mounts the usb disk in the pre-backup event, and unmount it in the post-backup event (it could permit to handle specific situations)
- let the admin specify his choice with an additional option in the usb backup setting panel, and modify the scripts consequently.

The second way needs a little modifications of the backup script.

Comments and opinions ?

N.B. I see no reason about the failure of the second backup. After the mount, script is the same for usb and  other types of backup support.

« Last Edit: September 09, 2007, 09:53:23 PM by jpl »

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #50 on: September 09, 2007, 10:47:50 PM »
jpl

I wonder if you could advise on the following situation please, ie how best to resolve it.

...

Ray,

I think there is nothing else to do... than keeping an eye on the backups, to see if this error happens again for several days and should prevent the backup to succeed.

I think there is no way to accept in a valuable backup a file which changes during the backup. dar choice to cleanly stop the backup and preserve only safely backuped files is the way to go. But the documentation is not clear about what can cause an error 11 and the way dar detects a file change.

If dar detects a change between the moment it reads the beginning of file, and the moment it reads it's end, there is little chance this happens in two consecutive backups... But if dar consider files which are changed beetween the beginning and the end of the backup job, things are more problematic. In such case, perhaps should we consider the possibility to relaunch immediatly a backup ended with an error 11. But is it a safe way ?


Offline jester

  • *
  • 496
  • +1/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #51 on: September 11, 2007, 11:44:11 AM »
JPL,
An option in the server-manager panel to mount/unmount would be fine for me but maintaining the 'Keep it simple' credo of SME Server in mind the most elegant solution would be to have some logic in the background doing this automagically.

Besides the mount/unmount issue i would still be interested in being able to eject (in my usb REV-drive case) after the unmount/backup to not by accident overwrite a (full) backup. An option for this would be very much appreciated.

Regards.

Offline nowhinjing

  • *
  • 32
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #52 on: September 11, 2007, 03:50:07 PM »
Hi,

I seem to have a severe problem with this.

My old server has given up the ghost, no problem I thought since I have taken daily backups on to my FreeNAS box, I can just restore it !

So I have built a new server, installed a fresh version of 7.2, configured the remote access and local networks to allow server manager, and installed the two required rpm via yum localinstall.

I then configured the network backup to point to the share on the FreeNAS box and started the restore on the latest (27mb) backup.

Lots of lovely messages telling me that files are restored, followed by

Code: [Select]
--------------------------------------------
9497 inode(s) restored
0 inode(s) not restored (not saved in archive)
0 inode(s) ignored (excluded by filters)
0 inode(s) less recent than the one on filesystem
0 inode(s) failed to restore (filesystem error)
0 inode(s) deleted
--------------------------------------------
Total number of inode considered: 9497
--------------------------------------------
Restore complete

You must reboot the server to activate any configuration changes that were made as a result of this restore.

Rebooted, went back to look at the server and NOTHING has been restored - no ibays and no files in them.

Thinking I must have done something wrong, I have repeated the install but gone to look at the file structure using WinSCP before the reboot. Same result, no restored files, NADA !

Tried to look at var/log/messages - not initialised.

Please, Please, what do I do next - I really do not fancy rebuilding 5 sites from scratch.

NWJ
...

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #53 on: September 11, 2007, 09:08:09 PM »
Dar saying "restore complete" and doing no restore sounds strange to me.

Strange to me also is the last message about the reboot. I didn't remember about such a message from dar, and the restore action beeing made through the server-manager I thought that the reboot action was requested by the sever-manager panel. But perhaps I forgot this message.

I think you should first verify your backup.

Two actions are available to verify your backup :
- the first one is not yet possible in your situation : use workstation backup verify option from theserver-manager panel,
- the second is always possible : you can use selective restore panel to verify if the backup really contains the files you need (it must show them), and test if restore of one of them via selective restore is possible.

Please report the result, you must first be sure of the content of your backup.

Offline nowhinjing

  • *
  • 32
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #54 on: September 12, 2007, 10:59:04 AM »
Hi, JPL,

Thanks for your response.

I have done as you suggested. The files in the backup appear to be OK. I have restored small sections with no problems, but if I select a large number it just says "Restore Complete" without listing or indeed restoring any files.

There seems to be an issue with the number of files to be restored at one time, so I gradually increased the number that I was trying to restore until I got the message "Error while calling execvp:Argument list too long".

I hope this gives you a handle on the problem.

NWJ
...

Offline Jim Holland

  • 2
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #55 on: September 12, 2007, 03:50:18 PM »
Hi jpl

You wrote:

No. Even temporary files are on the backup system.

I have 140 GB to backup, which takes around 18 hours for a full backup.  I see the temporary files appearing in /mnt/smb, which is on the source disk.  Only when complete are they moved to the target USB drive directory of /media/usbdisk.  That is ideal behaviour for me as it means that the maximum utilisation of the USB drive can be achieved.

About the possibilty to do the mount, I think we should take in account the two situations : admin keeps his usb disk always mounted, or admin wants the usb disk is mounted before each backup, and unmounted after each backup.

If we want to provide the two possibilities, we have two solutions :
- keep things unchanged and resquire the admin to add an action which mounts the usb disk in the pre-backup event, and unmount it in the post-backup event (it could permit to handle specific situations)
- let the admin specify his choice with an additional option in the usb backup setting panel, and modify the scripts consequently.

The second way needs a little modifications of the backup script.

Comments and opinions ?

I would be happy to have only the option of automatic mounting followed by auto unmounting at the end of the backup, although at present I leave it mounted all the time.  We don't need too many options.

I am glad to report that subsequent attempts have achieved one successful full backup and one incremental backup.  Now for a test restore on a spare machine!

Regards

Jim

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #56 on: September 13, 2007, 09:16:04 PM »
Hi jpl
 I see the temporary files appearing in /mnt/smb, which is on the source disk.

It's an error in my scripts, which I will look for. The temp backup should be on the backup host, not on the local server, even with usb disk. This is a security reason : if you program a daily full backup (probably the most frequent use of backup) and the transfer to the backup workstation or usb disk fails, you have no more available backup. But when the tmp backup is made on the backup workstation, if it fails the preceding backup is always available.

Yes, you need more space on the backup disk, but in the other solution you need more space on the server disk AND the backup process is far less reliable.

Quote
I would be happy to have only the option of automatic mounting followed by auto unmounting at the end of the backup, although at present I leave it mounted all the time.  We don't need too many options.

I also think it's the way to go, and I will do more tests in this direction.

Offline ianbsuki

  • 1
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #57 on: October 01, 2007, 06:10:33 PM »
Hi JPL,
On a Restore from Workstation:
I also have "nowhinjing" 's problem, and this message in server-manager :-
After the restore completes you must reboot the server. Your restore is complete when /etc/smbpasswd has been restored and the words "Restore complete" appear at the bottom of your screen.
 
The backup files are being restored to eg.
Restoring file: /etc/e-smith/web/panels/manager/cgi-bin/sys

At the end of the restore, the message:-
Restore complete

You must reboot the server to activate any configuration changes that were made as a result of this restore.
 
The directory structure is in:- /etc/e-smith/web/panels/manager/cgi-bin/
But, only the /home and /root directory's have any files in them.
These files are not being moved to their correct locations after the reboot

With a selective restore:-
Files are being restored, I have phpmyadmin and eventum in /opt/
these are not being backed-up, they are not in the "40go-into" template.

Any info on how to get a full backup and restore, would be welcome.
TIA


Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #58 on: October 04, 2007, 09:39:52 PM »

The directory structure is in:- /etc/e-smith/web/panels/manager/cgi-bin/
But, only the /home and /root directory's have any files in them.
These files are not being moved to their correct locations after the reboot


Default backuped files and directories are the same as with tape or desktop restore.
/opt directory is not saved by default.
But hopefully e-smith-backup wit dar is templated, so you can add directories and files you want to backup/restore.

Copy /etc/e-smith/templates/etc/dar/Dailybackup.dcf/40go-into to /etc/e-smith/templates-custom/etc/dar/Dailybackup.dcf/40go-into and modify /etc/e-smith/templates-custom/etc/dar/Dailybackup.dcf/40go-into to add needed files and directories.

Then expand the template with :
/sbin/e-smith/expand-template etc/dar/Dailybackup.dcf

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #59 on: October 05, 2007, 03:50:29 AM »
jpl

Just a minor correction, in case it was not obvious to readers
Dailybackup.dcf should read DailyBackup.dcf

Here's a corrected & slightly modified procedure:

mkdir -p /etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf

cp /etc/e-smith/templates/etc/dar/DailyBackup.dcf/40go-into /etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf
(the above is all on one line)

to add needed files and directories modify
/etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf/40go-into

Then expand the template with :
expand-template /etc/dar/DailyBackup.dcf
...