Koozali.org: home of the SME Server

[UPDATE 7.2] e-smith-backup with dar

Offline jpl

  • *
  • 112
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« on: July 25, 2007, 03:56:38 PM »
What is e-smith-backup with dar ?

e-smith-backup with dar is an upgrade of current e-smith-backup sme core rpm which provides panels for daily full or incremental backup on LAN workstation (via nfs, smbfs or cifs) or local usb disk, and full or selective restore  with use of dar program.

What provides e-smith-backup with dar ?

e-smith-backup with dar completes core e-smith-backup with four panels :
    - configure daily full or incremental backup to smbfs/cifs/nfs share on a LAN workstation or on local usb disk
    - verify backup on LAN workstation or local usb disk
    - restore full backup from LAN workstation or local usb disk
    - selective restore of files or directories from LAN workstation or local usb disk

New backup options are provided with additional perl functions included in backup panel script, a new perl backup script /etc/e-smith/events/actions/workstation-backup-dar, and a new backupwk record in configuration db. All are written with the same logic as present backup scripts.

What are specific problems of SME Server LAN backups managed by e-smith with dar ?

With experience of more than four years backuping sme servers with dar on LAN workstations (in production environments) the most important of encountered problems is that, when the backup session is started, you cannot rely upon a perfect working of the LAN or the distant workstation.
Inevitably, one day the LAN is too busy and/or the connection to the distant share hacks. After the failure sme may keep zombies backup or mount processes, and with some contribs you can be left with incomplete or buggy backup until hopped next good daily backup. These problems are problems of real world with LAN, that backup scripts must take into account.
Another common problem with backup through a LAN is that your lan disponibility for backup flow is not 24/24, but more probably something like "only from 20h to 6h, except on sunday where nobody works on the LAN..." So in case of large volume of data to backup, beeing able to backup all the datas during the week-end and doing only incremental backup on the week days really makes sense.
All these problems are considered with e-smith-backup with dar.
Scripts verify the absence of old share mounts, and alert admin about such situation. Test is made that the distant  backup share is available and writable. Of course with daily backup a maximum backup session time of only 24 h is available, but if this limit occurs e-smith-backup with dar cleanly stops the backup session  (maybe all is ok and the backup has not enough time to complete), the lan share is unmounted, and daily process can restart anyway. If you have enabled incremental backup and the last backup (full or incremental) was not completed because of a timeout, the backup can then safely restart and complete in an incremental way.
The admin can also set a timeout limit which keeps the session inside a specified number of hours, excluding traffic of LAN backup sessions when users work on the LAN.
Admin user can also set backup configuration so that this time limit occurs only on incremental backup sets. Then if you have a lot of data and your full backup time exceeds time of LAN inactivity, you can always backup all your data each end of week and preserve LAN bandwith for users. If you  program a full backup on saturday night, you get 24 hours to do full backup and even if the full backup time exceeds 24 hours, the first incremental backup on sunday night will backup remaining data. You begin your week with full data saved, and incremental backup session time is limited to 6 or 8 hours during week days.
Full backup sessions can also be allowed on every day, or limited to a specific day of week.

Are other new functions provided ?

Yes. Aside use of session timeout, the first is incremental backup.
This means that you can backup and restore data for the period of time you want : one day, three days, one week, one month, 100 days... and restore your system at any state it was during this period of time. This probably has no utility to do full restore of the system as it was one month ago, but restoring a file lost by a user two or three weeks ago can be useful. And restoring a safe system more than one day old can be needed.

The second function is keeping more than one set of backup (a set is full backup data and all datas of next daily incremental backups) with automatic rotation. e.g. you can do only nightly full backups but keep three sets of backup for security reason  (as being able to restore the system as it was 72h ago).

The third function is selective restore of any backuped file or directory, exactly as it was for any of the backuped days in your sets. Not only you can restore a lost file at his last state, but also say : make restore of the most recent version of the file before this given date... Selective restore is not an easy thing to manage by hand, and providing this in a simple way with panels is useful. Dar permits to manage selective restore and e-smith-backup with dar panels tries to keep this function as simple as possible to use.

Where to find e-smith-backup with dar ?

Presently beta version of e-smith with dar can be uploaded at http://beta.free-eos.org/sme/RPMS/e-smith-backup-1.14.0-16eos1.el4.sme.noarch.rpm

SRPMS available at http://beta.free-eos.org/sme/RPMS/e-smith-backup-1.14.0-16eos1.el4.sme.src.rpm
Dar rpm is available at http://beta.free-eos.org/sme/RPMS/dar-2.3.2-1.i386.rpm or mirrored at http://mirror.contribs.org/smeserver/contribs/dmay/smeserver/7.x/dar-2.3.2-1.i386.rpm

I read about problems about smbfs or cifs support with sme, and nfs packages are not on my sme...

Before sme 7.2, smb and kernel versions of sme have bugs in smb mount and cifs support.
For smb mount bug, e-smith-backup with dar includes necessary work-around.
For cifs support, corrected 2.6.9-55 kernel is packaged with sme 7.2.
For nfs support, e-smith-backup doesn't need install of any nfs package on the sme server (not even portmap).

Preventing automatic upgrades from sme yum updates...
You can prevent sme yum upgrade system from upgrading e-smith-backup with dar with new e-smith-backup core version.
Just enter as root :
Code: [Select]

# echo e-smith-backup > /etc/e-smith/db/yum_repositories/defaults/smeupdates/Exclude
# /etc/e-smith/events/actions/initialize-default-databases
# /sbin/e-smith/expand-template /etc/yum.conf

Manual yum upgrade is always available.

Comments are welcomed.

/edit/
sme 7.2 version
Jul 26 : corrected rpms release and link

Offline Tib

  • *
  • 571
  • +0/-0
    • http://www.tibors.net
[UPDATE 7.2] e-smith-backup with dar
« Reply #1 on: July 26, 2007, 05:20:24 AM »
I was reluctant to update my server because prior breakages for this great contrib ... Now I can update all servers.

Great Job jpl ... I love your contrib.

Regards,

Tib

Offline jpl

  • *
  • 112
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #2 on: July 26, 2007, 10:22:24 AM »
Seems there where a problem while generating rpm.
Updated version available in a few moments...

Offline jpl

  • *
  • 112
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #3 on: July 26, 2007, 11:30:51 AM »
Corrected rpms are now available for download (release 16eos1).
Some scripts were no more included in first 16eos release, so only upgrades from e-smith-backup with dar older releases where OK, not upgrades from core e-smith-backup.
Should be now corrected. Apologies...

Offline Elliott

  • ***
  • 150
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #4 on: July 26, 2007, 04:42:02 PM »
I think you need to update your dar links... I found one at http://mirror.contribs.org/smeserver/contribs/dmay/smeserver/7.x/dar-2.3.2-1.i386.rpm

Also, are there any docs pages or pages more clearly explaining the options on the workstation backup page. I'm not sure I understand exactly what the # of sets and dailys in a set mean.

I *think* that by setting it to 4 sets with 7 dailys per set with Friday set as the only full day should give me 4 sets of 7 files with one large file per set bing the Friday full and the rest being daily incrementals.
Elliott

Offline jpl

  • *
  • 112
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #5 on: July 26, 2007, 05:55:20 PM »
Thanks for the dar link.

In fact, Darrel originally  made a copy of my rpm, and  I have not seen that he provides now an updated version  :)

You are right about sets.

Sorry for the doc, it has yet to be made on the wiki. But panels are almost (almost only I know) self explainatory.

Offline jpl

  • *
  • 112
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #6 on: July 26, 2007, 06:42:12 PM »
Quote from: "jpl"
Thanks for the dar link.

In fact, Darrel originally  made a copy of my rpm, and  I have not seen that he provides now an updated version  :)


Sorry again, it's not a new rpm, it's the one I provided to Darrel.

Just my links that are wrong, as you say. I correct them.

Offline Elliott

  • ***
  • 150
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #7 on: July 27, 2007, 03:34:06 PM »
Ok... so I got this:

Quote
==================================
DAILY BACKUP TO WORKSTATION REPORT
==================================
Backup started at Thu Jul 26 20:15:02 2007
Backup of mysql databases has been done.
Mounting backup shared directory netshare/backup
*** No backup allowed or error during backup ***
Error while mounting netshare:backup :
mount error: could not find target server. TCP name netshare/backup not found  rc = 134526947
No ip address specified and hostname not found


My settings are:
Quote
Backup is enabled.
Backup is made on LAN workstation netshare via cifs
Destination backup share folder is backup
Login is backup
Password is ********
Number of rotating backup sets is 4
Number of daily backups contained in each set is 7
Compression level (0-7) of backup is 4
Daily backup occurs at 20:15
Each daily backup session is cleanly timed out after 10h except full backups which are cleanly timed out after 24 hours
Full backup session (new backup sets) is allowed only on Friday


This SME Server is 10.11.12.14 on my LAN and "netshare" is 10.11.12.3 on the same segment. I changed the workstation name to 10.11.12.3 for tonights attempt at a backup. Can anyone suggest anything else or possibly a way to test it now without actually starting a backup since I've got users hitting it?

Thanks,
E
Elliott

Offline jpl

  • *
  • 112
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #8 on: July 27, 2007, 10:50:04 PM »
Not at home for eigth days, and don't have acess to server.

You can look at /etc/e-smith/events/actions/workstation-backup-dar script, find the command which connect your cifs target, and try to debug it directly as a shell command.

I think it is :

/bin/mount -t cifs $smbhost:$smbshare $mntdir -o user=$login,pass=$password

in your case :

/bin/mount -t cifs netshare:backup /mnt/smb -o user=backup,pass=backup

You can also look at your /var/log/messages log for some more information about your cifs mount.

Offline gerd

  • ***
  • 107
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #9 on: August 03, 2007, 04:43:55 PM »
..I must be blind today...
I have made a fresh install of the SME7.2 and installed the "dar-2.3.2-1.i386.rpm" contrib.
But is my understanding correct that for this contrib there is no user-interface displayed in the web-panel??

Offline Elliott

  • ***
  • 150
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #10 on: August 03, 2007, 04:48:41 PM »
Quote from: "gerd"
But is my understanding correct that for this contrib there is no user-interface displayed in the web-panel??


It adds more options to the standard backup panel.
Elliott

Offline gerd

  • ***
  • 107
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #11 on: August 03, 2007, 06:10:42 PM »
Elliot,
a stupid question: which contribs have to installed in order to see
the see the additional these options??? For me here nothing has changed, strange ?? (& I used signal-event post-upgrade // signal-event reboot)

thanx

gerd

Offline gerd

  • ***
  • 107
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #12 on: August 03, 2007, 06:10:59 PM »
Elliot,
a stupid question: which contribs have to installed in order to see
the see the additional these options??? For me here nothing has changed, strange ?? (& I used signal-event post-upgrade // signal-event reboot)

thanx

gerd

Offline Elliott

  • ***
  • 150
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #13 on: August 03, 2007, 06:31:12 PM »
Follow the directions at the start of this post. That's it.

Then go to the "Backup or restore" tab on the SME Server Console. That's it. The new options are in the drop down list on the bottom.
Elliott

Offline gerd

  • ***
  • 107
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #14 on: August 04, 2007, 06:11:37 PM »
...yep, indeed - I was blind....it works perfect.

thanx

gerd

Offline gerd

  • ***
  • 107
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #15 on: August 11, 2007, 01:25:44 PM »
...finally, it doesn't work "that perfect" for me:

- the usb drive is ext3 formatted.
- the usb drive is automatically recognized by the SME7.2 server (media/usbdisk1)
- when I launch the workstation-backup_setconf I select local usb drive
- then I start the backup, but the files are not written on the local usb drive as expected - they are written into the directory /mnt/smb/; and that means in fact the files are written on the internal harddisk array. And this is basically not what I intended to do....

Can anyone enlighten me about this behaviour? As I am a newbie in Linux
my guess is that I have made somewhere a mistake - but no idea where to start with my investigations??

Just to check I installed the contrib "smeserver-dar2-0.0.1-3dmay.noarch.rpm, for the local usb drive I entered media/usbdisk1 and the backup worked perfect.


thanx

gerd

Offline jpl

  • *
  • 112
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #16 on: August 11, 2007, 08:39:54 PM »
The first goal of e-smith-backup with dar is to backup the server on an Lan smffs, cifs or nfs workstation share. A bug is always possible, specialy on usb backup which is not frequently used.

Quote
then I start the backup, but the files are not written on the local usb drive as expected - they are written into the directory /mnt/smb/


e-smith-backup with dar always do backup to /mnt/smb.

The question is : is the usb drive mounted on /mnt/smb, and if not why ?

Can you make some tests about it

Offline jester

  • *
  • 496
  • +1/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #17 on: August 12, 2007, 11:11:05 AM »
JPL,

If i would want to remove/undo this contrib what would need to be done and would it revert back to the default SME Server installation so that upgrading could cause no problems with future releases of SME Server?


jester.

Offline jpl

  • *
  • 112
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #18 on: August 12, 2007, 01:09:12 PM »
Nothing special. e-smith-backup with dar does not modify configuration needed for core e-smith-backup.

If you upgrade with new core e-smith-backup version, all should be OK... Just verify no cron job left in /etc/cron.d.backupwk

If you want to restore actual core e-smith-backup, rpm -e e-smith-backup ,reinstall core e-smith-backup rpm, and post-upgrade event should do the trick.

Offline gerd

  • ***
  • 107
  • +0/-0
[UPDATE 7.2] e-smith-backup with dar
« Reply #19 on: August 12, 2007, 09:35:17 PM »
@jpl

Quote
The question is : is the usb drive mounted on /mnt/smb, and if not why ?

When I connect the USB drive to the SME7.2 server, the USB drived is automatically mounted under /media/usbdisk. I made the test with three
different SME7.2 servers - with the same result. :(  :(

Offline syscom

  • 14
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #20 on: August 21, 2007, 02:19:47 AM »
Hello !

I have update my SME to 7.2 and installed the rpm for DAR backup below but after restart, it stills doesnt appear in server-manager.

Any suggests ?

When I try to reinstall, it says "already installed"


Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #21 on: August 21, 2007, 01:11:27 PM »
syscom

Answered earlier in this thread

Q. But is my understanding correct that for this contrib there is no user-interface displayed in the web-panel??

A. It adds more options to the standard backup panel.
...

Offline syscom

  • 14
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #22 on: August 21, 2007, 11:50:15 PM »
syscom

Answered earlier in this thread

Q. But is my understanding correct that for this contrib there is no user-interface displayed in the web-panel??

A. It adds more options to the standard backup panel.

Sorry but it DOES NOT add more options to the standard backup panel.

:P

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #23 on: August 22, 2007, 04:42:23 AM »
gerd, syscom & anyone else

Quote
...I installed the contrib smeserver-dar2-0.0.1-3dmay.noarch.rpm...

To avoid confusion, that's a totally different contrib than the e-smith-backup that this thread is talking about. It's a standalone backup system that adds a seperate server manager panel.

This thread refers to e-smith-backup (with dar) rpm which is a replacement rpm for the standard e-smith-backup rpm, and it adds extra settings to the standard server manager backup panel.
Note that as this contrib uses dar to perform backups, then you need to install both
e-smith-backup-1.14.0-16eos1.el4.sme.noarch.rpm
and
dar-2.3.2-1.i386.rpm

Note also that doing a normal upgrade using the software installer panel or yum update, will most likely overwrite the special version of e-smith-backup that needs to be installed in order to have the extra functionality.
There are commands given earlier in this thread that you need to enter to prevent yum from updating the e-smith-backup rpm automatically (which would be undesirable as you will lose the added functionality). ie
 
echo e-smith-backup > /etc/e-smith/db/yum_repositories/defaults/smeupdates/Exclude
/etc/e-smith/events/actions/initialize-default-databases
/sbin/e-smith/expand-template /etc/yum.conf

The author of the e-smith-backup (with dar) rpm, will I assume continue to update the rpm and release a new version as and when any changes in the base sme server rpm occur. In that case users of e-smith-backup (with dar) rpm will need to manually update that particular rpm.

Note the rpm version (without dar support) that is installed on a standard sme7.2 server is
e-smith-backup-1.14.0-16.el4.sme.noarch.rpm

If you are having problems then check the rpm version actually installed
rpm -qa e-smith-backup
rpm -qa dar

To install e-smith-backup with dar
download the correct version rpms to a clean folder on your sme server and do
rpm -Uvh dar*.rpm
rpm -Uvh e-smith-backup*.rpm

Open server manager & select the Backup or restore panel
Extra choices should have been added to the pull down menu for Select an action
ie
Configure workstation backup
Verify workstation backup
Restore from workstation
Selective file restore from workstation

Select the required backup type and configure it etc.
« Last Edit: August 22, 2007, 04:47:53 AM by RayMitchell »
...

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #24 on: August 30, 2007, 11:48:55 AM »
jpl

I added this to the Wiki
http://wiki.contribs.org/Backup_with_dar

It probably needs some further clarification re configuring backup sets and the number of daily backups in the set & associated practical usage concepts.
...

Offline Karsa

  • 12
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #25 on: September 01, 2007, 11:47:55 AM »
If you are having problems then check the rpm version actually installed
rpm -qa e-smith-backup
rpm -qa dar

After checking I got this reply:
e-smith-backup-1.14.0-16.el4.sme and dar-2.3.2-1
And I did follow the wiki instructions.

The problem seems to be the order of installation. Yum install of dar reverts e-smith-backup to original specs :?

I only had to run » rpm -Uvh e-smith-backup*.rpm and it was up and running.
Thanks Ray.
« Last Edit: September 01, 2007, 11:58:44 AM by Karsa »

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #26 on: September 01, 2007, 03:10:14 PM »
Karsa

Perhaps yum localinstall does not overwrite the existing rpm of the same name.
I'll check that. For now I changed the install order in the Wiki, but that may not be sufficient.

I appreciate you said you followed the wiki, but please confirm that you did actually download the rpms with wget and do a yum localinstall. Thanks

...

Offline Karsa

  • 12
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #27 on: September 01, 2007, 04:08:42 PM »
I made a directory (backupwdar), downloaded the files (wget) and made a localinstall with yum.

Followed with signal-event post-upgrade // signal-event reboot and after reboot...
Code: [Select]
echo e-smith-backup > /etc/e-smith/db/yum_repositories/defaults/smeupdates/Exclude
/etc/e-smith/events/actions/initialize-default-databases
expand-template /etc/yum.conf

I'm now trying to setup a lan backup...

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #28 on: September 01, 2007, 04:17:35 PM »
Karsa

This is required, I fixed the Wiki
wget http://mirror.contribs.org/smeserver/contribs/dmay/smeserver/7.x/dar-2.3.2-1.i386.rpm
wget http://beta.free-eos.org/sme/RPMS/e-smith-backup-1.14.0-16eos1.el4.sme.noarch.rpm
yum localinstall dar*.rpm
yum localinstall e-smith-backup*.rpm

Thanks
...

Offline Karsa

  • 12
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #29 on: September 01, 2007, 08:49:37 PM »
Still stuck at "Configure workstation backup"  :-(

I'm using SME Server 7.2 and the clients are using XP Pro SP2.
Any help would be welcome... I'm lost looking at the connection types (cifs, nfs,...yes total nood).   :?

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #30 on: September 01, 2007, 09:44:07 PM »
I'm using SME Server 7.2 and the clients are using XP Pro SP2.
Any help would be welcome... I'm lost looking at the connection types (cifs, nfs,...yes total nood).   :?

smbfs or cifs should be OK.

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #31 on: September 02, 2007, 02:06:07 AM »
jpl

Quote
e-smith-backup with dar is an upgrade of current e-smith-backup sme core rpm which provides panels for daily full or incremental backup on LAN workstation (via nfs, smbfs or cifs) or local usb disk, and full or selective restore with use of dar program.

Good job, an impressive add on.

What is your longer term goal with this rpm ie is it still in beta stage, are you going to continue to support it with each sme update/release etc ?

Have you or do you plan to submit it to the sme developers for possible inclusion in the base code ?

Regards
...

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #32 on: September 02, 2007, 11:36:02 AM »
jpl

What is your longer term goal with this rpm ie is it still in beta stage, are you going to continue to support it with each sme update/release etc ?

Have you or do you plan to submit it to the sme developers for possible inclusion in the base code ?


I have done the first release of this package about one year ago, following creation of the very lively bug 1880 on the bugtrack. Code is stable for about nine months. I had in mind inclusion in the base code and tried to develop dar integration in respect of core e-smith-backup code. It seems to me that I have asked about the question of submiting it for inclusion (and how to submit it) but had no response.

I think the question of inclusion is still very topical, but I don't know the correct way to ask about it.

I know that releasing a modified core package is not satisfactory... but doing a contrib from my work before getting sme team point of view is not a better choice and I have not given up inclusion.

But if things don't go further (or if core team doesn't want an inclusion) I think the viable solution is making a contrib permitting use and update of unmodified core e-smith-backup. But it leads to add another specific panel to existing backup panel, and I don't like this idea. I think at least core team should modify backup panel to provide (and specify) the way to plug other backup solutions, but it is not so trivial.

Opinions and suggestions are welcomed...

Offline Karsa

  • 12
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #33 on: September 02, 2007, 03:26:52 PM »
smbfs or cifs should be OK.

Thanks, but something isn't working, I don't get any backup file...
Probably my shared directory because I get "ERR_NO_HOST_DIRteste.teste.local" when trying to verify the backup (that doesn't exist but I should be able to log to workstation.

My settings:
Backup is enabled.
Backup is made on LAN workstation 10.1.1.7 via cifs
Destination backup share folder is backup
Login is admin
Password is *******
Number of rotating backup sets is 2
Number of daily backups contained in each set is 7
Compression level (0-7) of backup is 5
Daily backup occurs at 22:30
Each daily backup session is cleanly timed out after 10h
Full backup session (new backup sets) is allowed only on Saturdays

My SME server IP's 10.1.1.6. Any ideas?

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #34 on: September 02, 2007, 04:36:18 PM »
"teste.teste.local" seems a strange UNC name. Are you sure it is resolved by your SME WINS ?
You should try to use the IP of your backup workstation in place of his workstation name.

Before, you could try to mount your cifs share directly from the shell as a debugging way.

Verify you have no mount left on /mnt/smb and enter :

# /bin/mount -t cifs smbhost/backup /mnt/smb -o username=admin,pass=*****

Use first IP of your workstation as smbhost, and if succeeds, then try UNC host name (it is possible than only ip is ok).

Same tests could be done with smbfs and command :

# /bin/mount -t smbfs //smbhost/backup /mnt/smb -o username=admin,password=*****,fmask=777,ip=ip_of_smbhost

Don't forget to umount /mnt/smb when mount succeeds.

Offline Karsa

  • 12
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #35 on: September 02, 2007, 05:02:09 PM »
Tried smbfs command and it didn't work, with cifs I managed to made the mount. But I still fail to get any backup file and I get the same error message.

"teste.teste.local" is the SME server name.  :-(

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #36 on: September 02, 2007, 10:04:42 PM »
Are you able to write and create directories on /mnt/smb after the cifs mount ?

If not perhaps admin has not appropriate rights on the share.
« Last Edit: September 02, 2007, 10:10:13 PM by jpl »

Offline Karsa

  • 12
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #37 on: September 02, 2007, 10:59:10 PM »
No problem with that, I made directories both in the client and the server  :???:

I must be missing some step... probably a obvious one :roll:

Offline jester

  • *
  • 496
  • +1/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #38 on: September 03, 2007, 04:28:17 PM »
JPL,
I'm working with an Iomega REV drive (USB) that uses cartridges, that need to get mounted before a backup and unmounted afterwards preferably even ejected after an backup if possible. Is there a possibility for this?

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #39 on: September 04, 2007, 08:16:52 AM »

I must be missing some step... probably a obvious one :roll:

Maybe...

I think the best way should be to trace and debug the perl script /etc/e-smith/events/actions/workstation-backup-dar whick does the effective backup. You could get some useful information about what is wrong.

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #40 on: September 04, 2007, 08:28:22 AM »
JPL,
I'm working with an Iomega REV drive (USB) that uses cartridges, that need to get mounted before a backup and unmounted afterwards preferably even ejected after an backup if possible. Is there a possibility for this?

Two events, launched by daily SME backup, are available to customize such things : pre-backup and post-backup events. I think you could add actions scripts to these events for your specific needs.

Look for example at /etc/e-smith/events/actions/S50rewind-tape, lauched by pre-backup event, for such a script which rewind tape if the backup is a tape backup.

Offline Jim Holland

  • 2
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #41 on: September 06, 2007, 06:14:05 AM »
Hi jpl

You  wrote to another poster:

The first goal of e-smith-backup with dar is to backup the server on an Lan smffs, cifs or nfs workstation share. A bug is always possible, specialy on usb backup which is not frequently used.

e-smith-backup with dar always do backup to /mnt/smb.

The question is : is the usb drive mounted on /mnt/smb, and if not why ?

Can you make some tests about it

I appreciate your contribution as it is exactly what I am looking for.  However I also find that it is not quite ready for use for backups to USB.  My understanding of SME Server is that when you connect a USB drive, the HAL daemon will automatically create the directory usbdisk in /media, but will not actually mount the drive in that directory (you have to do that manually).  The directory is removed when the USB drive is disconnected.  When configuring e-smith-backup with dar to backup to USB (using "Configure workstation backup") the software seems to get confused if the USB drive is connected but not mounted: it sees the usbdisk directory and allows you to select it, but when the backup is run it creates the backup in "/media/usbdisk " where a trailing space is added to the target directory.  That directory is of course on the local server and not on the USB drive.

If the USB drive is connected and manually mounted when configuring the backup then the first backup is successful, and is created in /media/usbdisk/hostname on the USB drive.  However the next backup attempt then fails, with error:

FATAL error, aborting operation
Badly formatted terminator, cannot extract catalogue location
Dar error during backup : 2
Backup terminated: backup failed - status: 7424

If the /media/usbdisk/hostname directory on the USB drive is renamed then successive backups work OK.

My conclusion is that there are two issues: the software needs to understand how to mount/unmount/test for mounting of the USB drive correctly, and it also seems to have a minor bug involving adding a space to the directory name in some situations.  The "Backup share" selection in the configuration screen is also a little misleading as it is a misnomer - when backup is to USB then it is a mount point.  However as the user does not have any options to change the default of usbdisk, it would be preferable to change the wording slightly and make this read-only, eg: "Mount point usbdisk".

I think that the /mnt/smb directory is used for the temporary files.  That means that the full capacity of the USB drive is available for storing the backup.

On my wish list would be a request for an option to restore files to a new location, eg in an i-bay - ideally with a picklist for the final directory.  This would give the user the option to check the files before doing a restore to original location.

Thanks again for this great contribution.


Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #42 on: September 06, 2007, 06:25:05 AM »
Jim Holland

Quote
FATAL error, aborting operation
Badly formatted terminator, cannot extract catalogue location
Dar error during backup : 2
Backup terminated: backup failed - status: 7424

The Dar exit code is:
2 - Error due to a hardware problem or a lack of memory.
...

Offline Ptah

  • 3
  • +0/-0
  • so long and thanks for all the disk space...
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #43 on: September 06, 2007, 04:43:47 PM »
Quote
Follow the directions at the start of this post. That's it.

Then go to the "Backup or restore" tab on the SME Server Console. That's it. The new options are in the drop down list on the bottom.

I did all of that, installed the rpm's and confirmed the installation and did the signal-event post-upgrade // signal-event reboot but still no extra options in the server management panel and I am running SME server 7.2..

Any ideas?
...light travels faster than sound. That's why some people appear bright until you hear them speak.
:)

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #44 on: September 06, 2007, 05:53:48 PM »
Ptah
Quote
...no extra options in the server management panel...

No extra panels are aded to server manager.
Extra options are added to the existing Backup or restore panel.
Read Reply #23 and
http://wiki.contribs.org/Backup_with_dar
...

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #45 on: September 06, 2007, 06:15:25 PM »
jpl

I wonder if you could advise on the following situation please, ie how best to resolve it.

I successfully did a full backup of approx 95GB of data (87 file parts) to a removable 120Gb drive on a workstation. I then did two more daily incremental backups OK (4 parts & 1 part).
For the next three nights the backup has terminated with exit code 11, and messages saying that user mail folders were changed during reading.
WARNING! File modified while reading it for backup

The incomplete backup files (4 parts) are left in the tmp folder on the backup drive each night, and these get overwritten the next night, only to be left there again when the backup terminates again the following night.

Reading dar literature this appears to be correct behaviour, as the file being read is in an unknown state, but it certainly does not achieve the objective of completing the backup.

There are users email accounts that are automatically receiving & sending messages, so it's possible that email folders could change at any time.
At this rate I don't know when the system will complete an incremental backup again.

Any advice/comments/suggestion, thanks.

Emailed reports below:
No terminal found for user interaction. All questions will be assumed a negative answer (less destructive choice), which most of the time will abort the program.
WARNING! File modified while reading it for backup: /home/e-smith/files/users/xxxxxx/home/email/xxx yyy - Inbox.dbx


 --------------------------------------------
 9422 inode(s) saved
 with 75 hard link(s) recorded
 1 inode(s) changed at the moment of the backup
 628563 inode(s) not saved (no inode/file change)
 0 inode(s) failed to save (filesystem error)
 254 inode(s) ignored (excluded by filters)
 2445 inode(s) recorded as deleted from reference backup
 --------------------------------------------
 Total number of inode considered: 640685
 --------------------------------------------
Dar error during backup : 11
Backup terminated: backup failed - status: 7424

==================================
DAILY BACKUP TO WORKSTATION REPORT
==================================
Backup started at Thu Sep  6 23:55:14 2007
Backup of mysql databases has been done.
Mounting backup shared directory stationxx/backup
Backup temp directory /mnt/smb/tmp_dir/server.xxxxxxxx.xxx.xx is mounted and is writable
Using set number 0 of 1
Attempt to incremental backup number 3 of 7
Backup base file name is inc-003-20070906
Making backup on temporary dir...
using a backup session timeout of : 17970 seconds
*** No backup allowed or error during backup ***
Dar error during backup : 11


Thanks
...

Offline Ptah

  • 3
  • +0/-0
  • so long and thanks for all the disk space...
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #46 on: September 07, 2007, 11:04:02 AM »
Geez, excuse me as I am very new to SME server.

I get what you saying about it not being in the server manager. Without having someone laugh at me now but how do you access the existing Backup or restore panel. As far as my knowledge stretch(erm..) there is two management gui's. One from the ssh-client with login as admin and the other is the webfront server management page. Am I missing something here??

 :?
...light travels faster than sound. That's why some people appear bright until you hear them speak.
:)

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #47 on: September 07, 2007, 04:05:43 PM »
Ptah

I think you really need to read all of the manuals from start to finish, linked at the top of the forum as your knowledge and understanding seems weak.

I'm referring to what you call the "webfront server management page", which is commonly (and always as far as I know) called server manager, with a URL https://www.yourdomain.com/server-manager
and which by default is only accessible from your local network unless you enable remote access & enter your remote host IP.

Open server manager and click on the heading for Backup or restore, then select one of the options from the pull down menu (which should now have extra options if you have installed the correct e-smith-backup rpm).

With that knowledge please go and read the earlier posts and the wiki entry again, perhaps they will make more sense to you now.
...

Offline jester

  • *
  • 496
  • +1/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #48 on: September 07, 2007, 05:41:26 PM »
JPL,
an actionscript to mount and unmount alone will not do the trick. If there is no tape in the drive (or in case of an usbdisk not being present) the mount will fail, so then the backup has to be canceled and a warning mail be sent that there was no tape in the drive. I don't think this can and i wouldn't know how this can be accomplished by an action-script.

I know the Backup2 contrib had a mount/unmount option (and if i remember correctly even an eject option) and it worked for my REV-drive. If it still works on SME7.2 this might be the better option for usb-REV-drives/usb-disks (although it is not maintained anymore) for now.

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #49 on: September 09, 2007, 09:51:36 PM »
Hi jpl

You  wrote to another poster:

I appreciate your contribution as it is exactly what I am looking for.  However I also find that it is not quite ready for use for backups to USB. ...


Hi Jim,

Thanks for your comments. I agree my job needs improvement for usb disk backup. Here are some further details.

When usb type backup is used, no mount or umount is made. Dar backup files are written on the directory specified in the server-manager panel.
So if you specify /media/usbdisk in your panel and you connect an usk disk whitout mounting it, backup is made on your server's local /media/usbdisk directory.

Present script cares about a real mount for smbfs, cifs, and nfs, but not for usb.

Quote
I think that the /mnt/smb directory is used for the temporary files
No. Even temporary files are on the backup system.

So, what could be done ?

We cannot trust about the fact that the usb mount point is named /media/usbdisk. It can be another name. The admin must know this name and specify it in the server-manager.
But I think the problem of the real mount can be solved by adding a test of /proc/mounts, to verify ther is a real mount (not only for usb) in the backup script.
About the possibilty to do the mount, I think we should take in account the two situations : admin keeps his usb disk always mounted, or admin wants the usb disk is mounted before each backup, and unmounted after each backup.

If we want to provide the two possibilities, we have two solutions :
- keep things unchanged and resquire the admin to add an action which mounts the usb disk in the pre-backup event, and unmount it in the post-backup event (it could permit to handle specific situations)
- let the admin specify his choice with an additional option in the usb backup setting panel, and modify the scripts consequently.

The second way needs a little modifications of the backup script.

Comments and opinions ?

N.B. I see no reason about the failure of the second backup. After the mount, script is the same for usb and  other types of backup support.

« Last Edit: September 09, 2007, 09:53:23 PM by jpl »

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #50 on: September 09, 2007, 10:47:50 PM »
jpl

I wonder if you could advise on the following situation please, ie how best to resolve it.

...

Ray,

I think there is nothing else to do... than keeping an eye on the backups, to see if this error happens again for several days and should prevent the backup to succeed.

I think there is no way to accept in a valuable backup a file which changes during the backup. dar choice to cleanly stop the backup and preserve only safely backuped files is the way to go. But the documentation is not clear about what can cause an error 11 and the way dar detects a file change.

If dar detects a change between the moment it reads the beginning of file, and the moment it reads it's end, there is little chance this happens in two consecutive backups... But if dar consider files which are changed beetween the beginning and the end of the backup job, things are more problematic. In such case, perhaps should we consider the possibility to relaunch immediatly a backup ended with an error 11. But is it a safe way ?


Offline jester

  • *
  • 496
  • +1/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #51 on: September 11, 2007, 11:44:11 AM »
JPL,
An option in the server-manager panel to mount/unmount would be fine for me but maintaining the 'Keep it simple' credo of SME Server in mind the most elegant solution would be to have some logic in the background doing this automagically.

Besides the mount/unmount issue i would still be interested in being able to eject (in my usb REV-drive case) after the unmount/backup to not by accident overwrite a (full) backup. An option for this would be very much appreciated.

Regards.

Offline nowhinjing

  • *
  • 32
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #52 on: September 11, 2007, 03:50:07 PM »
Hi,

I seem to have a severe problem with this.

My old server has given up the ghost, no problem I thought since I have taken daily backups on to my FreeNAS box, I can just restore it !

So I have built a new server, installed a fresh version of 7.2, configured the remote access and local networks to allow server manager, and installed the two required rpm via yum localinstall.

I then configured the network backup to point to the share on the FreeNAS box and started the restore on the latest (27mb) backup.

Lots of lovely messages telling me that files are restored, followed by

Code: [Select]
--------------------------------------------
9497 inode(s) restored
0 inode(s) not restored (not saved in archive)
0 inode(s) ignored (excluded by filters)
0 inode(s) less recent than the one on filesystem
0 inode(s) failed to restore (filesystem error)
0 inode(s) deleted
--------------------------------------------
Total number of inode considered: 9497
--------------------------------------------
Restore complete

You must reboot the server to activate any configuration changes that were made as a result of this restore.

Rebooted, went back to look at the server and NOTHING has been restored - no ibays and no files in them.

Thinking I must have done something wrong, I have repeated the install but gone to look at the file structure using WinSCP before the reboot. Same result, no restored files, NADA !

Tried to look at var/log/messages - not initialised.

Please, Please, what do I do next - I really do not fancy rebuilding 5 sites from scratch.

NWJ
...

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #53 on: September 11, 2007, 09:08:09 PM »
Dar saying "restore complete" and doing no restore sounds strange to me.

Strange to me also is the last message about the reboot. I didn't remember about such a message from dar, and the restore action beeing made through the server-manager I thought that the reboot action was requested by the sever-manager panel. But perhaps I forgot this message.

I think you should first verify your backup.

Two actions are available to verify your backup :
- the first one is not yet possible in your situation : use workstation backup verify option from theserver-manager panel,
- the second is always possible : you can use selective restore panel to verify if the backup really contains the files you need (it must show them), and test if restore of one of them via selective restore is possible.

Please report the result, you must first be sure of the content of your backup.

Offline nowhinjing

  • *
  • 32
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #54 on: September 12, 2007, 10:59:04 AM »
Hi, JPL,

Thanks for your response.

I have done as you suggested. The files in the backup appear to be OK. I have restored small sections with no problems, but if I select a large number it just says "Restore Complete" without listing or indeed restoring any files.

There seems to be an issue with the number of files to be restored at one time, so I gradually increased the number that I was trying to restore until I got the message "Error while calling execvp:Argument list too long".

I hope this gives you a handle on the problem.

NWJ
...

Offline Jim Holland

  • 2
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #55 on: September 12, 2007, 03:50:18 PM »
Hi jpl

You wrote:

No. Even temporary files are on the backup system.

I have 140 GB to backup, which takes around 18 hours for a full backup.  I see the temporary files appearing in /mnt/smb, which is on the source disk.  Only when complete are they moved to the target USB drive directory of /media/usbdisk.  That is ideal behaviour for me as it means that the maximum utilisation of the USB drive can be achieved.

About the possibilty to do the mount, I think we should take in account the two situations : admin keeps his usb disk always mounted, or admin wants the usb disk is mounted before each backup, and unmounted after each backup.

If we want to provide the two possibilities, we have two solutions :
- keep things unchanged and resquire the admin to add an action which mounts the usb disk in the pre-backup event, and unmount it in the post-backup event (it could permit to handle specific situations)
- let the admin specify his choice with an additional option in the usb backup setting panel, and modify the scripts consequently.

The second way needs a little modifications of the backup script.

Comments and opinions ?

I would be happy to have only the option of automatic mounting followed by auto unmounting at the end of the backup, although at present I leave it mounted all the time.  We don't need too many options.

I am glad to report that subsequent attempts have achieved one successful full backup and one incremental backup.  Now for a test restore on a spare machine!

Regards

Jim

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #56 on: September 13, 2007, 09:16:04 PM »
Hi jpl
 I see the temporary files appearing in /mnt/smb, which is on the source disk.

It's an error in my scripts, which I will look for. The temp backup should be on the backup host, not on the local server, even with usb disk. This is a security reason : if you program a daily full backup (probably the most frequent use of backup) and the transfer to the backup workstation or usb disk fails, you have no more available backup. But when the tmp backup is made on the backup workstation, if it fails the preceding backup is always available.

Yes, you need more space on the backup disk, but in the other solution you need more space on the server disk AND the backup process is far less reliable.

Quote
I would be happy to have only the option of automatic mounting followed by auto unmounting at the end of the backup, although at present I leave it mounted all the time.  We don't need too many options.

I also think it's the way to go, and I will do more tests in this direction.

Offline ianbsuki

  • 1
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #57 on: October 01, 2007, 06:10:33 PM »
Hi JPL,
On a Restore from Workstation:
I also have "nowhinjing" 's problem, and this message in server-manager :-
After the restore completes you must reboot the server. Your restore is complete when /etc/smbpasswd has been restored and the words "Restore complete" appear at the bottom of your screen.
 
The backup files are being restored to eg.
Restoring file: /etc/e-smith/web/panels/manager/cgi-bin/sys

At the end of the restore, the message:-
Restore complete

You must reboot the server to activate any configuration changes that were made as a result of this restore.
 
The directory structure is in:- /etc/e-smith/web/panels/manager/cgi-bin/
But, only the /home and /root directory's have any files in them.
These files are not being moved to their correct locations after the reboot

With a selective restore:-
Files are being restored, I have phpmyadmin and eventum in /opt/
these are not being backed-up, they are not in the "40go-into" template.

Any info on how to get a full backup and restore, would be welcome.
TIA


Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #58 on: October 04, 2007, 09:39:52 PM »

The directory structure is in:- /etc/e-smith/web/panels/manager/cgi-bin/
But, only the /home and /root directory's have any files in them.
These files are not being moved to their correct locations after the reboot


Default backuped files and directories are the same as with tape or desktop restore.
/opt directory is not saved by default.
But hopefully e-smith-backup wit dar is templated, so you can add directories and files you want to backup/restore.

Copy /etc/e-smith/templates/etc/dar/Dailybackup.dcf/40go-into to /etc/e-smith/templates-custom/etc/dar/Dailybackup.dcf/40go-into and modify /etc/e-smith/templates-custom/etc/dar/Dailybackup.dcf/40go-into to add needed files and directories.

Then expand the template with :
/sbin/e-smith/expand-template etc/dar/Dailybackup.dcf

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #59 on: October 05, 2007, 03:50:29 AM »
jpl

Just a minor correction, in case it was not obvious to readers
Dailybackup.dcf should read DailyBackup.dcf

Here's a corrected & slightly modified procedure:

mkdir -p /etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf

cp /etc/e-smith/templates/etc/dar/DailyBackup.dcf/40go-into /etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf
(the above is all on one line)

to add needed files and directories modify
/etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf/40go-into

Then expand the template with :
expand-template /etc/dar/DailyBackup.dcf
...

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #60 on: October 05, 2007, 09:38:10 AM »
jpl

Just a minor correction, in case it was not obvious to readers
Dailybackup.dcf should read DailyBackup.dcf


Great thanks for the correction  :???:

Matt Hamilton

Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #61 on: October 06, 2007, 12:07:45 AM »
JPL,
I was very excited to discover this contrib as I've been looking for this functionality to roll out to my customers.

Unfortunately....
I tested this on my home/test server with a scheduled backup to a local usb disk (which had been manually mounted) and the backup process created the temporary files in /mnt/smb on the server itself which completely filled the server disks and brought the server to its knees.
I managed to delete the files using an SCP connection but obviously this is not a great solution if so much free space is required on the server for a successful backup.

I have gathered from your previous posts that the temporary files 'should' be placed on the USB disk itself which would be a much better solution.

Alternatively, could the temporary files not be transferred after the creation of each file instead of building a complete backup set before moving them all?

Offline raem

  • *
  • 3,972
  • +4/-0
e-smith-backup with dar file size mod
« Reply #62 on: October 06, 2007, 04:57:36 AM »
To all

Here's the procedure to set the backup file size.
The default slice or part setting is 700Mb (to suit CD's).
I wanted to change it to suit 4.7Gb DVD's, by making 3 parts fit to a DVD, so the slice size is 1529Mb (allowing for overheads & real data size).


mkdir -p /etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf

cp /etc/e-smith/templates/etc/dar/DailyBackup.dcf/90slice /etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf
(the above is all on one line)

to change the file size modify (& save)
/etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf/90slice

to read like this

{
($backupwk{Slice} || '1529M') eq '1529M' ?  "--slice 1529M\n" : "--slice $backup
}


Then expand the template with
expand-template /etc/dar/DailyBackup.dcf

...

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #63 on: October 08, 2007, 01:18:44 PM »
JPL,
I was very excited to discover this contrib as I've been looking for this functionality to roll out to my customers.

Unfortunately....


After my time was taken to release a Jabber server contrib for SME, I am about to modify e-smith-backup and test a release which mounts/unmounts usb devices, like other mounts,  and I hope resolve these problems (but I not before next week).

But I think there is a difficult choice when we consider usb devices : usb keys have limited space and storing temporary backup files on usb can be difficult, usb disks have a lot of space for backuping large volume of server data and for such usb disks temporary backup files need to be stored on the usb disk. Maybe we should consider two usb logical devices for backup : usb key and usb disk...

Offline Stiven

  • **
  • 53
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #64 on: October 11, 2007, 01:27:08 AM »
Hi,

I successfully installed this release of e-smith-backup. However, I've got some problems with the display in the server-manager.

The new list items seems to be displayed as WORKSTN_CONFIGURE, WORKSTN_VERIFY and so on...same problem when I choose one of these new items.

For information, my SME is set up in french language.

Is there someone able to help me ?

Thanks in advance.

Bye.

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #65 on: October 11, 2007, 08:07:31 AM »
This is because e-smith-backup is a modified core package, and this core package has a french translation without addition of new features.

Due to smeserver choices for locale packaging in SME 7 (for core packages), translations other than en-us are not in the package itself, but put together in a package for each language.

Nothing to do before updating smeserver french locale rpm, but I have an updated french localized file for backup panel, and I'll put it in my contrib area at the end of week (not at home before).

New post when done.

Offline Stiven

  • **
  • 53
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #66 on: October 11, 2007, 09:59:53 PM »
Right.

Thanx for your reply.

I'll wait till week end. :smile:

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #67 on: October 14, 2007, 07:58:24 PM »
Here is an updated french localized backup file to replace /etc/e-smith/locale/fr/etc/e-smith/web/functions/backup.

I am sorry to tell you that it is only partially translated, but it should not be difficult to complete the translation. However you'll not get messages like WORKSTN_CONFIGURE or WORKSTN_VERIFY  :o

Offline slords

  • ****
  • 235
  • +3/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #68 on: October 15, 2007, 12:25:05 AM »
jpl,

I believe most of your work has been integrated into the e-smith-backup that can be found in the smetest repo.  Would you be willing to work with byte (Gavin) to ensure all necessary changes are made to get this package ready for release.  He is a great tester and will help verify that all functionality that you have worked on is there.
"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs,
and the Universe trying to produce bigger and better idiots. So far, the Universe is winning." -- Rich Cook

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #69 on: October 28, 2007, 05:17:25 AM »
jpl & others

I ran into issues with the size of removable backup disks (120Gb IDE on a workstation) and the resultant backup data size (120Gb of data compressing to approx 50Gb).
I was able to get one full backup onto a disk, and a weeks worth of daily incrementals, but the next weekly full backup failed, due to the need to save the new backup in the tmp folder on the removable disk, before deleting the old backup files. There was not enough space on the disk (size) I was using.

I created a cron job and put it in /etc/cron.d with a script in /opt/scripts/..., which mounts the drive and deletes the old backup just before the new full backup commences. It works quite OK.
I realise this deletes the old backup before the new one has completed, but I'm happy to forgo that in order to use the available hardware.

In /etc/cron.d/deletebkp1

30    1   *    *   3    root   . /opt/scripts/deletebkp1


In /opt/scripts/deletebkp1

umount /mnt/smb
/bin/mount -t smbfs //stationXX/bkp1 /mnt/smb -o username=XXXXXX,password=XXXXXXXXXXXXX
rm -f -R /mnt/smb/servername.yourdomain.com/
rm -f -R /mnt/smb/tmp_dir/

...

Offline edform

  • *
  • 178
  • +0/-0
    • http://www.workgroupsolutions.co.uk
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #70 on: November 05, 2007, 11:52:29 PM »
Quote
SRPMS available at http://beta.free-eos.org/sme/RPMS/e-smith-backup-1.14.0-16eos1.el4.sme.src.rpm

This link doesn't seem to work at the moment and when I trace down throught he server at EOS the branch isn't there either.

Has it gone somewhere else to live?

Ed Form

Offline byte

  • *
  • 2,183
  • +2/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #71 on: November 06, 2007, 12:18:18 AM »
Has it gone somewhere else to live?

We have incorporated JPL's excellent work in the e-smith-backup base version which you can find in smetest repo:

http://distro.ibiblio.org/pub/linux/distributions/smeserver/releases/7.2/smetest/i386/repodata/repoview/e-smith-backup-0-1.15.0-05.el4.sme.html

The more people who test and use this package means it can get in to base and be a standard package, to install you can do by typing:

yum upgrade --enable smetest e-smith-backup

Remember this is in smetest so there will/may be bugs so don't forget to open a bug and open a bug in the SME Server 7.x release as the package in smetest is not a contrib. Thanks.

PS. JPL would you be interested in working together to get this polished and in to the base ? Thanks.
--[byte]--

Have you filled in a Bug Report over @ http://bugs.contribs.org ? Please don't wait to be told this way you help us to help you/others - Thanks!

Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #72 on: November 07, 2007, 09:27:13 PM »
PS. JPL would you be interested in working together to get this polished and in to the base ? Thanks.

Sure, I was about to post an update to Gavin (byte). But if this is not the prefered way to do it, please send me the right way by mail.
« Last Edit: November 07, 2007, 09:30:58 PM by jpl »

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #73 on: November 10, 2007, 06:14:40 AM »
Dear All

The How to has been updated
http://wiki.contribs.org/Backup_with_dar
...

Offline edform

  • *
  • 178
  • +0/-0
    • http://www.workgroupsolutions.co.uk
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #74 on: November 28, 2007, 04:00:38 AM »
I've had a rather disturbing incident with this backup procedure today.

I had to replace a disk in my server because of persistent dropping offline so I decided to rely on my backups and remake the disk mirror from scratch.

Loaded e-smith, did basic setup in the server manager and then loaded the e-smith-backup with dar system using yum --enable smetest upgrade e-smith-backup. After the reboot at the end of the install, I also ran the software installer and accepted the full set of updates available followed by another reboot from the reconfigure button at the bottom of the yum results.

Then did a full restore from the latest backup - it's on a NAS device - and it ran right through and stopped but didn't put up the reboot notice at the end which the instructions indicated I should get. After a reboot there were apparently no files in the data store ibay!!!

Ran the full restore again and this time I noticed that it was putting the restored files in /etc/e-smith/web/panels/manager/cgi-bin. I've looked again at the configuration of the backup and I can't see anything I could have done to cause this so why isn't the program restoring to the original locations?

Any ideas anyone!

Ed Form

Offline jester

  • *
  • 496
  • +1/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #75 on: November 28, 2007, 01:20:58 PM »
I've updated e-smith-backup from the test repo and trying to backup to an Iomega usb REV (rrd) drive. The rev-drive is normally in /media/usbdisk and through the backup admin-panel i can select it when configuring a 'backup to workstation'. I've created an action for pre-backup to mount the REV-cartridge with (otherwise i presume the backup will end up in the /media/usbdisk/ directory in stead of on the rev-cartridge):
Code: [Select]
#!/bin/sh
exec /bin/mount /media/usbdisk

and two actions for post-backup to umount and eject the REV-cartridge:
Code: [Select]
#!/bin/sh
exec /bin/umount /media/usbdisk
and
Code: [Select]
#!/bin/sh
exec /usr/bin/eject /media/usbdisk

But i must be doing something wrong 'cause nothing appears on the rev-cartridge.... suggestions anyone?!

Thanx!
jester.


Offline jpl

  • *
  • 112
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #76 on: November 28, 2007, 01:50:20 PM »
If you use last available version of e-smith-backup (e-smith-backup-1.15.0-7, or even -6)), the mount is made by the backup script.

So if you mount /media/usbdisk through a pre-backup script, the backup script should normally fail.


Offline jester

  • *
  • 496
  • +1/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #77 on: November 28, 2007, 10:46:15 PM »
Thanks jpl, got things working now.... on with the testing.

Offline altair40

  • 1
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #78 on: December 12, 2007, 08:04:24 PM »
Hello!

I have an annoying problem on my server with this contrib.

When I try a sel_restore from the web interface of server-manager, response is pretty bad:

"Corrupted database :Not a zero terminated string in file"

What's exactly going on? i didn't read/find anything about this problem...

Thanks!

Offline SoftDux

  • ****
  • 240
  • +0/-0
    • http://www.SoftDux.com
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #79 on: December 19, 2007, 02:38:22 PM »
I didn't see any mention of this, but would I be able to use e-smith backup to automatically backup selected folders on each one of my Windows XP machines?

Let's say I have an sme server, running e-smith-backup, and 3 Windows XP (pro or home) machines, with some shared folders:
XP 1 has "My Documents", "Games" & "Client invoices"
XP 2 has "My Documents", "random stuff", "quotes", "client invoices", "client work"
XP 3 has "My Documents", "client invoices"

How would I use e-smith-backup to automatically backup all of these to say /home/e-smith/files/ibays/backups/{machine name}?
Cause then I can automatically backup the backups ibays to my USB backup drive

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #80 on: December 19, 2007, 06:03:47 PM »
SoftDux

e-smith-backup does not support that funtionality.

If you read earlier parts of this thread, you will see commands that allow you to mount a windows share, so if you created shares for those locations you could then mount them to /home/e-smith/files/ibays/backups/....
and they would be included in normal daily backups.

You would have to create seperate cron jobs/scripts to handle the mounting and unmounting.

There is the BackupPC contrib which is designed to do this.

Why not just point those locations on your windows PC's to use the users home drive (H:), or enable roamimg profiles, then that data is automtaically included in backups.
...

Offline SoftDux

  • ****
  • 240
  • +0/-0
    • http://www.SoftDux.com
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #81 on: December 19, 2007, 06:39:16 PM »
ok, thanx. I'll check out BackupPC

Offline raem

  • *
  • 3,972
  • +4/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #82 on: December 19, 2007, 07:15:01 PM »
SoftDux

Assuming you are using sme as a domain controller, and as you only have 3 workstations, it would be very easy to just click on My Documents/Properties and change the path to H:, and then that stuff will automatically be saved on the server in the users home drive, and will therefore be included in normal backups, nothing extra to install or configure.

Alternatively if you don''t use sme as a domain controller and don't have home drives, change My Documents to another shared drive eg P:\username which happens to be an ibay, that will also be included in normal backups without any extra effort.
...

Offline SoftDux

  • ****
  • 240
  • +0/-0
    • http://www.SoftDux.com
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #83 on: December 19, 2007, 07:25:21 PM »
I don't use sme as a domain controller, and I don't want to move the Documents folder to a share on the server. I'm looking for something to implement on all the networks that I administrate, some have Windows XP, some Windows 2000, Some MAC, some Linux - all use samba though, so the principle will stay the same.

Ideally, I'd prefer to "pull" the files from the sme server, so the mount option will work fine :)

Offline holck

  • ****
  • 317
  • +1/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #84 on: September 19, 2008, 11:32:43 PM »
I have a small problem with this contrib, which otherwise is really helpful. I choose to backup to a USB-disk attached to the server, and the script automatically (?) selects "/media/usbdisk" as the backup-device, but this disk is non-existent. I want it to use "/media/usbdisk1" which is where the external disk is.

I found out that I can change the name of the disk with
/sbin/e-smith/db configuration setprop backupwk SmbShare media/usbdisk1

But this is annoying to do every time I change the backup settings.

Any ideas / help?
« Last Edit: September 19, 2008, 11:37:43 PM by holck »
......

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #85 on: September 20, 2008, 03:48:51 AM »
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline holck

  • ****
  • 317
  • +1/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #86 on: September 20, 2008, 06:34:25 PM »
Thanks for the link :-)

I have now labeled my USB backup disk:
# /sbin/e2label /dev/sda3 USB-backup

The two relevant lines in /etc/fstab:
/dev/sda3          /media/usbdisk1        ext3   defaults
/dev/sda1          /media/usbdisk          ext3   pamconsole,noatime,sync,exec,noauto,managed 0 0

I changed to:
#/dev/sda3          /media/usbdisk1         ext3   defaults
LABEL=USB-backup  /media/usbdisk        ext3   defaults
/dev/sda1         /media/usbdisk     ext3    pamconsole,noatime,sync,exec,noauto,managed 0 0

I hope it works, but I am little concerned whether these changes to /etc/fstab will "survive" various events and system changes.
......

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: [UPDATE 7.2] e-smith-backup with dar
« Reply #87 on: September 21, 2008, 03:26:06 AM »
holck

Not everything in sme is templated.
I think fstab changes will survive reconfiguration.

You can easily prove this by doing
signal-event post-upgrade
signal-event reboot

and then see if your fstab changes are still there.
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.