Koozali.org: home of the SME Server

[beta] e-smith-backup with dar

Offline jpl

  • *
  • 112
  • +0/-0
Re: Kernel requirement for this Contrib?
« Reply #15 on: November 03, 2006, 06:49:35 PM »
Quote from: "ronnie_dk"
Hi there.
I have seen topic saying "in the smeupdates-testing repo is 2.6.9-42.0.2.EL" does this kernel suit my needs to make CIFS work for this Contrib?


No it doesn't.

cifs correct patch has been committed by Jason Baron (maintainer of RHEL4 kernel) in 2.6.9-42.10 release of RHEL4 kernel.

All preceding versions of RHEL4 kernel have problem with cifs when listing directories.

For a test server you can try to use lastest dev kernel available at http://people.redhat.com/~jbaron/rhel4/ but for a production server, you are advised to wait next stable CentOS kernel update before using cifs.

Offline ronnie_dk

  • 19
  • +0/-0
[beta] e-smith-backup with dar
« Reply #16 on: November 03, 2006, 06:57:07 PM »
JPL
Thanks for info...

Ronine
...

Offline dmay

  • *
  • 450
  • +0/-0
    • http://myezserver.com
[beta] e-smith-backup with dar
« Reply #17 on: November 04, 2006, 12:14:44 AM »
Quote from: "ronnie_dk"
Meanwhile I have updated the kernel. I will try to cut+paste the "CIFS mount denied message" if backup fails again tonight..
Best Regards Ronnie

Ronnie,

This code works for me for mounting using cifs to a W2K3 share:

Code: [Select]
/bin/mount -t cifs -o username=blah,password=blah //ipaddress/share /mnt/smb

Example:

Code: [Select]
/bin/mount -t cifs -o username=dar2,password=logical //10.1.0.20/dar2 /mnt/smb

See if it works for you. If it does try editing JPL's workstation-backup-dar to match and test. Something like this might work as long as you make sure you input //ipaddress in JPL's panel entry for smbhost:

Code: [Select]
/bin/mount -t cifs -o username=$login,password=$password //$smbhost/$smbshare $mntdir

Darrell

Offline jpl

  • *
  • 112
  • +0/-0
[beta] e-smith-backup with dar
« Reply #18 on: November 04, 2006, 11:44:40 AM »
Quote from: "dmay"

See if it works for you. If it does try editing JPL's workstation-backup-dar to match and test. Something like this might work as long as you make sure you input //ipaddress in JPL's panel entry for smbhost:

Code: [Select]
/bin/mount -t cifs -o username=$login,password=$password //$smbhost/$smbshare $mntdir


e-smith-backup with dar uses a more recent syntax, but Darrell's proposition should be supported the same way (this is the case for my tests).
Tell if it changes anything. I don't think it will, but I hope to be wrong.

You could also verify this bug related to cifs use with Windows 2003 Server : https://bugzilla.samba.org/show_bug.cgi?id=3850

Offline ronnie_dk

  • 19
  • +0/-0
Confused.. Yep
« Reply #19 on: November 04, 2006, 03:18:46 PM »
Hi again.. I have managed to log onto the 2003 server now and here it gets funny..
I have 4 new lines in panel "Administration/Backup or restore"

I´ll try to generate a backup job:

It looks like this
NB: I have my 2003 server in hosts file so 192.168.0.2 wich is my 2003 server is named "filserver"

Quote

Configure Workstation Backup
This panel displays the present workstation backup configuration. You can change it in this panel and the next one.

Backup is enabled.
Backup is made on LAN workstation filserver via cifs
Destination backup share folder is Serverbackup
Login is username "used local superuser account"
Password is XXXX
Number of rotating backup sets is 1
Number of daily backups contained in each set is 1
Compression level (0-7) of backup is 4
Daily backup occurs at 14:00
Each daily backup session is cleanly timed out after 1h
Full backup sessions (new backup set) are allowed everyday


at 14:00 the following comes up in messages
Quote

14:00:01 router /sbin/e-smith/do_backupwk[17561]: /home/e-smith/db/backups: OLD 1162645201=(undefined)
Nov  4 14:00:01 router /sbin/e-smith/do_backupwk[17561]: /home/e-smith/db/backups: NEW 1162645201=backup_record
Nov  4 14:00:01 router /sbin/e-smith/do_backupwk[17561]: /home/e-smith/db/backups: OLD 1162645201=backup_record
Nov  4 14:00:01 router /sbin/e-smith/do_backupwk[17561]: /home/e-smith/db/backups: NEW 1162645201=backup_record|StartEpochTime|1162645201
Nov  4 14:00:01 router /sbin/e-smith/do_backupwk[17561]: /home/e-smith/db/backups: OLD 1162645201=backup_record|StartEpochTime|1162645201
Nov  4 14:00:01 router /sbin/e-smith/do_backupwk[17561]: /home/e-smith/db/backups: NEW 1162645201=backup_record|BackupType|workstation|StartEpochTime|1162645201
Nov  4 14:00:01 router esmith::event[17562]: Processing event: pre-backup  
Nov  4 14:00:01 router esmith::event[17562]: Running event handler: /etc/e-smith/events/pre-backup/S10mysql-delete-dumped-tables
Nov  4 14:00:01 router esmith::event[17562]: S10mysql-delete-dumped-tables=action|Event|pre-backup|Action|S10mysql-delete-dumped-tables|Start|1162645201 773364|End|1162645201 778396|Elapsed|0.005032
Nov  4 14:00:01 router esmith::event[17562]: Running event handler: /etc/e-smith/events/pre-backup/S20mysql-dump-tables
Nov  4 14:00:01 router esmith::event[17562]: S20mysql-dump-tables=action|Event|pre-backup|Action|S20mysql-dump-tables|Start|1162645201 779083|End|1162645201 893544|Elapsed|0.114461
Nov  4 14:00:01 router esmith::event[17562]: Running event handler: /etc/e-smith/events/pre-backup/S50rewind-tape
Nov  4 14:00:02 router esmith::event[17562]: S50rewind-tape=action|Event|pre-backup|Action|S50rewind-tape|Start|1162645201 894239|End|1162645202 174943|Elapsed|0.280704
Nov  4 14:00:04 router /sbin/e-smith/do_backupwk[17561]: /home/e-smith/db/backups: OLD 1162645201=backup_record|BackupType|workstation|StartEpochTime|1162645201
Nov  4 14:00:04 router /sbin/e-smith/do_backupwk[17561]: /home/e-smith/db/backups: NEW 1162645201=backup_record|BackupType|workstation|EndEpochTime|1162645204|StartEpochTime|1162645201
Nov  4 14:00:04 router /sbin/e-smith/do_backupwk[17561]: /home/e-smith/db/backups: OLD 1162645201=backup_record|BackupType|workstation|EndEpochTime|1162645204|StartEpochTime|1162645201
Nov  4 14:00:04 router /sbin/e-smith/do_backupwk[17561]: /home/e-smith/db/backups: NEW 1162645201=backup_record|BackupType|workstation|EndEpochTime|1162645204|Result|backup:7424|StartEpochTime|1162645201


But no backupfile is made on 2003 server... hmmmm...
OK another way to make backups is to use the other New button in my SME panel "Backup Disk ARchive"
I have managed to create a job called "stdbackup"
I fired it up by manually clicking "Backup" and the result is here.. fine

Quote

Nov  4 14:06:52 router e-smith-bg: Creating /mnt/dar2/stdbackup dir
Nov  4 14:06:52 router e-smith-bg: Connecting /mnt/dar2/stdbackup to //filserver/Serverbackup
Nov  4 14:06:52 router e-smith-bg: Expanding /etc/dar2/stdbackup template
Nov  4 14:06:52 router e-smith-bg: Creating destination /router.linwin.dk/stdbackup dir
Nov  4 14:06:52 router e-smith-bg: Expiring old backups
Nov  4 14:06:52 router e-smith-bg: Executing pre-backup event
Nov  4 14:06:52 router esmith::event[17800]: Processing event: pre-backup  
Nov  4 14:06:52 router esmith::event[17800]: Running event handler: /etc/e-smith/events/pre-backup/S10mysql-delete-dumped-tables
Nov  4 14:06:52 router esmith::event[17800]: S10mysql-delete-dumped-tables=action|Event|pre-backup|Action|S10mysql-delete-dumped-tables|Start|1162645612 649623|End|1162645612 654883|Elapsed|0.00526
Nov  4 14:06:52 router esmith::event[17800]: Running event handler: /etc/e-smith/events/pre-backup/S20mysql-dump-tables
Nov  4 14:06:52 router esmith::event[17800]: S20mysql-dump-tables=action|Event|pre-backup|Action|S20mysql-dump-tables|Start|1162645612 655594|End|1162645612 737923|Elapsed|0.082329
Nov  4 14:06:52 router esmith::event[17800]: Running event handler: /etc/e-smith/events/pre-backup/S50rewind-tape
Nov  4 14:06:52 router esmith::event[17800]: S50rewind-tape=action|Event|pre-backup|Action|S50rewind-tape|Start|1162645612 738683|End|1162645612 920405|Elapsed|0.181722
Nov  4 14:06:52 router e-smith-bg: Verbose logging is off
Nov  4 14:06:52 router e-smith-bg: Performing stdbackup backup
Nov  4 14:06:52 router e-smith-bg: No terminal found for user interaction. All questions will be assumed a negative answer (less destructive choice), which most of the time will abort the program.
Nov  4 14:07:28 router e-smith-bg:
Nov  4 14:07:28 router e-smith-bg:
Nov  4 14:07:28 router e-smith-bg:  --------------------------------------------
Nov  4 14:07:28 router e-smith-bg:  1770 inode(s) saved
Nov  4 14:07:28 router e-smith-bg:  with 0 hard link(s) recorded
Nov  4 14:07:28 router e-smith-bg:  0 inode(s) changed at the moment of the backup
Nov  4 14:07:28 router e-smith-bg:  0 inode(s) not saved (no inode/file change)
Nov  4 14:07:28 router e-smith-bg:  0 inode(s) failed to save (filesystem error)
Nov  4 14:07:28 router e-smith-bg:  240 inode(s) ignored (excluded by filters)
Nov  4 14:07:28 router e-smith-bg:  0 inode(s) recorded as deleted from reference backup
Nov  4 14:07:28 router e-smith-bg:  --------------------------------------------
Nov  4 14:07:28 router e-smith-bg:  Total number of inode considered: 2010
Nov  4 14:07:28 router e-smith-bg:  --------------------------------------------
Nov  4 14:07:28 router e-smith-bg:
Nov  4 14:07:28 router e-smith-bg: Listing stdbackup archives on target:
Nov  4 14:07:28 router e-smith-bg: total 104932
Nov  4 14:07:28 router e-smith-bg: -rw-rw-rw-  1 root admin 107450147 Nov  4  2006 2006.11.04.1.dar
Nov  4 14:07:28 router e-smith-bg:
Nov  4 14:07:28 router e-smith-bg: Total target disk space usage:
Nov  4 14:07:28 router e-smith-bg: Filesystem            Size  Used Avail Use% Mounted on
Nov  4 14:07:28 router e-smith-bg: //filserver/Serverbackup
Nov  4 14:07:28 router e-smith-bg:                       112G  106G  6.2G  95% /mnt/dar2/stdbackup
Nov  4 14:07:28 router e-smith-bg:
Nov  4 14:07:28 router e-smith-bg: Executing post-backup event
Nov  4 14:07:29 router esmith::event[17829]: Processing event: post-backup  
Nov  4 14:07:29 router esmith::event[17829]: Running event handler: /etc/e-smith/events/post-backup/S10mysql-delete-dumped-tables
Nov  4 14:07:29 router esmith::event[17829]: S10mysql-delete-dumped-tables=action|Event|post-backup|Action|S10mysql-delete-dumped-tables|Start|1162645649 492291|End|1162645649 497845|Elapsed|0.005554
Nov  4 14:07:29 router esmith::event[17829]: Running event handler: /etc/e-smith/events/post-backup/S50rewind-tape
Nov  4 14:07:29 router esmith::event[17829]: S50rewind-tape=action|Event|post-backup|Action|S50rewind-tape|Start|1162645649 498562|End|1162645649 674327|Elapsed|0.175765
Nov  4 14:07:29 router e-smith-bg: Dismounting /mnt/dar2/stdbackup
Nov  4 14:07:29 router e-smith-bg:
Nov  4 14:07:29 router e-smith-bg: Sat Nov  4 14:06:51 CET 2006 - backup started
Nov  4 14:07:29 router e-smith-bg: Sat Nov  4 14:07:29 CET 2006 - backup finished


Year i have finally created a Dar file on my 2003 server with the following name:"2006.11.04.1.dar" filesize among 107 MB..

Now I want to verify my backup with Panel "Administration/Backup or restore" click "Verify Workstation Backup"
stdbackup() is preselected and i click "verify"
Result is here:

Quote

The following files are contained in the backup file:
Aborting program. User refused to continue while asking: /mnt/smb/router.linwin.dk/stdbackup/2006.1.dar is required for further operation, please provide the file.
Error while reading files from : /mnt/smb/router.linwin.dk/stdbackup/2006

What is wrong with the filename here.
Hmm . Is the mountpoint missing? nope

Quote

[root@router router.linwin.dk]# ls
dar-catalog  stdbackup
[root@router router.linwin.dk]# cd stdbackup/
[root@router stdbackup]# ls
2006.11.04.1.dar
[root@router stdbackup]#


I now have 2 new folders on SME . /mnt/dar2 and /mnt/smb..
One of them is mounted to my 2003 server /mnt/smb and the other one /mnt/dar2 has been umounted by the backupscript..

How can i verify this backup ?
What am I doing wrong...?
I´m getting a little confused here...

Darrel/jpl:
Pls bear with me as I'm a newbe and doesn't know where to edit those files.

NB: My mtab is growing "1 line" each time i try to perform the verify:

Quote

/dev/mapper/main-root / ext3 rw,usrquota,grpquota 0 0
none /proc proc rw 0 0
none /sys sysfs rw 0 0
none /dev/pts devpts rw,gid=5,mode=620 0 0
usbfs /proc/bus/usb usbfs rw 0 0
/dev/md1 /boot ext3 rw 0 0
none /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
//filserver/Serverbackup /mnt/smb cifs rw,mand 0 0
//filserver/Serverbackup /mnt/smb cifs rw,mand 0 0
//filserver/Serverbackup /mnt/smb cifs rw,mand 0 0
//filserver/Serverbackup /mnt/smb cifs rw,mand 0 0
//filserver/Serverbackup /mnt/smb cifs rw,mand 0 0
//filserver/Serverbackup /mnt/smb cifs rw,mand 0 0
//filserver/Serverbackup /mnt/smb cifs rw,mand 0 0
...

Offline jpl

  • *
  • 112
  • +0/-0
Re: Confused.. Yep
« Reply #20 on: November 04, 2006, 04:24:47 PM »
Quote from: "ronnie_dk"
Hi again.. I have managed to log onto the 2003 server now and here it gets funny..


Few important things according to your tests :

1. e-smith-backup with dar and smeserver-dar2 are two different ways to do backup, even they share use of dar. You cannot verify (as you tried)  smeserver-dar2 made backup with verify option provided by e-smith-with dar.

2. I don't advise to simultaneously tests the two programs, or you can be confused.

3. smeserver-dar2 and e-smith-backup with dar are not equal in functionalities (incremental backups, selective restore, security, etc.). Just look at what are your needs and select only one of them for backup.

4. Concerning e-smith-backup with dar, one explaination is surely that there are zombies left in memory from preceeding tests (it is what suggest your mtab file). So in this case e-smith-backup see there is already a mount and doesn't do backup.

5. You normally are informed about by reading mail send to admin. What do these mails say ?

6. Before trying a new backup, you must kill all remaining  smbd zombies in memory. A simple way to do it is to reboot the server.

Offline dmay

  • *
  • 450
  • +0/-0
    • http://myezserver.com
Re: Confused.. Yep
« Reply #21 on: November 04, 2006, 04:57:47 PM »
Quote from: "ronnie_dk"
Hi again.. I have managed to log onto the 2003 server now and here it gets funny..

Did you make the changes I suggested to JPL's workstation-backup-dar?
Did this work or fail for you?

As JPL and I have stated before, our two contribs are completely separate. Separate rpms, panels and they do not share any code. The only thing in common is they both use 'dar' which in itself is a separate rpm.

As you have tested and reported my contrib and code works, I've simply suggested you update JPL's workstation-backup-dar script with my code and see if this fixes things and permits his script to work. We know it will permit the mount to function. After that there may be more changes required. What you really should do is open a Bug Tracker ticket on this issue and we can work together with JPL there to find a resolution.

Darrell

Offline ronnie_dk

  • 19
  • +0/-0
[beta] e-smith-backup with dar
« Reply #22 on: November 04, 2006, 05:10:15 PM »
Dmay Wrote:
Quote

Did you make the changes I suggested to JPL's workstation-backup-dar?
Did this work or fail for you?

Actually i don´t know where to edit what... so my answer is no.. i hope it will solve the problem when edited..
Quote

What you really should do is open a Bug Tracker ticket on this issue and we can work together with JPL there to find a resolution.

I will do that if it fails again after reboot

jpl Wrote:
Quote

6. Before trying a new backup, you must kill all remaining smbd zombies in memory. A simple way to do it is to reboot the server.

I will reboot the server later this evening. Time is now 5 PM here in Denmark so i will get back with the test results later this evening..
...

Offline dmay

  • *
  • 450
  • +0/-0
    • http://myezserver.com
[beta] e-smith-backup with dar
« Reply #23 on: November 04, 2006, 05:26:20 PM »
Quote from: "ronnie_dk"

Actually i don´t know where to edit what... so my answer is no.. i hope it will solve the problem when edited..

This is even more reason to open a Bug Tracker ticket. It is easier to share code in the Bug Tracker so please open a ticket here:

http://bugs.contribs.org/enter_bug.cgi?product=SME%20Contribs
Component: e-smith-backup update to use DAR

Darrell

Offline ronnie_dk

  • 19
  • +0/-0
TKS everybody...
« Reply #24 on: November 04, 2006, 07:24:34 PM »
Hi again.
I have rebooted the SME and guess what... I should have done that first.. The thing is actually working  :idea:  .. verify backup is ok too....Whining happily  :o
Gee man. I must admit that rebooting servers is not one of my favorites but the reboot made it work.
Thanks for helping me finding the right way to archive perfect backups, and replying to my foolish questions..
I could have waited for a howto on this contrib, but no i just fooled around with it and waisted Your time instead..
Sorry and thanks again  :D
...

Offline Tillebeck

  • ***
  • 76
  • +0/-0
A few findings
« Reply #25 on: November 08, 2006, 02:21:42 PM »
I have tested a bit on my a test server at home. I have not tried to do a restore yet. Hope to have time to play around with that this weekend.

I have these two findings and an entry to the wish-list:

Findings/Issues:
1) For the first many days backup failed for me. It was due to a " in th the password for the backup share. So I changed this example password fgrt!"3B to fgrt!23B and backup started to run smoothly.
2) I started a backupjob with two sets and full backup tuesday and incrementals the other days. I later changed full backup only to be allowed on sundays but that seems not to affect the current setup as a full backup still occours tuesday. I did try to disable and reenable the backup but no change. Maybe if I delete all backups in the backup share and restart the backup job it will work. This has not yet been tested. I guess this is not a bug but there may be needed a little info on how to reset the backupsettings (e.g. disbal backup, rename folder with server name on backup destination folder, reconfigure backup, reenable backup- if that is the way to do it)

Wish-list:
1) It would be really nice to have some sort of exclude list of folders not to be backed up. My testserver contains nearly 90GB opf data where 60-70 GB are static data not needed to be backed up.

BR. Anders

Offline jpl

  • *
  • 112
  • +0/-0
Re: A few findings
« Reply #26 on: November 08, 2006, 08:52:00 PM »
Quote from: "Tillebeck"

Wish-list:
1) It would be really nice to have some sort of exclude list of folders not to be backed up. My testserver contains nearly 90GB opf data where 60-70 GB are static data not needed to be backed up.

BR. Anders


You can exclude files as you want because the backup configuration uses templating.
To exclude files you must use a custom template and create a file /etc/e-smith/custom-templates/etc/dar/DailyBackup.dcf/50prune to satisfy your needs. You must write one prune line for each excluded file or directory.

Syntax (from man dar) is :

--prune <path>  
Do not consider file or directory sub-tree given by the path. The path mask is not applied only to the filename, but also include  the  path. Moreover it applies also to directories. Note that <path> may contains wild-cards like * or ? see glob(7) man page for more informations.


Dar template expansion is made automagically before each backup.

Offline Tillebeck

  • ***
  • 76
  • +0/-0
Re: A few findings
« Reply #27 on: November 09, 2006, 11:23:53 AM »
Quote from: "jpl"

You can exclude files as you want because the backup configuration uses templating.


Thanks jpl
That is really great and usefull info. I will try to exclude some folders.

BR. Anders

Offline jester

  • *
  • 496
  • +1/-0
[beta] e-smith-backup with dar
« Reply #28 on: November 17, 2006, 10:13:53 AM »
Is it also possible to use this contrib to backup to an usb REV drive?!

The drive is automagically being mounted on /media/cdrom (after reboot), but using the backup to 'local usb disk' option i get a 'no drives found' sort of error.


Can anyone shed some light?!

Offline jpl

  • *
  • 112
  • +0/-0
[beta] e-smith-backup with dar
« Reply #29 on: November 17, 2006, 02:17:33 PM »
Quote from: "jester"
Is it also possible to use this contrib to backup to an usb REV drive?!

The drive is automagically being mounted on /media/cdrom (after reboot), but using the backup to 'local usb disk' option i get a 'no drives found' sort of error.


Can anyone shed some light?!


"The drive is automagically being mounted on /media/cdrom"

Here is the explaination. e-smith-backup excludes and doesn't show /media/cdrom... because a cdrom should not be writable !

If you cannot change the mount name, and if your usb drive mounts as /media/cdrom, you could try to comment line 1565 ("next if /cdrom/;") from /etc/e-smith/web/functions/backup script.