Koozali.org: home of the SME Server

Affa failure

Offline Brenno

  • ****
  • 208
  • +0/-0
Affa failure
« on: December 19, 2008, 02:49:24 PM »
I installed the latest version of Affa on my 7.4 box, configured per the wiki, to backup to a 500GB USB drive.  When I ran the job the first time, it failed to mount the drive.  When I mounted the drive manually and re-ran the job, it worked fine and reported success.

Now, when I try to run subsequent jobs, either manually or via Affa's own scheduler, it fails:

Code: [Select]
Excerpt from log file /var/log/affa/backup.log:
Fri_Dec_19_08:24:01[START.11110]:_Affa_1.0.0-3:_Running_/sbin/e-smith/affa_--run_backup
Job_configuration:
__AutoUnmount=yes
__AutomountDevice=/dev/sdb1
__AutomountOptions=-t_ext3
__AutomountPoint=/mnt/affadevice
__ConnectionCheckTimeout=120
__Debug=no
__Description=XXXXXX_Backup
__DiskSpaceWarn=strict
__EmailAddresses=brent@XXXXXXX
__RPMCheck=no
__RootDir=/mnt/affadevice
__SMEServer=yes
__TimeSchedule=0630,1830
__Watchdog=no
__chattyOnSuccess=1094
__dailyKeep=7
__doneDaily=2008354
__doneMonthly=200812
__doneWeekly=200850
__doneYearly=2008
__monthlyKeep=12
__remoteHostName=localhost
__remoteOS=centos
__rsync--inplace=yes
__rsyncCompress=yes
__rsyncTimeout=900
__rsyncdMode=no
__rsyncdModule=AFFA
__rsyncdPassword=<not_shown>
__rsyncdUser=affa
__scheduledKeep=3
__sendStatus=weekly
__sshPort=22
__status=enabled
__type=job
__weeklyKeep=4
__yearlyKeep=1
Fri_Dec_19_08:24:01[11110]:_Affa_1.0.0-3:_Starting_job_backup_scheduled_(localhost)
Fri_Dec_19_08:24:01[11110]:_Description:_XXXXXX_Backup
Fri_Dec_19_08:24:01[11110]:_signaling_pre-backup_event_on_localhost
Exec_Out:_exitstatus=0
Fri_Dec_19_08:24:02[11110]:_Running_rsync...
Exec_Out:_rsync:_writefd_unbuffered_failed_to_write_4092_bytes_[sender]:_Broken_pipe_(32)
Exec_Out:_rsync:_mkdir_"/mnt/affadevice/backup/scheduled.running"_failed:_Read-only_file_system_(30)
Exec_Out:_rsync_error:_error_in_file_IO_(code_11)_at_main.c(576)_[receiver=3.0.4]
Exec_Out:_rsync:_connection_unexpectedly_closed_(9_bytes_received_so_far)_[sender]
Exec_Out:_rsync_error:_error_in_rsync_protocol_data_stream_(code_12)_at_io.c(632)_[sender=3.0.4]
Exec_Out:_exitstatus=12
Fri_Dec_19_08:24:02[11110]:_signaling_post-backup_event_on_localhost
Exec_Out:_exitstatus=0
Fri_Dec_19_08:24:02[11110]:_Error:_No_data_transferred._Check_include/exclude_settings.
Fri_Dec_19_08:24:02[11110]:_Error_537:_rsync_failed_with_status_1012.

Curiously, when I check to see if the drive is still mounted, it apparently is:

Code: [Select]
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/main-root
              ext3    266G   29G  224G  12% /
/dev/sda1     ext3     99M   21M   74M  22% /boot
none         tmpfs    4.0G     0  4.0G   0% /dev/shm
/dev/sdb1     ext3    459G   27G  409G   7% /mnt/affadevice

But if I go to view the contents of /mnt/affadevice, I get:

Code: [Select]
[root@mail affadevice]# ls -l
total 0
[root@mail affadevice]#

Looking for a little help on why the USB drive is considered read only and why I can't get a file listing from it.  Maybe this is a simple thing and I just don't know where to look.

Any help appreciated.

Offline Brenno

  • ****
  • 208
  • +0/-0
Re: Affa failure
« Reply #1 on: December 19, 2008, 08:54:34 PM »
OK, just as a followup, I unmounted the USB drive, re-mounted it and then ran the Affa job manually with success.  I then modified the Affa config to not manually mount and unmount the job as part of the backup and left the USB drive mounted.

If this works out, I may modify /etc/fstab so that this drive is permanently mounted.  I had heard that this was less desirable than having it mounted only when required.  But, because having a backup is more important right now, that's my primary concern.

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Re: Affa failure
« Reply #2 on: December 19, 2008, 09:06:36 PM »
I am not a fan of USB drives mounted directly on the server but YMMV. The mount and unmount issue becomes a much larger problem when you are swapping drives to rotate backups.

The essence of the problem is that the usb drive does not always unmount correctly especially if it is unplugged without manually unmounting it first. This leaves a mount point that looks like the drive but actually is pointing to the local disk. Backups will happen with no error in this situation as the mount point is discovered by the backup software. The problem is that the backup is going right back on the local hard drive. If the hard drive fails before this situation is discovered there is no backup on any other drive and you are just plain out of luck.

See this http://wiki.contribs.org/USBDisks for more explanation of the problem and some possible work arounds.

My recommendation for what it is worth: Do not use USB disks for backup.


Offline Brenno

  • ****
  • 208
  • +0/-0
Re: Affa failure
« Reply #3 on: December 19, 2008, 09:15:18 PM »
I agree with the points you've made regarding the unsuitability of the USB drive as a long-term solution.  Having just come out of a very large failure which was compounded by a lack of error notifications, I'm sensitive to those situations at the moment : )

The USB disk is a temporary backup strategy until we devise something more robust.  I just needed something quick and easy and had used Affa before.  At this point in time, we have neither spare hardware or storage space on our network, so the USB drive will give us some protection until we can devise a better solution.

Offline mercyh

  • *
  • 824
  • +0/-0
    • http://mercyh.org
Re: Affa failure
« Reply #4 on: December 19, 2008, 09:19:03 PM »
Brenno,

I know some are using it and it is working fine. I am scared of it (maybe unreasonably so). I sleep better when I can trust my backup logs.......

Test restores will also help your sleep.


Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Re: Affa failure
« Reply #5 on: December 20, 2008, 11:53:16 AM »
Please report all problems with contribs in the Contribs section of the bug tracker.

Offline Brenno

  • ****
  • 208
  • +0/-0
Re: Affa failure
« Reply #6 on: December 22, 2008, 01:28:03 PM »
OK, so now I have removed the auto-mount option on Affa and have the USB drive mounted all the time.  Affa is reporting a successful backup, but doesn't actually seem to be making new backups.  The only backups listed there are from the two manually run jobs.  From the email it sends:

Code: [Select]
+-----------------------+----------------+--------------+--------+-------------+
| Date                  | Archive        |        Files |   Size |  Disk usage |
+-----------------------+----------------+--------------+--------+-------------+
| Tue 2008 Dec 16 10:04 | scheduled.1    |       361171 |   26GB |     27GB/6% |
| Fri 2008 Dec 19 13:55 | scheduled.0    |       363330 |   26GB |     27GB/6% |
+-----------------------+----------------+--------------+--------+-------------+

This is verified by looking at the contents of /mnt/affadevice/backup:

Code: [Select]
[root@mail backup]# ls -l
total 12
drwxr-xr-x  5 root root 4096 Dec 19 13:55 scheduled.0
drwxr-xr-x  5 root root 4096 Dec 16 10:04 scheduled.1
drwxr-xr-x  5 root root 4096 Dec 19 18:30 scheduled.running

Is the backup failing because I have a scheduled.running directory "stuck" there?  If I manually remove this, might that get this working again?

Charlie, will happily report this to the bug tracker once I'm more certain it's a bug.  Until then, it's probably just an error on my part. : )

Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Re: Affa failure
« Reply #7 on: December 22, 2008, 09:25:41 PM »
Charlie, will happily report this to the bug tracker once I'm more certain it's a bug.  Until then, it's probably just an error on my part. : )

It's already a suspected bug, and all suspected bugs should be reported.

The contrib should be preventing you from making errors.

Offline Brenno

  • ****
  • 208
  • +0/-0
Re: Affa failure
« Reply #8 on: December 22, 2008, 09:52:40 PM »
I decided to see what happened if I renamed the "scheduled.running" to "x-scheduled.running".  I couldn't rename the folder because the filesystem was read-only (this corresponds to one of the errors in the original post) so I unmounted the drive, remounted the drive, and successfully ran the job manually, verifying that there was a new backup present.

Checking the perms on /mnt/affadevice post-backup shows:

Code: [Select]
[root@mail affadevice]# ls -la
total 28
drwxr-xr-x  4 root root  4096 Dec 16 09:10 .
drwxr-xr-x  4 root root  4096 Dec 15 18:30 ..
drwx------  5 root root  4096 Dec 22 15:32 backup
drwx------  2 root root 16384 Dec 15 14:33 lost+found

It seems I can run the backup fine only once and only on a freshly-mounted drive.  Once the backup has run, the drive changes to be read-only and subsequent backups fail.  Is this Affa causing this, or is it something to do with SME/Linux itself?  I'm not familiar with mounting/unmounting USB devices because I never had to bother with it before.

Offline m

  • ****
  • 276
  • +0/-0
  • Peet
Re: Affa failure
« Reply #9 on: December 22, 2008, 10:19:35 PM »
Brenno,
I guess there is a hardware problem with your uSB drive, like temporarily disconnecting, corrupted filesystem or similar.
Affa never mounts readonly. Try another USB drive.

Offline Brenno

  • ****
  • 208
  • +0/-0
Re: Affa failure
« Reply #10 on: December 23, 2008, 01:58:11 AM »
You could very well be correct.  I unmounted/mounted the drive before I left and the subsequent scheduled backup failed:

Code: [Select]
Exec_Out:_rsync:_mkdir_"/mnt/affadevice/backup/scheduled.running"_failed:_Read-only_file_system_(30)

Sadly, I don't have a spare USB drive large enough to contain the backup, so I suppose I'll have to buy one on prospect and try it that way.  But, any ideas as to why it runs manually sometimes?

Offline cactus

  • *
  • 4,880
  • +3/-0
    • http://www.snetram.nl
Re: Affa failure
« Reply #11 on: December 23, 2008, 09:14:06 AM »
Please do as Charlie stated:

Please report all problems with contribs in the Contribs section of the bug tracker.

Locking this thread, please post you issue in the bugtracker and follow up (only) there.
Be careful whose advice you buy, but be patient with those who supply it. Advice is a form of nostalgia, dispensing it is a way of fishing the past from the disposal, wiping it off, painting over the ugly parts and recycling it for more than its worth ~ Baz Luhrmann - Everybody's Free (To Wear Sunscreen)