Koozali.org: home of the SME Server

Affa - rsync errors

Offline jonash

  • ***
  • 120
  • +0/-0
Affa - rsync errors
« on: November 22, 2012, 09:22:00 PM »
Hi,
I have used affa as my backup solution on SME7 for "many" years. Some moths ago, I upgraded to SME8, and re-installed affa, but I cant get it to work...
Code: [Select]
[root@fs ~]# db affa show
AffaGlobalDisable=no
DefaultAffaConfig=default
    sendStatus=weekly
    status=enabled
fs=job
    AutoUnmount=yes
    AutomountDevice=/dev/sda1
    AutomountOptions=
    AutomountPoint=/media/USB
    ConnectionCheckTimeout=120
    Debug=no
    Description=
    DiskSpaceWarn=strict
    EmailAddresses=
    Exclude[0]=
    Include[0]=
    RPMCheck=no
    RetryAfter=600
    RetryAttempts=3
    RetryNotification=yes
    RootDir=/media/USB
    SMEServer=yes
    SambaShare=yes
    TimeSchedule=2230
    Watchdog=yes
    chattyOnSuccess=1
    dailyKeep=7
    doneDaily=2012284
    doneMonthly=201210
    doneWeekly=201241
    doneYearly=2012
    localNice=0
    monthlyKeep=12
    postJobCommand=
    preJobCommand=
    remoteHostName=fs.mimer.gotdns.org
    remoteNice=0
    rsync--inplace=yes
    rsync--modify-window=0
    rsyncCompress=no
    rsyncTimeout=900
    scheduledKeep=1
    sshPort=22
    status=enabled
    weeklyKeep=4
    yearlyKeep=2

When I run the command these processes run:
Code: [Select]
oot     31897 31810  0 21:06 pts/0    00:00:00 sh -c /usr/bin/rsync --archive --stats --delete-during --ignore-errors --delete-excluded --relative --partial   --inplace --timeout=900  --numeric-ids --rsync-path='/usr/bin/rsync'  --link-dest='/media/USB/fs/daily.0'    "/home/e-smith" "/etc/e-smith/templates-custom" "/etc/e-smith/templates-user-custom" "/etc/ssh" "/root" "/etc/sudoers" "/etc/passwd" "/etc/shadow" "/etc/group" "/etc/gshadow" "/etc/samba/secrets.tdb" "/etc/samba/smbpasswd" /media/USB/fs/scheduled.running/ 2>&1; echo ${PIPESTATUS}
root     31898 31897  1 21:06 pts/0    00:00:11 /usr/bin/rsync --archive --stats --delete-during --ignore-errors --delete-excluded --relative --partial --inplace --timeout=900 --numeric-ids --rsync-path=/usr/bin/rsync --link-dest=/media/USB/fs/daily.0 /home/e-smith /etc/e-smith/templates-custom /etc/e-smith/templates-user-custom /etc/ssh /root /etc/sudoers /etc/passwd /etc/shadow /etc/group /etc/gshadow /etc/samba/secrets.tdb /etc/samba/smbpasswd /media/USB/fs/scheduled.running/
root     31899 31898  1 21:06 pts/0    00:00:11 /usr/bin/rsync --archive --stats --delete-during --ignore-errors --delete-excluded --relative --partial --inplace --timeout=900 --numeric-ids --rsync-path=/usr/bin/rsync --link-dest=/media/USB/fs/daily.0 /home/e-smith /etc/e-smith/templates-custom /etc/e-smith/templates-user-custom /etc/ssh /root /etc/sudoers /etc/passwd /etc/shadow /etc/group /etc/gshadow /etc/samba/secrets.tdb /etc/samba/smbpasswd /media/USB/fs/scheduled.running/
root     31900 31899  0 21:06 pts/0    00:00:06 /usr/bin/rsync --archive --stats --delete-during --ignore-errors --delete-excluded --relative --partial --inplace --timeout=900 --numeric-ids --rsync-path=/usr/bin/rsync --link-dest=/media/USB/fs/daily.0 /home/e-smith /etc/e-smith/templates-custom /etc/e-smith/templates-user-custom /etc/ssh /root /etc/sudoers /etc/passwd /etc/shadow /etc/group /etc/gshadow /etc/samba/secrets.tdb /etc/samba/smbpasswd /media/USB/fs/scheduled.running/

But after some time I get a mail:
Code: [Select]
Excerpt from log file /var/log/affa/fs.log:
Affa_2.0.0-rc4:_Running_/sbin/e-smith/affa_--watchdog=86400_--run_fs
Job_configuration:
__AutoUnmount=yes
__AutomountDevice=/dev/sda1
__AutomountPoint=/media/USB
__ChunkSize=921600
__ChunkThresholdSize=2
__ConnectionCheckTimeout=120
__Debug=no
__DiskSpaceWarn=strict
__EmailAddresses=admin
__RPMCheck=no
__RetryAfter=600
__RetryAttempts=3
__RetryNotification=yes
__RootDir=/media/USB
__SMEServer=yes
__SambaShare=yes
__TimeSchedule=2230
__Watchdog=yes
__chattyOnSuccess=1
__dailyKeep=7
__doneDaily=2012284
__doneMonthly=201210
__doneWeekly=201241
__doneYearly=2012
__monthlyKeep=12
__remoteHostName=localhost
__remoteOS=centos
__rsync--inplace=yes
__rsyncCompress=no
__rsyncTimeout=900
__rsyncdMode=no
__rsyncdModule=AFFA
__rsyncdPassword=<not_shown>
__rsyncdUser=affa
__scheduledKeep=1
__sendStatus=weekly
__sshPort=22
__status=enabled
__type=job
__weeklyKeep=4
__yearlyKeep=2
##################
#_Affa_2.0.0-rc4_#
##################
Starting_job_fs_scheduled_(localhost)
Description:_
Check_mounted:_/dev/sda1_/media/USB._Result:_no_
Mounting_/dev/sda1_to_/media/USB
Exec_Out:_exitstatus=0
Using_link_destination_daily.0
signaling_pre-backup_event_on_localhost
Exec_Out:_exitstatus=0
Running_rsync...
Exec_Out:_rsync:_writefd_unbuffered_failed_to_write_4092_bytes_to_socket_[sender]:_Broken_pipe_(32)
Exec_Out:_[sender]_io_timeout_after_30_seconds_--_exiting
Exec_Out:_rsync_error:_timeout_in_data_send/receive_(code_30)_at_io.c(140)_[sender=3.0.9]
Exec_Out:_30
Exec_Out:_rsync:_writefd_unbuffered_failed_to_write_7_bytes_to_socket_[generator]:_Broken_pipe_(32)
Exec_Out:_exitstatus=rsync_error:_received_SIGUSR1_(code_19)_at_main.c(1298)_[generator=3.0.9]
signaling_post-backup_event_on_localhost
Exec_Out:_exitstatus=0
Error_603_in_'main':_rsync_failed_with_status_rsync_error:_received_SIGUSR1_(code_19)_at_main.c(1298)_[generator=3.0.9].
Check_mounted:_/dev/sda1_/media/USB._Result:_yes
Unmounting_/media/USB
Exec_Out:_exitstatus=0
Starting_re-run_1_of_3_in_the_background.
Total_execution_time:__0h19m17s

I have search on different places, but I cant find anything.. Maybe my USB disk is to slow for writing, so the rsync command stops because of that? Maybe alter rsyncTimeout=900 to something else?

It does not create any files/directories on the USB at all.

Jonas
« Last Edit: November 22, 2012, 09:25:31 PM by jonash »
.........

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: Affa - rsync errors
« Reply #1 on: November 22, 2012, 10:02:08 PM »
searching for "_received_SIGUSR1_(code_19)_at_main.c(1298)_[generator=3.0.9" with google I see many recent post.. and many distros affected.. maybe a bug?

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: Affa - rsync errors
« Reply #2 on: November 23, 2012, 02:32:10 AM »
jonash

Quote
I have used affa as my backup solution on SME7 for "many" years. Some moths ago, I upgraded to SME8, and re-installed affa, but I cant get it to work...
    AutomountDevice=/dev/sda1
    AutomountPoint=/media/USB
    remoteHostName=fs.mimer.gotdns.org
It does not create any files/directories on the USB at all.


Your settings & comments imply you are backing up to a locally connected USB drive from your local server, but you have also specified a remote location server hostname as the source to backup from

So assuming you do actually want to backup your local sme server to a locally connected USB drive, then change the setting in /root/fs.pl to the following
remoteHostName=localhost

then run
/root/fs.pl
and it should then work.

I have that version of Affa (Affa_2.0.0-rc4) working OK on a local sme8 backing up to a locally connected USB, so the contrib is inherently OK (except for other unknown or alluded to bugs occasionally commented upon in the forums which I think are maybe user error).

After you have resolved this issue, you might want to try the newer version 3 which is being maintained (although the contrib version is not the cause of your current problem).
From
http://wiki.contribs.org/Affa
it links you to
http://affa.sourceforge.net/
« Last Edit: November 23, 2012, 02:35:30 AM by mary »
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline jonash

  • ***
  • 120
  • +0/-0
Re: Affa - rsync errors
« Reply #3 on: November 24, 2012, 11:57:41 AM »
Hi,
Thanks for the suggestions, but there error remains.
Code: [Select]
Excerpt from log file /var/log/affa/fs.log:
Affa_2.0.0-rc4:_Running_/sbin/e-smith/affa_--watchdog=86400_--run_fs
Job_configuration:
__AutoUnmount=yes
__AutomountDevice=/dev/sda1
__AutomountPoint=/media/USB
__ChunkSize=921600
__ChunkThresholdSize=2
__ConnectionCheckTimeout=120
__Debug=no
__DiskSpaceWarn=strict
__EmailAddresses=admin
__RPMCheck=no
__RetryAfter=600
__RetryAttempts=3
__RetryNotification=yes
__RootDir=/media/USB
__SMEServer=yes
__SambaShare=yes
__TimeSchedule=2230
__Watchdog=yes
__chattyOnSuccess=1
__dailyKeep=7
__doneDaily=2012284
__doneMonthly=201210
__doneWeekly=201241
__doneYearly=2012
__monthlyKeep=12
__remoteHostName=localhost
__remoteOS=centos
__rsync--inplace=yes
__rsyncCompress=no
__rsyncTimeout=900
__rsyncdMode=no
__rsyncdModule=AFFA
__rsyncdPassword=<not_shown>
__rsyncdUser=affa
__scheduledKeep=1
__sendStatus=weekly
__sshPort=22
__status=enabled
__type=job
__weeklyKeep=4
__yearlyKeep=2
##################
#_Affa_2.0.0-rc4_#
##################
Starting_job_fs_scheduled_(localhost)
Description:_
Check_mounted:_/dev/sda1_/media/USB._Result:_no_
Mounting_/dev/sda1_to_/media/USB
Exec_Out:_exitstatus=0
Using_link_destination_daily.0
signaling_pre-backup_event_on_localhost
Exec_Out:_exitstatus=0
Running_rsync...
Exec_Out:_[sender]_io_timeout_after_900_seconds_--_exiting
Exec_Out:_rsync_error:_timeout_in_data_send/receive_(code_30)_at_io.c(140)_[sender=3.0.9]
Exec_Out:_30
Exec_Out:_rsync:_writefd_unbuffered_failed_to_write_7_bytes_to_socket_[generator]:_Broken_pipe_(32)
Exec_Out:_exitstatus=rsync_error:_received_SIGUSR1_(code_19)_at_main.c(1298)_[generator=3.0.9]
signaling_post-backup_event_on_localhost
Exec_Out:_exitstatus=0
Error_603_in_'main':_rsync_failed_with_status_rsync_error:_received_SIGUSR1_(code_19)_at_main.c(1298)_[generator=3.0.9].
Check_mounted:_/dev/sda1_/media/USB._Result:_yes
Unmounting_/media/USB
Exec_Out:_exitstatus=0
Starting_re-run_1_of_3_in_the_background.
Total_execution_time:__0h17m33s

Stefano, I also found these bug reports, but those I found were old...

I will try to find a non USB-disk which I can mount on the SME server to see if it works.

Jonas
.........

Offline jonash

  • ***
  • 120
  • +0/-0
Re: Affa - rsync errors
« Reply #4 on: November 24, 2012, 12:45:56 PM »
Now, I think I found the problem..

In my backup dir, there were a directory scheduled.running, which probably was from an old cancelled run. I removed it, and now it starts to write in a new scheduled.running directory.  I dont know if rsync compared the new files with the old files in scheduled.running, and got a time out, or if it was waiting for something to happen..

Thanks for all inputs.

Jonas
.........

Offline jonash

  • ***
  • 120
  • +0/-0
Re: Affa - rsync errors
« Reply #5 on: November 24, 2012, 10:29:05 PM »
Nope.. still fails after some hours. Maybe there is a corrupt file or diskblock somewhere. I will try to make rsync log more to see if I can find something.
.........

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: Affa - rsync errors
« Reply #6 on: November 25, 2012, 07:00:12 AM »
jonash

Quote
In my backup dir, there were a directory scheduled.running,

Where exactly did you see this backup dir, on the USB disk ? or on the hard disk (mount point) of sme server ?


Quote
....which probably was from an old cancelled run

It is from an unsuccessful or incompleted backup. The next time the backup job runs, it will recommence that scheduled backup from the point it was up to.
If the backup was already half done (say yesterday), and then stopped (for whatever reason) then only the second half of the backup will need to be done (say today).

The problem for you is that the backup seems to repeatedly stop each time it runs, for some reason yet to be determined.


Can you please confirm what you are trying to backup?
Is it a local sme server to a locally connected USB disk, and there is no backing up of workstations or remote sme server involved ?
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline jonash

  • ***
  • 120
  • +0/-0
Re: Affa - rsync errors
« Reply #7 on: November 25, 2012, 11:49:02 AM »
Quote
Where exactly did you see this backup dir, on the USB disk ? or on the hard disk (mount point) of sme server ?
The backup dir was on the USB disk.

I backup the whole SME-server to a local USB disk, mounted on /media/USB

When I start affa with the command from crontab: /sbin/e-smith/affa --watchdog=86400 --run fs these rsync processes start.
 
 
Code: [Select]
root      3762  3706  0 11:45 pts/0    00:00:00 sh -c /usr/bin/rsync --archive --stats --delete-during --ignore-errors --delete-excluded --relative --partial   --inplace --timeout=1900  --numeric-ids --progress --rsync-path='/usr/bin/rsync'  --link-dest='/media/USB/fs/daily.0'    "/home/e-smith" "/etc/e-smith/templates-custom" "/etc/e-smith/templates-user-custom" "/etc/ssh" "/root" "/etc/sudoers" "/etc/passwd" "/etc/shadow" "/etc/group" "/etc/gshadow" "/etc/samba/secrets.tdb" "/etc/samba/smbpasswd" /media/USB/fs/scheduled.running/ 2>&1; echo ${PIPESTATUS}
root      3763  3762  4 11:45 pts/0    00:00:02 /usr/bin/rsync --archive --stats --delete-during --ignore-errors --delete-excluded --relative --partial --inplace --timeout=1900 --numeric-ids --progress --rsync-path=/usr/bin/rsync --link-dest=/media/USB/fs/daily.0 /home/e-smith /etc/e-smith/templates-custom /etc/e-smith/templates-user-custom /etc/ssh /root /etc/sudoers /etc/passwd /etc/shadow /etc/group /etc/gshadow /etc/samba/secrets.tdb /etc/samba/smbpasswd /media/USB/fs/scheduled.running/
root      3764  3763  7 11:45 pts/0    00:00:03 /usr/bin/rsync --archive --stats --delete-during --ignore-errors --delete-excluded --relative --partial --inplace --timeout=1900 --numeric-ids --progress --rsync-path=/usr/bin/rsync --link-dest=/media/USB/fs/daily.0 /home/e-smith /etc/e-smith/templates-custom /etc/e-smith/templates-user-custom /etc/ssh /root /etc/sudoers /etc/passwd /etc/shadow /etc/group /etc/gshadow /etc/samba/secrets.tdb /etc/samba/smbpasswd /media/USB/fs/scheduled.running/
root      3765  3764  0 11:45 pts/0    00:00:00 /usr/bin/rsync --archive --stats --delete-during --ignore-errors --delete-excluded --relative --partial --inplace --timeout=1900 --numeric-ids --progress --rsync-path=/usr/bin/rsync --link-dest=/media/USB/fs/daily.0 /home/e-smith /etc/e-smith/templates-custom /etc/e-smith/templates-user-custom /etc/ssh /root /etc/sudoers /etc/passwd /etc/shadow /etc/group /etc/gshadow /etc/samba/secrets.tdb /etc/samba/smbpasswd /media/USB/fs/scheduled.running/

No other backups are involved.

The status from today.. (Is has worked before  :-? )

Code: [Select]
Archive lists of all jobs:
Affa version 2.0.0-rc4 on fs.mimer.gotdns.org (192.168.1.1)
+------------------------------------------------------------------------------+
| Job: fs                                                                      |
| Directory: /media/USB/fs/                                                    |
| Hostname: localhost                                                          |
| AutomountDevice: /dev/sda1                                                   |
| AutomountPoint: /media/USB                                                   |
| AutoUnmount: yes                                                             |
| Email: admin                                                                 |
+-------+-----------------------+-----------+----------------+--------+--------+
| Run   | Completion date       | Exec Time |          Files |   Size |  Recvd |
+-------+-----------------------+-----------+----------------+--------+--------+
| W 1   | Sun 2012 Sep 23 22:33 |  0h03m41s |         192127 |  202GB |  8.3kB |
| W 0   | Sun 2012 Sep 30 22:36 |  0h06m07s |         192339 |  202GB |  7.8kB |
+-------+-----------------------+-----------+----------------+--------+--------+
| D 6   | Wed 2012 Oct 03 22:58 |  0h28m20s |         190482 |  197GB |   15kB |
| D 5   | Thu 2012 Oct 04 23:07 |  0h37m04s |         191105 |  198GB |   24kB |
| D 4   | Fri 2012 Oct 05 22:39 |  0h09m37s |         192238 |  198GB |   29kB |
| D 3   | Sat 2012 Oct 06 22:36 |  0h06m11s |         192288 |  198GB |  7.8kB |
| D 2   | Sun 2012 Oct 07 22:36 |  0h05m59s |         192354 |  198GB |  9.4kB |
| D 1   | Mon 2012 Oct 08 22:34 |  0h04m15s |         192401 |  198GB |  8.4kB |
| D 0   | Tue 2012 Oct 09 22:36 |  0h06m00s |         192462 |  198GB |  8.4kB |
+-------+-----------------------+-----------+----------------+--------+--------+

generated on Sun Nov 25 05:00:03 2012


Jonas
« Last Edit: November 25, 2012, 11:52:54 AM by jonash »
.........

Offline jonash

  • ***
  • 120
  • +0/-0
Re: Affa - rsync errors
« Reply #8 on: November 25, 2012, 12:25:15 PM »
I added the flag --progress to rsync and got this:
Code: [Select]
Exec_Out:_/home/e-smith/files/users/marie/Maildir/cur/1353794835.15285.fs:2,S
Exec_Out:_________4646_100%____0.00kB/s____0:00:00
________4646_100%____0.00kB/s____0:00:00_(xfer#50,_to-check=1109/178182)
Exec_Out:_/home/e-smith/files/users/marie/Maildir/new/
Exec_Out:_/home/e-smith/files/users/marie/Maildir/tmp/
Exec_Out:_/home/e-smith/files/users/marie/home/PC_Avaio/Från_Mina_dokument/Mina_bilder/Kort_-_Sommar_2010/100MSDCF/
Exec_Out:_[sender]_io_timeout_after_1920_seconds_--_exiting
Exec_Out:_rsync_error:_timeout_in_data_send/receive_(code_30)_at_io.c(140)_[sender=3.0.9]
Exec_Out:_rsync:_writefd_unbuffered_failed_to_write_957_bytes_to_socket_[generator]:_Broken_pipe_(32)
Exec_Out:_30
Exec_Out:_exitstatus=rsync_error:_received_SIGUSR1_(code_19)_at_main.c(1298)_[generator=3.0.9]
signaling_post-backup_event_on_localhost
Exec_Out:_exitstatus=0
Error_604_in_'main':_rsync_failed_with_status_rsync_error:_received_SIGUSR1_(code_19)_at_main.c(1298)_[generator=3.0.9].
Starting_re-run_1_of_3_in_the_background.
Total_execution_time:__0h34m41s
.........

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: Affa - rsync errors
« Reply #9 on: November 25, 2012, 05:04:34 PM »
jonash

Maybe there is something odd with files in that folder with the unusual character ???

I would disable affa temporarily and manually mount the USB drive and then instigate a manual rsync command that backs up /home/e-smith to the USB. The content there should be sufficiently large so the backup will take some time to complete, thus testing the rsync process over a long time period. Use a minimal set of switches.
man rsync
The success or otherwise of a manual limited backup may give clues as to where the problem lies and identify whether affa is part of the problem or not.

I'd also try a different USB port & cable, a different USB drive and check your main sme servers drive(s) for errors with the manufacturers diagnostic software. Also check the USB drive.
« Last Edit: November 26, 2012, 01:17:44 AM by mary »
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline jonash

  • ***
  • 120
  • +0/-0
Re: Affa - rsync errors
« Reply #10 on: November 25, 2012, 09:06:07 PM »
Hi,
I suddenly noticed that the rsync command do copy some files, before it stops.
I added the flag --progress to find out what happens during the run.
I just re-ran it a couple of times, and suddenly I got the mail with the subject: Success: 'fs scheduled' completed

There were a lot of new image files on the source file system, so it looks like there is some timing issue; slow USB disk maybe, which causes the error.
The backup will run tonight again, so lets hope for the best.
.........

Offline elmarconi

  • ***
  • 139
  • +0/-0
Re: Affa - rsync errors
« Reply #11 on: November 26, 2012, 11:59:42 PM »
I had something similar, turned out to be a huge movie MP4 larger than 2 GB, syncing with rsync over NFS mounted ext2 disk.
To the local USBdisk it worked fine though...


...

Offline jonash

  • ***
  • 120
  • +0/-0
Re: Affa - rsync errors
« Reply #12 on: November 27, 2012, 04:38:38 PM »
Hi,
I just want to say that the backup works.. until someone puts up a lot of big files again :)
Thank you Mary for your help!

Jonas
.........

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: Affa - rsync errors
« Reply #13 on: November 28, 2012, 01:25:44 AM »
jonash

Try using the standard default backup to USB (in server manager backup or restore panel).
It works fine, so see if it handles those big files any better than affa rsync, or is more tolerant of speed issues.
Select Configure workstation backup (rather than desktop) and then select local removable disk

Apart from that there may be issues with your hardware.
Have you checked that the USB port is fast ie v2 or better & tested it to prove data transfer rate ?
Have you tested your drives, main & USB ?
http://wiki.contribs.org/Monitor_Disk_Health
« Last Edit: November 28, 2012, 01:38:58 AM by mary »
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.