Koozali.org: home of the SME Server

[ANNOUNCEMENT] Turn SME7 into a dedicated backup server

Offline mmccarn

  • *
  • 2,626
  • +10/-0
[ANNOUNCEMENT] Turn SME7 into a dedicated backup server
« Reply #60 on: May 23, 2007, 05:43:38 PM »
Michael - Is there a better place than this ever-growing topic to ask questions?

I can't figure out what 'Disk Usage' indicates in affa --status - my one backup using a USB drive occupies 27GB of disk, but 'Disk Usage' says 27MB.  My other two backups (backing up to /var/affa) occupy ~1.4GB but report 'Disk Usage' of 3.1MB.

Another comment/suggestion:  On my first attempt to run 'affa-job.pl' I didn't notice the entry for $jobname at the top of the file.  I ended up with a config db entry with a blank  job name...  I couldn't figure out how to delete it using 'db affa ...' so I ended up deleting it manually.  Perhaps it would be useful to have an error exit for no job name (the way you do for 'remoteHostName is not set').

Yet another comment/suggestion: add a db entry for "Nervous" or "Chatty" - if "Nervous" is enabled, send emails on success as well as on failure; or perhaps if 'Nervous' is '5', send 5 'success' notices after startup or after any failure.  (I would use this during setup & testing, then fall back to your default mode of error notification only).

I am also working on a secure way to allow my client to access their backups using http, webdav or ftp, but I suspect you don't plan to build this into affa.

Thanks for all your work - I really enjoy being able to setup a backup solution in 30 minutes or less!  Feel free to tell me to keep my suggestions to myself...

Offline m

  • ****
  • 276
  • +0/-0
  • Peet
[ANNOUNCEMENT] Turn SME7 into a dedicated backup server
« Reply #61 on: May 23, 2007, 10:23:55 PM »
Quote from: "mmccarn"


I can't figure out what 'Disk Usage' indicates in affa --status - my one backup using a USB drive occupies 27GB of disk, but 'Disk Usage' says 27MB.  My other two backups (backing up to /var/affa) occupy ~1.4GB but report 'Disk Usage' of 3.1MB.

I have added some more info to the Wiki.
Are the Disk Usage'  values really MB, not GB? If so, it appears to be a bug. What values shows df -h?

Quote from: "mmccarn"

Another comment/suggestion:  On my first attempt to run 'affa-job.pl' I didn't notice the entry for $jobname at the top of the file.  I ended up with a config db entry with a blank  job name...  I couldn't figure out how to delete it using 'db affa ...' so I ended up deleting it manually.  Perhaps it would be useful to have an error exit for no job name (the way you do for 'remoteHostName is not set').

This has been already fixed in the /usr/lib/affa/jobconfig-sample.pl of the current release.

Quote from: "mmccarn"

Yet another comment/suggestion: add a db entry for "Nervous" or "Chatty" - if "Nervous" is enabled, send emails on success as well as on failure; or perhaps if 'Nervous' is '5', send 5 'success' notices after startup or after any failure.  (I would use this during setup & testing, then fall back to your default mode of error notification only).

Great idea. I think about it.

Quote from: "mmccarn"

I am also working on a secure way to allow my client to access their backups using http, webdav or ftp, but I suspect you don't plan to build this into affa.

No such plans.

Offline mmccarn

  • *
  • 2,626
  • +10/-0
[ANNOUNCEMENT] Turn SME7 into a dedicated backup server
« Reply #62 on: May 24, 2007, 05:18:10 AM »
Quote from: "mweinber"
This has been already fixed in the /usr/lib/affa/jobconfig-sample.pl of the current release.
Oops.  I was looking at the version I had modified from 0.3.2...

Quote from: "mweinber"
What values shows df -h

Code: [Select]
# affa --status
+----------+---------+--------+-------+--------+------------+----------------+
| Job      | Enabled | Last   | Next  |   Size | Disk usage | N of s,d,w,m,y |
+----------+---------+--------+-------+--------+------------+----------------+
| family   | yes     | 22:31  | 22:30 |  1.3GB |  3.4MB/11% | 2,7,1,0,0      |
| hcm      | yes     | running (pid 19892)     |   36MB/13% | 2,1,0,0,0      |
| sme-mmsi | yes     | 22:30  | 22:30 |  157MB |  3.4MB/11% | 2,7,1,0,0      |
+----------+---------+--------+-------+--------+------------+----------------+


Code: [Select]
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/main-root
                       33G  3.5G   28G  12% /
/dev/md1               99M   20M   75M  21% /boot
none                  125M     0  125M   0% /dev/shm
/dev/sdc1             294G   37G  243G  14% /home/e-smith/files/ibays/hcmbackup/html
Yes, I know I'm not supposed to use /home/e-smith as the RootDir for the backup, but I don't think this is my problem.  The remote data set is approx. 25GB, and as of now the backup has not finished successfully due to a scheduled daily reboot of the source computer (which has now been disabled).  Also NONE of the 3 configured backup jobs have SMEServer set to 'yes', so I'm not planning to use 'rise' at this time...

Quote from: "Affa Wiki"
If all jobs are located in the same RootDir, identical disk usage is shown for all jobs. If RootDir is /var/affa, the usage of the SME server system partition is shown
Clearly, "Disk Usage" does not necessarily represent 'Size' at all; I just don't understand exactly what it does represent.

Offline m

  • ****
  • 276
  • +0/-0
  • Peet
[ANNOUNCEMENT] Turn SME7 into a dedicated backup server
« Reply #63 on: May 24, 2007, 09:33:34 AM »
Quote from: "mmccarn"
Clearly, "Disk Usage" does not necessarily represent 'Size' at all; I just don't understand exactly what it does represent.


It simply shows the disk usage of the partition where the archives are located. This also includes all other data on this partition. When you run a decticated backup server it shows on the whole the size of all archives including the small space used by the SME system, which can be safely ignored.

Please update to the new release 0.4.1. The disk usage should now display correct values, according to the df command.

Offline mmccarn

  • *
  • 2,626
  • +10/-0
[ANNOUNCEMENT] Turn SME7 into a dedicated backup server
« Reply #64 on: May 24, 2007, 03:42:59 PM »
Quote from: "mweinber"
It simply shows the disk usage
Now I understand... and with 0.4.1 it's so obvious!

I tried to create a 'preJobCommand' and 'postJobCommand' to mount & unmount a samba share on my (remote) source system (without reading the wiki) and discovered that if the 'postJobCommand' fails it drops into a loop where it sends a failure message, then re-tries the command, then sends another failure message, etc, etc...  In case it matters, the postJobCommand I had configured using:
Code: [Select]
db affa setprop hcm postJobCommand "/usr/bin/ssh <remote.host.name> -p 2222 '/bin/umount //backup/backup'"Note: I'm not looking for help fixing the command; after reading the manual I'm good on that score.  I'm just trying to alert you to what happened when 'postJobCommand' returned an error...

Also, I just got the following error trying to do a manual backup:
Code: [Select]
[root@office ~]# affa --stat
Affa version 0.4.1 on office.obfuscated.local (192.168.200.1)
+----------+---------+--------+-------+--------+------------+----------------+
| Job      | Enabled | Last   | Next  |   Size | Disk usage | N of s,d,w,m,y |
+----------+---------+--------+-------+--------+------------+----------------+
| family   | yes     | 22:31  | 22:30 |  1.3GB |  3.5GB/11% | 2,7,1,0,0      |
| hcm      | yes     | failed | 15:30 |      ? |   38GB/13% | 1,2,0,0,0      |
| sme-mmsi | yes     | 22:30  | 22:30 |  157MB |  3.5GB/11% | 2,7,1,0,0      |
+----------+---------+--------+-------+--------+------------+----------------+
[root@office ~]# affa --backup hcm
Argument "rsync error: timeout in data send/receive (code 30) at i..." isn't numeric in numeric eq (==) at /sbin/e-smith/affa line 383.
Use of uninitialized value in substitution (s///) at /sbin/e-smith/affa line 1255.
Use of uninitialized value in division (/) at /sbin/e-smith/affa line 1256.
[root@office ~]# affa --stat
Affa version 0.4.1 on office.obfuscated.local (192.168.200.1)
+----------+---------+--------+-------+--------+------------+----------------+
| Job      | Enabled | Last   | Next  |   Size | Disk usage | N of s,d,w,m,y |
+----------+---------+--------+-------+--------+------------+----------------+
| family   | yes     | 22:31  | 22:30 |  1.3GB |  3.5GB/11% | 2,7,1,0,0      |
| hcm      | yes     | 08:50  | 15:30 |      ? |   38GB/13% | 1,2,0,0,0      |
| sme-mmsi | yes     | 22:30  | 22:30 |  157MB |  3.5GB/11% | 2,7,1,0,0      |
+----------+---------+--------+-------+--------+------------+----------------+

Offline m

  • ****
  • 276
  • +0/-0
  • Peet
[ANNOUNCEMENT] Turn SME7 into a dedicated backup server
« Reply #65 on: May 24, 2007, 07:57:24 PM »
Quote from: "mmccarn"


if the 'postJobCommand' fails it drops into a loop where it sends a failure message, then re-tries the command, then sends another failure message, etc, etc...  

ok, endless looping is really a bad behavior. I have to look for this.

Quote from: "mmccarn"

Argument "rsync error: timeout in data send/receive (code 30) at i..." isn't numeric in numeric eq (==) at /sbin/e-smith/affa line 383.

Rsync timed out for some reason and (should) return exitcode 30, but Affa saw something non-numeric. Can you read the exitstatus from the log?

Quote from: "mmccarn"

Use of uninitialized value in substitution (s///) at /sbin/e-smith/affa line 1255.
Use of uninitialized value in division (/) at /sbin/e-smith/affa line 1256.

These warnings indicate, that the df command failed for the filessystem where the job archives are located on. Any ideas why?
Regardless the reason, Affa should be able to deal with bad return values and needs to be fixed.

Offline SoftDux

  • ****
  • 240
  • +0/-0
    • http://www.SoftDux.com
[ANNOUNCEMENT] Turn SME7 into a dedicated backup server
« Reply #66 on: May 28, 2007, 06:44:47 PM »
how well does it work with dabases, like MySQL, PostgresSQL and Oracle?

Offline mmccarn

  • *
  • 2,626
  • +10/-0
[ANNOUNCEMENT] Turn SME7 into a dedicated backup server
« Reply #67 on: May 29, 2007, 01:56:58 PM »
Quote from: "SoftDux"
how well does it work with dabases, like MySQL, PostgresSQL and Oracle?
If the affa configuration for the remote host (the source) contains 'SMEserver' not equal to 'no', then affa runs 'signal-event pre-backup' on the source server.  On a SME box this includes a dump of the mysql databases.  If you're also using PostgresSQL & Oracle you'll need to generate a custom process to dump the databases before backup (I think).  If the 'source' is a SME server, you could customize the 'pre-backup' event; if the 'source' is non-SME you can create scripts to be run before & after the backup which can (using ssh) execute commands on the 'source'.  Check the manual for preJobCommand and postJobCommand, and look at the sample scripts in  /usr/lib/affa

Quote from: "mweinber"
Rsync timed out for some reason and (should) return exitcode 30, but Affa saw something non-numeric. Can you read the exitstatus from the log?
Error: rsync failed with status 255
Quote from: "mweinber"
the df command failed for the filessystem where the job archives are located
Possibly a timeout caused by trying to use lame USB 1.1?

Michael - I think I've been fighting two or three separate issues - one related to large files (>2GB) accessed using smbfs on the 'source' server, one related to slow backup disk access (USB 1.1) on my local backup device, and one related to "successfully" backing up an empty mount-point...

I finally found the "lfs" option for "mount -t smbfs" which provides "large file support" - this let me complete two manual 'rsync' runs with no problems.  I later changed the mount command on the source server to use "mount -t cifs", which seems to a) not have problems with large files and b) allow the backup to run faster.

I have also installed a USB 2.0 controller in my backup server...

I also, at one point, while playing with 'preJobCommand' and 'postJobCommand' created a successful backup of the empty mount point on the remote host.

Between these three problems, I became very confused about how the rsync --link-dest function worked.  In an effort to straighten things out I ran the affa rsync command manually, using the largest (most successful?) local backup as the --link-dest (daily.3) and changing the destination folder (scheduled.0, scheduled.1, daily.0) until I had a) run rsync successfully several times in a row, and b) had several successful local copies...  After that I changed the value for 'keepDaily' and ran 'affa --cleanup <job>' to remove the under-sized backups.  Now I've re-enabled the affa schedule and will see what happens.

Offline SoftDux

  • ****
  • 240
  • +0/-0
    • http://www.SoftDux.com
[ANNOUNCEMENT] Turn SME7 into a dedicated backup server
« Reply #68 on: May 29, 2007, 02:57:21 PM »
Ok, if you say you have issues with large files. How large are the files you are working with?

And how does it perform on USB 2.0?

Offline m

  • ****
  • 276
  • +0/-0
  • Peet
[ANNOUNCEMENT] Turn SME7 into a dedicated backup server
« Reply #69 on: May 29, 2007, 04:37:28 PM »
Quote from: "SoftDux"
how well does it work with dabases, like MySQL, PostgresSQL and Oracle?


Affa does exactly that, what a backup to tape or desktop does (on the source server).

Offline m

  • ****
  • 276
  • +0/-0
  • Peet
[ANNOUNCEMENT] Turn SME7 into a dedicated backup server
« Reply #70 on: May 29, 2007, 04:51:41 PM »
Quote from: "mmccarn"

Between these three problems, I became very confused about how the rsync --link-dest function worked.


when a job starts, affa first rename the scheduled.0 (if it exists and is complete)  to scheduled.1 and then it runs a rsync with link-dest against the scheduled.1 to create a new scheduled.0. All files present in scheduled.1 and unmodified on the source are just hardlinked from scheduled.1 to the scheduled.0. Only modified files are transferred from the source (of course only the chunks, that differs).
If you have accidentally lost scheduled.0 and scheduled.1 a full tranfer from the source will be done. If you still have an older archive e.g. scheduled.2 or daily.0, you can copy it to scheduled.0 using 'cp -al' The -l option for hardlinks is important!

Offline m

  • ****
  • 276
  • +0/-0
  • Peet
Release 0.4.2
« Reply #71 on: May 29, 2007, 05:20:44 PM »
A new release 0.4.2 with some bug fixes (Changelog) and new property 'chattyOnSuccess ' is available.

Offline mmccarn

  • *
  • 2,626
  • +10/-0
[ANNOUNCEMENT] Turn SME7 into a dedicated backup server
« Reply #72 on: May 29, 2007, 05:46:17 PM »
Quote from: "SoftDux"
Ok, if you say you have issues with large files. How large are the files you are working with?
The problem I had with large files was not related to affa or rsync, but stemmed instead from a known issue with samba.  Samba is involved only because I'm using affa to backup a remote Windows system that is only accessible via samba at the remote end.  Adding the "lfs" option to the 'mount' command on my source computer solved the large file issue.  Changing from 'mount -t smbfs...' to 'mount -t cifs...' on the source system solved the file size issue and made the backup run faster.

The specific file that consistently trashed my 'rsync...' is 3.6GB.

Quote from: "SoftDux"
And how does it perform on USB 2.0?
40 times better, of course...  

I have had no rsync errors since upgrading my USB target device from 1.1 to 2.0 and changing the source mount command from '-t smbfs' to '-t cifs'.  I have not run a scheduled affa backup since then, but now that rsync works properly I expect affa to work properly, too.

To clarify: None of my problems were due to affa; I only reported them here in case Michael Weinberger sees something in my fumbling that he wants to address in affa's error handling...

Offline mmccarn

  • *
  • 2,626
  • +10/-0
[ANNOUNCEMENT] Turn SME7 into a dedicated backup server
« Reply #73 on: May 30, 2007, 03:18:33 PM »
A question about .AFFA-REPORT:
The last 3 lines of .AFFA-REPORT say:
Code: [Select]
RootDirFilesystemUsed: 54360144 kbytes
RootDirFilesystemAvail: 237667596 kbytes
RootDirFilesystemUsage: 22.9 %

This is generated (I believe) from this invocation of 'df' in /sbin/e-smith/affa:
Code: [Select]
   390                 (my $fs_type, my $fs_desc, my $used, my $avail) = df $job{'RootDir'};
    391                 print AS "Date: " . Date::Format::time2str("%Y%m%d%H%M",time()) ."\n";
    392                 print AS "$ExecCmdOutout";
    393                 print AS "Exit status: $status\n";
    394                 # Versions <= 0.4.0 printed kbytes values but bytes unit
    395                 printf( AS "RootDirFilesystemUsed: %d kbytes\n", $used );
    396                 printf( AS "RootDirFilesystemAvail: %d kbytes\n", $avail );
    397                 printf( AS "RootDirFilesystemUsage: %.1f %%\n", $used/$avail*100 );


I suspect 'RootDirFilesystemUsage' should really be '$used/$total*100' instead of '$used/$avail*100' (Of course you'd have to create & set $total; I have no idea how to do that...)

'$used/$avail' will over-report usage -- when the drive is 50% full, $used & $avail will be equal and .AFFA-REPORT will report 100% usage...

Offline m

  • ****
  • 276
  • +0/-0
  • Peet
[ANNOUNCEMENT] Turn SME7 into a dedicated backup server
« Reply #74 on: May 30, 2007, 09:11:56 PM »
Quote from: "mmccarn"


I suspect 'RootDirFilesystemUsage' should really be '$used/$total*100' instead of '$used/$avail*100' (Of course you'd have to create & set $total; I have no idea how to do that...)

'$used/$avail' will over-report usage -- when the drive is 50% full, $used & $avail will be equal and .AFFA-REPORT will report 100% usage...


a bug. Correct calculation is: $used/($avail+$used)*100
But not a serios issue. RootDirFilesystemUsage in .AFFA-REPORT is just for information (or future use) and not used yet.
Thanks for the report.