Koozali.org: home of the SME Server

backup2

Offline dmay

  • *
  • 450
  • +0/-0
    • http://myezserver.com
backup2
« on: August 15, 2006, 10:39:16 PM »
ALPHA RELEASE: Use with caution. Highly experimental!!!

If you are not familiar with this contrib, please do not bother installing this alpha release. If you are familiar and have a test server available, please test and report your results. General comments to the forum are fine. Bugs to BugZilla please for tracking.

http://mirror.contribs.org/smeserver/contribs/dmay/smeserver/7.x/alpha
smeserver-backup2-1.0.0-a1dmay.noarch.rpm

Complete FormMagick rewrite.

Darrell

Offline Tib

  • *
  • 571
  • +0/-0
    • http://www.tibors.net
backup2
« Reply #1 on: August 16, 2006, 12:42:26 PM »
Nice ... I did a simple backup about 200 odd meg and it went well on my test machine ... I'll try a more indepth backup tomorrow hopefully.

I just backed up a users complete home dirve to a windows share.

I haven't tried a restore yet so I can't comment on that yet either.

Regards,


Tib

icpix

backup2
« Reply #2 on: August 18, 2006, 02:52:47 PM »
Darrell----

Ref: smeserver-backup2-1.0.0-a2dmay.noarch.rpm

Trying it out on a live data SME7 fileserver (not public).
Updated over the top of existing Backup2 (e-smith version).
Inherited my old set of jobs (a very comprehensive set!) but
did not use them, other than to copy/paste the source path
into the FOREVER TOO SMALL form box (hint) in the alpha.

Have tried not to stress the alpha too much except to run
concurrent ie multiple realtime manually invoked jobs.
I haven't yet tried any jobs using the timer settings.
All to the W2kPro SP4 (workstation) work perfectly;~)
All to the MSS (a NAS type of box) don't... as you intimated.

Note: production SME6 using Backup2 (e-smith version) has
always successfully completed its own concurrent multiple
jobs to the MSS thing on its own overnight timered runs.

The MSS thing is a networked attached Maxtor Shared Storage
300GB box but unfortunately Maxtor spoiled a really good piece
of kit by inflicting a pointless 10/100Mbps interface on it instead
of a more real world gigabit interface;~/ Its proprietory setup
includes having no authentication as it's on my intranet. Though
I don't think my old MSS is of the new flavour ie a 'Plus' item:
http://www.maxtorsolutions.com/en/catalog/MSS_Plus/
Anyway all concurrent jobs to the MSS end up as 'aborted'.
Seems to mess up SME7's /mnt/ area too... Other transfers
ie the good ones to the workstation appear to proceed slowly.
Attempting to manually delete the stuck mount in /mnt/ seems
to do the trick - transfers then get completed faster.

Intranet is an unmanaged gigabit switch (3com), I'm the only user.

Production SME6 (awaiting some courage to move to SME7).

Much prefer your new clean interface form;~)

Mystified by the 'explanatory' help note about the Disaster slot...
Think I know what you mean or mean to say but it's not clear.

The entry box on the form for the source path. It's always been
too small for the endless /home/e-smith... path mantra out to all
the iBays. When I then add the hierarchy of the 2TB RAID5 stuff
on to an iBay the deficit is somewhat risible. It might fit in if the
data slot for the source path were to be twice as long... (hint)

Typo: No compression dramitcally improves backup speed.
No compression dramatically improves backup speed.

What did you do to get it all working more quickly? I think  
that Backup2 (the smeserver version not the e-smith version)
is working more quickly ie transfers complete in shorter periods.

Heartfelt thanks for the mod giving user choice of job names;~)

Are the logs rotated and so forth?

Could each job line on the interface include a URL to 'its' log?

Any chance that Backup2 might include some incremental or
otherwise rdiff sort of functionality? My site suffers massive hits
appropriate to the unnecessary duplication in day-to-day backups.

----best wishes, Robert

Offline dmay

  • *
  • 450
  • +0/-0
    • http://myezserver.com
backup2
« Reply #3 on: August 18, 2006, 06:53:19 PM »
Tib/Robert

Thanks for the feedback. Will try to incorporate a few of your suggestions into future builds.

Any difference experienced in accessing your MSS box would be samba related. Different samba version between 6.x/7.x. Check with Maxtor for samba 3.x issues.

Logs are not rotated. They are overwritten with each job run.

Darrell

icpix

backup2
« Reply #4 on: August 20, 2006, 02:18:52 AM »
Darrell----

Backup2 isn't the only thing on my SME7 whose operation exceeds its own
logistical window (daily backups take longer than a day). ClamAV cannot
complete its daily scan in a day (nowhere near). Despite appearing to have
successfully pruned ClamAV's scope...
Code: [Select]

[query scope allowed to ClamAV]
db configuration getprop clamav FilesystemScanExclude

Code: [Select]

[set apparent default scope allowed to ClamAV]
db configuration setprop clamav FilesystemScanExclude /proc,/sys,/usr/share,/var

...I now realise that the highly prolonged ClamAV activity is
due to it running through /mnt/ during the Backup2 runs;~/
I think it may also been a factor in some failed Backup2 runs.
Have now disassociated ClamAV activity from /mnt/...
Code: [Select]

[set own scope allowed to ClamAV]
db configuration setprop clamav FilesystemScanExclude /proc,/sys,/usr/share,/var,/mnt,/home/e-smith/files/ibays/storage

...where the 'storage' iBay area is the mount point of the 2TB array.

During the sleuthing I found it really difficult to either stop the back up
activities or to properly determine whatever backup run was active.
While Backup2 is 'doing its thing' could there be an activity indicator?
(text colour change in the JobName column or a blob or something)

On the subject of stopping the backup activities (pause? abort?) I found
it awkward properly 'clearing up' the 'lost' mounts in the /mnt/ area.
Could Backup2 ROBUSTLY clear up any lost mounts (due to aborts etc)?

----best wishes, Robert

Offline raem

  • *
  • 3,972
  • +4/-0
backup2
« Reply #5 on: August 20, 2006, 10:20:11 AM »
icpix

>...I found it really difficult to either stop the back up
> activities or to properly determine whatever backup run was active.

top -i
will show what's running

kill pidnumber
will kill the process
...

icpix

backup2
« Reply #6 on: August 20, 2006, 12:04:42 PM »
Ray----
Thank you that should be useful.
I usually do have <top -s i> running on both servers (prod/SME6
filelib/SME7) but it's pretty much impossible to determine which
back up job is whatever PID.

Darrell---
FWIW the two concurrently launched (manually) runs completed
successfully overnight after having persuaded ClamAV not to check
the /mnt/ path. The two concurrent runs, 400CANON (58GB/20k files)
and 500CANON (90GB/22k files), were done in about 7hrs. 150GB in
7hrs over an unloaded Gigabit intranet to a Maxtor 'OneTouch series 1'
250GB (external FireWire) - not the problematic (Samba?) network
attached Maxtor MSS box (NAS/100Mbps interface). The resulting maths
(very approximately 1.5TB in 70hrs?) highlights the unsustainable daily
back up window logistics;~/ It doesn't seem to be the intranet or the
target drives but the actual back up mechanism that is the bottleneck.
BTW no compression has been involved, those CANON file directories
hold compressed files in any case, though I do have plenty of horsepower
(dual 3.6GHz Xeons) and memory (4GB) too if only Backup2 could usefully
be configured to so use it advantageously.
----best wishes, Robert

Offline dmay

  • *
  • 450
  • +0/-0
    • http://myezserver.com
backup2
« Reply #7 on: August 20, 2006, 05:25:45 PM »
I would suggest you create a bug report for ClamAV recommending /mnt be a default exclusion.

As for speed, try a simple test. Create a mount point and simply copy a large dir tree from the command line. I expect the speed to be the same as a backup2 job of the same source and the limiting factor being your network and external device throughput.

Darrell

icpix

backup2
« Reply #8 on: August 20, 2006, 07:24:39 PM »
Darrell----

Noted.
http://bugs.contribs.org/show_bug.cgi?id=1867
(default exclusion for ClamAV)

As for the speed test... I hear you and understand the need but
I just don't know how to do as you have suggested;~/ Fill me in
and I'll willingly go for it.

Empiracal test done the other way around (least I know how to
do it that way) - from same drive on w2kpro workstation to the
hardware RAID5 array (it's mounted in fstab) and transferring
the Backup2 .RAR files originally deposited for the back up job
(400CANON comprising 94 .RAR files and amounting to 58.2GB)
was completed by the Windows/Explorer GUI (ie not command
line) in 49mins.

The Backup2 job that originally deposited those very files was
vying with a concurrent and similar job for 500CANON. The pair
took 7hrs overnight during the maintenance period of activities
(ie network might be busy - workstation ditto).

I estimate that Backup2 does 400CANON at ~18GB/hr. So,
very approximately, the bloated Windows GUI manages to
pass across the intranet those 94 .RAR files amounting to
58.2GB some three times quicker than Backup2 seemingly
can manage from its own Linux environment. This isn't a
statistic about which I am terribly happy;~/

I agree it's not like for like...  there's the obvious difference
between the reading and writing of the FireWire external drives
but so to with the RAID5 array... also prioritisation algorithms
in the 3com unmanaged Gigabit intelligent switch may play a
part in favouring a transfer in a particular manner. However
the gist of it all is that I've never been entirely confident that
the back up runs are going as quickly as possible hence my
pleas for either a turbo version (!) or incremental backup
functionality or even both;~)

Tell me how to replicate your speed test scenario and I will
gladly perform the test/s to the very hilt.

----best wishes, Robert

Offline raem

  • *
  • 3,972
  • +4/-0
backup2
« Reply #9 on: August 20, 2006, 08:19:07 PM »
icpix

>...it's pretty much impossible to determine which
back up job is whatever PID.

If you have multiple backups running then they just display as a rar process
Use
htop
that will show the full process name and backup job number
...

icpix

backup2
« Reply #10 on: August 20, 2006, 08:32:39 PM »
Ray----
Now THAT's more than a little useful, tks;~)
Finally get to see what BOTH processors are doing!
You couldn't conjure up some LM stuff into kernel
so that the CPU core temperatures show up too...
----best wishes, Robert

Offline raem

  • *
  • 3,972
  • +4/-0
backup2
« Reply #11 on: August 20, 2006, 08:41:50 PM »
icpix

I seem to recall that Charlie Brady described htop as "top on steroids" !!
...

Offline EdelingF

  • ****
  • 215
  • +0/-0
backup2
« Reply #12 on: August 21, 2006, 02:19:28 PM »
Could backup2 exist next to backuppc on one SME server?

I'm still searching for the perfect backup-solution, but every program seems to miss something I want; or I'm not flexible enough.....

I desperately need an backup program to backup ALL data to a (Win XP) workstation with a removable internal HD and if possible something to burn a rescue CD/DVD to quickly restore my server after I've done some stupid things.
...

icpix

backup2
« Reply #13 on: August 21, 2006, 02:57:24 PM »
EdlingF----

Believe I had Backup2's previous version (Backup2ws
ie Backup To Workstation) existing next to backuppc.
Obviously you need to tell Backup2 not to back up the
place where backuppc puts its data library stuff.

I agree, every programme seems to miss something I
want and I'm not flexible (aka experienced) enough to
run with the solutions that need compiling and other
clever trickery;~/

On a very simple understanding (my favourite) Backup2
seems to back up the server to everywhere else. The
downside is that it doesn't 'do' incrementals/partials,
everything specified is backed up each time. The only
intelligence in that respect is that it will abide by a
designated number of generations of that storage.
Bit tough if you're short on excess storage, working
bandwidth and time to complete the logistics of it all.
It DOES the job though and you can't ask much more
from a back up procedure.

Whereas backuppc seems to back up everything else
to the server, which isn't at all appropriate for my site.
There's a laudable degree of intelligence and
permutations available. However I did not get on with
the cygwin/Rsync stuff necessary on my w2kpro
workstation. Somewhat inexplicably cygwin went
wild and hogged the processor until I managed to kill it
(well, I think I've killed it). So I'm out of backuppc now
particularly as it's not very logical to relocate where
backuppc holds its data. It seems an admirable
programme but I was unable to get with it
appropriately enough.

Backup2 will certainly run your server stuff to a workstation.

Not sure about your thinking on the rescue stuff... I'm
of a mindset that you've already GOT a rescue CD ~
the install ISO. The rest is up to whatever grey cells
remain with you after your feared 'stupid' incident;~)

----best wishes, Robert

Offline dmay

  • *
  • 450
  • +0/-0
    • http://myezserver.com
backup2
« Reply #14 on: August 21, 2006, 09:42:33 PM »
Quote from: "EdelingF"
Could backup2 exist next to backuppc on one SME server?

backup2 will co-exist with backuppc. Just make sure to schedule your jobs appropriately to avoid overburdening the server hard drive read/write cycles. Backuppc first, backup2 last would seem most logical.

Quote from: "EdelingF"
I desperately need an backup program to backup ALL data to a (Win XP) workstation with a removable internal HD and if possible something to burn a rescue CD/DVD to quickly restore my server after I've done some stupid things.

Backup2 will backup to your WinXP drive. A standard SME iso can be your rescue CD -or- you could go fancy and build a Mondo rescue CD

http://forums.contribs.org/index.php?topic=33240.0

Darrell