Topic: backup2

[dmay]

ALPHA RELEASE: Use with caution. Highly experimental!!!

If you are not familiar with this contrib, please do not bother installing this alpha release. If you are familiar and have a test server available, please test and report your results. General comments to the forum are fine. Bugs to BugZilla please for tracking.

http://mirror.contribs.org/smeserver/contribs/dmay/smeserver/7.x/alpha
smeserver-backup2-1.0.0-a1dmay.noarch.rpm

Complete FormMagick rewrite.

Darrell

[Tib]

Nice ... I did a simple backup about 200 odd meg and it went well on my test machine ... I'll try a more indepth backup tomorrow hopefully.

I just backed up a users complete home dirve to a windows share.

I haven't tried a restore yet so I can't comment on that yet either.

Regards,


Tib

[icpix]

Darrell----

Ref: smeserver-backup2-1.0.0-a2dmay.noarch.rpm

Trying it out on a live data SME7 fileserver (not public).
Updated over the top of existing Backup2 (e-smith version).
Inherited my old set of jobs (a very comprehensive set!) but
did not use them, other than to copy/paste the source path
into the FOREVER TOO SMALL form box (hint) in the alpha.

Have tried not to stress the alpha too much except to run
concurrent ie multiple realtime manually invoked jobs.
I haven't yet tried any jobs using the timer settings.
All to the W2kPro SP4 (workstation) work perfectly;~)
All to the MSS (a NAS type of box) don't... as you intimated.

Note: production SME6 using Backup2 (e-smith version) has
always successfully completed its own concurrent multiple
jobs to the MSS thing on its own overnight timered runs.

The MSS thing is a networked attached Maxtor Shared Storage
300GB box but unfortunately Maxtor spoiled a really good piece
of kit by inflicting a pointless 10/100Mbps interface on it instead
of a more real world gigabit interface;~/ Its proprietory setup
includes having no authentication as it's on my intranet. Though
I don't think my old MSS is of the new flavour ie a 'Plus' item:
http://www.maxtorsolutions.com/en/catalog/MSS_Plus/
Anyway all concurrent jobs to the MSS end up as 'aborted'.
Seems to mess up SME7's /mnt/ area too... Other transfers
ie the good ones to the workstation appear to proceed slowly.
Attempting to manually delete the stuck mount in /mnt/ seems
to do the trick - transfers then get completed faster.

Intranet is an unmanaged gigabit switch (3com), I'm the only user.

Production SME6 (awaiting some courage to move to SME7).

Much prefer your new clean interface form;~)

Mystified by the 'explanatory' help note about the Disaster slot...
Think I know what you mean or mean to say but it's not clear.

The entry box on the form for the source path. It's always been
too small for the endless /home/e-smith... path mantra out to all
the iBays. When I then add the hierarchy of the 2TB RAID5 stuff
on to an iBay the deficit is somewhat risible. It might fit in if the
data slot for the source path were to be twice as long... (hint)

Typo: No compression dramitcally improves backup speed.
No compression dramatically improves backup speed.

What did you do to get it all working more quickly? I think  
that Backup2 (the smeserver version not the e-smith version)
is working more quickly ie transfers complete in shorter periods.

Heartfelt thanks for the mod giving user choice of job names;~)

Are the logs rotated and so forth?

Could each job line on the interface include a URL to 'its' log?

Any chance that Backup2 might include some incremental or
otherwise rdiff sort of functionality? My site suffers massive hits
appropriate to the unnecessary duplication in day-to-day backups.

----best wishes, Robert

[dmay]

Tib/Robert

Thanks for the feedback. Will try to incorporate a few of your suggestions into future builds.

Any difference experienced in accessing your MSS box would be samba related. Different samba version between 6.x/7.x. Check with Maxtor for samba 3.x issues.

Logs are not rotated. They are overwritten with each job run.

Darrell

[icpix]

Darrell----

Backup2 isn't the only thing on my SME7 whose operation exceeds its own
logistical window (daily backups take longer than a day). ClamAV cannot
complete its daily scan in a day (nowhere near). Despite appearing to have
successfully pruned ClamAV's scope...

Code: [Select]


[query scope allowed to ClamAV]
db configuration getprop clamav FilesystemScanExclude


Code: [Select]


[set apparent default scope allowed to ClamAV]
db configuration setprop clamav FilesystemScanExclude /proc,/sys,/usr/share,/var

...I now realise that the highly prolonged ClamAV activity is
due to it running through /mnt/ during the Backup2 runs;~/
I think it may also been a factor in some failed Backup2 runs.
Have now disassociated ClamAV activity from /mnt/...

Code: [Select]


[set own scope allowed to ClamAV]
db configuration setprop clamav FilesystemScanExclude /proc,/sys,/usr/share,/var,/mnt,/home/e-smith/files/ibays/storage

...where the 'storage' iBay area is the mount point of the 2TB array.

During the sleuthing I found it really difficult to either stop the back up
activities or to properly determine whatever backup run was active.
While Backup2 is 'doing its thing' could there be an activity indicator?
(text colour change in the JobName column or a blob or something)

On the subject of stopping the backup activities (pause? abort?) I found
it awkward properly 'clearing up' the 'lost' mounts in the /mnt/ area.
Could Backup2 ROBUSTLY clear up any lost mounts (due to aborts etc)?

----best wishes, Robert

[raem]

icpix

>...I found it really difficult to either stop the back up
> activities or to properly determine whatever backup run was active.

top -i
will show what's running

kill pidnumber
will kill the process

[icpix]

Ray----
Thank you that should be useful.
I usually do have <top -s i> running on both servers (prod/SME6
filelib/SME7) but it's pretty much impossible to determine which
back up job is whatever PID.

Darrell---
FWIW the two concurrently launched (manually) runs completed
successfully overnight after having persuaded ClamAV not to check
the /mnt/ path. The two concurrent runs, 400CANON (58GB/20k files)
and 500CANON (90GB/22k files), were done in about 7hrs. 150GB in
7hrs over an unloaded Gigabit intranet to a Maxtor 'OneTouch series 1'
250GB (external FireWire) - not the problematic (Samba?) network
attached Maxtor MSS box (NAS/100Mbps interface). The resulting maths
(very approximately 1.5TB in 70hrs?) highlights the unsustainable daily
back up window logistics;~/ It doesn't seem to be the intranet or the
target drives but the actual back up mechanism that is the bottleneck.
BTW no compression has been involved, those CANON file directories
hold compressed files in any case, though I do have plenty of horsepower
(dual 3.6GHz Xeons) and memory (4GB) too if only Backup2 could usefully
be configured to so use it advantageously.
----best wishes, Robert

[dmay]

I would suggest you create a bug report for ClamAV recommending /mnt be a default exclusion.

As for speed, try a simple test. Create a mount point and simply copy a large dir tree from the command line. I expect the speed to be the same as a backup2 job of the same source and the limiting factor being your network and external device throughput.

Darrell

[icpix]

Darrell----

Noted.
http://bugs.contribs.org/show_bug.cgi?id=1867
(default exclusion for ClamAV)

As for the speed test... I hear you and understand the need but
I just don't know how to do as you have suggested;~/ Fill me in
and I'll willingly go for it.

Empiracal test done the other way around (least I know how to
do it that way) - from same drive on w2kpro workstation to the
hardware RAID5 array (it's mounted in fstab) and transferring
the Backup2 .RAR files originally deposited for the back up job
(400CANON comprising 94 .RAR files and amounting to 58.2GB)
was completed by the Windows/Explorer GUI (ie not command
line) in 49mins.

The Backup2 job that originally deposited those very files was
vying with a concurrent and similar job for 500CANON. The pair
took 7hrs overnight during the maintenance period of activities
(ie network might be busy - workstation ditto).

I estimate that Backup2 does 400CANON at ~18GB/hr. So,
very approximately, the bloated Windows GUI manages to
pass across the intranet those 94 .RAR files amounting to
58.2GB some three times quicker than Backup2 seemingly
can manage from its own Linux environment. This isn't a
statistic about which I am terribly happy;~/

I agree it's not like for like...  there's the obvious difference
between the reading and writing of the FireWire external drives
but so to with the RAID5 array... also prioritisation algorithms
in the 3com unmanaged Gigabit intelligent switch may play a
part in favouring a transfer in a particular manner. However
the gist of it all is that I've never been entirely confident that
the back up runs are going as quickly as possible hence my
pleas for either a turbo version (!) or incremental backup
functionality or even both;~)

Tell me how to replicate your speed test scenario and I will
gladly perform the test/s to the very hilt.

----best wishes, Robert

[raem]

icpix

>...it's pretty much impossible to determine which
back up job is whatever PID.

If you have multiple backups running then they just display as a rar process
Use
htop
that will show the full process name and backup job number

[icpix]

Ray----
Now THAT's more than a little useful, tks;~)
Finally get to see what BOTH processors are doing!
You couldn't conjure up some LM stuff into kernel
so that the CPU core temperatures show up too...
----best wishes, Robert

[raem]

icpix

I seem to recall that Charlie Brady described htop as "top on steroids" !!

[EdelingF]

Could backup2 exist next to backuppc on one SME server?

I'm still searching for the perfect backup-solution, but every program seems to miss something I want; or I'm not flexible enough.....

I desperately need an backup program to backup ALL data to a (Win XP) workstation with a removable internal HD and if possible something to burn a rescue CD/DVD to quickly restore my server after I've done some stupid things.

[icpix]

EdlingF----

Believe I had Backup2's previous version (Backup2ws
ie Backup To Workstation) existing next to backuppc.
Obviously you need to tell Backup2 not to back up the
place where backuppc puts its data library stuff.

I agree, every programme seems to miss something I
want and I'm not flexible (aka experienced) enough to
run with the solutions that need compiling and other
clever trickery;~/

On a very simple understanding (my favourite) Backup2
seems to back up the server to everywhere else. The
downside is that it doesn't 'do' incrementals/partials,
everything specified is backed up each time. The only
intelligence in that respect is that it will abide by a
designated number of generations of that storage.
Bit tough if you're short on excess storage, working
bandwidth and time to complete the logistics of it all.
It DOES the job though and you can't ask much more
from a back up procedure.

Whereas backuppc seems to back up everything else
to the server, which isn't at all appropriate for my site.
There's a laudable degree of intelligence and
permutations available. However I did not get on with
the cygwin/Rsync stuff necessary on my w2kpro
workstation. Somewhat inexplicably cygwin went
wild and hogged the processor until I managed to kill it
(well, I think I've killed it). So I'm out of backuppc now
particularly as it's not very logical to relocate where
backuppc holds its data. It seems an admirable
programme but I was unable to get with it
appropriately enough.

Backup2 will certainly run your server stuff to a workstation.

Not sure about your thinking on the rescue stuff... I'm
of a mindset that you've already GOT a rescue CD ~
the install ISO. The rest is up to whatever grey cells
remain with you after your feared 'stupid' incident;~)

----best wishes, Robert

[dmay]

Could backup2 exist next to backuppc on one SME server?

backup2 will co-exist with backuppc. Just make sure to schedule your jobs appropriately to avoid overburdening the server hard drive read/write cycles. Backuppc first, backup2 last would seem most logical.

I desperately need an backup program to backup ALL data to a (Win XP) workstation with a removable internal HD and if possible something to burn a rescue CD/DVD to quickly restore my server after I've done some stupid things.

Backup2 will backup to your WinXP drive. A standard SME iso can be your rescue CD -or- you could go fancy and build a Mondo rescue CD

http://forums.contribs.org/index.php?topic=33240.0

Darrell

[dmay]

Uploaded beta release:

smeserver-backup2-1.0.0-b1dmay.noarch.rpm

Available here:

http://mirror.contribs.org/smeserver/contribs/dmay/smeserver/7.x/testing/

Would appreciate users testing restore functionality.

Darrell

[EdelingF]

Backup2 is running now and it looks fine, a bit slow but sufficient for the data in Ibays.
Is Mondo allready save to install? I have a DVD-writer in the server since SME6 but still haven't found any use for it. I was hoping to test Mondo with it sometime.
On my Windows machines I use Ghost and I'm searching for a sort of Linux equivelant that's easy to use.

Freek

[dmay]

Issue with ClamAV file system scans has been reported. Roberts initial bug entry shows the fix:

http://bugs.contribs.org/show_bug.cgi?id=1867

Darrell

[dmay]

Uploaded beta release:

smeserver-backup2-1.0.0-b2dmay.noarch.rpm

Available here:

http://mirror.contribs.org/smeserver/contribs/dmay/smeserver/7.x/testing/

[root@sme ~]# rpm -q --changelog smeserver-backup2
* Tue Aug 22 2006 Darrell May <dmay@myezserver.com>
- panel textarea support added. Requires perl-CGI-FormMagick-0.92-08
- [1.0.0-b2dmay]

http://bugs.contribs.org/show_bug.cgi?id=1088

Darrell

[Tib]

dmay

Wouldn't be able to push it to a diff mirror could you?

Ibiblio has been down for days

This mirror still works ...

ftp://ftp.oss.cc.gatech.edu/pub/linux/distributions/smeserver/contribs

Regards,

Tib

[icpix]

Somebody changed the title of the thread?
Might explain why I got no message updates.

Darrell----
I cannot seem to reach/see the b2 update.
Suggest you use the email address in bugzilla?
Or make it available as a download on your myEZs... site?
----best wishes, Robert

postedit: OK now see the b2 and perl files

[icpix]

Darrell----

Thanks for the (very slightly) longer box for source path entry.
Bemused as to why you've made it multiple lines too though.
It still needs to be longer...
Why on earth is there so much 'white space' onscreen left?

Compression help typo "dramitcally/dramatically" uncorrected.

The 'music' back up still runs at about 18GB/hr so it's no faster.

*NOW* I understand the help notes on the Disaster back up.
FWIW I had originally interpreted it quite wrongly.
Believe the logical 'normal' position should be disabled for
the simple reason that I make a bunch of backups (having to
toggle it each time) whereas I make a single 911/disaster
backup per session.

Code: [Select]

[mantra]longer_entry_boxes_for_the_smeserver's_inflicted_legacy_of_whacking_great_long_filename_paths_that_just_go_on_and_on_seemingly_forever_/home/e-smith/files/ibays/storage/files/raid5/pix/__canon/__100/[/mantra]
My point being that these long paths appear to be necessary
therefore adequately long entry boxes should be coded so
that data entry does not HAVE to be made in permanently
scrolling undersized entry boxes! Less user (path) errors too.
(PS: I'll delete the scrolling segment above after you've read it)

You wanted some testing of the RESTORE functionality.
Normally I just get lost/deleted/overwritten files back
using the WinRAR interface and shoehorn them back in.
So I tried your restore functionality...

Notwithstanding that none of my paths fitted the entry boxes
I thought I had understood the explained logic but the result
showed that seemingly I had not interpreted it correctly as it
all ended up with a double path structure. I specified...

Code: [Select]

Restore 1: *.*
Restored to: /home/e-smith/files/ibays/storage/files/raid5/pix/__canon/__100/
...and what I actually got back was...

Code: [Select]

/home/e-smith/files/ibays/storage/files/raid5/pix/__canon/__100/home/e-smith/files/ibays/storage/files/raid5/pix/__canon/__100/
/home/e-smith/files/ibays/storage/files/raid5/pix/__canon/__100/home/e-smith/files/ibays/storage/files/raid5/pix/__canon/__100/146.dvdr
/home/e-smith/files/ibays/storage/files/raid5/pix/__canon/__100/home/e-smith/files/ibays/storage/files/raid5/pix/__canon/__100/147.dvdr
/home/e-smith/files/ibays/storage/files/raid5/pix/__canon/__100/home/e-smith/files/ibays/storage/files/raid5/pix/__canon/__100/148.dvdr
/home/e-smith/files/ibays/storage/files/raid5/pix/__canon/__100/home/e-smith/files/ibays/storage/files/raid5/pix/__canon/__100/198CANON
/home/e-smith/files/ibays/storage/files/raid5/pix/__canon/__100/home/e-smith/files/ibays/storage/files/raid5/pix/__canon/__100/199CANON
...the returned data was usable by my image workflow system.

The example 'Note:' (referring to included spaces) confused
me initially, I don't think it should've included the leading '/'.

Now that I've seen/done a restore (using Backup2) I'd
like to amend a previous wishlist request... the final
onscreen column should link to logs for backup/restore.
So simple to code and so user-friendly;~)

It's not a bug but I noticed that the 'mount' thingummy
for the restore functionality appears to hark back to the
previous (e-smith) iteration of this backup programme...
/mnt/backup2ws.100CANON

----best wishes, Robert

[icpix]

Darrell----

Whilst I've dropped my immediate use of backuppc I've
maintained a lurking interest in its home newsgroup...
gmane.comp.sysutils.backup.backuppc.general
...and I noticed some stuff in one recent thread <slow transfer smb>:

Code: [Select]

Mark> I changed the config to use smb and connect directly to the
Mark> client host, I then ran a full backup of the host, but it has
Mark> been running for close to 24hours now and seems to have only
Mark> gotten halfway through the smb shares?  How can I make the
Mark> transfer rate faster? It is running across gigabit Ethernet and
Mark> shouldn't never take this long.

I am having this same problem.

I have noticed that if I access the share outside of BackupPC at the
same time, for example at a prompt on the BackupPC computer I run "ls
-lR /mnt/computername/sharename/", then the BackupPC access
immediately begins to run fast -- you can see this in top.

I searched for different samba optimization parameters which I will
try some options to smbmount to see if I can fix it.  If that doesn't
work, I am considering writing something that accesses the smb share
while backuppc is accessing it, to keep the transfer from slowing
down.

Hope this might be of some help to you/backup2.

----best wishes, Robert

[Mjohnson]

I am getting a installation failure due to the Formmagick 0.92-08 dependancy.  I checked the bugtraker and it seems the FormMagick rpm was allegedly uploaded to the smetesting repository a couple of weeks ago, a repository I do not seem to have in my list.

Wonder if someone can point me in the right direction.

Thanks.

MJ

[icpix]

Mjohnson----

Your check of the Bug Tracker served you very poorly.

Go to the contribs section...
http://no.longer.valid/phpwiki/index.php/SME7Contribs
...and move down to the backup2 contrib.

Click the included Bug Tracker section's All Bugs...
http://bugs.contribs.org/buglist.cgi?query_format=specific&order=relevance+desc&bug_status=__all__&product=SME+Contribs&component=smeserver-backup2&content=
...and view the only one in red (referring to dependencies...)

----best wishes, Robert

[Mjohnson]

Hi Robert,

Thanks for the reply, but I was at that site and links do not address my issue directly.

I am using yum as the installer.  I have also located and downloaded the FormMagick 0.92-08 rpm

When I run:  yum localinstall smeserver-backup2-1.0.0-b2dmay.noarch.rpm, I recieve:  Error Missing Dependancy: perl-CGI-FormMagick >= 0.92-08 is needed...

I have the perl-CGI-Formmagick-0.92-08.noarch.rpm in the localinstall directory, which yum apparently does not see.

When I attempt to install the perl-CGI.......FormMagick....rpm I receive the error message:  Transaction Check Error:  Package already installed.

When I attempt to yum remove perl-CGI....FormMagick, I recieve the message that the package is not installed.

I seem to be caught in the middle.  

Thanks.

MJ

[icpix]

Mjohnson----

!?...guessing here.
Manually uninstall your existing perl stuff ie leave out the numbers.
Then, with everything in a separate directory, you could try
yum localinstall *
ie don't be specific and let yum sort out the dependancies.

----best wishes, Robert

[Mjohnson]

Interesting... I did an install of backup2 on a differnet server last evening.  It went with no problems what so ever.  The only difference I can come up with between the two servers is that one was a clean install of SME 7 and the other was upgraded over serveral months from SME 7 RC1 to the current version.  The later was the one giving me fits.

Also to note, the bugtracker for backup2 inditcates that the SME-testing repository is one of 4 default repositories activated (or at least available).  That is not the case on my "giving me fits" server.  That repository is neither listed nor available.

Just pointing this out.  The tests so far on the contrib have been flawless.  Thanks for the hard work.

[peterhocking]

Hi

I've looked everywhere for this version to download, but can't find it.

CXan anyone tell me where I can download it?

Thanks,

Peter

(SME Serrver & linux newbie)

[raem]

peterhocking

> I've looked everywhere for this version to download, but can't find it.

It's bad practice to change the thread topic, I nearly answered you for backup2


Re
perl-CGI-FormMagick 0.92-08

http://mirror.contribs.org/smeserver/releases/7/updates-testing/i386/RPMS/

[meofcourse]

I used your backup2 program.

Works well, just one comment.

How about adding support for a fully qualified domain name rather than just the hostname?

ie.  //computername.domain.name/share

instead of //computername/share

both should work

[dmay]

Because in both a domain and non domain environment, hostname works.

Darrell

[owen]

Hi guys,

Just wondering if there is a new url to download backup2 from.

The url mentioned in posts above (http://mirror.contribs.org/smeserver/contribs/dmay/smeserver/7.x/testing/ ) doesn't seem to contain the rpm.

The same url is listed on the contribs wiki (http://no.longer.valid/phpwiki/index.php/SME7Contribs) as the location of the rpm.

Maybe I've missed something obvious but if someone could help me anyway that would be great.

Many Thanks,

Owen

[dmay]

Replaced by smeserver-dar2

http://forums.contribs.org/index.php?topic=34041.0

Darrell

[steever]

I've installed the new dar backup.  Does anyone know how to remove the backup2 contrib?

Steve

[jmartin]

I've installed the new dar backup.  Does anyone know how to remove the backup2 contrib?

Steve

Here is an example but make sure to check which version numbers you have installed.

rpm -e smeserver-backup2-0.0.1-b2dmay
rpm -e rar-3.5.1-7dmay

[raem]

jmartin & steever

You only need to do

rpm -e smeserver-backup2
rpm -e rar

You usually leave the numbers off when removing rpms.

[jmartin]

Thanks Ray....good to know....I'd never thought to try that and it'll save some typing.