Koozali.org: home of the SME Server

Keine gezielte Wiederherstellungen aus dem Backup mehr möglich

Offline funky

  • 10
  • +0/-0
Hallo zusammen,

wir haben bei uns folgendes Szenario:

Der Hauptserver SME9.2 sichert täglich über ein Workstation Backup (Cifs) in ein Share eines zweiten Servers (ebenfalls SME 9.2). Will man gezielt eine Datei aus den Sicherungssets wieder herstellen, funktioniert das nicht mehr. Die einzelnen Backups werden nicht mehr angezeigt und wenn man alle Backups auswählt werden keine Dateien aufgelistet.

Das Szenario existiert an zwei Standorten.

Wir haben dann auf dem Sicherungsserver den Ordner server.mycompany.local mit allen Backups gelöscht und danach ein Vollbackup durchlaufen lassen. Nach dem Vollbackup kann man wieder einzelne Dateien aus dem Backup wiederherstellen.

Sobald das erste inkrementelle Backup einen Tag später gelaufen ist, tritt oben beschriebenes Problem wieder auf.

Kann da eventuell jemand weiterhelfen? Mein englisch reicht leider nicht um den Vorgang im Detail zu beschreiben.

VG Frank
« Last Edit: June 17, 2019, 11:54:58 AM by funky »

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #1 on: June 17, 2019, 04:36:08 PM »
Sorry only English

You will find a list of reported issues that go a long way back covering various things however,
Although a sme10 bug it does have a possible solution that may also apply to sme9, have a read:

https://bugs.contribs.org/show_bug.cgi?id=9563

around line 3082
# /etc/e-smith/web/functions/backup
--
qui scribit bis legit

Offline funky

  • 10
  • +0/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #2 on: June 19, 2019, 09:22:33 AM »
Thank you for your help - selective file restore workstation backup does not list backups or files


I changed (removed the nounix option)

-        return ( qx(/bin/mount -t cifs "//$host/$share" $mountdir -o credentials=/etc/dar/CIFScredentials,nounix 2>&1) );
+       return ( qx(/bin/mount -t cifs "//$host/$share" $mountdir -o credentials=/etc/dar/CIFScredentials 2>&1) );

My problem was not resolved.

my Versions:

[root@fileserver functions]# cat /etc/e-smith-release
SME Server release 9.2

[root@fileserver functions]# uname -r
2.6.32-754.15.3.el6.x86_64

[root@fileserver functions]# rpm -qa e-smith-backup
e-smith-backup-2.4.0-45.el6.sme.noarch

in /var/log/httpd/admin_error_log:

[Wed Jun 19 09:00:25 2019] [error] [client 127.0.0.1] No terminal found for user interaction. All questions will be assumed a negative answer (less destructive choice), which most of the time will abort the program., referer: https://192.168.3.11/server-manager/cgi-bin/backup

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #3 on: June 19, 2019, 11:02:42 AM »
[root@fileserver functions]# rpm -qa e-smith-backup
e-smith-backup-2.4.0-45.el6.sme.noarch

That version is from updates-testing..maybe roll back to e-smith-backup-2.4.0-44.el6.sme.noarch

Perhaps roll back to e-smith-backup-2.4.0-44 and see what you get
--
qui scribit bis legit

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #4 on: June 19, 2019, 11:39:42 AM »
Further to this, what is the size of the backup?

I have a system with a backup of 400+gb that is exhibiting similar behaviour, although I can see that the action is running..have to see if it will complete
--
qui scribit bis legit

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #5 on: June 19, 2019, 11:48:12 AM »
A consideration : https://bugs.contribs.org/show_bug.cgi?id=7035
and
https://bugs.contribs.org/show_bug.cgi?id=8271

Sorry, all I've got, will be trying those on server mentioned above

Added: OK, selective restore ran for almost 15 mins and then closed, did not display files, this is 400+gb of data, seems fine on a much smaller backup set, more testing etc needed
« Last Edit: June 19, 2019, 11:57:55 AM by TerryF »
--
qui scribit bis legit

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #6 on: June 19, 2019, 12:02:25 PM »
and looks to be the timeout issue:
See Bug https://bugs.contribs.org/show_bug.cgi?id=8271

[Wed Jun 19 19:46:57 2019] [warn] [client 127.0.0.1] Timeout waiting for output from CGI script /etc/e-smith/web/panels/manager/cgi-bin/backup, referer: https://192.168.11.19/server-manager/cgi-bin/backup
[Wed Jun 19 19:46:57 2019] [error] [client 127.0.0.1] (70007)The timeout specified has expired: ap_content_length_filter: apr_bucket_read() failed, referer: https://192.168.11.19/server-manager/cgi-bin/backup

--
qui scribit bis legit

Offline funky

  • 10
  • +0/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #7 on: June 19, 2019, 12:38:39 PM »
Further to this, what is the size of the backup?

I have a system with a backup of 400+gb that is exhibiting similar behaviour, although I can see that the action is running..have to see if it will complete

I have roll back

[root@fileserver ~]# rpm -qa e-smith-backup
e-smith-backup-2.4.0-44.el6.sme.noarch

The problem is the same. After the first full backup, the files are displayed. The first incremental backup after full backup no longer displays files.

The backup is 224 gb.

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #8 on: June 19, 2019, 12:59:38 PM »
Try and delete a file or two and see if you get a result..
--
qui scribit bis legit

Offline funky

  • 10
  • +0/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #9 on: June 19, 2019, 01:16:45 PM »
Deleting files from the backup set performed. The problem is the same.


« Last Edit: June 19, 2019, 01:30:46 PM by funky »

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #10 on: June 19, 2019, 02:04:48 PM »
I do think its this issue https://bugs.contribs.org/show_bug.cgi?id=8271

See https://bugs.contribs.org/show_bug.cgi?id=8271#c10

Going to see if it can resolve my own issues as well, tomorrow for an update
--
qui scribit bis legit

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #11 on: June 20, 2019, 01:57:46 AM »
Nope, tried all the suggested workarounds, I am sure it is a an issue with disconnect of the running script and server-manager updating, there is also a file number limit that I recall associated with dar

I was able to verify my 400gb+ backup no problems albeit it took some time to list the full backup.

The restore of a full backup also looks to be OK, so its just the selective, going to need some heavy hitters to look at it, there are bugs current, sorry cant help anymore than yet as yet.

Added:  See Bug https://bugs.contribs.org/show_bug.cgi?id=8271
« Last Edit: June 20, 2019, 02:16:20 AM by TerryF »
--
qui scribit bis legit

Offline funky

  • 10
  • +0/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #12 on: June 20, 2019, 08:44:45 AM »
Thank you very much for your efforts.

The first full backup is displayed in the backup list and the files are also listed for targeted recovery.

As soon as the first incremental backup has been run one day after the first full backup, no backups will be displayed during selective restore. Neither the full backup nor the incremental backup. As a result, no files are listed.

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #13 on: June 20, 2019, 10:45:06 AM »
This may also be a factor, areas for investigation:  https://bugs.contribs.org/show_bug.cgi?id=7035
--
qui scribit bis legit

Offline funky

  • 10
  • +0/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #14 on: June 20, 2019, 01:41:08 PM »
I can not explain where the mistake is. Until a few months ago, there were no problems with the SME 9.2.

Today we once again deleted all backups and created a full backup on the bash with do_backupwk. After the full backup, the backup is shown when recovering files and all files are displayed in the backup.

Then another do_backupwk at the bash, which went through without error. The incremental backup can also be seen in the share.

If you then want to do a targeted restore of files through the server manager, neither the full backup nor the incremental backup is displayed for selection.


Offline funky

  • 10
  • +0/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #15 on: June 20, 2019, 01:44:05 PM »
Can you create a bug report for the developers in perfect English?

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #16 on: June 20, 2019, 02:54:31 PM »
Yes, I have found the dar catalog is being corrupted for reasons as yet unknown...

There have been similar issues in the past https://bugs.contribs.org/show_bug.cgi?id=6950

Working on it
--
qui scribit bis legit

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #17 on: June 20, 2019, 03:04:11 PM »
Also this was an issue here as well, OS and version specific:  https://bugs.contribs.org/show_bug.cgi?id=6052#c34
--
qui scribit bis legit

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #18 on: June 20, 2019, 03:09:03 PM »
Before I raise a bug I want to run a few more scenarios. I will post Bug number back here when done, thanks

You are backing up to another smeserver, I am backing up to a share on a windows 10 box, cant get much more different than that, which would suggest issue is with the sme9.2
« Last Edit: June 20, 2019, 03:15:34 PM by TerryF »
--
qui scribit bis legit

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #19 on: June 20, 2019, 04:39:06 PM »
OK, looks like we have found a fix, there was a patch to sme10 in late 2016 https://bugs.contribs.org/show_bug.cgi?id=9563 to fix the same issue,

It was not backported to sme9

The fix listed in that bug was not the final fix..just need to do a little more testing
and in particular try it on a big backup like my NFP 400gb one,
--
qui scribit bis legit

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #20 on: June 20, 2019, 05:10:06 PM »
Bug raised against sme9.2 patch will be forthcoming https://bugs.contribs.org/show_bug.cgi?id=10781
--
qui scribit bis legit

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #21 on: June 21, 2019, 03:41:17 AM »
see Bug 10781, rpm has been patched now verifying etc, in paerticular this, noticed it last night in early tests, make sure to read comment by Ian Wells in Bug 7035 :

"I have found that restoring a directory seems to restore all the files that are present in all the backups that are held..this needs a some more testing and maybe adding to the wiki"
--
qui scribit bis legit

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #22 on: June 21, 2019, 06:16:26 AM »
Small sets are fine, don't know the point at which below comes in.

On large backup sets, tested with 400+gb using selective file restore NO files or dirs are listed and the routine eventually times out..

var/log/httpd/admin_error_log: Viewed at Fri 21 Jun 2019 01:54:32 PM AEST.

[Fri Jun 21 13:40:11 2019] [warn] [client 127.0.0.1] Timeout waiting for output from CGI script /etc/e-smith/web/panels/manager/cgi-bin/backup, referer: https://192.168.11.19/server-manager/cgi-bin/backup

[Fri Jun 21 13:40:11 2019] [error] [client 127.0.0.1] (70007)The timeout specified has expired: ap_content_length_filter: apr_bucket_read() failed, referer

Work needs to be done - https://bugs.contribs.org/show_bug.cgi?id=10782
--
qui scribit bis legit

Offline funky

  • 10
  • +0/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #23 on: June 21, 2019, 12:01:36 PM »
Great, thank you for your effort. Now we are on the right path. I will wait until there is an update for the rpm package.

I have big backups over 400 GB on another machine with SME9.2. The behavior and the error is the same as on the machine with the small 224 GB backup.


Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #24 on: June 21, 2019, 01:12:00 PM »
G'day..

With the updated rpm I am able to see all the backups including all inc on large and small backup setc, select last backup inc or full on smaller backups it will display dirs. and files etc on the test 400+gb it times out see error from logs last post, this is what we have to fix, it may NOT be fixable, I have tried increasing timeouts in httpd.conf but still errors,

I do remember in the dim mists of time a discussion about just this scenario cant find the bloody thing now :-( it will come out eventually.. :-)
« Last Edit: June 21, 2019, 01:16:04 PM by TerryF »
--
qui scribit bis legit

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #25 on: June 21, 2019, 01:14:30 PM »
If you want to play at the pointy end :-) see the Bug 10781 on where to yum the update from or wait until it appears in updates, that could be a few days away..

Just on the smaller backup as I am hopeful this will fix the issue there..good luck
--
qui scribit bis legit

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #26 on: June 26, 2019, 01:59:46 AM »
Just a further note, the displaying of backup sets to select so as to list files/dirs to display in the server-manager window is working with the updated rpm.

I have been able to use dar-manager from the cli to display the list of files/dirs that the selective restore produces to be displayed in server-manager. This included a 400+gb backup, this list in most cases is quite small, also dumped the output to a small file.  So the actual cli action seems to not produce an error even with the very large backup.

Small backup sets upto at least 100gb (limit of my testing) all display a list of files/dirs in the selective restore window of server-manager.

Backup set of 400+gb (all I have to test with) times out in displaying files/dirs in the server-manager window, a smarter person than I will find the why and we can fix it. 

The "pipe/process/whatever" that sits between the perl script and server-manager (apache or whatever) breaks with the bigger backup

See Bug 8271 there were a couple of other bugs that were dupes these have been closed and consolidated in this bug
« Last Edit: June 26, 2019, 02:01:18 AM by TerryF »
--
qui scribit bis legit

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Keine gezielte Wiederherstellungen aus dem Backup mehr möglich
« Reply #27 on: July 05, 2019, 05:46:43 AM »
Until workstation selective restore can be fixed, on the system with the very large backup set of 400+gb I have switched to using Affa and an external USB disk - https://wiki.contribs.org/Affa#Use Affa to backup to a NFS-mounted NAS or a local attached USB drive.. same USB disk as I was using with workstation backup, formatted to ntfs. Reetp is in process of updating the affa contrib, see Bug 10783, however for just a localhost backup is fine.
--
qui scribit bis legit