Koozali.org: home of the SME Server

smeserver-affa

Offline ReetP

  • *
  • 3,722
  • +5/-0
smeserver-affa
« on: September 22, 2021, 11:01:19 AM »
So, I have started working on this again.

Affa is a huge piece of code and pretty complicated. I have cleaned up a lot of it, tried to refine some of it, and fix some of it. And I am no coder.

It probably needs breaking into some separate libraries, and I am pretty sure there is a lot of duplication in there.

Nonetheless there is some code to test. It is a LONG way from prime time.

I will make this quite clear, right here, and right now.

DO NOT USE THIS IN PRODUCTION.

DO NOT USE THIS WITH v9.

Get two v10 VMs and test (see the note below about Windows testing)

I think there are probably some issues in there that may totally destroy your backup, or during a restore.

So, for those who have claimed they "can't code but want to help test" now is the time to put your money where your mouth is.

The main import bug is https://bugs.koozali.org/show_bug.cgi?id=11024

The wiki page, which lists associated bugs at the bottom is here.

https://wiki.koozali.org/Affa

It is badly out of date and will need a lot of rewriting. Volunteers to assist are required. If you don't have wiki editing rights then ask us.

You will need to do something like this to install affa - do NOT use 'update' or 'upgrade' with smedev. It will break your server every time.

You will need the openfusion repo.

Code: [Select]
yum --enablerepo=smedev,openfusion install smeserver-affa smeserver-systemd-control
It has several linked bugs. Each needs testing and verifying. Use the bug tracker to make practical comments and notes - bugzilla is not a place for idle chatter. If you want to waffle, do it here.

Make sure you get logs of any errors. /var/log/messages and /var/log/affa/*

Make sure you comment on the correct bug. Try and stay focused and on topic.

Note that this is NOT a 'how do I use affa' session. This is real testing.

If you are verifying, use a verification template.

See this page for a template to use:

https://wiki.koozali.org/SME_Server:Documentation:QA:Verification

I am in the process of modifying the code to use RSA keys instead of DSA. I hope that I will push that to CVS for testing later today.

http://bugs.contribs.org/show_bug.cgi?id=10783

Please go though each affa option carefully and see what works, and what does not work. Be patient and methodical and MAKE NOTES so you, and I, can repeat the issue.

If you find a potential issue don't open another bug and until we have had a look.

There is an option to run rsyncd on Windows - I have nothing to test this on so will need assistance.

You can ask questions here, or ask for if you really want to get involved in helping properly then DM me and ask for a Rocket.Chat account on my server and you can chat to us real time.

Please read these guides to help us:

https://www.chiark.greenend.org.uk/~sgtatham/bugs.html
http://www.catb.org/esr/faqs/smart-questions.html
http://xyproblem.info/

Remember, this is open source. Don't sit there and complain. Get involved. Learn. Those who help tend to get helped......
« Last Edit: September 22, 2021, 11:29:43 AM by ReetP »
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: smeserver-affa
« Reply #1 on: September 22, 2021, 02:07:32 PM »
Newer version 3.3.1-4 in smedev.

Updates keys from DSA to RSA 4096. (No it does not try to migrate old ones - do a new send-key)

* Wed Sep 22 2021 John Crisp <jcrisp@safeandsoundit.co.uk> 3.3.1-4
- Change ssh keys from DSA to RSA 4096 [SME: 10783]
- Fix missing /var/affa store dir
Tagged with: smeserver-affa-3_3_1-4_el7_sme

/usr/bin/plague-client build smeserver-affa smeserver-affa-3_3_1-4_el7_sme contribs10
Package smeserver-affa enqueued.  Job ID: 3329.

Looking forward to the testing and verifications.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: smeserver-affa
« Reply #2 on: September 22, 2021, 03:47:52 PM »
I used epel repo as well :-)

# yum install smeserver-affa --enablerepo=smedev,epel,openfusion
--
qui scribit bis legit

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: smeserver-affa
« Reply #3 on: September 22, 2021, 04:51:12 PM »
So that broke things. This should fix it for now.

All the SSH & Rsync commands need refactoring so I have untidied the SSH commands for now.

* Wed Sep 22 2021 John Crisp <jcrisp@safeandsoundit.co.uk> 3.3.1-5
- Modify ssh key links [SME: 10783]
- Untidy the ssh commands so I can see them and tidy them more easily
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Gary Douglas

  • *
  • 68
  • +1/-0
Re: smeserver-affa
« Reply #4 on: September 24, 2021, 11:48:41 AM »
'Looking forward to the testing and verifications'
'If you want to waffle, do it here'

so far so good!

# yum --enablerepo=smedev update smeserver-affa
---> Package smeserver-affa.noarch 0:3.3.1-1.el7.sme will be updated
---> Package smeserver-affa.noarch 0:3.3.1-5.el7.sme will be an update
---> Package smeserver-systemd-control.noarch 0:0.1-2 will be installed
---> Package python3.x86_64 0:3.6.8-18.el7 will be installed
---> Package python3-libs.x86_64 0:3.6.8-18.el7 will be installed
---> Package python3-pip.noarch 0:9.0.3-8.el7 will be installed
---> Package python3-setuptools.noarch 0:39.2.0-10.el7 will be installed
# signal-event post-upgrade; signal-event reboot


SUCCESS  sshPort=4567
SUCCESS  chattyOnSuccess=9
SUCCESS  SMEServer=yes
SUCCESS  Watchdog=yes
SUCCESS  RPMCheck=yes
Other properties not checked (yet)
 BandwidthLimit=0
 dedup=no
 DiskSpaceWarn=risky


SUCCESS  Delete folder from targetsvr user /home directory, check folder deleted on backup server
SUCCESS  Restore folder to targetsvr using  export RDIR=/path/ rsync $RDIR method
SUCCESS  Reduce scheduled archives, check archives reduced and deleted
SUCCESS  Increase scheduled archives, check scheduled archives
SUCCESS  bkupsvr missing RPM's compared to targetsvr notified by email

SUCCESS   affa --run JOB
SUCCESS   affa --configcheck
SUCCESS   affa --make-cronjobs
SUCCESS   affa --send-key [JOB]
SUCCESS   affa --check-connections [JOB JOB ...]
          affa --resume-interrupted  NOT TESTED
SUCCESS   affa --full-restore [--preserve-newer=no] [--delete=yes] JOB [ARCHIVE]
SUCCESS   affa --list-archives [--csv] [JOB JOB ...]
SUCCESS   affa --status [--csv]
SUCCESS   affa --show-config-pathes [--csv] [JOB JOB ...]
SUCCESS   affa --show-default-config
SUCCESS   affa --show-schedule [-15]
SUCCESS   affa --show-property PROPERTY
SUCCESS   affa --log-tail [JOB]
SUCCESS   affa --send-status
SUCCESS   affa --disk-usage [--csv]
SUCCESS   affa --cleanup JOB
SUCCESS   affa --rename-job JOB NEWNAME
SUCCESS   affa --move-archive JOB NEWROOTDIR
SUCCESS   affa --delete-job JOB
SUCCESS   affa --revoke-key JOB
SUCCESS   affa --kill JOB
          affa --killall  NOT TESTED
SUCCESS   affa --mailtest JOB
          affa --nrpe [JOB JOB ...]  NOT TESTED
SUCCESS   affa --version
SUCCESS   affa --warranty
SUCCESS   affa --license
SUCCESS   affa --help

-----------------------
SUCCESS   affa --rise --all

pre-rise on targetsvr;
systemctl stop qpsmtpd.service
systemctl stop sqpsmtpd.service
systemctl stop crond.service
systemctl stop dovecot.service
systemctl stop httpd-e-smith.service
systemctl stop smb.service
systemctl stop qmail.service
systemctl stop pop3.service
systemctl stop pop3s.service
systemctl stop ftp.service

SUCCESS   affa --undo-rise

SUCCESS   affa --full-restore  (after deleting a few user folders on targetsvr)
gives;    Error (1766): Executing script SME/signal-post-upgrade-reboot on targetsvr failed; but targetsvr rebooted and folders restored.
options [--preserve-newer=no] [--delete=yes] NOT TESTED

-----------------------
FAIL   affa --init-nrpe   (never used this option but do use nrpe)

bkupsvr:/etc/nagios# affa --init-nrpe
sh: /etc/init.d/nrpe: No such file or directory

bkupsvr:/etc/nagios# systemctl status nrpe.service
nrpe.service - Nagios Remote Program Executor
-----------------------

Offline mauro

  • ***
  • 101
  • +0/-0
Re: smeserver-affa
« Reply #5 on: September 24, 2021, 02:53:03 PM »
Quote
There is an option to run rsyncd on Windows - I have nothing to test this on so will need assistance.
I have some win clients with rsyncd running (for backups with BackupPC), I can give it a try.
Any documentation available on how the rsync option should work?
All parts should go together without forcing. You must remember that the parts you are reassembling were disassembled by you. Therefore, if you can't get them together again, there must be a reason. By all means, do not use a hammer.
-- IBM maintenance manual (1975)

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: smeserver-affa
« Reply #6 on: September 24, 2021, 03:18:25 PM »
so far so good!

Nice!! And thank you for stepping up to the plate.

Sorts the men from the boys.

Code: [Select]
SUCCESS   affa --full-restore  (after deleting a few user folders on targetsvr)
gives;    Error (1766): Executing script SME/signal-post-upgrade-reboot on targetsvr failed; but targetsvr rebooted and folders restored.
options [--preserve-newer=no] [--delete=yes] NOT TESTED

OK - I'll take a look. Interesting as it must have run pre-restore correctly, but failed after. Maybe it did execute but doesn't correctly understand the response.

Let me know the conf settings please so we can vaguely replicate it.


Quote
dedup=no

I have a test binary build for dedup which I will import to contribs but my lappy here is messing me about so will do it when I get home next week.

https://www.reetspetit.com/smetest/7/repoview/freedup.html

Feel free to download and test.

Code: [Select]
FAIL   affa --init-nrpe   (never used this option but do use nrpe)

bkupsvr:/etc/nagios# affa --init-nrpe
sh: /etc/init.d/nrpe: No such file or directory

bkupsvr:/etc/nagios# systemctl status nrpe.service
nrpe.service - Nagios Remote Program Executor


--init-nrpe will need NRPE installed (part of Nagios)

I have no idea on the situation with that on v10 at the minute.

A couple of other options that cold be tested.

1. A normal Linux box.
Set conf
SMEServer=no

2. A windows box - I think you can do that with Rsycnd as there is some mentions in the code.
Set conf
SMEServer=no
rsyncdMode=yes

There are some Samba options in there too:

SambaShare
setupSamba

But the confs seem to go here:

my $affasmb = "/etc/samba/Affa-Job-$jobname.conf";
Any other variables in the Affa page.


This is the code but I haven't really figure it out yet - this gets called after setting up the watchdog. I think the Samba settings need to be put in the Global Conf but it doesn't say that in the Wiki page!

This is the setup section - if I have time I'll step through the code and try to figure it out.

Code: [Select]
sub setupSamba() {
    lg("Configuring Samba");
    if ( $smbdStatus ne 'enabled' ) {
        lg("Samba service is not installed or not properly configured.");
        return;
    }

    my %inc;
    foreach my $jobname ( $cfg->Sections() ) {
        next if $jobname eq 'GlobalAffaConfig';
        my %job = getJobConfig($jobname);
        if ( not $job{'_SambaValidUser'} and $job{'SambaShare'} eq 'yes' ) {
            lg( "Job $jobname: No Valid Users defined. Samba has been access disabled."
            );
        }
        my $affasmb = "/etc/samba/Affa-Job-$jobname.conf";
        if ( $job{'SambaShare'} eq 'yes' and $job{'_SambaValidUser'} ) {
            open( FD, ">$affasmb" )
                or affaErrorExit("Could not create $affasmb");
            print FD "[$jobname]\n";
            print FD "path = $job{'RootDir'}/$jobname\n";
            print FD "comment = Affa archive: $job{'Description'}\n";
            print FD "valid users = $job{'_SambaValidUser'}\n";
            print FD "force user = root\n";
            print FD "read only = yes\n";
            print FD "writable = no\n";
            print FD
                "veto files = /.$jobname-setup.ini/,/.doneDates/,/.AFFA3-REPORT/,/.AFFA-REPORT/.AFFA-TRASH/\n\n";
            close(FD);
            $inc{"include = $affasmb"} = 1;

All comments welcome!
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: smeserver-affa
« Reply #7 on: September 24, 2021, 03:20:07 PM »
I have some win clients with rsyncd running (for backups with BackupPC), I can give it a try.
Any documentation available on how the rsync option should work?

Ooohhhh - might be fun then.

Defo set 'SMEServer no' .

See the wiki page and this for a bit of guidance, but it is a bit of trial and error!

http://affa.sourceforge.net/AffaPdfMan.pdf

I'll try and figure a bit more out next week but I am trying to get some other stuff fixed too!!
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: smeserver-affa
« Reply #8 on: September 24, 2021, 03:21:07 PM »
And all, please document stuff so we can repeat/test.

It is VITALLY important.


Thanks.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline mauro

  • ***
  • 101
  • +0/-0
Re: smeserver-affa
« Reply #9 on: September 27, 2021, 03:35:12 PM »
I can confirm that I've been able to backup a directory of a windows machine via affa through rsyncd without particular issues.
All parts should go together without forcing. You must remember that the parts you are reassembling were disassembled by you. Therefore, if you can't get them together again, there must be a reason. By all means, do not use a hammer.
-- IBM maintenance manual (1975)

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: smeserver-affa
« Reply #10 on: September 27, 2021, 06:05:14 PM »
Cool. Restore?

Can you detail that for us for the wiki?

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline mauro

  • ***
  • 101
  • +0/-0
Re: smeserver-affa
« Reply #11 on: September 28, 2021, 11:56:45 AM »
For restore you'd need also a sshd server running on the windows machine, which I don't have.
I tried with Freesshd but I could not get the exchange of keys to work...

Of course I can share my notes on the setup used for rsyncd backup.

All parts should go together without forcing. You must remember that the parts you are reassembling were disassembled by you. Therefore, if you can't get them together again, there must be a reason. By all means, do not use a hammer.
-- IBM maintenance manual (1975)

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: smeserver-affa
« Reply #12 on: September 28, 2021, 03:44:10 PM »
For restore you'd need also a sshd server running on the windows machine, which I don't have.
I tried with Freesshd but I could not get the exchange of keys to work...

OK - be nice to see if someone can get the restore to work somehow!!

Quote
Of course I can share my notes on the setup used for rsyncd backup.

If you can list them here - without passwords/server names - that would be good. You can also drop any notes in the SME Doc channel on Rocket and James will pick them up - he's cleaning up the page currently.

A note of your conf file etc too.

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline mauro

  • ***
  • 101
  • +0/-0
Re: smeserver-affa
« Reply #13 on: September 28, 2021, 05:02:52 PM »
I sent my notes on the Rocket chat.
All parts should go together without forcing. You must remember that the parts you are reassembling were disassembled by you. Therefore, if you can't get them together again, there must be a reason. By all means, do not use a hammer.
-- IBM maintenance manual (1975)

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: smeserver-affa
« Reply #14 on: September 29, 2021, 03:17:57 AM »
Haveing a go at getting a restore going..thank you for your work
--
qui scribit bis legit