Koozali.org: home of the SME Server

Standby machine for disaster recovery restore ?

Online david000

  • ****
  • 193
  • +0/-0
Standby machine for disaster recovery restore ?
« on: November 08, 2018, 10:58:15 AM »
Hi all,

I use SME 9.2 for our office email and keeping a little data in the i-bays.  It's a small setup of say 5 users with a few years email.

It's sitting on an oldish PC and works very well, however I'd like to have an 'easy' disaster recovery plan in place.  I've set up a 'twin' machine of the same type and specification (Both Dells of the same model) with the idea that I should be able to switch on the spare machine and restore directly from the NAS back ups I run.  This machine has SME9.x plus the same contribs. 

If a recovery is required it's certain that the install on the spare machine will be out of date, so an update process will be needed at some point in the process. This is something I'd usually do from the web based /server-manager/ and copy paste the contribs update commend from the alert email.  Ideally I'd do this post restore.

My question is, is this plausible ?

I'm out of office a fair bit so need to try and get something in place that's relatively easy to do should I not be around.

Thanks for your thoughts

Best

David




Online ReetP

  • *
  • 3,722
  • +5/-0
Re: Standby machine for disaster recovery restore ?
« Reply #1 on: November 08, 2018, 12:35:03 PM »
It depends on how much down time you are prepared to suffer.

If you only turn on the back up machine periodically you will have to wait for it to boot, run updates, and the restore from your backups,k and can you do that all remotely ??

That will take as long as it takes.

If you want a faster restore then run the backup machine as an Affa backup. You can dictate how often it runs backups, and if the main machine goes down you can 'Rise' the Affa box in minutes. It means the machine is on and you can update it as you go. You can shell in and access/rise at a command prompt if required - that may not be so easy with your proposed method.

(I think some people actually have the Affa box on a timer to wake it, run a backup, and put it to sleep again, if you don't want it running 24/7 - question is whether you can wake it remotely when you need to do a restore)

You pays your money and you takes your choice....
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline brianr

  • *
  • 988
  • +2/-0
Brian j Read
(retired, for a second time, still got 2 installations though)
The instrument I am playing is my favourite Melodeon.
.........

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Standby machine for disaster recovery restore ?
« Reply #3 on: November 08, 2018, 12:48:21 PM »
Affa
I have not used it for years, but Affa was practically magical for this use case when I did use it:
- Full system backup (including snapshots) maintained on a second system
- One line command to convert the backup system into a running clone of the active server for disaster recovery

Properly configured, all you need is remote access to the network to convert the backup server into the "live" server.

Restore during setup
If your situation can handle the recovery time, you could schedule a regular backup of the primary SME server to a USB drive, then practice regular system restores from "bare metal" (boot the install disk, restore during setup).  You would need to identify and document separate backup procedures anything for that is not included in this process, since it is possible to add software or web apps to your SME server that are not included in the USB backup by default.

Roll your own
You can also create your own manual backup using rsync or scp.  I still use a variation of smeclone.sh script in Bug  5701 to migrate a SME server to new hardware.  That script is intended for upgrading rather than backup -- but it includes notes that may be useful if you want to roll your own disaster recovery system.

Online david000

  • ****
  • 193
  • +0/-0
Re: Standby machine for disaster recovery restore ?
« Reply #4 on: November 08, 2018, 05:39:24 PM »
Thanks All,

Simplicity probably trumps reduced down time, especially if it's something I can do remotely say via a logmein style login to one of the client machines here.

In the past I have restored the system 'long hand' and have notes to that,  but it would be nice to have something less reliant on me being physically present in place.

One of the problems I recall with turning on the backup box 'as is' is that it shares the same IP as the 'live' one which causes issues, but I guess if I am only pulling down updates that can be changed, as long as it's in the correct ip range. 

Affa seems to offer some options - I'll have a read.

Online david000

  • ****
  • 193
  • +0/-0
Re: Standby machine for disaster recovery restore ?
« Reply #5 on: November 09, 2018, 02:47:01 PM »
Affa looks a good solution :)

My main concern is be that by Linux skills could be graded at 'script kiddy' level and the set up looks relatively complex.

If I understand it correctly it's a case of setting up a fresh SME server with the required contribs and installing Affa as per here:

https://wiki.contribs.org/Affa#tab=For_SME9

The Rise function looks straight forward, I think my concern will be setting up the backup so that everything gets backed up.

What do you reckon ?

Online ReetP

  • *
  • 3,722
  • +5/-0
Re: Standby machine for disaster recovery restore ?
« Reply #6 on: November 09, 2018, 06:02:27 PM »
It's not that bad, and if you ask nicely you'll get some help ;-)

It really is one of the outstanding features of SME, and it really should have a bigger name for itself - it would be nice to do a panel for it. The time to rise it is measured in a few minutes.

Particularly as you don't have much data you can easily test it. Just disconnect the new backup affa box before you do a 'rise' so you don't end up with two identical machines on the network :-)

So, the wiki page looks bad but it isn't really, though it is well worth gently reading through a few times just to get your bearings.

Essentially:

Code: [Select]
yum install smeserver-affa --enablerepo=smecontribs
Code: [Select]
mkdir /var/affa
Reboot

Add in your contribs at some point.

Code: [Select]
touch /etc/affa/backup-smeserver.conf
Add your remote server details - just swap the IP address to match

Add in extra dirs  you want included eg dehydrated/letsencrypt

Make sure your remote server to back up has ssh enabled and you can login with a password to send the ssh key (test first)

Test run your backup - the fisrt will take a while. Thereafter it will be very quick as it only copies the changed files.

That's about it - tweak away.....

NB. Make sure you have enough disk space for all the 'Keeps'

You can go back and pull individual files from each backup if needed which is handy when they delete something they later wanted...... :-)

Enjoy.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Online david000

  • ****
  • 193
  • +0/-0
Re: Standby machine for disaster recovery restore ?
« Reply #7 on: November 12, 2018, 06:11:33 PM »
ReetP Thank you.

I'll have a play and see how I get on. I may indeed ask for some help  8-)

Just for clarification, when you say "Add in your contribs at some point." do you mean install them afresh via yum or include them in some way in the Affa backup ?

Quote
Make sure your remote server to back up has ssh enabled and you can login with a password to send the ssh key (test first)

I think more reading required but I'll have a go and ask here later with the inevitable questions.

Thank you again :)





 


Online ReetP

  • *
  • 3,722
  • +5/-0
Re: Standby machine for disaster recovery restore ?
« Reply #8 on: November 12, 2018, 06:24:48 PM »
When you set up your affa box, at some point add in any of your existing contribs so they are there ready in case you 'Rise' it. You can do that any time.

If you read the manual and have a go, no one will mind helping answer sensible questions :-)
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Online david000

  • ****
  • 193
  • +0/-0
Re: Standby machine for disaster recovery restore ?
« Reply #9 on: November 13, 2018, 01:24:07 PM »
When you set up your affa box, at some point add in any of your existing contribs so they are there ready in case you 'Rise' it. You can do that any time.

If you read the manual and have a go, no one will mind helping answer sensible questions :-)

Thanks. I will do :)

Online david000

  • ****
  • 193
  • +0/-0
Re: Standby machine for disaster recovery restore ?
« Reply #10 on: November 29, 2018, 04:43:58 PM »
A quick update.

I fired up the 'spare' machine and ran the

Code: [Select]
yum clean all
yum update

commend as it had not been on for 12 months or so. It turned out that I hadn't installed any contribs so I quickly added these (Roundcube, diskusage and fetchmail) to match my production server.

Affa installed, I added and amended the config file by swapping the IP addresses as above, sent the key and now seem to have completed a back up.

It really is very easy and the documentation page on the money. My biggest problem was working out how to create and edit the config file  :lol:

A couple of questions -

Do I need to set up a cron job to schedule repeat runs of the job, or does the config take care of this ?

Reading the notes, it looks like I can test the machine with a 'Raise', and then a 'Undo-rise'.   Once undone it would seem it reverts to being a back-up box with no further action needed ?


Thanks again for everyone's comments on this.

Online ReetP

  • *
  • 3,722
  • +5/-0
Re: Standby machine for disaster recovery restore ?
« Reply #11 on: November 29, 2018, 05:30:47 PM »
Yes make sure you keep your affa box reasonably up to date.

I'll try and modify the 'edit conf file' to clarify (I can see it says 'touch' to create, but not how to edit !)

You don't need to setup a cron as Affa does that for you.

Make sure you set your time schedule in the conf file as per the notes:

Code: [Select]
mcedit /etc/affa/my_server.conf
Or

Code: [Select]
nano /etc/affa/my_server.conf
Quote
TimeSchedule   HHMM   yes   2230   

The time the job will run. Use exactly 4 digits, no colon, no point.
Important: Using the proper format HHMM is essential. Badly formatted TimeSchedule will cause strange Perl errors. Multiple runs can be achieved by repeating the line for each run time.

So like this:

Code: [Select]
TimeSchedule 0601
TimeSchedule 1001
TimeSchedule 2001

Then make the jobs with this (it has some double documentation I'll fix)

Quote
affa --make-cronjobs
Configures the cronjobs as scheduled in the jobs records. By default this command will by executed by 'cronjobs' every 15 minutes. So you don't need to run this command manually, you can also just wait max. 15 minutes before your updates job configurations become effective.

You can use this to check jobs:

Quote
affa --configcheck
Checks the syntax and values in all configuration files found in /etc/affa/. Run this command after modifying the configuration. Lower/Upper case errors in property names are corrected automatically.

Yes, Rise and Undo-Rise should work but YMMV :-)

Please test and let us know if you have any issues.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Online david000

  • ****
  • 193
  • +0/-0
Re: Standby machine for disaster recovery restore ?
« Reply #12 on: November 29, 2018, 06:07:06 PM »
ReetP.  Thank you.

Yes, it was trying to edit in touch which confused me, although in fairness the page does say to use an editor such as nano.

I'll work through your notes and test out the install.  The current TimeSchedule setting, copied form the example is:

Code: [Select]
TimeSchedule=0630 which passes the configcheck, the difference to your example above being the '='.  I'll check to see if it's backed up again tomorrow morning.

I checked the admin email account on the back up and there is an alert from affa:
Code: [Select]
*
* The following packages are installed on (production server IP),
* but they are missing on this backup host:
*
libmcrypt-2.5.8-9.el6
perl-Data-Uniqid-0.12-2.el6
perl-TermReadKey-2.30-13.el6
perl-File-Copy-Recursive-0.38-4.el6
perl-Parse-RecDescent-1.965-1.el6
php-mcrypt-5.3.3-5.el6
perl-Test-Pod-1.40-1.el6
perl-Unicode-String-2.09-12.el6
php-pear-Mail-mimeDecode-1.5.5-1.el6
perl-Mail-IMAPClient-3.37-1.el6
perl-NTLM-1.09-1.el6
perl-IO-Tee-0.64-1.el6
imapsync-1.684-1.el6

*
* The following packages are installed on both,
* the source (Production Server IP) and on this backup host,
* but the version does not match:
*
gpg-pubkey-xxxxxxxxx-xxxxxx
sqlite-3.7.17-9.el6.sme

I assume it may be legacy stuff as they are not RPMs\Contribs I've knowingly installed as stand alone items. Is it reasonable to ignore ?

On the time schedule, I'll want to keep the existing daily back up arrangement to NAS for now so I guess best to try and avoid an over lap. Belt and Braces and all that. 

« Last Edit: November 29, 2018, 06:18:34 PM by david000 »

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: Standby machine for disaster recovery restore ?
« Reply #13 on: November 29, 2018, 11:01:48 PM »
david

Yes Affa backup server (with rise function) works great, the backup server can even be at a remote location for added security. Watch out for data transfer/speed limits & time it takes to complete a backup. If you have Gigabytes worth of files changing, then backup can take a long time & maybe even exceed monthly bandwidth allowance.

If the backup server is local (on LAN) no problems.

One word of warning/catch22 issue, is that if your sme server acts as gateway, & it goes down/fails, then you may not have remote access via ssh or whatever to do rising etc from a remote location.
ie if your sme server is down then you will not have incoming internet access for remote control & configuration etc.

You can create a step by step set of instructions for the most technically literate person to follow, &/or you can talk them through the steps over the phone.

Nothing worse than a server going down when you are away on holiday so be well prepared & test test test beforehand to make sure it all really works.
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Online ReetP

  • *
  • 3,722
  • +5/-0
Re: Standby machine for disaster recovery restore ?
« Reply #14 on: November 29, 2018, 11:09:17 PM »
Good points well made :-)

The first sync is the longest. After that it only syncs changes so should be much faster.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation