Koozali.org: home of the SME Server

Backup Configuration - workstation

Offline smnirosh

  • ****
  • 329
  • +0/-0
  • Learning never ends
Backup Configuration - workstation
« on: November 20, 2015, 10:49:49 AM »
Number of rotating backup sets: ?
Daily backups in each set: ?
Workstation backup time of day: 10:00 PM
Optional backup session timeout: ?
Don't timeout full backup sessions: ?
Backup compression level: 6
Full backup is allowed on: Sunday

Hi guys, I want to schedule a workstation backup on sme server 9.
Can some one explain above ?s for me?

Offline Stefano

  • *
  • 10,839
  • +2/-0
Re: Backup Configuration - workstation
« Reply #1 on: November 20, 2015, 10:56:36 AM »
Did you search the wiki? Please do, then come here if you have doubts, thank you

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: Backup Configuration - workstation
« Reply #2 on: November 20, 2015, 02:25:16 PM »
smnirosh

example here
http://wiki.contribs.org/Backup_with_dar

You could experiment & find out for yourself what each setting does.
Not a hard thing to do.
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline smnirosh

  • ****
  • 329
  • +0/-0
  • Learning never ends
Re: Backup Configuration - workstation
« Reply #3 on: November 20, 2015, 02:34:50 PM »
thanks very much janet. I will refer this doc.

Offline Mirfster

  • **
  • 61
  • +0/-0
Re: Backup Configuration - workstation
« Reply #4 on: November 26, 2015, 02:11:40 PM »
FYI, it is a little confusing regarding the routines.  Especially where it talks about values of 7.  I swear I had to read it like 10 times to get the jist of it to determine how I wanted it to work with the desired "Full" and "Incremental" backups.

Maybe it was just me though...  :P

Offline warren

  • *
  • 293
  • +0/-0
Re: Backup Configuration - workstation
« Reply #5 on: November 27, 2015, 12:28:37 AM »
FYI, it is a little confusing regarding the routines.  Especially where it talks about values of 7.  I swear I had to read it like 10 times to get the jist of it to determine how I wanted it to work with the desired "Full" and "Incremental" backups.

Maybe it was just me though...  :P

I need to fess up !! I still have to re-read it a couple of times myself  :-(

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: Backup Configuration - workstation
« Reply #6 on: November 27, 2015, 02:19:47 AM »
Mirfster

I think some of the "mumbo jumbo"
ie the part that says:
"The number of increments in a set is only equal to the "Daily backups in each set" if a multiple of 7. For instance choose 27, and you only get 21 increments, whereas 28 gives 28 increments. Or choose 3, and it gives 7 increments. (This assumes that "Full backup is allowed on" is not set to everyday)",
is really relating to unusual combination of settings where the number of incrementals does not match or equate to the full backup period eg 7 incrementals matches a full backup every 7 days, so don't be confused by that.
Use settings that are relative to each other eg weekly (7 day) full backup with 7 incrementals.

Hopefully the following simplifies it, keeping in mind the choice of full backup on a specified day or every day will also interact with these settings.

If you wanted to do a full backup every day, & only retain 1 historical backup, then select
Number of daily backups contained in each set is 1.
Number of rotating backup sets is 1.

If you wanted to do a full backup once every 7 days, then daily incremental backups every other day, & only retain 1 historical backup, select
Number of daily backups contained in each set is 7.
Number of rotating backup sets is 1.

If you wanted to do a full backup once every 14 days, then daily incremental backups every other day, & only retain 1 historical backup, select
Number of daily backups contained in each set is 14.
Number of rotating backup sets is 1.

If you wanted to do a full backup once every 28 days, then daily incremental backups every other day, & only retain 1 historical backup, select
Number of daily backups contained in each set is 28.
Number of rotating backup sets is 1.

For any of the above, if you want to retain more historical backup sets, then change the
Number of rotating backup sets to 2 or 3 or 4 as desired


So let's say the full backup is set to run on Friday only, & the backup time is set for 11pm.
On the first Friday night at 11pm a full backup is done, this may take many hours or even time out if there is a lot of data to backup.

On the first Saturday night at 11pm if the full backup did not complete the night before (timed out as per time out setting or reached 24 hours run time), then the full backup is continued on from where it left off.
If the full backup did complete, then an incremental backup is done, so only the additions & changes of data are added to the first full backup.
These are added as incremental backups, in a seperate incremental backup file with a date identifier.

Then every day of the backup period (being the Number of daily backups contained in each set), ie 7, 14 or 28 days, an incremental backup is done.

Then on the Friday (being either day 8, 15 or 29 in examples above) at 11pm a new full backup is done.
After that incremental backups are done each day, & so the cycle continues on.

Note, when a full backup is done:
If Number of rotating backup sets is set to 1, then the original (first) backup is deleted.
If Number of rotating backup sets is set to 2, then the original (first) backup set is retained & a new backup set is created/started.
If Number of rotating backup sets is set to 3, then the first & second backup sets are retained & a new backup set is created/started.
If Number of rotating backup sets is set to 4, then the first, second & third backup sets are retained & a new backup set is created/started.
.... & so on.

Note that if a full restore is needed, the full & incremental backups are used, so all data is restored.


A "practical" backup approach may be to configure for a Full backup every 7 days & daily incremental backups in between, with 2 rotating sets, so a new full backup is created each Friday & you retain a complete 7 days worth of backups for historical purposes.
You could even extend that to retaining 4 sets of historical backups, in case you wanted to restore some individual files from 25 days ago.

Another approach is to have 2 backup USB disks, & swap the disk every Friday (during the day) so the new full backup (on Friday night) is done to the alternate USB drive, and that alternate drive is used for the rest of the 7 days period for daily incremental backups.

How often you want to create a full backup & how many daily incrementals you want to have in a set & how many historical sets & whether you swap disks off site etc, is really a part of your overall backup protocol or strategy for the site/server in question.

Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline smnirosh

  • ****
  • 329
  • +0/-0
  • Learning never ends
Re: Backup Configuration - workstation
« Reply #7 on: November 27, 2015, 07:41:40 AM »
Dear janet, great. This example is really good for a person like me who has poor knowledge of sme server. Thanks very much

Offline Mirfster

  • **
  • 61
  • +0/-0
Re: Backup Configuration - workstation
« Reply #8 on: November 27, 2015, 02:27:50 PM »
Thanks Janet.  Some great information there.  Perhaps the DAR Backup page could be updated to include various routines to assist others?

For one company, they wanted to deviate from the * 7 routine and be able to go back up to ~ 1 year.  Their initial "Full Backup" contained ~ 500 GB of data and "Incremental Backups" average 1 -3 GB.

Based on my interpretation, I offered the following possible routines:

Full Backup ~ Monthly (Each Set will have 31 days of data) – Results in 372 Days Total
Number of Rotating Backup Sets = 12
Daily Backups in Each Set = 31
Workstation Backup Time of Day (Hour/Min) = 0:01 (* This is basically 12:01 AM)
Don’t Time Out Full Backup Sessions = Checked
Backup Compression Level = 6
Full Backup is Allowed on = Everyday (Have to since it is impossible to know what actual weekday the last day will fall on)

Full Backup ~ Quarterly/Every 3 Months (Each Set will have 122 days of data) – Results in 366 Days Total
Number of Rotating Backup Sets = 3
Daily Backups in Each Set = 122
Workstation Backup Time of Day (Hour/Min) = 0:01 (* This is basically 12:01 AM)
Don’t Time Out Full Backup Sessions = Checked
Backup Compression Level = 6
Full Backup is Allowed on = Everyday (Have to since it is impossible to know what actual weekday the last day will fall on)

Full Backup ~ Tri-Annual/Every 4 Months (Will have 91 days of data) – Results in 364 Days Total
Number of Rotating Backup Sets = 4
Daily Backups in Each Set = 91 (** Note: This is also a multiple of 7)
Workstation Backup Time of Day (Hour/Min) = 0:01 (* This is basically 12:01 AM)
Don’t Time Out Full Backup Sessions = Checked
Backup Compression Level = 6
Full Backup is Allowed on = Everyday (Have to since it is impossible to know what actual weekday the last day will fall on)

Full Backup ~ Bi-Annual/Every 6 Months (Will have 182 days of data) – Results in 364 Days Total
Number of Rotating Backup Sets = 2
Daily Backups in Each Set = 182 (** Note: This is also a multiple of 7)
Workstation Backup Time of Day (Hour/Min) = 0:01 (* This is basically 12:01 AM)
Don’t Time Out Full Backup Sessions = Checked
Backup Compression Level = 6
Full Backup is Allowed on = Everyday (Have to since it is impossible to know what actual weekday the last day will fall on)
 
The Owner decided to go with the "Full Backup ~ Bi-Annual/Every 6 Months" option; since he really wanted the ability to restore file(s) up to a year back.

Keep in mind that other Disaster Recovery methods are in-play at the same time:
1) SME is running within ESXi and periodic "Snapshots" are being taken as well as manually before each Server Update (may want to copy the Snapshots or even entire VM elsewhere also...)
2) ESXi Data Volume is really a RAID6 (so there is redundancy there)
3) AFFA is running on an "off-site" Server (so a "Rise" can always be done)
4) DAR Backups are housed on a FreeNas Server (Currently being done via CIFs in the DAR Backup)
5) FreeNas Data Volume is a Pool of 2 RAIDZ2 vDevs with "Periodic Snapshots" of the Data Volumes occurring
6) Finally, FreeNas Data Set housing the SME Backups will be RSYNC'd to a re-purposed Drobo (TBD)

I still think that there may be better methods/scenarios to use that could streamline/simplify the process, but for now this is what they wanted.

If you see a flaw in my logic regarding the DAR Backup Routines (or anywhere else for that matter); please feel free to let me know.  After all, in the end it is really about providing the best possible solution.

*** Forgot to mention that the CIFs connection is over a dedicated 10GB Virtual Network between SME and the FreeNas Server...
« Last Edit: November 27, 2015, 02:38:06 PM by Mirfster »

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: Backup Configuration - workstation
« Reply #9 on: November 28, 2015, 12:39:30 AM »
Mirfster

Well that sounds like a very comprehensive backup system.
You do not mention the physical storage size of the FreeNAS.
IIRC  keep in mind that the backup "disk" requires spare space twice the size of a backup, as a tmp file is created during the backup & then the tmp file is saved to a permanent filename just at the end of the backup. This will be more of an issue when a full backup is done especially when you already have a number of full backups already saved on the disk.

Also related to backup disk size, is the consideration of how many backups will fit on one disk. This may limit the time period you can cover.

Also I would say when retaining a full backup for 6 months you should really be using rotating backup  disks, so you are not reliant on one disk eg  probably for your level of requirement 4 to 6 rotating disks would be appropriate (2 would theoretically be enough). I would also rotate (swap) these disks frequently (eg daily), & that way you would have all of your data on all of the disks, so lots of redundancy, except for the last few days incrementals which would be limited to the disk last used on those days. Start the process with a full backup to each disk on consecutive days, then change the settings to daily incrementals.

I would also ask/add if you are doing backup to a Affa dedicated server, then when is the main server having time to run multiple full backups. It is not good to have different backups running that overlap each other. Is the comprehensive arrangement you describe already functioning (& smoothly), or is it a proposed system yet to be implemented ?

Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline Mirfster

  • **
  • 61
  • +0/-0
Re: Backup Configuration - workstation
« Reply #10 on: November 28, 2015, 05:15:09 PM »
You do not mention the physical storage size of the FreeNAS.

Sorry about that, the current allocated storage size on FreeNas for SME Backups is 15 TB.  However, there actually is 19.9 TB of space on the Volume.  Since the pool is composed of two vdevs (six disks in each vdev), each vdev can be increased by replacing all six drives with larger ones (when/if needed).  However, that would entail replacing the drives one at a time and re-silvering in between.  Thoughts are by the time that is needed prices on Enterprise Hard Drives would have decreased significantly.  Currently, all the drives are 3TB and by the same time next year 4TB drives will be about the same price (was that way for 2TB/3TB drives).

I would also ask/add if you are doing backup to a Affa dedicated server, then when is the main server having time to run multiple full backups. It is not good to have different backups running that overlap each other. Is the comprehensive arrangement you describe already functioning (& smoothly), or is it a proposed system yet to be implemented ?

Secondary SME Server (AFFA) is currently being rebuilt for its role.  We wanted to let the new setup have a couple weeks to ensure everything else was working as desired.  So AFFA is not yet "in-place".  Once the Server rebuild is completed, I will do the initial sync on-site.  After that the thinking for the AFFA time is to start @ 8:00 PM.  This should provide a four hour window before the DAR Backup kicks off (@ 12:01 AM).  However, really will be dependent on the connection speed of where the Server will end up being housed.  Right now, they are wanting to have it located at one of the Co-Owner's house.  So I will be eventually delivering it there and finalizing the configurations.

As always, thanks for your input.

Offline Stefano

  • *
  • 10,839
  • +2/-0
Re: Backup Configuration - workstation
« Reply #11 on: November 28, 2015, 06:46:07 PM »
Quote
Perhaps the DAR Backup page could be updated to include various routines to assist others?

Mirfster, please feel free to join wiki team and amend the related pages, thank you

Offline Stefano

  • *
  • 10,839
  • +2/-0
Re: Backup Configuration - workstation
« Reply #12 on: November 28, 2015, 06:52:14 PM »
Once the Server rebuild is completed, I will do the initial sync on-site.  After that the thinking for the AFFA time is to start @ 8:00 PM.  This should provide a four hour window before the DAR Backup kicks off (@ 12:01 AM).  However, really will be dependent on the connection speed of where the Server will end up being housed.  Right now, they are wanting to have it located at one of the Co-Owner's house.  So I will be eventually delivering it there and finalizing the configurations.

As always, thanks for your input.

please, take a look at
http://bugs.contribs.org/show_bug.cgi?id=9127

having concurrent (even potentially) backup routines running can lead to problems..

Offline Stefano

  • *
  • 10,839
  • +2/-0
Re: Backup Configuration - workstation
« Reply #13 on: November 28, 2015, 06:55:21 PM »

For one company, they wanted to deviate from the * 7 routine and be able to go back up to ~ 1 year.  Their initial "Full Backup" contained ~ 500 GB of data and "Incremental Backups" average 1 -3 GB.

----zac!-----

inpressive :-)

Quote
*** Forgot to mention that the CIFs connection is over a dedicated 10GB Virtual Network between SME and the FreeNas Server...

try NFS too.. many years ago, I tested SME's backup toward a Freenas, using Cifs and then NFS.. NFS was at least 3 times faster (without changing anything, same hw, same lan infra)

HTH

Offline Mirfster

  • **
  • 61
  • +0/-0
Re: Backup Configuration - workstation
« Reply #14 on: November 28, 2015, 10:53:48 PM »
Mirfster, please feel free to join wiki team and amend the related pages, thank you

Just saw that the page was updated (http://wiki.contribs.org/Backup_with_dar) by ‎RayMitchell yesterday (Thanks Ray!).  Will still see about helping by joining the Wiki Team though.

Took a look at the bug you listed and will be sure to keep that in mind.  Don't want to end up creating more issues.

Still need to test the NFS route, but may also look into SAN or iSCSI.  Haven't gotten to it yet.  :)

smnirosh, sorry if this hijacked your post.  Wasn't my intention and I will "cease and desist".