rsync Digest, Vol 162, Issue 18

Thu Jun 23 01:20:22 UTC 2016

Folks,

Am 20.06.2016 um 22:01 schrieb Larry Irwin (gmail):
  > The scripts I use analyze the rsync log after it completes and then 
sftp's a summary to the root of the just completed rsync.
  > If no summary is found or the summary is that it failed, the folder 
rotation for that set is skipped and that folder is re-used on the 
subsequent rsync.
  > The key here is that the folder rotation script runs separately from 
the rsync script(s).

That is what we found to be important some years ago - do most of the 
management outside of the scripts that do rsync.

In particular, our scripts prepare a target backup directory with a name 
of the form:
	slash.20151122.seq.1379.current
   - the name of the area being backed-up is obvious, as is the date.
We use Tower of Hanoi management (see the update at 
https://en.wikipedia.org/wiki/Backup_rotation_scheme#Tower_of_Hanoi), so 
embed a sequence number in the backup directory name, and the suffix 
.current to indicate a current backup target.

we can keep on trying the rsync backups until we get success, and the 
scripts then remove the .current suffix.  This works because the rsync 
includes the --delete option, used since the backup target can be a 
recycled hard-linked directory: also recommended.

  > For each entity I want to rsync, I create a named folder to identify 
it and the rsync'd data is held in sub-folders:
  > daily.[1-7] and monthly.[1-3]
  > When I rsync, I rsync into daily.0 using daily.1 as the link-dest.
  > Then the rotation script checks daily.0/rsync.summary - and if it 
worked, it removes daily.7 and renames the daily folders.
  > On the first of the month, the rotation script removes monthly.3, 
renames the other 2 and makes a complete hard-link copy of daily.1 to 
monthly.1
  > It's been running now for about 4 years and, in my environment, the 
10 copies take about 4 times the space of a single copy.
  > (we do complete copies of linux servers - starting from /)
  > If there's a good spot to post the scripts, I'd be glad to put them up.

I'd recommend Tower of Hanoi, as referenced above, for managing sets of 
backups - far simpler than dealing with days and dates, etc.  For 
example, the above mentions special actions on the first of the month - 
what if that is missed because of some failure - more special case 
scripting? Tower of Hanoi is self-healing....

Regards

Rob.

Dr Robert C. Bell
HPC National Partnerships | Scientific Computing
CSIRO Information Management and Technology
T +61 3 9545 2368 (Docklands and Clayton)
T +61 3 9669 8102 (Docklands only)
Mob +61 428 108 333
Robert.Bell at csiro.au<mailto:Robert.Bell at csiro.au> | www.csiro.au
https://wiki.csiro.au/display/ASC/Scientific+Computing+Homepage

Also: Bureau of Meteorology Scientific Computing Services

Street: CSIRO SC Lvl 11, 700 Collins Street, Docklands Vic 3008, Australia
Postal: CSIRO SC Lvl 11, GPO Box 1289, Melbourne Vic 3001, Australia
Street: CSIRO IMT, Rivett Room RNW 2.03, Clayton Vic 3168, Australia
Postal: CSIRO IMT, Private Bag 10, Clayton South Vic 3169, Australia
Parcels: CSIRO IMT, Gate 3 Normanby Road, Clayton Vic 3168, Australia

PLEASE NOTE
The information contained in this email may be confidential or
privileged.  Any unauthorised use or disclosure is prohibited.  If
you have received this email in error, please delete it immediately
and notify the sender by return email. Thank you.  To the extent
permitted by law, CSIRO does not represent, warrant and/or
guarantee that the integrity of this communication has been
maintained or that the communication is free of errors, virus,
interception or interference.

Please consider the environment before printing this email.

On 22/06/2016 10:00 pm, rsync-request at lists.samba.org wrote:
> Send rsync mailing list submissions to
> 	rsync at lists.samba.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://lists.samba.org/mailman/listinfo/rsync
> or, via email, send a message with subject or body 'help' to
> 	rsync-request at lists.samba.org
>
> You can reach the person managing the list at
> 	rsync-owner at lists.samba.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of rsync digest..."
>
>
>
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
> ---------------------------------------
>
>
>
> Today's Topics:
>
>    1. Re: rsync script for snapshot backups (Dennis Steinkamp)
>    2. Re: rsync script for snapshot backups (Petros Angelatos)
>
>
>
> _______________________________________
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
> rsync mailing list
> rsync at lists.samba.org
> https://lists.samba.org/mailman/listinfo/rsync
>