rsync Digest, Vol 162, Issue 18

Thu Jun 23 14:59:35 UTC 2016

I agree that the Tower of Hanoi distribution is a good method.
It ensures a better depth. And with 10 copies, it would really, really 
be good depth.
With accounting systems though, it's good to have a backup of the data 
just -before- they close the fiscal period...
(the call we get is -- OMG we forgot to post that $250K before we closed 
-- admin is going to kill us -- can we go back?)
We've only had a couple of connectivity issues with 2 or 3 clients over 
4 years that caused it to miss one of the monthly rotations.
(the auto-retry off business hours is what has increased the resiliency 
of the process at sites with less than perfect internet service)

-- 
Larry Irwin
Email: lrirwin at alum.wustl.edu

On 06/22/2016 09:20 PM, Bell, Robert (CSIRO IM&T, Docklands) wrote:
> Folks,
>
> Am 20.06.2016 um 22:01 schrieb Larry Irwin (gmail):
>  > The scripts I use analyze the rsync log after it completes and then 
> sftp's a summary to the root of the just completed rsync.
>  > If no summary is found or the summary is that it failed, the folder 
> rotation for that set is skipped and that folder is re-used on the 
> subsequent rsync.
>  > The key here is that the folder rotation script runs separately 
> from the rsync script(s).
>
> That is what we found to be important some years ago - do most of the 
> management outside of the scripts that do rsync.
>
> In particular, our scripts prepare a target backup directory with a 
> name of the form:
>     slash.20151122.seq.1379.current
>   - the name of the area being backed-up is obvious, as is the date.
> We use Tower of Hanoi management (see the update at 
> https://en.wikipedia.org/wiki/Backup_rotation_scheme#Tower_of_Hanoi), 
> so embed a sequence number in the backup directory name, and the 
> suffix .current to indicate a current backup target.
>
> we can keep on trying the rsync backups until we get success, and the 
> scripts then remove the .current suffix.  This works because the rsync 
> includes the --delete option, used since the backup target can be a 
> recycled hard-linked directory: also recommended.
>
>  > For each entity I want to rsync, I create a named folder to 
> identify it and the rsync'd data is held in sub-folders:
>  > daily.[1-7] and monthly.[1-3]
>  > When I rsync, I rsync into daily.0 using daily.1 as the link-dest.
>  > Then the rotation script checks daily.0/rsync.summary - and if it 
> worked, it removes daily.7 and renames the daily folders.
>  > On the first of the month, the rotation script removes monthly.3, 
> renames the other 2 and makes a complete hard-link copy of daily.1 to 
> monthly.1
>  > It's been running now for about 4 years and, in my environment, the 
> 10 copies take about 4 times the space of a single copy.
>  > (we do complete copies of linux servers - starting from /)
>  > If there's a good spot to post the scripts, I'd be glad to put them 
> up.
>
> I'd recommend Tower of Hanoi, as referenced above, for managing sets 
> of backups - far simpler than dealing with days and dates, etc.  For 
> example, the above mentions special actions on the first of the month 
> - what if that is missed because of some failure - more special case 
> scripting? Tower of Hanoi is self-healing....
>
>
>
> Regards
>
> Rob.
>
> Dr Robert C. Bell
> HPC National Partnerships | Scientific Computing
> CSIRO Information Management and Technology
> T +61 3 9545 2368 (Docklands and Clayton)
> T +61 3 9669 8102 (Docklands only)
> Mob +61 428 108 333
> Robert.Bell at csiro.au<mailto:Robert.Bell at csiro.au> | www.csiro.au
> https://wiki.csiro.au/display/ASC/Scientific+Computing+Homepage
>
> Also: Bureau of Meteorology Scientific Computing Services
>
> Street: CSIRO SC Lvl 11, 700 Collins Street, Docklands Vic 3008, 
> Australia
> Postal: CSIRO SC Lvl 11, GPO Box 1289, Melbourne Vic 3001, Australia
> Street: CSIRO IMT, Rivett Room RNW 2.03, Clayton Vic 3168, Australia
> Postal: CSIRO IMT, Private Bag 10, Clayton South Vic 3169, Australia
> Parcels: CSIRO IMT, Gate 3 Normanby Road, Clayton Vic 3168, Australia
>
>
> PLEASE NOTE
> The information contained in this email may be confidential or
> privileged.  Any unauthorised use or disclosure is prohibited.  If
> you have received this email in error, please delete it immediately
> and notify the sender by return email. Thank you.  To the extent
> permitted by law, CSIRO does not represent, warrant and/or
> guarantee that the integrity of this communication has been
> maintained or that the communication is free of errors, virus,
> interception or interference.
>
> Please consider the environment before printing this email.
>
> On 22/06/2016 10:00 pm, rsync-request at lists.samba.org wrote:
>> Send rsync mailing list submissions to
>>     rsync at lists.samba.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>     https://lists.samba.org/mailman/listinfo/rsync
>> or, via email, send a message with subject or body 'help' to
>>     rsync-request at lists.samba.org
>>
>> You can reach the person managing the list at
>>     rsync-owner at lists.samba.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of rsync digest..."
>>
>>
>>
>> To unsubscribe or change options: 
>> https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>>
>> ---------------------------------------
>>
>>
>>
>> Today's Topics:
>>
>>    1. Re: rsync script for snapshot backups (Dennis Steinkamp)
>>    2. Re: rsync script for snapshot backups (Petros Angelatos)
>>
>>
>>
>> _______________________________________
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>>
>> rsync mailing list
>> rsync at lists.samba.org
>> https://lists.samba.org/mailman/listinfo/rsync
>>
>