Backup scripts

Kevin Korb kmk at
Tue Sep 9 18:45:42 MDT 2014

Hash: SHA1

Have you considered more advanced methods such as subvolume snapshots
provided by ZFS and BTRFS?  At work we were forced to abandon rsync
- --link-dest because of the amount of time it takes to delete old
backups when the data is primarily many millions of small files
(shared web hosting company).

Essentially, instead of a new target and a reference to the previous
on each run your target is the same on each run and after the run
finishes you tell the filesystem to make a snapshot of it.  Then when
it is time to delete an old backup it is a single filesystem operation
rather than a massive recursive rm.

On 09/09/2014 08:32 PM, Henri Shustak wrote:
> Hello,
> The --link-dest approach is the way LBackup works and this is also
> what allows the lcd command (part of LBackup) to move you back and
> forth in time while you remain in the same directory relative
> directory. This makes finding a particular data quite easy during a
> partial restore.
> In particular, having a full directory makes restore from a point
> in time easy and (as Kevin pointed out) there is no need to merge
> various incremental backups. Each point it time simply appears as a
> full file system tree. Although, the space required is lowered
> somewhat due to the use of hard links.
> --------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution
> On 5/09/2014, at 9:44 AM, Kevin Korb <kmk at> wrote:
> That is how to do an incremental backup.  But rsync can do a
> snapshot with --link-dest instead of --backup.  The big difference
> is that with --link-dest each backup will contain the entire tree
> not just the different files.  Therefore if you need to restore an
> old version you can do so without merging the current with the
> old.
> Also, no -z on a local copy ;)
> On 09/04/2014 05:41 PM, Greg Deback (rsync) wrote:
>>>> Hi,
>>>> I think the words used are not correct, that's why you (and
>>>> Kevin) are a bit confused. You are in fact asking for
>>>> "incremental" backup, not a "snapshot" : 
>>>> In
>>>> this case, you can use rsync with a few flags to delete (or
>>>> move in another directory) the modified/deleted files
>>>> regarding the last "backup".
>>>> This would look something like this ((h)uman, do not
>>>> (x)cross filesystem boundaries, (P)rogress, (H)ard-links,
>>>> (a)rchive, aka 
>>>> recursive+symlinks+perms+time+owner+group+specials and
>>>> g(z)ip):
>>>> # Cloning your data to a distant folder with timestamp rsync 
>>>> -hxPHaz --exclude=<exclude_list> <source> <target>/$NOW/ # 
>>>> Incremental backup only rsync -hxPHaz
>>>> --exclude=<exclude_list> --delete --backup
>>>> --backup-dir=<backup> <source> <target>/latest/ # You can
>>>> even "suffix" the modified/deleted files rsync -hxPHaz 
>>>> --exclude=<exclude_list> --delete --backup
>>>> --backup-dir=<backup> --suffix=~$NOW <source>
>>>> <target>/latest/
>>>> Hope that will help, Greg
>>>> On Thu, Sep 4, 2014 at 11:21 PM, Kevin Korb
>>>> <kmk at <mailto:kmk at>> wrote:
>>>> Yes, you have different scripts now at least.  But I still
>>>> don't see any snapshotting.  The only difference in these is
>>>> that the "full" one rsyncs to a date+time stamped directory
>>>> while the "snapshot" one rsyncs to a directory named
>>>> snapshot.  As far as I can tell both will be a complete copy
>>>> with no relationship to the other.
>>>> On 09/04/2014 05:17 PM, Chris wrote:
>>>>> On Thu, 2014-09-04 at 17:00 -0400, Kevin Korb wrote:
>>>>>> Do you actually have any snapshots currently?  From the 
>>>>>> scripts you posted it seems to just be rsyncing to the
>>>>>> same dir every run and only claiming to be making
>>>>>> snapshots.  If you do have snapshots now then something
>>>>>> else is happening in addition to this script.
>>>>> It was pointed out to me in a direct message that I had 
>>>>> inadvertently posted the same scripts (full backup) in
>>>>> pastebin. Here are the correct one.
>>>>> Full
>>>>> Snapshot
>>>>>> On 09/04/2014 04:58 PM, Chris wrote:
>>>>>>> On Thu, 2014-09-04 at 12:46 -0400, Kevin Korb wrote:
>>>>>>>> The scripts you posted look the same to me.  And I
>>>>>>>> don't see any form of snapshotting.  The $NOW
>>>>>>>> variable is set and is echoed but it is never
>>>>>>>> actually used.  For an rsync snapshot I would expect
>>>>>>>> to see either rsync --link-dest or a cp -al depending
>>>>>>>> on the age of the script.
>>>>>>>> The only reason to need two scripts would be because
>>>>>>>> the snapshot would need 3 dirs to work with (the
>>>>>>>> source, the target, and the previous backup) but that
>>>>>>>> could also be handled with 1 script and a simple
>>>>>>>> existing check.
>>>>>>>> On 09/04/2014 12:30 PM, Chris wrote:
>>>>>>>>> I have two scripts that a kind soul on this list
>>>>>>>>> wrote for me over 4yrs ago. I got to looking at
>>>>>>>>> them the other day because my old box crashed and
>>>>>>>>> had to build a new one also got a new backup USB
>>>>>>>>> drive since I'm still copying over things from the
>>>>>>>>> old one. The first one is for a full backup:
>>>>>>>>> Works great, does exactly what it's supposed to do.
>>>>>>>>> The second is for a 'snapshot' which is where I get
>>>>>>>>> a bit confused. I would think that a 'snapshot'
>>>>>>>>> would be just the changed files either since the
>>>>>>>>> last full backup or since the last 'snapshot' the
>>>>>>>>> night before. It seems though that it's actually
>>>>>>>>> the same as a full backup. I don't profess to be a
>>>>>>>>> script person so I have no idea if it's doing what
>>>>>>>>> it should or something needs to be changed. Below
>>>>>>>>> is the 2nd script:
>>>>>>>>> Any advice would be appreciated.
>>>>>>>>> Chris
>>>>>>> Thanks Kevin, I guess for now I'll leave them as they
>>>>>>> are until I can get smart on scripting.
>>>>>>> Chris
>>>> -- Please use reply-all for most replies to avoid omitting
>>>> the mailing list. To unsubscribe or change options: 
>>>> Before
>>>> posting, read:
>> -- Please use reply-all for most replies to avoid omitting the
>> mailing list. To unsubscribe or change options:
>> Before posting,
>> read:

- -- 
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at  (work)
	Orlando, Florida		kmk at (personal)
	Web page:
	PGP public key available on web site.
Version: GnuPG v2


More information about the rsync mailing list