Backup scripts
Kevin Korb
kmk at sanitarium.net
Tue Sep 9 18:45:42 MDT 2014
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Have you considered more advanced methods such as subvolume snapshots
provided by ZFS and BTRFS? At work we were forced to abandon rsync
- --link-dest because of the amount of time it takes to delete old
backups when the data is primarily many millions of small files
(shared web hosting company).
Essentially, instead of a new target and a reference to the previous
on each run your target is the same on each run and after the run
finishes you tell the filesystem to make a snapshot of it. Then when
it is time to delete an old backup it is a single filesystem operation
rather than a massive recursive rm.
On 09/09/2014 08:32 PM, Henri Shustak wrote:
> Hello,
>
> The --link-dest approach is the way LBackup works and this is also
> what allows the lcd command (part of LBackup) to move you back and
> forth in time while you remain in the same directory relative
> directory. This makes finding a particular data quite easy during a
> partial restore.
>
> In particular, having a full directory makes restore from a point
> in time easy and (as Kevin pointed out) there is no need to merge
> various incremental backups. Each point it time simply appears as a
> full file system tree. Although, the space required is lowered
> somewhat due to the use of hard links.
>
> --------------------------------------------------------------------
>
>
This email is protected by LBackup, an open source backup solution
> http://www.lbackup.org
>
> On 5/09/2014, at 9:44 AM, Kevin Korb <kmk at sanitarium.net> wrote:
>
> That is how to do an incremental backup. But rsync can do a
> snapshot with --link-dest instead of --backup. The big difference
> is that with --link-dest each backup will contain the entire tree
> not just the different files. Therefore if you need to restore an
> old version you can do so without merging the current with the
> old.
>
> Also, no -z on a local copy ;)
>
> On 09/04/2014 05:41 PM, Greg Deback (rsync) wrote:
>>>> Hi,
>>>>
>>>> I think the words used are not correct, that's why you (and
>>>> Kevin) are a bit confused. You are in fact asking for
>>>> "incremental" backup, not a "snapshot" :
>>>> http://en.wikipedia.org/wiki/Snapshot_(computer_storage). In
>>>> this case, you can use rsync with a few flags to delete (or
>>>> move in another directory) the modified/deleted files
>>>> regarding the last "backup".
>>>>
>>>> This would look something like this ((h)uman, do not
>>>> (x)cross filesystem boundaries, (P)rogress, (H)ard-links,
>>>> (a)rchive, aka
>>>> recursive+symlinks+perms+time+owner+group+specials and
>>>> g(z)ip):
>>>>
>>>> # Cloning your data to a distant folder with timestamp rsync
>>>> -hxPHaz --exclude=<exclude_list> <source> <target>/$NOW/ #
>>>> Incremental backup only rsync -hxPHaz
>>>> --exclude=<exclude_list> --delete --backup
>>>> --backup-dir=<backup> <source> <target>/latest/ # You can
>>>> even "suffix" the modified/deleted files rsync -hxPHaz
>>>> --exclude=<exclude_list> --delete --backup
>>>> --backup-dir=<backup> --suffix=~$NOW <source>
>>>> <target>/latest/
>>>>
>>>> Hope that will help, Greg
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Sep 4, 2014 at 11:21 PM, Kevin Korb
>>>> <kmk at sanitarium.net <mailto:kmk at sanitarium.net>> wrote:
>>>>
>>>> Yes, you have different scripts now at least. But I still
>>>> don't see any snapshotting. The only difference in these is
>>>> that the "full" one rsyncs to a date+time stamped directory
>>>> while the "snapshot" one rsyncs to a directory named
>>>> snapshot. As far as I can tell both will be a complete copy
>>>> with no relationship to the other.
>>>>
>>>> On 09/04/2014 05:17 PM, Chris wrote:
>>>>> On Thu, 2014-09-04 at 17:00 -0400, Kevin Korb wrote:
>>>>>> Do you actually have any snapshots currently? From the
>>>>>> scripts you posted it seems to just be rsyncing to the
>>>>>> same dir every run and only claiming to be making
>>>>>> snapshots. If you do have snapshots now then something
>>>>>> else is happening in addition to this script.
>>>>
>>>>> It was pointed out to me in a direct message that I had
>>>>> inadvertently posted the same scripts (full backup) in
>>>>> pastebin. Here are the correct one.
>>>>
>>>>> Full http://pastebin.com/dEk7kBip
>>>>
>>>>> Snapshot http://pastebin.com/H7SuABN1
>>>>
>>>>>>
>>>>>> On 09/04/2014 04:58 PM, Chris wrote:
>>>>>>> On Thu, 2014-09-04 at 12:46 -0400, Kevin Korb wrote:
>>>>>>>> The scripts you posted look the same to me. And I
>>>>>>>> don't see any form of snapshotting. The $NOW
>>>>>>>> variable is set and is echoed but it is never
>>>>>>>> actually used. For an rsync snapshot I would expect
>>>>>>>> to see either rsync --link-dest or a cp -al depending
>>>>>>>> on the age of the script.
>>>>>>>>
>>>>>>>> The only reason to need two scripts would be because
>>>>>>>> the snapshot would need 3 dirs to work with (the
>>>>>>>> source, the target, and the previous backup) but that
>>>>>>>> could also be handled with 1 script and a simple
>>>>>>>> existing check.
>>>>>>>>
>>>>>>>> On 09/04/2014 12:30 PM, Chris wrote:
>>>>>>>>> I have two scripts that a kind soul on this list
>>>>>>>>> wrote for me over 4yrs ago. I got to looking at
>>>>>>>>> them the other day because my old box crashed and
>>>>>>>>> had to build a new one also got a new backup USB
>>>>>>>>> drive since I'm still copying over things from the
>>>>>>>>> old one. The first one is for a full backup:
>>>>>>>>>
>>>>>>>>> http://pastebin.com/XF6Zm42A
>>>>>>>>>
>>>>>>>>> Works great, does exactly what it's supposed to do.
>>>>>>>>> The second is for a 'snapshot' which is where I get
>>>>>>>>> a bit confused. I would think that a 'snapshot'
>>>>>>>>> would be just the changed files either since the
>>>>>>>>> last full backup or since the last 'snapshot' the
>>>>>>>>> night before. It seems though that it's actually
>>>>>>>>> the same as a full backup. I don't profess to be a
>>>>>>>>> script person so I have no idea if it's doing what
>>>>>>>>> it should or something needs to be changed. Below
>>>>>>>>> is the 2nd script:
>>>>>>>>>
>>>>>>>>> http://pastebin.com/MkBzJnux
>>>>>>>>>
>>>>>>>>> Any advice would be appreciated.
>>>>>>>>>
>>>>>>>>> Chris
>>>>>>>>>
>>>>>>>>
>>>>>>> Thanks Kevin, I guess for now I'll leave them as they
>>>>>>> are until I can get smart on scripting.
>>>>>>>
>>>>>>> Chris
>>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>> -- Please use reply-all for most replies to avoid omitting
>>>> the mailing list. To unsubscribe or change options:
>>>> https://lists.samba.org/mailman/listinfo/rsync Before
>>>> posting, read:
>>>> http://www.catb.org/~esr/faqs/smart-questions.html
>>>>
>>>>
>
>> -- Please use reply-all for most replies to avoid omitting the
>> mailing list. To unsubscribe or change options:
>> https://lists.samba.org/mailman/listinfo/rsync Before posting,
>> read: http://www.catb.org/~esr/faqs/smart-questions.html
>
- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. Kevin at FutureQuest.net (work)
Orlando, Florida kmk at sanitarium.net (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iEYEARECAAYFAlQPnzUACgkQVKC1jlbQAQeiTwCg06TFxD4WC6IrvQOc1y1pwvIh
8r0AoMTdCafNM3QsbQiH670IdHtO0wwq
=Q8cI
-----END PGP SIGNATURE-----
More information about the rsync
mailing list