Is there a howto/tutorial on backups/rsync that covers the use of hard and soft links?

Kevin Korb kmk at sanitarium.net
Tue Jan 22 16:14:26 MST 2013


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Also, if you put dates and times in the file names instead of .01,
.02, etc you don't have to do any mv's, you can easily tell when each
backup was run, and ls can tell you which the newest and oldest are.

On 01/22/13 18:12, Kevin Korb wrote:
> That is the old way that pre-dates --link-dest.  Instead of cp -al 
> daily.02 daily.01 you can do a mkdir daily.01 then an rsync ... 
> --link-dest=../daily.02 daily.01
> 
> Rsync then doesn't need any --delete and you don't bother making
> any hard links that will eventually be replaced.  Plus the linking
> happens while rsync is running so it is usually much faster.
> 
> On 01/22/13 18:10, Ashley M. Kirchner wrote:
>> Joe, this is specific to having a backup with rsync.  The way I 
>> use links for rsync is by not using the link (ln) command at
>> all, but instead using cp's build-in -l (link) option.  It looks 
>> something like this:
> 
>> 1) delete the oldest backup (simple 'rm' command) 2) shift the
>> rest (with 'rm') by 1, so if I have a 7 day daily backup, it
>> moves daily.06 -> 07, daily.05 -> 06, etc., etc.  This
>> effectively removed daily.01 as that is now .02. 3) make a linked
>> copy of .02 back into .01, so cp -al daily.02 daily.01 - this
>> links only, it doesn't physically copy anything. 4) run rsync
>> against that .01 backup.  I run rsync with --delete and
>> --delete-excluded options in case something got removed from the
>> live system, it gets deleted, but only from this newest backup.
>> If I need to recover it 5 days from now, I still can by looking
>> in daily.05.
> 
>> This setup works well in that only one folder will have actual 
>> physical files (and thus the largest of them all.)  The rest of 
>> them are links back to it and are very small.  The neat thing
>> is, when a file gets deleted from the newest backup, .01, it
>> breaks the link from the others, and the physical file remains in
>> the next older backup, in this case .02,  (till it "expires" out
>> of the cycle.)
> 
>> I have 12 servers backing up gigabytes worth of data every night 
>> like this, maintaining anywhere between 3 to 6 weeks worth for
>> each one. They're all going to a single 1 terabyte drive.
> 
>> A
> 
> 
> 
>> On Tue, Jan 22, 2013 at 3:31 PM, Fran￧ois <daitheflu at free.fr 
>> <mailto:daitheflu at free.fr>> wrote:
> 
>> Hi Joe,
> 
>> If you want to understand hard-links, just take a look at
>> Wikipedia : http://en.wikipedia.org/wiki/Hard_link#Example
> 
>> I think it's pretty easy to understand.
> 
>> To understand how hard-links (and rsync) can help you make strong
>>  incremental backups, head over 
>> http://blog.interlinked.org/tutorials/rsync_time_machine.html
> 
>> Cheers,
> 
>> -- Fran￧ois
> 
> 
>> Le mardi 22 janvier 2013 23:12:38, Joe a ←crit :
>>> Thanks for the reply.  I know what hard and soft links are and
>> have some
>>> idea of how they relate to backup.
>>> 
>>> What I need is a tutorial on how all of that works with rsync.
>>> I can see that there are a lot of considerations as to which 
>>> options to use for different situations and maybe some general 
>>> strategies on how to build something like an incremental  or 
>>> differential backup.
>>> 
>>> I use rsync now on a one directory tree to one directory tree
>> basis now,
>>> but I'd like to have more than one backup on my backup device 
>>> without doubling the storage which is what hard links will
>>> help me do once I know how they work in more detail.
>>> 
>>> Joe
>>> 
>>> On 01/22/2013 02:48 AM, Tony Abernethy wrote:
>>>> This may help:  (man ln)
>>>> 
>>>> A hard link to a file is indistinguishable from the original 
>>>> directory entry; any changes
>> to a
>>>> file are effectively independent of the name used to
>>>> reference
>> the file.
>>>> Hard links may not normally refer to directories and may not
>> span file
>>>> systems.
>>>> 
>>>> Assuming you do many backups and many of the files do not 
>>>> change, hard links are your friend.
>>>> 
>>>> Backing up soft links: Do you back up the link or what the
>>>> link points to? (Even that simple thing has interesting ways
>>>> to get complicated.)
>>>> 
>>>> -----Original Message----- From:
>>>> rsync-bounces at lists.samba.org
>> <mailto:rsync-bounces at lists.samba.org>
>>>> [mailto:rsync-bounces at lists.samba.org
>> <mailto:rsync-bounces at lists.samba.org>] On Behalf Of Joe Sent: 
>> Tuesday,
>>>> January 22, 2013 1:32 AM To: rsync at lists.samba.org 
>>>> <mailto:rsync at lists.samba.org> Subject: Is there a 
>>>> howto/tutorial on backups/rsync that covers
>> the use
>>>> of hard and soft links?
>>>> 
>>>> There have been a lot of posts on the list lately about
>>>> issues
>> with hard
>>>> links.  It has been very interesting, but I don't understand
>>>> it very thoroughly.  I haven't used hard links for anything
>>>> yet. I've used symlinks - not for backups, of course - and
>>>> have seen them get
>> broken or
>>>> deleted in backups.
>>>> 
>>>> Is there a tutorial anywhere that will explain how this
>>>> works
>> (assuming
>>>> that the reader understands the basic concepts of backups
>>>> and
>> knows how
>>>> to program, but doesn't really understand how to use links
>>>> to create things like incremental or differential backups)?
>>>> It seems like
>> there
>>>> are a lot of fine points to consider, some of which can
>>>> really
>> bite you
>>>> if you don't take them into account.
>>>> 
>>>> I'm working on my own personal backup system using bash and
>> rsync.  When
>>>> it's done it will be pretty good, but it would be *much*
>>>> better if I rewrote it to have more backup versions using
>>>> hard links to save
>> space.
>>>> 
>>>> I am writing my own because (aside from learning a lot) I
>>>> have
>> only seen
>>>> two types of backup utilities - those that are very
>>>> simplistic
>> and won't
>>>> let me do what I want and those which are enterprise level
>>>> and I
>> can't
>>>> figure out how to get them to do anything without extensive
>> study.  (I
>>>> did experiment with areca (I think it uses rsync libraries 
>>>> under the hood) which would probably do everything I want,
>>>> but I got stuck too many times and couldn't get enough
>>>> support on their forum to keep going.)
>>>> 
>>>> TIA
>>>> 
>>>> Joe
>>>> 
>>>> -- Please use reply-all for most replies to avoid omitting
>>>> the
>> mailing list.
>>>> To unsubscribe or change options: 
>>>> https://lists.samba.org/mailman/listinfo/rsync Before
>>>> posting, read:
>>>> http://www.catb.org/~esr/faqs/smart-questions.html
>> -- Please use reply-all for most replies to avoid omitting the 
>> mailing list. To unsubscribe or change options: 
>> https://lists.samba.org/mailman/listinfo/rsync Before posting, 
>> read: http://www.catb.org/~esr/faqs/smart-questions.html
> 
> 
> 
> 
> 
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlD/HVIACgkQVKC1jlbQAQd2OwCgwF8H2EjiCmKcTboG5rK8w7Vm
Sv4AoIIknAaOcuMLOj9dKlykKV1Wvu1F
=kLZP
-----END PGP SIGNATURE-----


More information about the rsync mailing list