Is there a howto/tutorial on backups/rsync that covers the use of hard and soft links?

Kevin Korb kmk at sanitarium.net
Tue Jan 22 16:12:56 MST 2013


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

That is the old way that pre-dates --link-dest.  Instead of cp -al
daily.02 daily.01 you can do a mkdir daily.01 then an rsync ...
- --link-dest=../daily.02 daily.01

Rsync then doesn't need any --delete and you don't bother making any
hard links that will eventually be replaced.  Plus the linking happens
while rsync is running so it is usually much faster.

On 01/22/13 18:10, Ashley M. Kirchner wrote:
> Joe, this is specific to having a backup with rsync.  The way I
> use links for rsync is by not using the link (ln) command at all,
> but instead using cp's build-in -l (link) option.  It looks
> something like this:
> 
> 1) delete the oldest backup (simple 'rm' command) 2) shift the rest
> (with 'rm') by 1, so if I have a 7 day daily backup, it moves
> daily.06 -> 07, daily.05 -> 06, etc., etc.  This effectively 
> removed daily.01 as that is now .02. 3) make a linked copy of .02
> back into .01, so cp -al daily.02 daily.01 - this links only, it
> doesn't physically copy anything. 4) run rsync against that .01
> backup.  I run rsync with --delete and --delete-excluded options in
> case something got removed from the live system, it gets deleted,
> but only from this newest backup.  If I need to recover it 5 days
> from now, I still can by looking in daily.05.
> 
> This setup works well in that only one folder will have actual
> physical files (and thus the largest of them all.)  The rest of
> them are links back to it and are very small.  The neat thing is,
> when a file gets deleted from the newest backup, .01, it breaks the
> link from the others, and the physical file remains in the next
> older backup, in this case .02,  (till it "expires" out of the
> cycle.)
> 
> I have 12 servers backing up gigabytes worth of data every night
> like this, maintaining anywhere between 3 to 6 weeks worth for each
> one. They're all going to a single 1 terabyte drive.
> 
> A
> 
> 
> 
> On Tue, Jan 22, 2013 at 3:31 PM, Fran￧ois <daitheflu at free.fr 
> <mailto:daitheflu at free.fr>> wrote:
> 
> Hi Joe,
> 
> If you want to understand hard-links, just take a look at Wikipedia
> : http://en.wikipedia.org/wiki/Hard_link#Example
> 
> I think it's pretty easy to understand.
> 
> To understand how hard-links (and rsync) can help you make strong 
> incremental backups, head over 
> http://blog.interlinked.org/tutorials/rsync_time_machine.html
> 
> Cheers,
> 
> -- Fran￧ois
> 
> 
> Le mardi 22 janvier 2013 23:12:38, Joe a ←crit :
>> Thanks for the reply.  I know what hard and soft links are and
> have some
>> idea of how they relate to backup.
>> 
>> What I need is a tutorial on how all of that works with rsync.  I
>> can see that there are a lot of considerations as to which
>> options to use for different situations and maybe some general
>> strategies on how to build something like an incremental  or
>> differential backup.
>> 
>> I use rsync now on a one directory tree to one directory tree
> basis now,
>> but I'd like to have more than one backup on my backup device
>> without doubling the storage which is what hard links will help
>> me do once I know how they work in more detail.
>> 
>> Joe
>> 
>> On 01/22/2013 02:48 AM, Tony Abernethy wrote:
>>> This may help:  (man ln)
>>> 
>>> A hard link to a file is indistinguishable from the original
>>> directory entry; any changes
> to a
>>> file are effectively independent of the name used to reference
> the file.
>>> Hard links may not normally refer to directories and may not
> span file
>>> systems.
>>> 
>>> Assuming you do many backups and many of the files do not
>>> change, hard links are your friend.
>>> 
>>> Backing up soft links: Do you back up the link or what the link
>>> points to? (Even that simple thing has interesting ways to get
>>> complicated.)
>>> 
>>> -----Original Message----- From: rsync-bounces at lists.samba.org
> <mailto:rsync-bounces at lists.samba.org>
>>> [mailto:rsync-bounces at lists.samba.org
> <mailto:rsync-bounces at lists.samba.org>] On Behalf Of Joe Sent:
> Tuesday,
>>> January 22, 2013 1:32 AM To: rsync at lists.samba.org
>>> <mailto:rsync at lists.samba.org> Subject: Is there a
>>> howto/tutorial on backups/rsync that covers
> the use
>>> of hard and soft links?
>>> 
>>> There have been a lot of posts on the list lately about issues
> with hard
>>> links.  It has been very interesting, but I don't understand it
>>> very thoroughly.  I haven't used hard links for anything yet.
>>> I've used symlinks - not for backups, of course - and have seen
>>> them get
> broken or
>>> deleted in backups.
>>> 
>>> Is there a tutorial anywhere that will explain how this works
> (assuming
>>> that the reader understands the basic concepts of backups and
> knows how
>>> to program, but doesn't really understand how to use links to
>>> create things like incremental or differential backups)?  It
>>> seems like
> there
>>> are a lot of fine points to consider, some of which can really
> bite you
>>> if you don't take them into account.
>>> 
>>> I'm working on my own personal backup system using bash and
> rsync.  When
>>> it's done it will be pretty good, but it would be *much* better
>>> if I rewrote it to have more backup versions using hard links
>>> to save
> space.
>>> 
>>> I am writing my own because (aside from learning a lot) I have
> only seen
>>> two types of backup utilities - those that are very simplistic
> and won't
>>> let me do what I want and those which are enterprise level and
>>> I
> can't
>>> figure out how to get them to do anything without extensive
> study.  (I
>>> did experiment with areca (I think it uses rsync libraries
>>> under the hood) which would probably do everything I want, but
>>> I got stuck too many times and couldn't get enough support on
>>> their forum to keep going.)
>>> 
>>> TIA
>>> 
>>> Joe
>>> 
>>> -- Please use reply-all for most replies to avoid omitting the
> mailing list.
>>> To unsubscribe or change options: 
>>> https://lists.samba.org/mailman/listinfo/rsync Before posting,
>>> read: http://www.catb.org/~esr/faqs/smart-questions.html
> -- Please use reply-all for most replies to avoid omitting the
> mailing list. To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync Before posting,
> read: http://www.catb.org/~esr/faqs/smart-questions.html
> 
> 
> 
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlD/HPgACgkQVKC1jlbQAQcfZgCeO25bLuBD+TdloH72nXdSJG/3
zXIAoMPt6W5ouK1V8bM7S6oenAPq7jyG
=3Irt
-----END PGP SIGNATURE-----


More information about the rsync mailing list