Is there a howto/tutorial on backups/rsync that covers the use of hard and soft links?

Tue Jan 22 16:10:11 MST 2013

Joe, this is specific to having a backup with rsync.  The way I use links
for rsync is by not using the link (ln) command at all, but instead using
cp's build-in -l (link) option.  It looks something like this:

1) delete the oldest backup (simple 'rm' command)
2) shift the rest (with 'rm') by 1, so if I have a 7 day daily backup, it
moves daily.06 -> 07, daily.05 -> 06, etc., etc.  This effectively removed
daily.01 as that is now .02.
3) make a linked copy of .02 back into .01, so cp -al daily.02 daily.01 -
this links only, it doesn't physically copy anything.
4) run rsync against that .01 backup.  I run rsync with --delete and
--delete-excluded options in case something got removed from the live
system, it gets deleted, but only from this newest backup.  If I need to
recover it 5 days from now, I still can by looking in daily.05.

This setup works well in that only one folder will have actual physical
files (and thus the largest of them all.)  The rest of them are links back
to it and are very small.  The neat thing is, when a file gets deleted from
the newest backup, .01, it breaks the link from the others, and the
physical file remains in the next older backup, in this case .02,  (till it
"expires" out of the cycle.)

I have 12 servers backing up gigabytes worth of data every night like this,
maintaining anywhere between 3 to 6 weeks worth for each one.  They're all
going to a single 1 terabyte drive.

A

On Tue, Jan 22, 2013 at 3:31 PM, François <daitheflu at free.fr> wrote:

> Hi Joe,
>
> If you want to understand hard-links, just take a look at Wikipedia :
> http://en.wikipedia.org/wiki/Hard_link#Example
>
> I think it's pretty easy to understand.
>
> To understand how hard-links (and rsync) can help you make strong
> incremental
> backups, head over
> http://blog.interlinked.org/tutorials/rsync_time_machine.html
>
> Cheers,
>
> --
> François
>
>
> Le mardi 22 janvier 2013 23:12:38, Joe a écrit :
> > Thanks for the reply.  I know what hard and soft links are and have some
> > idea of how they relate to backup.
> >
> > What I need is a tutorial on how all of that works with rsync.  I can
> > see that there are a lot of considerations as to which options to use
> > for different situations and maybe some general strategies on how to
> > build something like an incremental  or differential backup.
> >
> > I use rsync now on a one directory tree to one directory tree basis now,
> > but I'd like to have more than one backup on my backup device without
> > doubling the storage which is what hard links will help me do once I
> > know how they work in more detail.
> >
> > Joe
> >
> > On 01/22/2013 02:48 AM, Tony Abernethy wrote:
> > > This may help:  (man ln)
> > >
> > > A hard link to a file is
> > > indistinguishable from the original directory entry; any changes to a
> > > file are effectively independent of the name used to reference the
> file.
> > > Hard links may not normally refer to directories and may not span file
> > > systems.
> > >
> > > Assuming you do many backups and many of the files do not change,
> > > hard links are your friend.
> > >
> > > Backing up soft links:
> > > Do you back up the link or what the link points to?
> > > (Even that simple thing has interesting ways to get complicated.)
> > >
> > > -----Original Message-----
> > > From: rsync-bounces at lists.samba.org
> > > [mailto:rsync-bounces at lists.samba.org] On Behalf Of Joe Sent: Tuesday,
> > > January 22, 2013 1:32 AM
> > > To: rsync at lists.samba.org
> > > Subject: Is there a howto/tutorial on backups/rsync that covers the use
> > > of hard and soft links?
> > >
> > > There have been a lot of posts on the list lately about issues with
> hard
> > > links.  It has been very interesting, but I don't understand it very
> > > thoroughly.  I haven't used hard links for anything yet.  I've used
> > > symlinks - not for backups, of course - and have seen them get broken
> or
> > > deleted in backups.
> > >
> > > Is there a tutorial anywhere that will explain how this works (assuming
> > > that the reader understands the basic concepts of backups and knows how
> > > to program, but doesn't really understand how to use links to create
> > > things like incremental or differential backups)?  It seems like there
> > > are a lot of fine points to consider, some of which can really bite you
> > > if you don't take them into account.
> > >
> > > I'm working on my own personal backup system using bash and rsync.
>  When
> > > it's done it will be pretty good, but it would be *much* better if I
> > > rewrote it to have more backup versions using hard links to save space.
> > >
> > > I am writing my own because (aside from learning a lot) I have only
> seen
> > > two types of backup utilities - those that are very simplistic and
> won't
> > > let me do what I want and those which are enterprise level and I can't
> > > figure out how to get them to do anything without extensive study.  (I
> > > did experiment with areca (I think it uses rsync libraries under the
> > > hood) which would probably do everything I want, but I got stuck too
> > > many times and couldn't get enough support on their forum to keep
> > > going.)
> > >
> > > TIA
> > >
> > >     Joe
> > >
> > > --
> > > Please use reply-all for most replies to avoid omitting the mailing
> list.
> > > To unsubscribe or change options:
> > > https://lists.samba.org/mailman/listinfo/rsync Before posting, read:
> > > http://www.catb.org/~esr/faqs/smart-questions.html
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20130122/b1971528/attachment.html>