Is there a howto/tutorial on backups/rsync that covers the use of hard and soft links?

Ashley M. Kirchner ashley at pcraft.com
Tue Jan 22 16:57:52 MST 2013 

Yeah I know, there's more I need to do to optimize.  That script is
probably nearing ten years old.  It's been running without a single problem
so I never bothered to revisit it, even after updating machines, I just
copy it over and keep in trucking.


On Tue, Jan 22, 2013 at 4:14 PM, Kevin Korb <kmk at sanitarium.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Also, if you put dates and times in the file names instead of .01,
> .02, etc you don't have to do any mv's, you can easily tell when each
> backup was run, and ls can tell you which the newest and oldest are.
>
> On 01/22/13 18:12, Kevin Korb wrote:
> > That is the old way that pre-dates --link-dest.  Instead of cp -al
> > daily.02 daily.01 you can do a mkdir daily.01 then an rsync ...
> > --link-dest=../daily.02 daily.01
> >
> > Rsync then doesn't need any --delete and you don't bother making
> > any hard links that will eventually be replaced.  Plus the linking
> > happens while rsync is running so it is usually much faster.
> >
> > On 01/22/13 18:10, Ashley M. Kirchner wrote:
> >> Joe, this is specific to having a backup with rsync.  The way I
> >> use links for rsync is by not using the link (ln) command at
> >> all, but instead using cp's build-in -l (link) option.  It looks
> >> something like this:
> >
> >> 1) delete the oldest backup (simple 'rm' command) 2) shift the
> >> rest (with 'rm') by 1, so if I have a 7 day daily backup, it
> >> moves daily.06 -> 07, daily.05 -> 06, etc., etc.  This
> >> effectively removed daily.01 as that is now .02. 3) make a linked
> >> copy of .02 back into .01, so cp -al daily.02 daily.01 - this
> >> links only, it doesn't physically copy anything. 4) run rsync
> >> against that .01 backup.  I run rsync with --delete and
> >> --delete-excluded options in case something got removed from the
> >> live system, it gets deleted, but only from this newest backup.
> >> If I need to recover it 5 days from now, I still can by looking
> >> in daily.05.
> >
> >> This setup works well in that only one folder will have actual
> >> physical files (and thus the largest of them all.)  The rest of
> >> them are links back to it and are very small.  The neat thing
> >> is, when a file gets deleted from the newest backup, .01, it
> >> breaks the link from the others, and the physical file remains in
> >> the next older backup, in this case .02,  (till it "expires" out
> >> of the cycle.)
> >
> >> I have 12 servers backing up gigabytes worth of data every night
> >> like this, maintaining anywhere between 3 to 6 weeks worth for
> >> each one. They're all going to a single 1 terabyte drive.
> >
> >> A
> >
> >
> >
> >> On Tue, Jan 22, 2013 at 3:31 PM, Fran￧ois <daitheflu at free.fr
> >> <mailto:daitheflu at free.fr>> wrote:
> >
> >> Hi Joe,
> >
> >> If you want to understand hard-links, just take a look at
> >> Wikipedia : http://en.wikipedia.org/wiki/Hard_link#Example
> >
> >> I think it's pretty easy to understand.
> >
> >> To understand how hard-links (and rsync) can help you make strong
> >>  incremental backups, head over
> >> http://blog.interlinked.org/tutorials/rsync_time_machine.html
> >
> >> Cheers,
> >
> >> -- Fran￧ois
> >
> >
> >> Le mardi 22 janvier 2013 23:12:38, Joe a ￩crit :
> >>> Thanks for the reply.  I know what hard and soft links are and
> >> have some
> >>> idea of how they relate to backup.
> >>>
> >>> What I need is a tutorial on how all of that works with rsync.
> >>> I can see that there are a lot of considerations as to which
> >>> options to use for different situations and maybe some general
> >>> strategies on how to build something like an incremental  or
> >>> differential backup.
> >>>
> >>> I use rsync now on a one directory tree to one directory tree
> >> basis now,
> >>> but I'd like to have more than one backup on my backup device
> >>> without doubling the storage which is what hard links will
> >>> help me do once I know how they work in more detail.
> >>>
> >>> Joe
> >>>
> >>> On 01/22/2013 02:48 AM, Tony Abernethy wrote:
> >>>> This may help:  (man ln)
> >>>>
> >>>> A hard link to a file is indistinguishable from the original
> >>>> directory entry; any changes
> >> to a
> >>>> file are effectively independent of the name used to
> >>>> reference
> >> the file.
> >>>> Hard links may not normally refer to directories and may not
> >> span file
> >>>> systems.
> >>>>
> >>>> Assuming you do many backups and many of the files do not
> >>>> change, hard links are your friend.
> >>>>
> >>>> Backing up soft links: Do you back up the link or what the
> >>>> link points to? (Even that simple thing has interesting ways
> >>>> to get complicated.)
> >>>>
> >>>> -----Original Message----- From:
> >>>> rsync-bounces at lists.samba.org
> >> <mailto:rsync-bounces at lists.samba.org>
> >>>> [mailto:rsync-bounces at lists.samba.org
> >> <mailto:rsync-bounces at lists.samba.org>] On Behalf Of Joe Sent:
> >> Tuesday,
> >>>> January 22, 2013 1:32 AM To: rsync at lists.samba.org
> >>>> <mailto:rsync at lists.samba.org> Subject: Is there a
> >>>> howto/tutorial on backups/rsync that covers
> >> the use
> >>>> of hard and soft links?
> >>>>
> >>>> There have been a lot of posts on the list lately about
> >>>> issues
> >> with hard
> >>>> links.  It has been very interesting, but I don't understand
> >>>> it very thoroughly.  I haven't used hard links for anything
> >>>> yet. I've used symlinks - not for backups, of course - and
> >>>> have seen them get
> >> broken or
> >>>> deleted in backups.
> >>>>
> >>>> Is there a tutorial anywhere that will explain how this
> >>>> works
> >> (assuming
> >>>> that the reader understands the basic concepts of backups
> >>>> and
> >> knows how
> >>>> to program, but doesn't really understand how to use links
> >>>> to create things like incremental or differential backups)?
> >>>> It seems like
> >> there
> >>>> are a lot of fine points to consider, some of which can
> >>>> really
> >> bite you
> >>>> if you don't take them into account.
> >>>>
> >>>> I'm working on my own personal backup system using bash and
> >> rsync.  When
> >>>> it's done it will be pretty good, but it would be *much*
> >>>> better if I rewrote it to have more backup versions using
> >>>> hard links to save
> >> space.
> >>>>
> >>>> I am writing my own because (aside from learning a lot) I
> >>>> have
> >> only seen
> >>>> two types of backup utilities - those that are very
> >>>> simplistic
> >> and won't
> >>>> let me do what I want and those which are enterprise level
> >>>> and I
> >> can't
> >>>> figure out how to get them to do anything without extensive
> >> study.  (I
> >>>> did experiment with areca (I think it uses rsync libraries
> >>>> under the hood) which would probably do everything I want,
> >>>> but I got stuck too many times and couldn't get enough
> >>>> support on their forum to keep going.)
> >>>>
> >>>> TIA
> >>>>
> >>>> Joe
> >>>>
> >>>> -- Please use reply-all for most replies to avoid omitting
> >>>> the
> >> mailing list.
> >>>> To unsubscribe or change options:
> >>>> https://lists.samba.org/mailman/listinfo/rsync Before
> >>>> posting, read:
> >>>> http://www.catb.org/~esr/faqs/smart-questions.html
> >> -- Please use reply-all for most replies to avoid omitting the
> >> mailing list. To unsubscribe or change options:
> >> https://lists.samba.org/mailman/listinfo/rsync Before posting,
> >> read: http://www.catb.org/~esr/faqs/smart-questions.html
> >
> >
> >
> >
> >
> >
>
> - --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>         Kevin Korb                      Phone:    (407) 252-6853
>         Systems Administrator           Internet:
>         FutureQuest, Inc.               Kevin at FutureQuest.net  (work)
>         Orlando, Florida                kmk at sanitarium.net (personal)
>         Web page:                       http://www.sanitarium.net/
>         PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.19 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlD/HVIACgkQVKC1jlbQAQd2OwCgwF8H2EjiCmKcTboG5rK8w7Vm
> Sv4AoIIknAaOcuMLOj9dKlykKV1Wvu1F
> =kLZP
> -----END PGP SIGNATURE-----
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20130122/2fa86d9c/attachment.html>

More information about the rsync mailing list