Is there a howto/tutorial on backups/rsync that covers the use of hard and soft links?
Kevin Korb
kmk at sanitarium.net
Tue Jan 22 16:12:56 MST 2013
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
That is the old way that pre-dates --link-dest. Instead of cp -al
daily.02 daily.01 you can do a mkdir daily.01 then an rsync ...
- --link-dest=../daily.02 daily.01
Rsync then doesn't need any --delete and you don't bother making any
hard links that will eventually be replaced. Plus the linking happens
while rsync is running so it is usually much faster.
On 01/22/13 18:10, Ashley M. Kirchner wrote:
> Joe, this is specific to having a backup with rsync. The way I
> use links for rsync is by not using the link (ln) command at all,
> but instead using cp's build-in -l (link) option. It looks
> something like this:
>
> 1) delete the oldest backup (simple 'rm' command) 2) shift the rest
> (with 'rm') by 1, so if I have a 7 day daily backup, it moves
> daily.06 -> 07, daily.05 -> 06, etc., etc. This effectively
> removed daily.01 as that is now .02. 3) make a linked copy of .02
> back into .01, so cp -al daily.02 daily.01 - this links only, it
> doesn't physically copy anything. 4) run rsync against that .01
> backup. I run rsync with --delete and --delete-excluded options in
> case something got removed from the live system, it gets deleted,
> but only from this newest backup. If I need to recover it 5 days
> from now, I still can by looking in daily.05.
>
> This setup works well in that only one folder will have actual
> physical files (and thus the largest of them all.) The rest of
> them are links back to it and are very small. The neat thing is,
> when a file gets deleted from the newest backup, .01, it breaks the
> link from the others, and the physical file remains in the next
> older backup, in this case .02, (till it "expires" out of the
> cycle.)
>
> I have 12 servers backing up gigabytes worth of data every night
> like this, maintaining anywhere between 3 to 6 weeks worth for each
> one. They're all going to a single 1 terabyte drive.
>
> A
>
>
>
> On Tue, Jan 22, 2013 at 3:31 PM, Franois <daitheflu at free.fr
> <mailto:daitheflu at free.fr>> wrote:
>
> Hi Joe,
>
> If you want to understand hard-links, just take a look at Wikipedia
> : http://en.wikipedia.org/wiki/Hard_link#Example
>
> I think it's pretty easy to understand.
>
> To understand how hard-links (and rsync) can help you make strong
> incremental backups, head over
> http://blog.interlinked.org/tutorials/rsync_time_machine.html
>
> Cheers,
>
> -- Franois
>
>
> Le mardi 22 janvier 2013 23:12:38, Joe a ←crit :
>> Thanks for the reply. I know what hard and soft links are and
> have some
>> idea of how they relate to backup.
>>
>> What I need is a tutorial on how all of that works with rsync. I
>> can see that there are a lot of considerations as to which
>> options to use for different situations and maybe some general
>> strategies on how to build something like an incremental or
>> differential backup.
>>
>> I use rsync now on a one directory tree to one directory tree
> basis now,
>> but I'd like to have more than one backup on my backup device
>> without doubling the storage which is what hard links will help
>> me do once I know how they work in more detail.
>>
>> Joe
>>
>> On 01/22/2013 02:48 AM, Tony Abernethy wrote:
>>> This may help: (man ln)
>>>
>>> A hard link to a file is indistinguishable from the original
>>> directory entry; any changes
> to a
>>> file are effectively independent of the name used to reference
> the file.
>>> Hard links may not normally refer to directories and may not
> span file
>>> systems.
>>>
>>> Assuming you do many backups and many of the files do not
>>> change, hard links are your friend.
>>>
>>> Backing up soft links: Do you back up the link or what the link
>>> points to? (Even that simple thing has interesting ways to get
>>> complicated.)
>>>
>>> -----Original Message----- From: rsync-bounces at lists.samba.org
> <mailto:rsync-bounces at lists.samba.org>
>>> [mailto:rsync-bounces at lists.samba.org
> <mailto:rsync-bounces at lists.samba.org>] On Behalf Of Joe Sent:
> Tuesday,
>>> January 22, 2013 1:32 AM To: rsync at lists.samba.org
>>> <mailto:rsync at lists.samba.org> Subject: Is there a
>>> howto/tutorial on backups/rsync that covers
> the use
>>> of hard and soft links?
>>>
>>> There have been a lot of posts on the list lately about issues
> with hard
>>> links. It has been very interesting, but I don't understand it
>>> very thoroughly. I haven't used hard links for anything yet.
>>> I've used symlinks - not for backups, of course - and have seen
>>> them get
> broken or
>>> deleted in backups.
>>>
>>> Is there a tutorial anywhere that will explain how this works
> (assuming
>>> that the reader understands the basic concepts of backups and
> knows how
>>> to program, but doesn't really understand how to use links to
>>> create things like incremental or differential backups)? It
>>> seems like
> there
>>> are a lot of fine points to consider, some of which can really
> bite you
>>> if you don't take them into account.
>>>
>>> I'm working on my own personal backup system using bash and
> rsync. When
>>> it's done it will be pretty good, but it would be *much* better
>>> if I rewrote it to have more backup versions using hard links
>>> to save
> space.
>>>
>>> I am writing my own because (aside from learning a lot) I have
> only seen
>>> two types of backup utilities - those that are very simplistic
> and won't
>>> let me do what I want and those which are enterprise level and
>>> I
> can't
>>> figure out how to get them to do anything without extensive
> study. (I
>>> did experiment with areca (I think it uses rsync libraries
>>> under the hood) which would probably do everything I want, but
>>> I got stuck too many times and couldn't get enough support on
>>> their forum to keep going.)
>>>
>>> TIA
>>>
>>> Joe
>>>
>>> -- Please use reply-all for most replies to avoid omitting the
> mailing list.
>>> To unsubscribe or change options:
>>> https://lists.samba.org/mailman/listinfo/rsync Before posting,
>>> read: http://www.catb.org/~esr/faqs/smart-questions.html
> -- Please use reply-all for most replies to avoid omitting the
> mailing list. To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync Before posting,
> read: http://www.catb.org/~esr/faqs/smart-questions.html
>
>
>
>
- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. Kevin at FutureQuest.net (work)
Orlando, Florida kmk at sanitarium.net (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlD/HPgACgkQVKC1jlbQAQcfZgCeO25bLuBD+TdloH72nXdSJG/3
zXIAoMPt6W5ouK1V8bM7S6oenAPq7jyG
=3Irt
-----END PGP SIGNATURE-----
More information about the rsync
mailing list