Problems with rsync 2.5.1pre1 and hardlinks

[Dave Dykstra]

On Mon, Dec 10, 2001 at 09:18:33PM +0100, birger at takatukaland.de wrote:
> Dave Dykstra schrieb am Mon, Dec 10, 2001 at 01:44:26PM -0600:
> * > I have to copy a structure of 70 directories where the data of these 
> * > directories are hardlinked to the data of the 1st directory.  Within this
> * > "orig data" directory, I have about 30.000 files, so the amount of files
> * > to sync is approx. 2.100.000.  The overall size is about 9.2GB.
> * 
> * ...  [runs out of memory] ...
> * 
> * In case you weren't aware, rsync uses a lot less memory overall if you can
> * split up the copies into smaller pieces, because it keeps some memory for
> * every file it touches in a run.  In your case, it may be especially
> * difficult to break up the copy into smaller pieces because of all the
> * hardlinks.
> 
> I thought of that possibility and it won't work exactly of the reason you
> described (the hardlink issue).
> 
> 
> * 
> * Ideas:
> *     1. Would it be possible to use symlinks instead of hardlinks?  That
> * 	would give you more flexibility to split things up however you
> * 	like.
> *     2. Perhaps you could break it up into ~70 copies, where each time you
> * 	give it the first directory that contains the data and another
> * 	one that contains one of the hardlinks.
> 
> Both alternatives will eat up huge amounts of disk space as the numbers
> above suggest.  I will therefore consider plugging in more mem/swap before
> trying them.

Why would symlinks eat up disk space?

I only suggested the second alternative because I thought that it would
end up with all the destination files hardlinked together as on the
original system.  I hadn't tested it, but now I did and it works:

    $ mkdir s s/d1 s/d2 s/d3 t
    $ touch s/d1/l1
    $ ln s/d1/l1 s/d2/l1
    $ ln s/d1/l1 s/d3/l1
    $ ls -li s/*/l1
	286226 -rw-rw-r--   3 dwd      dwd            0 Dec 10 14:20 s/d1/l1
	286226 -rw-rw-r--   3 dwd      dwd            0 Dec 10 14:20 s/d2/l1
	286226 -rw-rw-r--   3 dwd      dwd            0 Dec 10 14:20 s/d3/l1
    $ rsync -aH s/d1 s/d2 t
    $ rsync -aH s/d1 s/d3 t
    $ ls -li t/*/l1
	622728 -rw-rw-r--   3 dwd      dwd            0 Dec 10 14:20 t/d1/l1
	622728 -rw-rw-r--   3 dwd      dwd            0 Dec 10 14:20 t/d2/l1
	622728 -rw-rw-r--   3 dwd      dwd            0 Dec 10 14:20 t/d3/l1



> But I thank you for the hint anyway,


- Dave Dykstra