--hard-links performance

Chuck Wolber chuckw at quantumlinux.com
Tue Jun 5 18:11:27 GMT 2007


On Tue, 5 Jun 2007, Paul Slootman wrote:

> > In any case, what's the general consensus behind using the 
> > --hard-links option on large (100GB and above) images? Does it still 
> > use a ton of memory? Or has that situation been alleviated?
> 
> The size of the filesystem isn't relevant, the number of hard-linked 
> files is. It still uses a certain amount of memory for each hard-linked 
> file, but the situation is a lot better than with earlier rsync 
> versions. (As always, make sure you use the newest version.)

In our case, we store images as hardlinks and would like an easy way to 
migrate images from one backup server to another. We currently do it with 
a script that does a combination of rsync'ing and cp -al. Our layout is 
similar to:

image_dir
| -- img1
| -- img2 (~99% hardlinked to img1)
| -- img3 (~99% hardlinked to img2)
   .
   .
   .
` -- imgN (~99% hardlinked to img(N-1))


Each image in image_dir is hundreds of thousands of files. It seems to me 
that even a small amount of memory for each hardlinked file is going to 
clobber even the most stout of machines (at least by 2007 standards) if I 
tried a wholesale rsync of image_dir using --hard-links. No?

If so, then is a "hard link rich environment" an assumption that can be 
used to make an optimization of some sort?

..Chuck..



-- 
http://www.quantumlinux.com
 Quantum Linux Laboratories, LLC.
 ACCELERATING Business with Open Technology

 "The measure of the restoration lies in the extent to which we apply
  social values more noble than mere monetary profit." - FDR

09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0


More information about the rsync mailing list