File system usage

[Eric Whiting]

Rsync is great. (What is cp? What is tar? Since I started using rsync I
forgot how to use those tools.) I sync over 1,000,000 files (50G) to
multiple destinations every day.  Over medium speed links it takes about
3 hrs. 

The default mode of operation creates a hidden (.file) on the
destination while transferring. This is only done one file at a time so
there is very little overhead unless you have a 10G filesystem with a
single 9G file -- even then I think there are rsync flags to deal with
this situation...

If you run into a file count limitation then split it up do do the rsync
on a dir-by-dir basis. There are scripts in the mailing list archives. I
don't have enough memory/time to easily do the whole tree at once -- I
do each top-level subdir individually.

eric





> Duane Meyer wrote:
> 
> This is a simple question. How much file system overhead is there with
> this system? Is it only as large as the largest file transfered or
> could you potentially (even if configured correctly) end up with
> double what you started out with on the sending or receiving end?
> 
> The reason I need to be sure is that I have a file system that's
> literally several hundred thousand files. Complete it's around
> 25-30GB.. these are sparse files so there's lots of room for
> compression. ..tar'd and compressed it's 3-4GB.
> 
> I just need to be sure there will be enough room if no single file is
> larger than about 200K and I have about 5GB of space left of the
> partition.
> 
> It's just taking way too long to tar and compress the complete
> directory, transfer than uncompress and untar.. the last run took
> around 24 hours front to back. I'm hoping to cut that to around 4-5
> hours. :) (probably optimistic, but who knows!)
> 
> Anybody know of limitations on file count or size I might run into
> here?
> 
> Any help is appreciated for this rsync newbie.
> 
> Regards,
> Duane Meyer