RSync with /really/ long file lists
Brandon Knitter
knitterb at blandsite.org
Sat Dec 4 03:09:11 GMT 2004
Hi everyone!
rsync 2.5.7
linux RHES v3
We need to sync over 4 Million files, and when we run rsync we run out of
memory! :(
We were going to batch things, but the build in Batch mode still will use a
large amount of memory. Any manualy batching is proving to be harder than
you'd think, even with --include and --exclude flags. The flags actually make
it still have to recurse the whole tree.
We would sync things at the top level, one dir at a time, but the tree of files
is not evenly weighted and we still run out of memory.
Questions:
1) What strategies do other people employ?
2) Is there a better tool out there for moving large amounts of files?
3) Is there a way to save the filelist to a file instead of in memory?
Thanks,
--
-bk
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
More information about the rsync
mailing list