RSync with /really/ long file lists

[Brandon Knitter]

Hi everyone!

rsync 2.5.7
linux RHES v3

We need to sync over 4 Million files, and when we run rsync we run out of
memory! :(

We were going to batch things, but the build in Batch mode still will use a
large amount of memory.  Any manualy batching is proving to be harder than
you'd think, even with --include and --exclude flags.  The flags actually make
it still have to recurse the whole tree.

We would sync things at the top level, one dir at a time, but the tree of files
is not evenly weighted and we still run out of memory.

Questions:

1) What strategies do other people employ?
2) Is there a better tool out there for moving large amounts of files?
3) Is there a way to save the filelist to a file instead of in memory?

Thanks,


-- 
-bk








----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.