rsync takes way too long to perform this....
jhelfman at bizrate.com
Thu Nov 15 14:30:57 EST 2001
It is a reiserfs system on the client, and ext2 on the rsync server.
The file system is organized lovely. Just a ton of files.
On Thu, Nov 15, 2001 at 07:13:09AM +0100, Ph. Marek thus spat:
| >rsync -avnp remote::gif/ `find /home/www/html/ -maxdepth 1
| >-name "*.[j,g][pg,if]*"` /tmp/
| >If I run this on the local machine, the rsync server, it takes this
| >---> root at server (0.34)# time find /home/www/html/ -maxdepth 1
| >-name "*.[j,g][pg,if]*" -type f
| >real 0m0.015s
| >user 0m0.000s
| >sys 0m0.000s
| >However if I run it from a client, it will take forever. Too much to
| >run, it seems. Our directory structure has well over a million files.
| >And this is just one directory under /home/www/html. We can't afford the
| >cpu and system load to traverse everything, this is why I am using the
| >find command. Shouldn't this work? It does come back with retrieving the
| >list from the remote server.
| What OS are you running on both systems?? AFAIK linux with ext2/ext3 has
| (currently) severe problems with large directories (>5000 files).
| [Work is done to avoid that: see ext2 directory index patch at
| http://kernelnewbies.org/~phillips/ ]
| Maybe that's your problem.
| (In my - and strictly my - opinion, a directory with that many files is
| "unmaintainable". I'd do some partitioning - and if it's only sorting by
| filetype (.html, .gif, .jpg, ...))
Jason G Helfman
Fingerprint: DA13 C109 072B CC12 B568 8D84 E9A2 6A7D C479 BCFB
GnuPG http://www.gnupg.org Get Private! 1024D/D75E0A36
More information about the rsync