rsync takes way too long to perform this....

[Jason Helfman]

It is a reiserfs system on the client, and ext2 on the rsync server.

The file system is organized lovely. Just a ton of files.

On Thu, Nov 15, 2001 at 07:13:09AM +0100, Ph. Marek thus spat:
| >rsync -avnp remote::gif/ `find /home/www/html/ -maxdepth 1
| >-name "*.[j,g][pg,if]*"` /tmp/
| >
| >If I run this on the local machine, the rsync server, it takes this
| >long:
| >
| >---> root at server (0.34)# time find /home/www/html/ -maxdepth 1
| >-name "*.[j,g][pg,if]*" -type f
| >/home/www/html/comparestores_2.jpg
| >/home/www/html/home.jpg
| >/home/www/html/comparestores_3.jpg
| >/home/www/html/specialoffer_apparel.jpg
| >/home/www/html/bike_gary.gif
| >/home/www/html/gary_bike.gif
| >/home/www/html/none.gif
| > 
| >real    0m0.015s
| >user    0m0.000s
| >sys     0m0.000s
| >
| >However if I run it from a client, it will take forever. Too much to
| >run, it seems. Our directory structure has well over a million files.
| >And this is just one directory under /home/www/html. We can't afford the
| >cpu and system load to traverse everything, this is why I am using the
| >find command. Shouldn't this work? It does come back with retrieving the
| >list from the remote server.
| What OS are you running on both systems?? AFAIK linux with ext2/ext3 has
| (currently) severe problems with large directories (>5000 files).
| [Work is done to avoid that: see ext2 directory index patch at
| http://kernelnewbies.org/~phillips/ ]
| 
| Maybe that's your problem.
| 
| 
| (In my - and strictly my - opinion, a directory with that many files is
| "unmaintainable". I'd do some partitioning - and if it's only sorting by
| filetype (.html, .gif, .jpg, ...))
| 
| 
| Regards,
| 
| Phil

-- 
Jason G Helfman                                                                 
Network Administrator                                                           
BizRate.com, Co-Owner
310.754.1264 desk
310.466.2319 cell

	Fingerprint: DA13 C109 072B CC12 B568  8D84 E9A2 6A7D C479 BCFB
            GnuPG http://www.gnupg.org  Get Private!  1024D/D75E0A36