Max filesize for rsync?

[Shachar Shemesh]

Tony at ServaCorp.com wrote:

>I would be very happy to test any patches. (Assorted RedHat/Fedora i386)
>(Assume I am a total newbie, much safer that way)
>
>A few very large files regularly rsync'd in production.
>Seems like it sometimes gets somewhat stuck in the middle of something
>large.
>(The rsync is mostly staging area to staging area.
>Plenty of redundancy, so I'm unlikely to get hurt if I'm aware of problems.
>The way the targets are used, I will know about problems before any real
>damage is done.)
>The more important of the transfers are over occasionally very bad internet
>connections, so I'm pretty much in the situation of something to gain,
>nothing to lose.
>-rw-rw----    1 27       27       1187270120 Apr 13 03:24
>/home/rsync-sjs/mysql/sjs/dwf.MYD
>-rw-rw----    1 27       27       1098515060 Apr  8 07:34
>/home/rsync-sjs/mysql/srvs/dwf.MYD
>-rw-rw----    1 mysql    mysql    840374964 Apr 12 12:59
>/home/rsync-2bb/mysql/srv/dwf.MYD
>-rw-rw----    1 mysql    mysql    520216980 Apr 12 20:54
>/home/rsync-2bb/mysql/map/dwf.MYD
>-rw-rw----    1 mysql    mysql    221876208 Apr 11 14:21
>/home/rsync-2bb/mysql/ecp/dwf.MYD
>  
>
Actually, rsync's implementation only starts to give suboptimal results 
when we pass the 2.5GB area. Up until there one should see no noticeable 
difference in performance (well, my patch can allow using somewhat less 
memory for those cases, but no one would really miss 256KB of RAM these 
days, and I'm not sure how much impact the "CPU cache" effect is going 
to have in rsync's case. I guess that some effect will be seen anyways, 
but not as noticeable).

The performance bottleneck is due to hash table buckets load. Optimal 
load, as taught at computer science, is 80% (or 0.8). Your smallest file 
puts a load of 23%, while your largest file has a load of 53%. You 
shouldn't see any problems when using rsync (at least, not the type of 
problem I'm talking about).

          Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html