rsync feature request: conditional compression

[Seann Herdejurgen]

Thanks for the tip.

I was thinking about my suggestion, and I'm guessing that the --bwlimit feature is implemented as
an if statement like:

    if (data sent/time since last check > bwlimit) then sleep

There probably isn't a network buffer that is being checked.  Maybe this can be changed to:

    if (data sent/time since last check > bwlimit) then compress data

The problem then becomes, if bwlimit isn't specified, how could rsync determine what it should be?
Is there any way you can think of that would allow rsync to detect when it is on a fast network?
(as compared to CPU or disk subsystem)

I am not familiar with the inner workings of rsync.  I have a gut feel that if you were to
implement "conditional compression" that it would require asynchronous I/O.  Does rsync use AIO?

If a target file does not exist (for example, on the initial run of rsync), does it do --whole-file
by default?

-Seann

jw schultz wrote:
> 
> On Wed, Jan 15, 2003 at 06:47:09PM -0600, Seann Herdejurgen wrote:
> > rsync is optimized to conserve network resources,
> > sometimes at the expense of CPU resources.  As long as
> > CPU's are fast and networks are slow, rsync with
> > compression does a great job.  However, if I run rsync on
> > a slow box (166MHz Ultra SPARC) over a fast network
> > (100Mb/s), the compression kills the transfer rate.  I
> > turned compression off and I should be done rsyncing my
> > files (60GB) overnight.
> 
> If your network is faster than CPU and the disk subsystem
> you will find --whole-file speeds things up even more.
> The rsync algorithm is great but it does generate more than
> twice the disk I/O of --whole-file (subject to caching).
> 
> --
> ________________________________________________________________
>         J.W. Schultz            Pegasystems Technologies
>         email address:          jw at pegasys.ws
> 
>                 Remember Cernan and Schmitt