Feature Request: Multiple Streams

[Clint Byrum]

On Tue, 2004-03-09 at 14:15, Jim Salter wrote:
> Tim Conway wrote:
> 
> > for source in source1 source2 source3
> >         do
> >                 rsync -options $source destination:$source &
> >         done
> > wait
> > 
> > adapt as needed.
> 
> That will WORK, of course, but it does require that you do a file list 
> build and compare for each of the above, instead of doing a single list 
> build and then dissemination to multiple targets... which (I think?) is 
> what the original poster is asking for(?)
> 

I know thats what I'd want. I have one source server, that rsyncs the
same 3GB of random files to over 20 machines a few times a day. While we
can do something similar like this:

for dest in d1 d2 d3 ; do
  rsync -xxxx /source/dir $dest:/source/dir &
done


...the RAM on the source server isn't endless, and 20 copies of rsync
reading thousands of random files would probably hit the VM and
scheduler pretty hard as well.  We've thought of doing something similar
with tar...

for dest in d1 d2 d3 ; do
  mkfifo fifo_$dest
  ssh $dest "tar xf - -C /source" < fifo_$dest &
  FIFOS="$FIFOS fifo_dest"
done
tar cvf - -C /source dir | tee $FIFOS > /dev/null

The problems with this are
1) no rsync algorithm.. just slamming data out
2) tee isn't asynchronous or multithreaded/multiprocessed, so it waits
for each write.

Even with a smarter tee, we lose the rsync algorithm's adaptability.

With gigabit ethernet, we could easily send 85MB/s, and the boxes can
only write to disk at 20M/s, so in theory we could send to 4 machines at
once if rsync had multiple stream support. ;)

It would be nice to have it read the data once, and then sync it to all
of the destinations once. IIRC, there was a move to do this at some
point. Am I right?