[Bug 9624] New: rsync p2p extensions

[samba-bugs at samba.org]

https://bugzilla.samba.org/show_bug.cgi?id=9624

           Summary: rsync p2p extensions
           Product: rsync
           Version: 3.1.0
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: mark at ibiblio.org
         QAContact: rsync-qa at samba.org


Summary
=======
Add p2p enhancements to rsync to allow allow the rsync client to source the
file from multiple endpoints. The rsync server would track and nominate a list
of valid mirrors.

Background
==========
(Incredibly) large internet archive sites are moving towards phasing out
anonymous rsync and FTP in favour of BitTorrent P2P systems which offload the
majority of the transfer load to other sites. With the use of torrents then P2P
clients can also help with the traffic offloading. P2P also allows continued
functionality in the cases where network traffic isn't available to the
original server, due to network outages or attacks.

The torrent approach is being investigated now (see
http://www.terasaur.org/browse/category/software ) and the issues being ironed
out (
https://docs.google.com/a/ibiblio.org/document/d/1d2gJYsesalyXkYvrnY9AOornHYeGlwcLX1gjR8VMFlw/edit
)

The problem is when torrents are working adequately the intent is to phase out
rsync and FTP. This causes issues in the unix world with auto-mirroring from
cron, access via a cli, fetching files using regexps, accessing torrents from
behind corporate etc firewalls and reinventing the agility of rsync into the
bittorrent clients.

Enhancement
===========
rsync needs to extend its protocol to support more P2P functions.
 - mirrors
     - mirrors could nominate themselves when fetching content from the
       central site, possibly with timeouts for caching their existance
     - mirrors could download the mirror list using the P2P system and
       re-share it to P2P clients
 - clients
     - clients attach to the server and negotiate P2P fetching. They
       would then switch from classic rsync behaviour to P2P behaviour and
       spread the traffic load across a number of mirrors, possibly client
       peers in the future.

Searching google for "rsync p2p" returns a number of older projects which have
attempted this. It's possible a lot of those efforts could be leveraged to
reduce the development burden.

Justification
=============
Adding these and other P2P enhancements will future proof rsync so that archive
sites can utilize rsync more without worrying that excessive loads will
continue on the central site. The main traffic transfer cost can be shared with
sites around the world. rsync's value will remain high and those of us who rely
on it will continue to be offered it.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.