rsync with overlay tree

[tomr]

> On 31 Mar 2016, at 16:22, Marian Marinov <mm at yuhu.biz> wrote:
> 
> On 03/31/2016 07:40 AM, tomr wrote:
>> I maintain a directory structure containing dirs and files that I regularly push to ~50 hosts, which are divided into 3 groups that have slightly different needs (minor mods in a couple of files).
>> 
>> So ideally I would have 4 directories:
>>    /path/to/sync/common/   <- common files
>>    /path/to/sync/group1/   <- group1 specific only
>>    /path/to/sync/group2/   <- group2 specific only
>>    /path/to/sync/group3/   <- group3 specific only
>> 
>> Then I'd run an rsync like:
>>    rsync -av --overlay /path/to/sync/groupN \
>>     /path/to/sync/common remotehost:
>> 
>> Thinking in terms of a list of files to be transferred, I would like:
>> - Anything present in common/ added to the file list; then
>> - Anything present in groupN/ added to the list, clobbering if applicable (regardless of mtime)
>> - The destination directory to show no sign of the common / overlay structure
> 
> I really like the idea, however I have one question. How is rsync going to decide which files to be present in the final list?
> I mean, should rsync always prefer the files from the overlay dir instead of the files in the common dir, no matter is they differ or not?

Yes :-)

In my thinking rsync should never know or even check whether they differ: if a filename (or directory?) is present in the overlay path, it should clobber any matching filename (or directory?) in the common path.

An example for clarity.... I start with:

    common/config/settings.conf
    common/config/include.d/names.conf
    common/config/include.d/numbers.conf
    common/config/include.d/plugins.conf
    common/bin/someapp
    common/bin/someplugin
    common/lib/somelib
    common/someapp-release

Almost all of the config of someapp has remained unchanged in the latest major release.  I don't want to fork all that config, and maintain in two places until we're fully upgraded, and I want to be able to provision servers with the old and the new config easily in the mean time.  So I create (minimally):

    group1/config/include.d/plugins.conf
    group1/config/include.d/newfeature.conf
    group1/bin/someplugin
    group1/lib/somelib
    group1/someapp-release

An rsync with eg '--overlay-path group1/' would sync plugins.conf, newfeature.conf, someplugin, somelib and someapp-release from under group1 regardless of any fact other than that they exist.  The automation that produces names.conf and numbers.conf can continue running without having to know about the arrangement.

If I eventually want to collapse group1 into common I can just run 'cp -r group1/* common && rm -r group1/*'

I think ;)

best,
tom



>> I suspect I could populate the list myself and use '--files-from=<LIST>' but I would rather have rsync work it out if possible.  If all else fails, I will do the merge locally first.
>> 
>>    TEMPDIR=$(mktemp -d)
>>    cp -r /path/to/sync/common/* $TEMPDIR
>>    cp -r /path/to/sync/groupN/* $TEMPDIR
>>    rsync -av $TEMPDIR/* remotehost:
>>    rm -r $TEMPDIR
>> 
>> Thanks in advance,
>> tom
>> 
>> 
>> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.samba.org/pipermail/rsync/attachments/20160331/ae6447a5/signature.sig>