preserving Mac OS X metadata in rsync backups and restores

[Matt McCutchen]

On Sat, 2008-01-19 at 04:39 +0100, Moritz Heckscher wrote: 
> Good news indeed (for me at least)! Having read this FAQ on the  
> rsnapshot page in an early stage, without having much background info  
> about rsync and rsnapshot myself, I thought it was best to abandon  
> the idea of pushed backups with rsnapshot altogether:
> 
> [...]
> 
> Your suggestion does indeed take care of the "doubled disk space"  
> issue nicely. It's only really possible with your very recent patch  
> regarding the --no-tweak-hlinked option, so maybe now this FAQ entry  
> could be update with the above link to your description!

Getting in touch with the rsnapshot people to get the FAQ updated is on
my long to-do list.

> Regarding the other two points: If I have several clients, I guess I  
> should have several independent deamons, each with "max connections"  
> set to 1, and several independent "incoming" and "snapshots"  
> directories, to avoid conflicts: [...]

Yes.  Specifically:

- You definitely need a separate "incoming" module (including the lock
file used to enforce "max connections = 1"), rsnapshot configuration
file, and "snapshots" dir for each client.

- It is your choice whether to put all the modules in the same
rsyncd.conf (perhaps named "incoming.laptop0", etc.) or each in its own
rsyncd.laptop*.conf .  But note: if you plan to secure the setup by
having each client log in with a different ssh authorized key with a
forced command, you want the latter so you can limit each client to its
own module.

- I wouldn't make lots of copies of "kick-rsnapshot" if I could avoid it
by having it check which daemon/module called it.

In any event, I encourage you to write scripts/templates to
automatically generate the necessary configuration files so it's easy to
add or remove clients over time.  If you want to be really clever, you
can skip saving the files to disk and instead generate them on the fly
and pipe them to rsync and rsnapshot each time a client connects; if you
do this, put the magic in an ssh forced command so the client doesn't
have to specify a horrendous --rsync-path .

BTW, I just verified that the "max connections" limit covers daemon
processes that are in the post-xfer exec stage, which is important: we
don't want a client both pushing and rotating at the same time.

> Could there be any problems if laptop0 and laptop1 are both pushing  
> at the same time? Configurations are independent, and in that case  
> there should simply be two instances of rsnapshot running, right?

I don't foresee any problems with different clients pushing at the same
time.

> By they way, for the client to see if the backup operation succeeded,  
> this could be easily tested by looking at the return code from the  
> communication with the daemon, right?

The return code will tell you whether the push to "incoming" succeeded.
It will not reflect any failure in the rsnapshot rotation.  This is
probably fine from the client's perspective, as it can't really do
anything to handle such a failure.  Still, you might want to ensure that
you find out about any problems by adding code to "kick-rsnapshot" to
send you email if the rotation fails.

> > I would strongly discourage you from pushing the backup via a network
> > drive.  In addition to potential metadata mangling, you lose the  
> > benefit
> > of rsync's delta-transfer algorithm.

> Just out of curiosity: Why would rsync not use the delta-transfer in  
> that case? Is it because this would be the "local disk -> local disk"  
> transfer case for which a fast connection is assumed, so no resources  
> are wasted on trying to find differences?

The delta-transfer algorithm only reduces the data exchanged among the
rsync processes.  When all these processes are on the local machine, the
algorithm serves no purpose.  If the destination happens to be on a
network filesystem, the entire contents of each file rsync writes to the
network filesystem will be sent over the network regardless of how much
data changed; there is very little rsync can do about this.

> Well, thank you so very much. That post you've referred to was really  
> one of the most helpful texts for me! Thank you also for the nice  
> patch and all the hard work. I very much appreciate it!

Good luck!

Matt