[patch] read-devices

[Dave Dykstra]

Oh, syncing from a pipe on one end to a file on the other end is a
different story.  That I can see a use for.  However, it's not really what
the rsync command is designed for; it's designed for handling lots of
files.  I would think a simple tool based on librsync would be better
suited.  If we did want rsync to support it, I wouldn't lump it with
reading of other device files.  A syntax of specifying "-" as the source
file would make some sense, but a whole lot of options (especially most of
the ones inferred by "-a") won't make any sense in that situation, and the
destination has to be a file and cannot be a directory.  I think it would
probably have too great of an impact on the rsync code to be worth
supporting.

Syncing an entire raw partition is not a good idea unless you plan to have
it unmounted during the entire operation, because otherwise you're likely
to get some corruption in the copy.

- Dave

On Fri, Aug 30, 2002 at 02:48:20PM +0300, Eran Tromer wrote:
> Hi,
> 
> Thanks for your explanation. Before answering, I'll note that (as 
> mentioned in a followup) my patch has some unintended debugging cruft 
> and I'll gladly provide a clean patch if anyone's interested.
> 
> Well, here's my personal motivation. I have a remotely located server 
> with a 6GB disk, and also a newer machine a 80GB disk. I wish to keep a 
> full backup of the server filesystem on the other machine. A fairly 
> common setting, I imagine. Originally I rsynced the servers root 
> directory into a directory on the backup machine. This has two main 
> disadvantages:
> 
> 1. To get permissions and special files right, I had to run the receiver 
> side as root. From a security viewpoint, this is already nasty in the 
> extreme, but it's much worse than that, because now the backup machine 
> contained all sort of spooky device files and suid root binaries and 
> whatnot, created by rsync. If the server was compromised, in all 
> likelyhood the backup machine would go down next. Trusting permissions 
> on the parent directory of the rsync destination directory is not quite 
> enough for such stuff -- these things *have* been bypassed before.
> 
> 2. The target machine has a different filesystem. In my case it was only 
> a larger block size, which caused the backup to consume *much* more 
> diskspace than needed. In other cases, it could mean losing ACLs or 
> resource forks or whatever.
> 
> Now the beautiful thing is that tar handles all those things. All I 
> really want to do is to have an up-to-date "server.tar" file on the 
> backup machine, containing the output of running "tar cf - /" on the 
> server. But to do this with unpatched rsync, I needed to first create 
> the tar file on the server, which would *double* its diskspace requirements.
> 
> Nowadays, I pipe the output of "tar cf - /" into a pipe that's given as 
> as a filename to a patched sender rsync; the patched receiver rsync then 
> updates the corresponding huge file on the client.  I have a rotten ADSL 
> connection with terrible upstream speed so I increased the block size to 
> 10K, but the daily updates of these 6GB still take about 10 minutes (the 
> bottleneck is the server disk I/O time). Pretty fast, for something 
> completely stateless that's perfectly immune to wrong timestamps and 
> suchlike.
> 
> The SQL thing I mentioned is related, but here there isn't even a "just 
> rsync the files directly" alternative. Again, say you have a server 
> running some database server, and you want to take snapshots for backup 
> purposes, or whatever. Rsyncing the actual database files, if at all 
> possible for your RDBMS, will give you a corrupted database. However, if 
> you have a utility to dump the database into a flat file (e.g., 
> mysqldump of MySQL) then you can use that to get a consistent snapshot, 
> and rsync that snapshot. As before, to prevent the need for a huge 
> temporary file it's really nice to be able to pipe the output of the 
> dump utility directly into rsync.
> 
> Someone else had a different scenario, involving backing up a complete 
> disk partition at the block level. This also sounds quite useful, and 
> again quite impossible with unpatched rsync without a lot of extra 
> diskspace outside that partition.
> 
> As you can see, all of these backup-related scenarios make sense even 
> without the ability to update special files (which is obviously 
> problematic, though as I suggested a while ago, perhaps many common 
> cases *can* be efficiently handled pretty easily, especially data is 
> frequently updated but seldom shifted --- think /dev/hda).
> 
> Last but not least: this is the Unix way. It provides much power and 
> flexibility to the user, hence it is a Good Thing. I find my local 
> patched rsync very useful, and would experience eternal merriment if 
> everybody got that feature.
> 
> "device files" come into the picture simply because a pipe into stdin 
> (or any other FD) is seen a special file (on Linux, at least), and the 
> "--devices" command to rsync affects all special files, so 
> "--read-devices" appears appropriate. "--read-special" would be more 
> accurate, but less consistent.
> 
> BTW, I noticed that statistics reporting isn't working very well when 
> special files are read using my --read-devices patch, despite my efforts 
> to get some reasonable behavior (the problem is that the file size isn't 
> known until the whole file was read and sent). Probably fixable, and not 
> a major issue when someone needs the feature, methinks.
> 
>   Regards,
>     Eran
> 
> 
> 
> 
> Dave Dykstra wrote:
> >Similar patches have been submitted before but they've always been
> >rejected.  I'm sorry that you spent so much time on this one, although
> >perhaps it's been useful to you so far.
> >
> >As you know, rsync won't be able to write to such devices because it needs
> >to work on a temporary copy; that limits the usefulness of reading from
> >them greatly.  I'm not at all convinced that it's worths supporting it in
> >rsync.
> >
> >I can't read your examples in the rsync.fom right now because it's broken,
> >so I don't understand your motivating examples.  How can you rsync into a
> >tar file, and what do SQL databases have to do with device files?
> >
> >- Dave Dykstra
> 
> -- 
> To unsubscribe or change options: 
> http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html