dwd at bell-labs.com
Fri Aug 30 10:51:00 EST 2002
Oh, syncing from a pipe on one end to a file on the other end is a
different story. That I can see a use for. However, it's not really what
the rsync command is designed for; it's designed for handling lots of
files. I would think a simple tool based on librsync would be better
suited. If we did want rsync to support it, I wouldn't lump it with
reading of other device files. A syntax of specifying "-" as the source
file would make some sense, but a whole lot of options (especially most of
the ones inferred by "-a") won't make any sense in that situation, and the
destination has to be a file and cannot be a directory. I think it would
probably have too great of an impact on the rsync code to be worth
Syncing an entire raw partition is not a good idea unless you plan to have
it unmounted during the entire operation, because otherwise you're likely
to get some corruption in the copy.
On Fri, Aug 30, 2002 at 02:48:20PM +0300, Eran Tromer wrote:
> Thanks for your explanation. Before answering, I'll note that (as
> mentioned in a followup) my patch has some unintended debugging cruft
> and I'll gladly provide a clean patch if anyone's interested.
> Well, here's my personal motivation. I have a remotely located server
> with a 6GB disk, and also a newer machine a 80GB disk. I wish to keep a
> full backup of the server filesystem on the other machine. A fairly
> common setting, I imagine. Originally I rsynced the servers root
> directory into a directory on the backup machine. This has two main
> 1. To get permissions and special files right, I had to run the receiver
> side as root. From a security viewpoint, this is already nasty in the
> extreme, but it's much worse than that, because now the backup machine
> contained all sort of spooky device files and suid root binaries and
> whatnot, created by rsync. If the server was compromised, in all
> likelyhood the backup machine would go down next. Trusting permissions
> on the parent directory of the rsync destination directory is not quite
> enough for such stuff -- these things *have* been bypassed before.
> 2. The target machine has a different filesystem. In my case it was only
> a larger block size, which caused the backup to consume *much* more
> diskspace than needed. In other cases, it could mean losing ACLs or
> resource forks or whatever.
> Now the beautiful thing is that tar handles all those things. All I
> really want to do is to have an up-to-date "server.tar" file on the
> backup machine, containing the output of running "tar cf - /" on the
> server. But to do this with unpatched rsync, I needed to first create
> the tar file on the server, which would *double* its diskspace requirements.
> Nowadays, I pipe the output of "tar cf - /" into a pipe that's given as
> as a filename to a patched sender rsync; the patched receiver rsync then
> updates the corresponding huge file on the client. I have a rotten ADSL
> connection with terrible upstream speed so I increased the block size to
> 10K, but the daily updates of these 6GB still take about 10 minutes (the
> bottleneck is the server disk I/O time). Pretty fast, for something
> completely stateless that's perfectly immune to wrong timestamps and
> The SQL thing I mentioned is related, but here there isn't even a "just
> rsync the files directly" alternative. Again, say you have a server
> running some database server, and you want to take snapshots for backup
> purposes, or whatever. Rsyncing the actual database files, if at all
> possible for your RDBMS, will give you a corrupted database. However, if
> you have a utility to dump the database into a flat file (e.g.,
> mysqldump of MySQL) then you can use that to get a consistent snapshot,
> and rsync that snapshot. As before, to prevent the need for a huge
> temporary file it's really nice to be able to pipe the output of the
> dump utility directly into rsync.
> Someone else had a different scenario, involving backing up a complete
> disk partition at the block level. This also sounds quite useful, and
> again quite impossible with unpatched rsync without a lot of extra
> diskspace outside that partition.
> As you can see, all of these backup-related scenarios make sense even
> without the ability to update special files (which is obviously
> problematic, though as I suggested a while ago, perhaps many common
> cases *can* be efficiently handled pretty easily, especially data is
> frequently updated but seldom shifted --- think /dev/hda).
> Last but not least: this is the Unix way. It provides much power and
> flexibility to the user, hence it is a Good Thing. I find my local
> patched rsync very useful, and would experience eternal merriment if
> everybody got that feature.
> "device files" come into the picture simply because a pipe into stdin
> (or any other FD) is seen a special file (on Linux, at least), and the
> "--devices" command to rsync affects all special files, so
> "--read-devices" appears appropriate. "--read-special" would be more
> accurate, but less consistent.
> BTW, I noticed that statistics reporting isn't working very well when
> special files are read using my --read-devices patch, despite my efforts
> to get some reasonable behavior (the problem is that the file size isn't
> known until the whole file was read and sent). Probably fixable, and not
> a major issue when someone needs the feature, methinks.
> Dave Dykstra wrote:
> >Similar patches have been submitted before but they've always been
> >rejected. I'm sorry that you spent so much time on this one, although
> >perhaps it's been useful to you so far.
> >As you know, rsync won't be able to write to such devices because it needs
> >to work on a temporary copy; that limits the usefulness of reading from
> >them greatly. I'm not at all convinced that it's worths supporting it in
> >I can't read your examples in the rsync.fom right now because it's broken,
> >so I don't understand your motivating examples. How can you rsync into a
> >tar file, and what do SQL databases have to do with device files?
> >- Dave Dykstra
> To unsubscribe or change options:
> Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
More information about the rsync