compression and encryption
vdvo at vdvo.net
Mon Jul 28 13:33:13 EST 2003
I've been thinking about adding the capability to store files compressed
and/or encrypted on either side of the rsync transfer. Let's say there's
a storage space provider. I want to store my files on that server
compressed (so I don't use more paid space than I need), and encrypted
(so they can't read my data).
I don't know much about encryption, but I suppose that there are some
ciphers that are reasonably strong and don't have the same problem like
gzip (that a single changed byte in the middle of the file affects the
contents of the rest of the file)? Even if not, the same thing like with
gzip (--rsyncable) could probably be done. The goal is to do the
encryption on one side so that the storage provider doesn't ever see the
unencrypted content or the key.
As for compression, I was thinking of making rsync temporarily
uncompress the file, then update it, and then recompress it. This way,
the gzip rsyncable patch wouldn't be needed, and one could actually use
any other compressor, like bzip2.
One very general way to achieve things like this would be to add the
ability to filter every file through a given filter before and after the
transfer, like this:
rsync --filter-source="encrypt --key=xyzzy" --filter-dest-before=bunzip2
--filter-dest-after=bzip2 /source/directory host:
It would be extremely slow, of course, but it may be worth it in some
situations. (The --filter-source wouldn't actually need to be so slow if
the filter was guaranteed to produce exactly as many bytes as it consumes.)
Of course, there's at least one major problem with this: generally, the
server can't let the client run any command as the filter. But the
server side rsync could be modified to have a config file option with a
regexp for allowed commands, or something like that.
Obviously, rsync could also be amended with built-in encryption. Less
general solution, but worth considering, too.
Any thoughts or opinions? Any existing work that I should be aware of?
Would a patch in some of the directions outlined above be accepted?
More information about the rsync