rsync to iSCSI over WAN

[keropen]

On Fri, Jul 23, 2010 at 6:29 PM, Tom Christensen <pavera at live.com> wrote:

>
>
> > Date: Fri, 23 Jul 2010 10:02:26 +0200
> > From: ms at citd.de
>
> > To: pavera at live.com
> > CC: rsync at lists.samba.org
> > Subject: Re: rsync to iSCSI over WAN
> >
> > On 23.07.2010 00:30, Tom Christensen wrote:
> > >
> > > I am running rsync in cygwin on windows. I am attempting to backup a
> > > somewhat large data store (750GB) to a remote site. As its windows
> > > and preserving permissions exactly is important, I have an iSCSI drive
> > > mounted on the local system across a somewhat slow WAN link (IE, it
> > > would take about 3 months to copy the datastore over it).
> > > Unfortunately, since this appears as a "local" copy to rsync, it
> > > always copies whole files. Even though it is a "local" copy, I want
> > > to only send diffs, as we have large files that have small changes
> > > daily. Reading the man page, and everything I can find on the net I
> > > don't see an option to force diffs only/rsync protocol, is this
> > > possible?
> >
> > You have an abstraction error here, an iSCSI device is just a bit-bucket
> > like any localy connected HDD in that all filesystem and data processing
> > is done on the local side and raw block-data is send over the WAN. You
> > would have to run the other side of rsync on the machine that provides
> > the iSCSI-Device. For that to work the remote-machine would have to
> > mount the filesystem localy which in most cases means you would have to
> > unmount it from your Windows machine. This is because, except for
> > cluster-filesystems or a filesystem that is mounted read-only (on every
> > machine!), a given fileystem can only be mounted on 1 machine at a time.
> >
> >
> >
> >
> >
> >
> > Bis denn
> >
> > --
> > Real Programmers consider "what you see is what you get" to be just as
> > bad a concept in Text Editors as it is in women. No, the Real Programmer
> > wants a "you asked for it, you got it" text editor -- complicated,
> > cryptic, powerful, unforgiving, dangerous.
> >
>
> I am aware of that fact, it is precisely the fact that the filesystem is
> exposed directly to windows that I need iSCSI.  Mounting an ext3 partition
> shared via smb/cifs (another option on this NAS device) does not provide me
> with permissions fidelity (as ext3 does not support all of the ntfs
> permissions.  I was under the impression that rsync would calculate the
> diffs between the "local" copy and the "remote" (iSCSI) copy, and then only
> "send" the diffs, IE only the diffs would be written to disk (and therefore
> only the diffs would be sent via block-io over the WAN to the iSCSI).  But
> that very well could be a misunderstanding on my part of the way rsync
> functions at that low level.  Maybe on the remote side it reads through the
> whole file, writing it out to disk and inserting the diffs it receives?  In
> which case no matter what the whole file would be sent over block-io?  This
> functionality would also mean that even at the "network share" level
> (smb/cifs) at the remote side, you would gain nothing from --no-whole-file
> because the remote side would do the same thing IE write the whole file to
> the share anew resulting in the same amount of network traffic.
>
> At any rate, --no-whole-file appeared to improve performance on the backup
> last night, running the backup with --no-whole-file resulted in about 75%
> less data copied (according to the report at the end) and the backup ran in
> about half the time it was taking  previously.  Granted it could have just
> been a slow day in the office and maybe not that many files were changed...
> a sample size of 1 is not representative..
>
>
What kind of system is hosting the remote iSCSI? If you need NTFS, then you
could set up a iSCSI drive on on top of ZFS/FreeBSD - locally. Then setup a
similar system remotely. Do your backups against the local system NTFS-NTFS,
then send deltas/snapshot diffs to the remote system over WAN (using rsync
or "zfs send")

k
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20100726/f7827488/attachment.html>