always copy the whole file?

jw schultz jw at pegasys.ws
Mon Aug 4 18:19:01 EST 2003


On Mon, Aug 04, 2003 at 03:02:48PM +0800, AndyWu wrote:
> Thanks for your help.
> The linux.iso is 640MB.
> I guess you indicate this thread, "[RFC] dynamic checksum size".

I don't think so.  I think it is an earlier thread.  Try
googling on --block-size.

> I must review the whole archive again, since most files > 8GB in my case.
> 
> 
> I'm trying to add some realtime replication funciton to rsync.
> I also found that someone did it with the changedfiles package(http://freshmeat.net/projects/changedfiles/).
> My idea is to use the netlink socket plus a module to replace sys_close.
> 
> Any help/comment/pointer would be appreciated.

Lustre, AFS, coda, almost any other clustering project, a
vfs layer to provide change notification (like coda), or JBD
with NBD.  Perhaps librsync to optimise transfers although
doing it in the vfs or block layer compressing the local
diff will be more efficient.

> 
> 
> Andy
> 
> -----Original Message-----
> From: rsync-bounces+andywu=iei.com.tw at lists.samba.org [mailto:rsync-bounces+andywu=iei.com.tw at lists.samba.org]On Behalf Of jw schultz
> Sent: Monday, August 04, 2003 11:20 AM
> To: rsync mailing list
> Subject: Re: always copy the whole file?
> 
> 
> On Mon, Aug 04, 2003 at 10:55:55AM +0800, AndyWu wrote:
> > Here's how it works in my 2 computers.
> > 
> > First, on auk2
> > # rsync -a auk1::pub/linux.iso .
> > It takes about 4 minutes. Everything is fine.
> > 
> > However, after doing this on auk1,
> > # echo "hello" >> /pub/linux.iso
> > 
> > Then, on auk2
> > # rsync -a auk1::pub/linux.iso .
> > 
> > The execution time of the first command is still 4 minutes.
> > I think it's supposed to be much shorter than the first case?
> > The same results appear in local rsync, too.
> > Is there anything wrong with my commands?
> 
> Unlikely.
> 
> I'll bet that file is several hundred megabytes.  Rsync has
> been known that have some difficulty with checksum
> collisions on files that large.  If you run with -vv i'll
> bet it will report "redoing linux.iso(0)" or something very
> similar.
> 
> With released versions of rsync this is avoidable to some
> degree with the --block-size option.  At one time someone
> produced a list of suitable block/file sizes.  Search the
> archives if this matters to you.  This problem has already
> been addressed in CVS.
> 
> -- 
> ________________________________________________________________
> 	J.W. Schultz            Pegasystems Technologies
> 	email address:		jw at pegasys.ws
> 
> 		Remember Cernan and Schmitt
> -- 
> To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
> -- 
> To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw at pegasys.ws

		Remember Cernan and Schmitt



More information about the rsync mailing list