Large file - match process taking days

Rob Bosch robbosch at msn.com
Wed Jul 30 13:16:53 GMT 2008


I've been trying to figure out why some large files are taking a long time
to rsync (80GB file).  With this file, the match process is taking days.
I've added logging to verbose level 4.  The output from match.c is at the
point where it is writing out the "potential match at" message.  In a 9 hour
period the match verbiage has changed from:

potential match at 14993337175 i=2976 sum=7c07ae74
potential match at 14993834514 i=3517 sum=0956772e
potential match at 14994673480 i=3232 sum=9be33b55
potential match at 14994912897 i=4739 sum=7b87587a
potential match at 14996877980 i=1453 sum=b7715246
potential match at 14999624225 i=906 sum=d9d831c6
potential match at 14999951039 i=2235 sum=6ca97091
potential match at 15001174331 i=3866 sum=12f966ee
potential match at 15001209073 i=2080 sum=783c7750
potential match at 15001399336 i=4522 sum=87f122e0
potential match at 15001543265 i=1360 sum=85dee02c
potential match at 15001770789 i=1637 sum=c55912e6
potential match at 15002913113 i=2783 sum=3fdbf408
potential match at 15004011466 i=3552 sum=ea7d0f44
potential match at 15005784863 i=2758 sum=cf9e00d6

To

potential match at 19827231165 i=3880 sum=f0b58ab2
potential match at 19827785238 i=4099 sum=f3338531
potential match at 19827870435 i=1232 sum=6abf175c
potential match at 19829135485 i=4472 sum=1ed3674e
potential match at 19829758278 i=2705 sum=dc796cb7
potential match at 19830224336 i=2959 sum=f0bd8161
potential match at 19830896106 i=3185 sum=6f83947a
potential match at 19832087866 i=1306 sum=14b38acb
potential match at 19832536037 i=1411 sum=3de116db
potential match at 19833817328 i=102 sum=45a8d003
potential match at 19835208508 i=2706 sum=e326d8e4
potential match at 19836927143 i=1591 sum=e357d821
potential match at 19838869812 i=4324 sum=1b113e13
potential match at 19839194857 i=3894 sum=03e116c1
potential match at 19839789868 i=3285 sum=39139716

I believe this means that 4.8GB of the file has been processed in this 9
hour period?  Blocksize is currently manually set at 1149728, 4 times the
default value.  Any idea on why it would be taking so long to go through
this portion of the sync process?  Rsync version is 3.0.3 on both ends.

Rob 





More information about the rsync mailing list