is it possible to continue rsync after network interruption?

Matthias Meyer matthias.meyer at gmx.li
Tue Mar 31 21:03:37 GMT 2009


Larry Irwin wrote:

> If you check the output and see that it failed, you can run the process
> again and it will pick up (after re-building the lists) where it left off.
> There is some overhead for the list rebuild and checking the hash's of the
> previously updated files, but it is minimal.
> This is the method I use to generate the initial image of a client
> machine...
> Just keep running the rsync process over and over until the first image is
> completed.
> Then the process reduces to a minimal transfer/execution time from that
> point forward.
> Especially when using the --link-dest option...
> 
> Hope that helps!
> Larry Irwin
> 
Unfortunately no :-(
I have a remote client with round about 100GB and a 1MBit/s upload connection.
This client try to backup since some weeks.
Each day he write into the logfile something like:
The log starts with something like:
2009/03/30 23:54:43 [2564] sender finished ...
2009/03/30 23:54:43 [2564] send_files(1749, ...)
2009/03/30 23:54:43 [2564] send_files mapped ... of size 2207411
2009/03/30 23:54:43 [2564] calling match_sums ...
2009/03/30 23:55:12 [2564] rsync: writefd_unbuffered failed to write 4092 bytes [sender]: Connection reset by peer (104)
2009/03/30 23:55:12 [2564] rsync error: error in rsync protocol data stream (code 12) at /home/lapo/packaging/rsync-3.0.4-1/src/rsync-3.0.4/io.c(1541) [sender=3.0.4]
2009/03/30 23:55:12 [2564] _exit_cleanup(code=12, file=/home/lapo/packaging/rsync-3.0.4-1/src/rsync-3.0.4/io.c, line=1541): about to call exit(12)
2009/03/30 23:55:12 autossh[452]: ssh exited with error status 255; restarting ssh
2009/03/31 00:03:51 [5328] name lookup failed for 127.0.0.1: Unknown server error
2009/03/31 00:03:51 [5328] connect from UNKNOWN (127.0.0.1)
2009/03/31 00:03:51 [5328] rsync on . from UNKNOWN (127.0.0.1)
2009/03/31 00:03:51 [5328] [sender] add_rule(- /proc)
2009/03/31 00:03:51 [5328] [sender] add_rule(- /cygdrive)
2009/03/31 00:03:51 [5328] server_sender starting pid=5328
2009/03/31 00:03:51 [5328] building file list
2009/03/31 00:03:51 [5328] [sender] make_file(.,*,0)
2009/03/31 00:03:51 [5328] [sender] make_file(...,*,2)
2009/03/31 00:03:51 [5328] [sender] make_file(...,*,2)
: <this is another share, therefor another pid>
2009/03/31 00:08:18 [804] send_file_list done
2009/03/31 00:08:18 [804] send_files starting
2009/03/31 00:08:22 [804] send_files(1, BOOTSECT.BAK)
2009/03/31 00:08:22 [804] send_files mapped BOOTSECT.BAK of size 8192
2009/03/31 00:08:22 [804] calling match_sums BOOTSECT.BAK
2009/03/31 00:08:22 [804] built hash table
2009/03/31 00:08:22 [804] hash search b=2048 len=8192
2009/03/31 00:08:22 [804] match at 2048 last_match=0 j=1 len=2048 n=2048
2009/03/31 00:08:22 [804] match at 4096 last_match=4096 j=2 len=2048 n=0
2009/03/31 00:08:22 [804] match at 6144 last_match=6144 j=3 len=2048 n=0
2009/03/31 00:08:22 [804] done hash search
2009/03/31 00:08:22 [804] sending file_sum
2009/03/31 00:08:22 [804] false_alarms=0 hash_hits=3 matches=3
2009/03/31 00:08:22 [804] sender finished BOOTSECT.BAK
2009/03/31 00:08:22 [804] send_files(3, Boot/BCD)
:
2009/03/31 00:12:58 [804] match at 374784 last_match=374784 j=183 len=2048 n=0
2009/03/31 00:12:58 [804] match at 376832 last_match=376832 j=184 len=2048 n=0
2009/03/31 00:12:58 [804] match at 378880 last_match=378880 j=185 len=512 n=0
2009/03/31 00:12:58 [804] done hash search
2009/03/31 00:12:58 [804] sending file_sum
2009/03/31 00:12:58 [804] false_alarms=0 hash_hits=186 matches=186
2009/03/31 00:12:58 [804] sender finished ...
2009/03/31 00:12:58 [804] send_files(255, ...)
2009/03/31 00:12:58 [804] send_files mapped ... of size 5212160
2009/03/31 00:12:58 [804] calling match_sums ...
2009/03/31 00:14:26 [804] sending file_sum
2009/03/31 00:14:26 [804] false_alarms=0 hash_hits=0 matches=0
2009/03/31 00:14:26 [804] sender finished ...
:
2009/03/31 22:33:55 [4952] sender finished ...
2009/03/31 22:33:55 [4952] send_files(1752, ...)
2009/03/31 22:33:55 [4952] send_files mapped ... of size 2065979
2009/03/31 22:33:55 [4952] calling match_sums ...
2009/03/31 22:34:13 [4952] sending file_sum
2009/03/31 22:34:13 [4952] false_alarms=0 hash_hits=0 matches=0
2009/03/31 22:34:13 [4952] sender finished ...

In this 22 hours only 4400 files with a total of 2500MB are transfered.
I'm right, that "match at ..." means an identical file was found and nothing has to be transfered
and "send_files ..." means a new (part of) file will be transfered?

Should I make the backup as incremental backup? Would that be better?
Is it possible that it is a problem in backuppc?
Should I ask my question their?

Thanks,
Matthias
-- 
Don't Panic



More information about the rsync mailing list