Hi Paul,<div><br></div><div>Thanks for your inputs. I think I've got an idea of whats taking place. Please see if what I have mentioned makes sense.</div><div><br></div><div>The reason file transfer is taking a few secs in the partial transfer case is cause the file is highly compressible and rsync</div>
<div>is using that to transfer less bytes to replicate the file. The file contains lines of chars. The same char is repeated on each line.</div><div>And the pattern is repeated. I yanked multiple lines and posted them to increase the file size. My bad.. I used a wrong test file..</div>
<div><br></div><div>When I tarred the file the size is just about 197 bytes.. Will try transferring/syncing a tarred file to start with. Its probably the </div><div>uncompressing and replication of the file that increased the CPU usage of rsync on the client to 80-90% which I though was </div>
<div>due to increased transfer rate.</div><div><br></div><div>The tarred file size</div><div>---------------------------</div><div><div>ls -la foo.txt.tar.gz </div><div>-rw-r--r-- 1 root root 197 Jan 22 15:40 foo.txt.tar.gz</div>
</div><div><br></div><div>I am taking note of time on a watch/PC. I am syncing a file of size 6,615,128 bytes (~6.3MB) which should take about </div><div>646 secs to transfer at 10KBps. When the file is transferred completely then it does take a little less than 11 mins to </div>
<div>complete.</div><div><div><br></div><div>Test 1: transfer complete file</div><div><meta http-equiv="content-type" content="text/html; charset=utf-8">The rsync server and client logs also show that the time taken is little less than 11 mins when the complete file has to be transferred.</div>
<div><br></div><div>rsync server daemon logs which is running on the cloud</div><div>------------------------------------------------------------------------------</div><div><div>2011/01/22 14:32:13 [12454] forward name lookup for YYYYYY failed: No address associated with hostname</div>
<div>2011/01/22 14:32:13 [12454] connect from UNKNOWN (XXX.XXX.XXX.XXX)</div><div>2011/01/22 09:02:14 [12454] rsync on sessions/example-7-step-1/* from UNKNOWN (XXX.XXX.XXX.XXX)</div><div>2011/01/22 09:02:15 [12454] building file list</div>
<div>2011/01/22 09:13:02 [12454] sent 6,616,898 bytes received 583 bytes total size 6,615,128</div></div><div>--------------------------------------------------------------------------------------------------------------------------------------------</div>
<div><br></div><div>rsync client logs which is running on my PC</div><div><div>------------------------------------------------------------------------------</div></div><div>receiving incremental file list</div><div>data/</div>
<div>data/foo.txt</div><div>meta/</div><div><br></div><div>Number of files: 3 (reg: 1, dir: 2)</div><div>Number of created files: 3 (reg: 1, dir: 2)</div><div>Number of deleted files: 0</div><div>Number of regular files transferred: 1</div>
<div>Total file size: 6,615,128 bytes</div><div>Total transferred file size: 6,615,128 bytes</div><div>Literal data: 6,615,128 bytes</div><div>Matched data: 0 bytes</div><div>File list size: 80</div><div>File list generation time: 0.001 seconds</div>
<div>File list transfer time: 0.000 seconds</div><div>Total bytes sent: 578</div><div>Total bytes received: 6,616,878</div><div><br></div><div>sent 578 bytes received 6,616,878 bytes 10,188.54 bytes/sec</div><div>total size is 6,615,128 speedup is 1.00</div>
<div>-------------------------------------------------------------------------------</div><div><br></div><div><br></div><div>Test 2: Start with transferring complete file i.e. remove the file on the client from the previous run.</div>
<div> Midway I remove the ethernet cable or kill the rsync client.</div><div> The partial transfer is of size 655360 bytes. i.e. 5959768 bytes are pending</div><div> The remaining transfer should take around 582 seconds but when I start the rsync client again</div>
<div> it just takes about 4 to 6 seconds.</div><div><br></div><div><br></div><div><div>rsync server daemon logs which is running on the cloud</div><div>------------------------------------------------------------------------------</div>
<div></div></div><div><div>2011/01/22 15:05:19 [12615] forward name lookup for XXXXXX failed: No address associated with hostname</div><div>2011/01/22 15:05:19 [12615] connect from UNKNOWN (XXX.XXX.XXX.XXX)</div><div>2011/01/22 09:35:20 [12615] rsync on sessions/example-7-step-1/* from UNKNOWN (XXX.XXX.XXX.XXX)</div>
<div>2011/01/22 09:35:20 [12615] building file list</div><div><br></div><div><br></div><div>2011/01/22 09:36:33 [12615] rsync: [sender] write error: Broken pipe (32)</div><div>2011/01/22 09:36:33 [12615] rsync: connection unexpectedly closed (109 bytes received so far) [sender]</div>
<div>2011/01/22 09:36:33 [12615] rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(-1604) [sender=3.1.0dev]</div><div><br></div><div>2011/01/22 15:07:54 [12624] forward name lookup for XXXXXX failed: No address associated with hostname</div>
<div>2011/01/22 15:07:54 [12624] connect from UNKNOWN (XXX.XXX.XXX.XXX)</div><div>2011/01/22 09:37:54 [12624] rsync on sessions/example-7-step-1/* from UNKNOWN (XXX.XXX.XXX.XXX)</div><div>2011/01/22 09:37:55 [12624] building file list</div>
<div>2011/01/22 09:38:02 [12624] sent 39,793 bytes received 4,938 bytes total size 6,615,128</div></div><div><br></div><div>check size of partial data transfer after killing the rsync client/removing the ethernet cable</div>
<div>------------------------------------------------------------------------------------------------------------------------------</div><div><div>ls -la rsync-partial/foo.txt </div><div>-rw-r--r-- 1 root root 655360 Jan 22 15:06 rsync-partial/foo.txt</div>
</div><div>------------------------------------------------------------------------------------------------------------------------------</div><div><div><br></div><div>rsync client logs which is running on my PC</div><div>
<div>------------------------------------------------------------------------------</div></div></div><div>receiving incremental file list</div><div><div>data/</div><div>data/foo.txt</div><div><br></div><div><br></div><div>
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(611) [generator=3.1.0dev]</div><div>rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(611) [receiver=3.1.0dev]</div><div>rsync: [receiver] write error: Broken pipe (32)</div>
<div><br></div><div><br></div><div>receiving incremental file list</div><div>data/foo.txt</div><div><br></div><div>Number of files: 3 (reg: 1, dir: 2)</div><div>Number of created files: 1 (reg: 1)</div><div>Number of deleted files: 0</div>
<div>Number of regular files transferred: 1</div><div>Total file size: 6,615,128 bytes</div><div>Total transferred file size: 6,615,128 bytes</div><div>Literal data: 6,496 bytes</div><div>Matched data: 6,608,632 bytes</div>
<div>File list size: 80</div><div>File list generation time: 0.001 seconds</div><div>File list transfer time: 0.000 seconds</div><div>Total bytes sent: 4,933</div><div>Total bytes received: 39,773</div><div><br></div><div>
sent 4,933 bytes received 39,773 bytes 5,259.53 bytes/sec</div><div>total size is 6,615,128 speedup is 147.97</div></div><div>------------------------------------------------------------------------------------------------------------------------------</div>
<div><br></div><div>Regards,</div><div>Alistair</div></div><div><br></div><div><br></div><div><div class="gmail_quote">On Fri, Jan 21, 2011 at 6:45 PM, Paul Slootman <span dir="ltr"><<a href="mailto:paul%2Brsync@wurtel.net">paul+rsync@wurtel.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">On Fri 21 Jan 2011, Alistair Dsouza wrote:<br>
><br>
> I have set the bwlimit to 10 i.e. 10 KBps. When I try to sync files the<br>
> bwlimit is as what is<br>
> set. I have enabled storage of partial files. When a communication break<br>
> causes rsync to<br>
> time out during a transfer the partial file is stored in the partial<br>
> directory. I then try rsync<br>
> again when the link is up. At this point I see that the bwlimit of 10KBps is<br>
> not followed.<br>
> rsync transfers the data as fast as possible.<br>
<br>
</div>How did you measure this data rate?</blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Are you using the --progress output? Then you need to realize that what<br>
is shown there is not the data rate over the network, but the effective<br>
rate at which the file is transferred. As the first part of the file is<br>
in the partial directory, the data that needs to be transmitted is not<br>
much more than the block checksums plus a little overhead. So then it<br>
looks like the file is being transmitted a lot faster.<br>
<br>
The same thing can be seen when using -z while transferring a highly<br>
compressable file: --progress will show a much higher transfer rate than<br>
what you've configured with --bwlimit.<br>
<br>
This phenomenon can be easily verified by doing it over a dialup line,<br>
or other medium that physically can't go faster than a given rate:<br>
--progress will show higher rates than would be possible.<br>
<br>
<br>
<br>
Paul<br>
<font color="#888888">--<br>
Please use reply-all for most replies to avoid omitting the mailing list.<br>
To unsubscribe or change options: <a href="https://lists.samba.org/mailman/listinfo/rsync" target="_blank">https://lists.samba.org/mailman/listinfo/rsync</a><br>
Before posting, read: <a href="http://www.catb.org/~esr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/smart-questions.html</a><br>
</font></blockquote></div><br></div>