Trying to diagnose incomplete file transfer
Robin Lee Powell
robinleepowell at gmail.com
Sun Mar 5 00:17:36 UTC 2023
I think it's very hard to be sure what's going on with
--remove-source-files ; I think you should drop that option, look
for whether the problem continues, and if you need the files to be
cleaned up, do so in a separate step.
In particular as someone else suggested, are you *sure* the original
copies of the partial files are actually the size you think they
are?
I don't ever recall seeing a case where rsync just failed to
transfer a file and thought it had succeeded; there pretty much has
to be something else going on.
On Sat, Mar 04, 2023 at 12:39:52AM -0600, Albert Croft via rsync wrote:
> At $work I have an odd situation involving incomplete file transfers, but I
> am unsure where the issue may be occurring. Here is the scenario.
>
> Problem:
> Sometimes the file transfer seems to have completed, but the file size does
> not match that on the remote system.
>
>
> Details:
> I transfer a number of large (>1GB) Tar-Gzipped (.tgz) files via SSH tunnels
> from $customer. Because of some previous issues, sometimes the SSH tunnels
> may be terminated externally. As a result, I am currently using the 'split'
> command to break the files into 1-GB "chunks" (ex.: foo.tgz.aa, foo.tgz.ab,
> ...).
>
> For the rsync transfer, I am using the following options:
> rsync -az \
> -e "ssh ..." \
> --link-dest=/local/path1 \
> --link-dest=/local/path2 \
> --remove-source-files \
> user at remote:/path/to/files \
> /local/path1/
>
> where
> '-e "ssh ..."' is the set of SSH options (for tunneling, etc.).
> '--link-dest=/local/path1' refers to a local directory that might contain a
> copy of the file.
> '--link-dest=/local/path2' refers to a local directory that might contain a
> copy of the file.
>
> I am frequently encountering times where the file appears to have been
> transferred but is incomplete. (Example: foo.tgz.ab now exists on the local
> system, has been removed from the remote, but is incomplete.)
>
>
> Additional notes:
> To my knowledge I do not know if the 'gzip' '--rsyncable' option is being
> used (but I do not think so--I suspect the file is created using a command
> similar to 'tar czf foo.tgz ...').
>
> The rsync commands may be launched from command-line or cron, but use the
> same format and options in either case. As a result, there may be multiple
> rsync processes pulling files from the same remote path to the same local
> path.
>
> I know that when rsync transfers a file (ex.: foo.tgz.ab) that during the
> transfer process it is named '.foo.tgz.ab.??????' (where '.??????' is a
> 6-character unique extension), and that upon completion the file is renamed
> to 'foo.tgz.ab'. (So I may see .foo.tgz.ab.4e67d0 and .foo.tgz.ab.fa7325 in
> the directory while the transfers are going.)
>
>
> I am unsure if this is a result of the combination of options I am using, or
> where to begin troubleshooting. Any guidance or direction would be
> appreciated.
>
> -Albert C.
>
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
More information about the rsync
mailing list