files not moved immediately to final destination from temp location after rsync returns with success
Alistair Dsouza
alistaird at gmail.com
Wed Apr 20 13:44:02 MDT 2011
Hi,
I am using rsync version 3.0.7 on an arm linux based embedded device. The
device pulls data periodically from a rsync server and stores the files on
an SD card.
The partial, temp and final rsync destinations all reside on the SD card.
I came across an issue where it seems that the rsync call returned with a
success but the files that it pulled are not
moved immediately to its final destination. The issue points either to rsync
not moving data immediately to its final location
of some delay in the virtual file system. However a read system call would
flush the block buffer via the VFS.
I have a script which periodically calls the rsync client to check if data
is available on a server located on a cloud.
The script contains two separate rsync calls. The first checks periodically
if a file of a particular name is present.
The script has logic to know which file name is should be searching for
next. Basically the file name has a number
in the name and this rysc calls periodically checks for that file.
The file pulled by the first rsync call contains info on the data (multiple
files contained under a parent directory) that has to
be pulled from the server. The second rsync call comes into play when the
first rsync call succeeds in pulling the file.
When the second rsync call returns with success the name of the file that
has to be pulled by the first call is updated and
the cycle continues. In case either of the rsync calls fail, the script
starts with the first rsync call.
<snippet of the script>
while :
do
# get the name of the command file to pull next.
CURR_Q_NUM=get_counter
REMOTE_Q_FILE="${CURR_Q_NUM}.txt"
LOCAL_Q_FILE="${Q_FOLDER}/${CURR_Q_NUM}.txt"
# update the source and destination for rsync client
SRC=$OPT_RSYNC_USER_NAME@
$OPT_RSYNC_DEV_NAME::$RSYNC_Q_MODULE/$REMOTE_Q_FILE
DEST=$LOCAL_Q_FILE
rsync --update \
--verbose \
--stats \
--port=$OPT_RSYNC_PORT \
--partial-dir=$PARTIAL_XFER_FOLDER \
--temp-dir=$TEMP_FOLDER \
--bwlimit $CONFIG_PULL_RATE \
--timeout $CONFIG_TIMEOUT \
--contimeout $CONFIG_TIMEOUT \
$SRC $DEST 1>>$RSYNC_LOG 2>&1
if [ $? -eq 0 ]; then
process_command_file <<<< if the file was pulled completely then
pull the files mentioned in this file.
[ $? -eq 0 ] && { update_counter; } <<<< update_counter increments
the file name to the next file we should be looking for if all the data was
all pulled correctly
fi
sleep $CONFIG_POLL_PERIOD <<<<< this was configured as 180 seconds.
done
function process_command_file
{
# extract the source and destination from the file pulled by te
rsync call in the while loop.
SRC= ....
DEST= .....
rsync --update \
--verbose \
--stats \
--port=$OPT_RSYNC_PORT \
--partial-dir=$PARTIAL_XFER_FOLDER \
--temp-dir=$TEMP_FOLDER \
--recursive \
--bwlimit $CONFIG_PULL_RATE \
--timeout $CONFIG_TIMEOUT \
--contimeout $CONFIG_TIMEOUT \
$SRC $DEST 1>>$RSYNC_LOG 2>&1
local RET=$?
if [ "$RET" -eq 0 ]
then
# if the second rsync call succeeds then process the data
FILES=$DEST/*
for file in $FILES
do
# do some processing on the files *<<<<< for some reason in
one instance the files were not present immediately in the final location
even though rsync pulled the files and returned with success.*
done
fi
return $RET
}
*<snippet end>*
I noticed that the data is pulled by the second rsync call (in
process_command_file) but when the script tries to process the data (just
one instance was seen) it did not find it in its final location.
in this instance when the data was not processed (in the if condition in the
process_command_file function) the rsync logs shows the data pulled
successfully by both rsync calls but some of the logs of the second rsync
call appear after the logs on the next rsync call in the cycle.
*<snippet of rsync logs when the issue is seen>*
0000000328.txt * <<<<< corresponding to the rsync call in
the while loop the command file is pulled*
Number of files: 1
Number of files transferred: 1
Total file size: 108 bytes
Total transferred file size: 108 bytes
Literal data: 108 bytes
Matched data: 0 bytes
File list size: 28
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 53
Total bytes received: 213
sent 53 bytes received 213 bytes 76.00 bytes/sec
total size is 108 speedup is 0.41
receiving incremental file list *<<<<<<< corresponding to the rsync
call in process_command_file function*
data/
data/LGC130UB.zip
rsync: link_stat "/0000000329.txt" (in q) failed: No such file or directory
(2) *<<<<< corresponding to the rsync call in the while loop*
* <<<<<< since 0000000329.txt is not yet present on the server the script
will loop on the rsync call in the while loop till it finds 000000329.txt*
Number of files: 0
Number of files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 3
Total bytes sent: 15
Total bytes received: 8
sent 15 bytes received 8 bytes 9.20 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors)
(code 23) at main.c(1508) [Receiver=3.0.7]
meta/ *<<<<<<< corresponding to the rsync call in process_command_file
function which should have occurred before the rsync call above it*
Number of files: 3 *<<<<< 3 corresponds to data, data/LGC130UB.zip, meta*
Number of files transferred: 1
Total file size: 15284642 bytes
Total transferred file size: 15284642 bytes
Literal data: 15284642 bytes
Matched data: 0 bytes
File list size: 56
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 68
Total bytes received: 15286657
sent 68 bytes received 15286657 bytes 87103.85 bytes/sec
total size is 15284642 speedup is 1.00
rsync: link_stat "/0000000329.txt" (in q) failed: No such file or directory
(2) *<<<<< corresponding to the rsync call in the while loop*
Number of files: 0
Number of files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 3
Total bytes sent: 15
Total bytes received: 8
sent 15 bytes received 8 bytes 5.11 bytes/sec
total size is 0 speedup is 0.00
*<end of snippet>*
*
*
*
*
*<snippet of rsync logs when everything works fine>*
rsync: link_stat "/0000000330.txt" (in q) failed: No such file or directory
(2) *<<<<< corresponding to the rsync call in the while loop*
Number of files: 0
Number of files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 3
Total bytes sent: 15
Total bytes received: 8
sent 15 bytes received 8 bytes 9.20 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors)
(code 23) at main.c(1508) [Receiver=3.0.7]
rsync: link_stat "/0000000330.txt" (in q) failed: No such file or directory
(2) *<<<<< corresponding to the rsync call in the while loop*
Number of files: 0
Number of files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 3
Total bytes sent: 15
Total bytes received: 8
sent 15 bytes received 8 bytes 6.57 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors)
(code 23) at main.c(1508) [Receiver=3.0.7]
0000000330.txt *<<<<< corresponding to the rsync call in the while loop. we
can now process the contents of this file and pull the data.*
Number of files: 1
Number of files transferred: 1
Total file size: 108 bytes
Total transferred file size: 108 bytes
Literal data: 108 bytes
Matched data: 0 bytes
File list size: 28
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 53
Total bytes received: 213
sent 53 bytes received 213 bytes 59.11 bytes/sec
total size is 108 speedup is 0.41
receiving incremental file list *<<<<<<< corresponding to the rsync call in
process_command_file function*
data/
data/MDL130UB.zip
meta/
Number of files: 3 *<<<<< 3 corresponds to data, data/MDL130UB.zip, meta*
Number of files transferred: 1
Total file size: 13810445 bytes
Total transferred file size: 13810445 bytes
Literal data: 13810445 bytes
Matched data: 0 bytes
File list size: 60
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 68
Total bytes received: 13812284
sent 68 bytes received 13812284 bytes 91776.43 bytes/sec
total size is 13810445 speedup is 1.00
rsync: link_stat "/0000000331.txt" (in q) failed: No such file or directory
(2) *<<<<< corresponding to the rsync call in the while loop*
Number of files: 0
Number of files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 3
Total bytes sent: 15
Total bytes received: 8
sent 15 bytes received 8 bytes 9.20 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors)
(code 23) at main.c(1508) [Receiver=3.0.7]
rsync: link_stat "/0000000331.txt" (in q) failed: No such file or directory
(2) *<<<<< corresponding to the rsync call in the while loop*
Number of files: 0
Number of files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 3
Total bytes sent: 15
Total bytes received: 8
sent 15 bytes received 8 bytes 6.57 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors)
(code 23) at main.c(1508) [Receiver=3.0.7]
*<end of snippet>*
thanks,
Alistair
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20110421/3567032f/attachment.html>
More information about the rsync
mailing list