can't delete older file/directory links on another local partition after `snapshot' type backup script

Wed Mar 12 18:09:42 EST 2003

Thanks for looking at this, and developing rsync.  Really useful program that 
I rely on.  Bit of a newbie but I couldn't find anything on this in these 
archives.  Hopefully it's a really stupid question!

I'm trying to backup using cp -al and then rsync -auR --delete backup idea 
basically as per 
http://www.mikerubel.org/computers/rsync_snapshots/#Incremental.

I have a separate partition which I remount as rw, then cp -al yesterday's 
tree to a new one, say /todays/backup/directory.  Then

rsync -auR --delete --delete-excluded --ignore-errors --force 
--include-from=$includedfiles / /todays/backup/directory

to copy over any new files from the master (/) and delete any links to files 
that are no longer on the master.  This puts the root of master at 
/todays/backup/directory/ because of -R.

The problem: the links to old files and directories aren't getting deleted.  
As I understand --force shouldn't be needed, and I can't 
see why --ignore-errors would be necessary either in this case, but they 
don't make any difference.

It's all in a perl script (given below) run by cron.  Makes no difference 
whether run manually by root though.  I can remove directories or files from 
within the script using rm -r, so I don't think it is a permission or 
mounting problem?  I have it set up to use ssh by default (relevant?), but -e 
rsh doesn't make any difference.  I have tried doing this problem with small 
test directory trees on a single partition under a single user using the 
above command and that works fine.  Are there any file/directory deletion 
errors that could cause the deletion process to stop, even with 
--ignore-errors on?

rsync version 2.5.5, mandrake 8.2
Partition line from /etc/fstab: /dev/hdb1 /old ext3 defaults,ro 1 2

Any help appreciated --- I'm running out of disk space!

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
SCRIPT: relevant part is after 'now do rsync for real'

#!/usr/bin/perl -w
#
# ARCHIVE: script written to backup all info specified in archive.files
# Dalton Harvie, 24/2/03
#
# version 2: 26/2/03: include ls -l infor in archive.trans listing
#
# version 3: 10/3/03: including --force in rsync command to hopefully remove 
old directories
#
# wanted:
# 1) don't die without remounting /old as ro
# 2) gzip everthing and don't transfer unnecessary stuff if already there but 
zipped
#
#---------------------------------------------------------------------
# definitions

$version=3;

# define strings to define platform dependent functions
$CP='/bin/cp';
$MV='/bin/mv';
$LN='/bin/ln';
$LS='/bin/ls';
$RSYNC='/usr/bin/rsync';
$MOUNT='/bin/mount';

# $archivepart is a string specifying the partition which contains $archivedir
$archivepart='/old';

# $archivedir is the absolute path to archive structure
#$archivedir='/home/daltonh/tmp/old';  # defined globally
$archivedir='/old';  # so structure will be /old/old2003

# $log is a directory where the logfile archive.log should be stored
$log='/var/log';
#$log='/home/daltonh/tmp';

# $includedfiles is a filename which specifies in rsync format what files to 
transfer
#$includedfiles='/home/daltonh/perl/archive/archive.include';
$includedfiles='/root/archive/archive.include';

#---------------------------------------------------------------------
# open logfile in $log directory and append new info

open(LOG, ">>$log/archive.log");

print LOG "archive script version $version called at ".(scalar 
localtime)."\n";

#---------------------------------------------------------------------
# remount partition where the archive is in in rw mode

$systemcall="$MOUNT -w -o remount $archivepart";
(!(system($systemcall))) or do {print LOG "could not $systemcall\n"; die;};

#---------------------------------------------------------------------
# create backup directory if one hasn't already been specified

createdir($dir);  # call subroutine createdir to create the directory and 
possibly the tree too

print LOG "created directory for archive $dir\n";

#---------------------------------------------------------------------
# copy (hard link) files to directory from recent which has the previous tree 
(if it exists)

$recent="$archivedir/recent";

if (-e $recent and (-d $recent or -l $recent)) {  # have established that 
recent exists and that it's a link or directory

# copy (actually hard link) contents of $recent to new $dir, recursively, 
preserving atributes and copying but not following soft links

  $systemcall="$CP -al $recent/* $dir";  
  (!(system($systemcall))) or do {print LOG "could not $systemcall\n"; die;};

# now remove link ready for new one to be put in place

  unlink $recent or do {print LOG "could not unlink $recent\n"; die;};

}

# remove old archive.trans file if possible

if (-e "$dir/archive.trans") {unlink "$dir/archive.trans" or print LOG "could 
not unlink $dir/archive.trans\n";}

#---------------------------------------------------------------------
# rsync files to the new directory
# options: -a = -r (recursive) l (recreate symbolic links) ptgo (preserve 
permissions, time, group, ownership) D (copy devices)
#          -u = update
#          -R = relative filenames (eg /home/joeblogg -> old/home/joeblogg)
#          --delete = remove files from destination that aren't in source
#          --delete-excluded = remove files from destination that aren't 
included in transfer either
#          --include-from=$includedfiles = specify what to copy

# first check that $includedfiles exists

if (!(-f $includedfiles)) {print LOG "$includedfiles file does not exist\n"; 
die;}

# this bit modified for version 2
# open TEMP file and write out to this the transfer listing and some other 
info

open(TEMP, ">$dir.temp") or do {print LOG "could not open $dir.temp\n"; die;};

# assemble systemcall to run rsync for output purposes

$systemcall="$RSYNC -auRn --delete --delete-excluded 
--include-from=$includedfiles / $dir";  

print TEMP "archive script version $version called at ".(scalar 
localtime)."\n";
print TEMP "rsync command: $systemcall\n";
print TEMP "archive directory $dir\n";
print TEMP "this file contains a listing of files that were transferred for 
this backup\n";
print TEMP ('_'x120)."\n";
print TEMP `$systemcall`;
print TEMP ('_'x120)."\n";

close TEMP;

# now, open TEMP again, this time for reading

open(TEMP, "<$dir.temp") or do {print LOG "could not open $dir.temp\n"; die;};

# open TRAN, the destination file for this info, for writing

open(TRAN, ">$dir.trans") or do {print LOG "could not open $dir.trans\n"; 
die;};

# loop through initial lines, reading from TEMP and writing to TRAN

while (<TEMP>) {
  print TRAN $_;
  if (/^building file list/) {last;}  # this indicates the next line will be 
file listings
}

# now loop through file names, using ls -l to give file specifics

while (<TEMP>) {
  if (/^wrote/) {print TRAN $_; last;}  # this indicates that we have 
finished the file listings
  print TRAN `$LS -l $_` or print TRAN "problem listing $_";
}

# write out final lines
while (<TEMP>) {
  print TRAN $_;
}

close TRAN;
close TEMP;

# get rid of TEMP

unlink "$dir.temp" or print LOG "could not unlink $dir.temp\n";

#**********************
# now do rsync for real

# trying to work out why directories aren't deleted --- maybe permission 
problem? --- nah don't think so as rm -r works 
# shouldn't need --force or --ignore-errors according to me
$systemcall="$RSYNC -auR --delete --delete-excluded --ignore-errors --force 
--include-from=$includedfiles / $dir";  
(!(system($systemcall))) or do {print LOG "could not $systemcall\n"; die;};

# temp old
#$systemcall="$RSYNC -auR --delete --delete-excluded --force 
--include-from=$includedfiles / $dir";  
#(!(system($systemcall))) or do {print LOG "could not $systemcall\n"; die;};

# temp try just deleting something using this script --- works
#$systemcall="rm -r $dir/home/daltonh/balls/herbo/ir1";
#(!(system($systemcall))) or do {print LOG "could not $systemcall\n"; die;};

#**********************

#---------------------------------------------------------------------
# if possible move transfer file to directory

if (!(-e "$dir/archive.trans")) {    # this is an unlikely possibility if 
archive.trans already exists
  $systemcall="$MV $dir.trans $dir/archive.trans";
  (!(system($systemcall))) or print LOG "could not $systemcall\n";
}

#---------------------------------------------------------------------
# create a new soft link from the new directory to $recent

if (!(-e $recent))  # true if $recent doesn't exist
{
  $systemcall="$LN -s $dir $recent";  
  (!(system($systemcall))) or do {print LOG "could not $systemcall\n"; die;};
}
else
{
  print LOG "$recent link cannot be made because of existing file\n";
}

#---------------------------------------------------------------------
# remount partition where the archive is in in ro mode

$systemcall="$MOUNT -r -o remount $archivepart";
(!(system($systemcall))) or do {print LOG "could not $systemcall\n"; die;};

print LOG "archive finished\n".('_'x80)."\n";

#*******************************************************************************
# subroutine to check and create backup directory tree
# new directory (complete path from root) is given by $dir

sub createdir
{

# set date variables

  $year=(localtime)[5]+1900;        # portable date functions

$month=('jan','feb','mar','apr','may','jun','jul','aug','sep','oct','nov','dec')[(localtime)[4]];
  $day=(localtime)[3];

  $diryear=$archivedir.'/old'.$year;
  $dirmonth=$diryear.'/old'.$month.$year;
  $dirday=$dirmonth.'/old'.$day.$month.$year;

# loop through all required directories up to final checking and creating if 
necessary
# change directory permissions

  if (!(-d $archivedir)) {print LOG "archivedir $archivedir does not 
exist\n"; die;}

  foreach $dirc ($diryear,$dirmonth)
  {
    if (!(-d $dirc)) {mkdir ($dirc, 0755) or do {print LOG "can't make $dirc: 
$! \n"; die;};}
  }

# now loop through possible directory suffixes finding a directory that 
hasn't been created yet

  foreach $diradd ('','a'..'z','A'..'Z')
  {

    $dirtry=$dirday.$diradd;              # add a suffix to the default day 
string

    if (!(-d $dirtry))                    # test if that directory exists
    {
      mkdir ($dirtry, 0755) or do {print LOG "can't make $dirtry: $!\n"; 
die;};
      $dir=$dirtry;                       # must have made directory if we 
reach here
    }

    if ($dir) {last;}             # if $dir has been set then a directory has 
been creted

  }

# check that a directory actually was created

  if (!($dir))
  {
    print LOG "ran out of directory suffixes when attempting to create 
directory\n";
    die;
  }

}

#*******************************************************************************