feature request, hardlink progress......

[George Georgalis]

On Thu, Feb 22, 2007 at 05:09:06PM +0100, Paul Slootman wrote:
>On Thu 22 Feb 2007, George Georgalis wrote:
>> >>
>> >>Please upgrade -- hard-link handling is much improved in newer versions.
>> >
>> >Thanks, turns out there are closer to 500,000 files and 89
>> >snapshots (@ ~90% of the files). The process was to install
>
>> > root at target:/root $ rsync --numeric-ids -avHP source:/data/ /data/
>> >receiving file list ...
>> >ERROR: out of memory in inode_table [receiver]
>> >rsync error: error allocating core memory buffers (code 22) at util.c(115) [receiver=2.6.9]
>> 
>> I added more memory, same problem.
>> 
>> any recommendations to get around this? will the old version of
>> rsync use less memory to make all the hardlinks?
>
>Having done something similar recently (copied an archive with daily
>snapshots, unchanged files hardlinked) I can suggest the following:
>
>If your layout is something like this:
>/data/YYYYMMDD/ where this is a daily snapshot, you can do:
>
>$ rsync --numeric-ids -avHP source:/data/20060101/ /data/20060101/
>(if the first snapshot is 20060101 of course.)
>
>Then:
>$ cd /data
>$ x=20060101
>$ for i in 200601{02,03,...,31}; do
>>       rsync -avHP --link-dest=../$x source:/data/$i/ /data/$i/
>>       x=$i
>> done
>
>If you ensure that the (empty) snapshot directories are already on your
>new system, then you can replace the 200601{...} with 2006*, although
>you may need to make it skip the first one; I don't know what rsync will
>do with a link-dest that is the same as the target :-)

my snapshots are at 3 hours, daily, and weekly, I weed out as they
get old; I was planning to "find" all my directory names, sort,
and use a scripted approach like yours. but I really wanted an
easier solution with less chance of it not working correctly.

when I set ulimit memory size and wired memory to unlimited, I
made progress, but still failed.

in the end the tar cf - . | tar xpf - method worked, and quickly;
remember that old transport??? I have some broken uid:gid to deal
with but that's par for the course.

cheers,
// George


-- 
George Georgalis, systems architect, administrator <IXOYE><