Issue with hard links, please help!
Max Kipness
max at assuredata.com
Sat May 13 14:34:38 GMT 2006
> On Thu 11 May 2006, Max Kipness wrote:
>
> > [root at backup backup]# cp -al Latest/ mtest/
> > [root at backup backup]# du --max-depth=1 -h
> > 21G ./Latest
> > 8.7M ./mtest
> > 21G .
> > [root at backup backup]# rm mtest/ -rf
> > [root at backup backup]# cp -al Latest/ test/
> > [root at backup backup]# du --max-depth=1 -h
> > 21G ./test
> > 8.3M ./Latest
> > 21G .
> >
> > The last instance is the problem that happens quite often. Now when
I
>
> No, it's not a problem. It's just that now du encounters the "test"
> directory before finding the "Latest" directory. du only counts the
> blocks of hardlinked files once, and reports the size under the first
> directory such a file is in.
>
> The 8.3M (or 8.7M) is purely the disk blocks needed for the
directories
> and any symlinks if applicable, it is *not* related to storage of file
> contents.
>
> If you run "du -s -h test Latest" (or use the --count-links option)
you
> will see that each directory is handled separately, and both will have
> 21G.
>
> > perform an rsync as such:
> >
> > rsync /share/ /backup/Latest --stats --recursive --archive --times
> > --modify-window=1 --delete --ignore-errors --no-whole-file
> > --files-from=/var/www/html/new/var/backup_selections.txt
> > --exclude-from=/var/www/html/new/var/file-exclude --progress
> >
> > I get the following results:
> >
> > Number of files: 53911
> > Number of files transferred: 52223
> > Total file size: 21654476720 bytes
> > Total transferred file size: 21654476720 bytes
> > Literal data: 21651840443 bytes
> > Matched data: 0 bytes
> > File list size: 992872
> > Total bytes sent: 21657710607
> > Total bytes received: 1044480
> >
> > And a du gives me:
> >
> > [root at backup backup]# du --max-depth=1 -h
> > 21G ./test
> > 21G ./Latest
> > 41G .
> >
> > It appears that due to the cp -al command not working right as
stated
> > above, the literal changes needed was everything minus the 8.3mb,
when
> > in reality there were very few changes between 'Share' and 'Latest'.
>
> What's happened is that the files are updated, and the hard link is
> lost. Why the files are updated I can't say, it could be due to all
> sorts of reasons; perhaps using the --itemize-changes option will
help.
>
> Look into the --link-dest option, you can leave out your cp -al pass
in
> that case.
Thanks for the info. I think I understand better how the hard linking
works now.
I still can't seem to figure out why the hard links are breaking though.
And now I've noticed that I have similar issues on other server with
hard links (evidently).
In one instance, the server has maybe 600mb or so of changes per day,
and a total of about 19GB total files, yet each incremental directory
shows 5gb or so when doing a du. So does that mean that there are hard
links breaking daily?
du --max-depth=1 -h /backup
18G /backup/05-02-2006
5.1G /backup/04-26-2006
5.4G /backup/05-05-2006
5.0G /backup/04-23-2006
5.1G /backup/04-29-2006
5.1G /backup/04-27-2006
5.0G /backup/04-17-2006
5.0G /backup/04-13-2006
5.4G /backup/05-08-2006
3.8G /backup/05-06-2006
5.0G /backup/04-20-2006
3.9G /backup/05-07-2006
5.9G /backup/Current
5.1G /backup/05-03-2006
3.7G /backup/05-01-2006
5.0G /backup/04-21-2006
5.0G /backup/04-19-2006
5.0G /backup/04-25-2006
5.0G /backup/04-14-2006
3.6G /backup/04-15-2006
5.0G /backup/04-24-2006
3.7G /backup/04-28-2006
3.7G /backup/04-30-2006
3.9G /backup/05-09-2006
3.9G /backup/05-11-2006
3.6G /backup/04-16-2006
5.0G /backup/04-18-2006
3.9G /backup/05-10-2006
5.1G /backup/05-04-2006
146G /backup
[root at backup reports]# du -sh /backup/05-02-2006/ /backup/04-26-2006/
/backup/05-05-2006/ /backup/04-23-2006 /backup/04-29-2006
/backup/04-27-2006 /backup/04-17-2006 /backup/04-13-2006
/backup/05-08-2006 /backup/05-06-2006 /backup/04-20-2006
/backup/05-07-2006 /backup/Current /backup/05-03-2006 /backup/05-01-2006
/backup/04-21-2006 /backup/04-19-2006 /backup/04-25-2006
/backup/04-14-2006 /backup/04-15-2006 /backup/04-24-2006
/backup/04-24-2006 /backup/04-30-2006 /backup/05-09-2006
/backup/05-11-2006 /backup/04-16-2006 /backup/04-18-2006
/backup/05-10-2006 /backup/05-04-2006
18G /backup/05-02-2006/
18G /backup/04-26-2006/
19G /backup/05-05-2006/
18G /backup/04-23-2006
18G /backup/04-29-2006
18G /backup/04-27-2006
18G /backup/04-17-2006
18G /backup/04-13-2006
19G /backup/05-08-2006
19G /backup/05-06-2006
18G /backup/04-20-2006
19G /backup/05-07-2006
19G /backup/Current
18G /backup/05-03-2006
18G /backup/05-01-2006
18G /backup/04-21-2006
18G /backup/04-19-2006
18G /backup/04-25-2006
18G /backup/04-14-2006
18G /backup/04-15-2006
18G /backup/04-24-2006
18G /backup/04-24-2006
18G /backup/04-30-2006
19G /backup/05-09-2006
19G /backup/05-11-2006
18G /backup/04-16-2006
18G /backup/04-18-2006
19G /backup/05-10-2006
18G /backup/05-04-2006
Should each day/directory show around 600mb? I definitely don't think
that for 18GB of data, I should have a total of 146GB of storage total.
Here are the stats for the last backup. Should the Matched and Literal
equal the total file size?
Number of files: 50285
Number of files transferred: 113
Total file size: 16191157376 bytes
Total transferred file size: 4581163348 bytes
Literal data: 673846205 bytes
Matched data: 3905515136 bytes
File list size: 932272
Total bytes sent: 675118017
Total bytes received: 525414
sent 675118017 bytes received 525414 bytes 591890.87 bytes/sec
total size is 16191157376 speedup is 23.96
One thing to note is that the source data is coming from cifs mounted
windows shares. The rsync command I'm using for this one is:
/usr/bin/rsync /share/ /backup/Current/ --stats --recursive --partial
--archive --times --modify-window=1 --delete-after --delete-excluded
--ignore-errors --no-whole-file
--files-from=/var/www/html/backup/adlist.txt
--exclude-from=/scripts/file-exclude --log-format="%f %l %b"
And using Rsync 2.6.3 on this one.
Any additional information anyone has as to what might be my issue,
would be appreciated.
Thanks,
Max
More information about the rsync
mailing list