I don't get --link-dest, at all, Now I do - SOLVED
Scott Schappell
archon at silvertree.org
Tue Aug 11 15:55:29 MDT 2009
Maybe I should take up an afternoon coffee habit. I did some reading
on du, and found out that it only disregards a file with multiple hard
links if it has seen it before. Running du -hcd1 on /home/backup
resulted in all expected results.
[root at arthur /home/backup]# du -hcd1 .
2.0K ./home.0
2.0K ./home.1
2.0K ./home.2
2.0K ./home.3
2.0K ./home.4
2.0K ./home.5
2.0K ./home.6
2.0K ./home.7
2.0K ./home.8
2.0K ./home.9
2.0K ./home.10
2.0K ./home.11
2.0K ./home.12
2.7G ./home.13
38M ./home.14
2.0K ./home.15
2.0K ./home.16
2.0K ./home.17
2.0K ./home.18
2.0K ./home.19
2.0K ./home.20
2.0K ./home.21
2.0K ./home.22
2.7G .
2.7G total
Ah well, I guess if nothing else, this may be useful to someone
Googling in the future.
Sorry for the spam to the list.
Scott
On Aug 11, 2009, at 14:49:18, Scott Schappell wrote:
> And now df is reporting proper usage of 5.4 GiB (which is what I
> expected). Maybe I just wasn't being patient enough and there's some
> weird df lag or something. Anyway, seems like it's working OK, but
> if anyone has any pointers on doing this more efficiently, I'd be
> more than happy to hear it.
>
> I'm still puzzed as to why du shows 2.7 GiB in both directories,
> though.
>
> Scott
> On Aug 11, 2009, at 14:25:46, I wrote:
>
>> Hourly I have an rsync job backup /home to /home/backup. I have 24
>> directories (one for each hour):
>>
>> home.0
>> ...
>> home.23
>>
>> Here is the script I am running via cron:
>>
>> #! /usr/local/bin/bash
>> dest=`date +%k | sed 's/ //g'`
>> linkdir=`date -v-1H +%k | sed 's/ //g'`
>> chflags -R noschg /home/backup
>> rm -rf /home/backup/home.$dest
>> rsync -ahHP --numeric-ids --delete --stats --link-dest=../
>> home."$linkdir"\
>> --exclude=/backup/* /home/ /home/backup/home."$dest"/ > /var/
>> rsync.log
>> sleep 2
>> chflags -R schg /home/backup
>>
>> Per reading this list and other google finds, it was stated that
>> for link-dest to work, the destination must be empty, which I do.
>> However, let's say a 2PM backup runs. The 2PM backup has 2.7GiB
>> (the current size of /home). The 1PM backup directory is also
>> 2.7GiB.
>>
>> [root at arthur /home/backup/home.14]# du -hcd0
>> 2.7G .
>> 2.7G total
>> [root at arthur /home/backup/home.14]#
>>
>> [root at arthur /home/backup/home.13]# !du
>> du -hcd0
>> 2.7G .
>> 2.7G total
>> [root at arthur /home/backup/home.13]#
>>
>> For a specific file:
>>
>> [root at arthur /home/backup/home.14/archon/public_html]# ls -li d2x.jpg
>> 22963255 -rwxr-xr-x 2 archon archon 47318 Oct 29 2008 d2x.jpg
>> [root at arthur /home/backup/home.14/archon/public_html]#
>>
>> [root at arthur /home/backup/home.13/archon/public_html]# !ls
>> ls -li d2x.jpg
>> 22963255 -rwxr-xr-x 2 archon archon 47318 Oct 29 2008 d2x.jpg
>> [root at arthur /home/backup/home.13/archon/public_html]#
>>
>> Both files share the same inode number, so I'm assuming it's a
>> hardlink, but why is the home.14 directory still saying 2.7GiB
>> used? Shouldn't it only be using whatever the delta is between home.
>> 13 and home.14 (in this case, rsync reported copying over about 40
>> MiB of data).
>>
>> df -h shows that /home is 8.6GiB when only ~3 GiB is actually being
>> used, so it's still creating fresh copies.
>>
>> I'm guessing I'm misunderstanding what --link-dest does, as I
>> understood it to create a hardlink to the file in the link-dest
>> directory if the file does not exist in the destination directory.
>>
>> Here's a log from the 2PM script run:
>>
>> [root at arthur /var]# cat rsync.log
>> sending incremental file list
>> created directory /home/backup/home.14
>> archon/IMAPdir/Apple Mail To Do/
>> archon/IMAPdir/Deleted Messages/
>> archon/IMAPdir/Deleted Messages/bincimap-cache
>> 2.22K 100% 0.00kB/s 0:00:00 (xfer#1, to-check=1943/2336)
>> archon/IMAPdir/Deleted Messages/bincimap-uidvalidity
>> 75 100% 0.60kB/s 0:00:00 (xfer#2, to-check=1942/2336)
>> archon/IMAPdir/Deleted Messages/cur/
>> archon/IMAPdir/Deleted Messages/cur/
>> 1250024242.R2110010672M559524P93911Q41.arthur.silvertree.org:2,S
>> 24.44K 100% 194.07kB/s 0:00:00 (xfer#3, to-check=1921/2336)
>> archon/IMAPdir/Deleted Messages/new/
>> archon/IMAPdir/Deleted Messages/tmp/
>> archon/IMAPdir/Freecycle/
>> archon/IMAPdir/Freecycle/bincimap-cache
>> 876 100% 6.96kB/s 0:00:00 (xfer#4, to-check=1890/2336)
>> archon/IMAPdir/Freecycle/bincimap-uidvalidity
>> 74 100% 0.59kB/s 0:00:00 (xfer#5, to-check=1889/2336)
>> archon/IMAPdir/Freecycle/cur/
>> archon/IMAPdir/Freecycle/cur/
>> 1250023358.R784558821M677376P93911Q39.arthur.silvertree.org:2,
>> 5.20K 100% 40.95kB/s 0:00:00 (xfer#6, to-check=1879/2336)
>> archon/IMAPdir/Freecycle/new/
>> archon/IMAPdir/Freecycle/tmp/
>> archon/Maildir/
>> archon/Maildir/bincimap-cache
>> 1.13K 100% 2.14kB/s 0:00:00 (xfer#7, to-check=1041/3870)
>> archon/Maildir/bincimap-uidvalidity
>> 75 100% 0.14kB/s 0:00:00 (xfer#8, to-check=1040/3870)
>> archon/Maildir/cur/
>> archon/Maildir/cur/1250021559.7658.arthur.silvertree.org:2,S
>> 7.60K 100% 13.45kB/s 0:00:00 (xfer#9, to-check=1026/3870)
>> archon/Maildir/cur/1250022468.8904.arthur.silvertree.org:2,S
>> 7.76K 100% 13.71kB/s 0:00:00 (xfer#10, to-check=1025/3870)
>> archon/Maildir/cur/1250022542.9042.arthur.silvertree.org:2,ST =>
>> archon/IMAPdir/Deleted Messages/cur/
>> 1250024242.R2110010672M559524P93911Q41.arthur.silvertree.org:2,S
>> archon/Maildir/cur/1250023069.9684.arthur.silvertree.org:2,ST =>
>> archon/IMAPdir/Freecycle/cur/
>> 1250023358.R784558821M677376P93911Q39.arthur.silvertree.org:2,
>> archon/Maildir/cur/1250023397.10077.arthur.silvertree.org:2,S
>> 4.68K 100% 8.26kB/s 0:00:00 (xfer#11, to-check=1022/3870)
>> archon/Maildir/cur/1250023861.10654.arthur.silvertree.org:2,S
>> 6.45K 100% 11.38kB/s 0:00:00 (xfer#12, to-check=1021/3870)
>> archon/Maildir/new/
>> archon/Maildir/tmp/
>> backup/
>> pinnacle/IMAPdir/Apple Mail To Do/
>> romany/Maildir/new/
>> romany/Maildir/new/1250023382.10044.arthur.silvertree.org
>> 2.49K 100% 0.00kB/s 0:00:00 (xfer#13, to-check=1001/22172)
>> romany/Maildir/tmp/
>> scott.schappell/IMAPdir/Apple Mail To Do/
>> scott.schappell/Maildir/
>> scott.schappell/Maildir/bincimap-cache
>> 46.29K 100% 279.03kB/s 0:00:00 (xfer#14, to-check=1004/24511)
>> scott.schappell/Maildir/bincimap-uidvalidity
>> 75 100% 0.30kB/s 0:00:00 (xfer#15, to-check=1003/24511)
>> scott.schappell/Maildir/cur/
>> scott.schappell/Maildir/cur/1250023996.10809.arthur.silvertree.org:2,
>> 20.12K 100% 66.38kB/s 0:00:00 (xfer#16, to-check=1024/24943)
>> scott.schappell/Maildir/new/
>> scott.schappell/Maildir/tmp/
>> soucon/3.1/game/data/sc.gdbm
>> 34.97M 100% 16.10MB/s 0:00:02 (xfer#17, to-check=1014/28108)
>>
>> Number of files: 32052
>> Number of files transferred: 17
>> Total file size: 2.79G bytes
>> Total transferred file size: 35.10M bytes
>> Literal data: 35.10M bytes
>> Matched data: 0 bytes
>> File list size: 991.74K
>> File list generation time: 0.009 seconds
>> File list transfer time: 0.000 seconds
>> Total bytes sent: 36.10M
>> Total bytes received: 2.71K
>>
>> sent 36.10M bytes received 2.71K bytes 4.81M bytes/sec
>> total size is 2.79G speedup is 77.36
>>
>> rsync version:
>>
>> [root at arthur /var]# rsync --version
>> rsync version 3.0.6 protocol version 30
>> Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and
>> others.
>> Web site: http://rsync.samba.org/
>> Capabilities:
>> 64-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints,
>> socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
>> append, ACLs, xattrs, no iconv, symtimes
>>
>> Installed from FreeBSD 7.2 ports collection.
>>
>> Thanks for any help in enabling me to understand how this works.
>>
>> Scott
>>
>>
>> --
>> Please use reply-all for most replies to avoid omitting the mailing
>> list.
>> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
> --
> Please use reply-all for most replies to avoid omitting the mailing
> list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
More information about the rsync
mailing list