rsync and hfs+ compressed files damaged
Robert DuToit
rdutoit at comcast.net
Thu Jul 1 20:14:23 MDT 2010
Thanks Mike once again,
I'll test this out tomorrow.
Rob
On Jul 1, 2010, at 6:13 PM, Mike Bombich wrote:
> Well, it looks like the hfs_compression patch that I posted was the
> problem, and I've solved it since then. I can reproduce the issue
> with the patch, but not with my subsequent changes (my changes are
> posted in their entirety, vs. breaking out the hfs_compression-
> specific changes). Here is an updated patch, and I have verified
> that it resolves this issue.
>
> As an aside, the patch contains a couple bug fixes to the xattr code
> as well. I'll break those out and submit them separately in the
> near future.
>
> Mike
>
> <rsync_3.0.7-hfs_compression_20100701.diff>
>
>
> On Jul 1, 2010, at 6:16 AM, Robert DuToit wrote:
>
>> Hi Mike and All,
>>
>> I have been doing some research on the damaged HFS compressed files
>> and found some interesting clues. Looking at mike's patch so far I
>> can't find any reason for the damage (I have to learn more C
>> language ) but what still seems odd is that the damage only happens
>> when copying the whole /usr directory or /usr/bin /usr/share.
>> Individually copying one of the original files does not create the
>> damaged file.
>>
>> Pertaining to the use of Mike's patch on OSX 10.6.4 copying /usr/
>> bin and resulting damaged files
>>
>> In /usr/bin/ I found that the originals of the damaged files are
>> different from the other "normal" exec files. In this case I am
>> looking at the "find" exec.
>>
>> If I copy "find" from /usr/bin/ to the desktop via finder copy, it
>> is changed to resemble the "normal" execs. And indeed if I replace
>> the original with this copy in /usr/bin and copy /usr/bin with
>> patched rsync, the copied "find" is no longer damaged!
>>
>> So it appears that when the patched rsync encounters one of these
>> "odd" files ( and only when copying the whole /usr/bin/ directory
>> containing them ) it causes the problem.
>>
>> Here are results using hfsdebug (sorry about the amount of reading!)
>>
>>
>> ####################
>> original "file" exec - works ok but different from other execs in /
>> usr/bin. ( It shows the com.apple.decmpfs flag and data is stored
>> in Resource Fork)
>>
>> flags = 0000000000000110
>> . File has extended attributes.
>>
>> # Data Fork
>> logicalSize = 0 bytes
>> # Resource Fork
>> logicalSize = 101755 bytes
>> totalBlocks = 25
>>
>> # Attributes
>> <Attributes B-Tree node = 6754 (sector 0x294b0)>
>> # Attribute Key
>> keyLength = 46
>> pad = 0
>> fileID = 3904426
>> startBlock = 0
>> attrNameLen = 17
>> attrName = com.apple.decmpfs
>> # Inline Data
>> recordType = 0x10
>> reserved[0] = 0
>> reserved[1] = 0
>> attrSize = 16 bytes
>> attrData = 66 70 6d 63 04 00 00 00 b0 a0 03 00 00 00
>> 00 00
>> f p m c
>>
>> compression magic = cmpf
>> compression type = 4 (resource fork has compressed data)
>> uncompressed size = 237744 bytes
>>
>> ####################
>> A normal exec "find" in /usr/bin/ ( data stored in # Data Fork so
>> it is showing as uncompressed in finder)
>>
>> flags = 0000000000000010
>>
>> # BSD Info
>> ownerID = 0 (root)
>> groupID = 0 (wheel)
>> = 0
>> # Data Fork
>> logicalSize = 147680 bytes
>> totalBlocks = 37
>> clumpSize = 2445
>>
>> # Resource Fork
>> logicalSize = 0 bytes
>>
>>
>> ####################
>> the original "file" exec now copied via patched rsync within the
>> whole /usr/bin directory (Note: no data in Data Fork or Resource
>> Fork)
>>
>> First note that the xattrs show up ( not supposed to when seen
>> from 10.6)
>>
>> robert-dutoits-macbook-pro:~ astrid$ xattr -l /Users/astrid/Desktop/
>> copied_bin_test/bin/file
>> com.apple.ResourceFork:
>> com.apple.decmpfs:
>>
>> and the hfsdebug report:
>>
>> flags = 0000000000000110
>> . File has extended attributes.
>>
>> # Data Fork
>> logicalSize = 0 bytes
>> # Resource Fork
>> logicalSize = 1 bytes
>> totalBlocks = 1
>>
>> # Attributes
>> <Attributes B-Tree node = 5258 (sector 0x23730)>
>> # Attribute Key
>> keyLength = 46
>> startBlock = 0
>> attrNameLen = 17
>> attrName = com.apple.decmpfs
>> # Inline Data
>> recordType = 0x10
>> reserved[0] = 0
>> reserved[1] = 0
>> attrSize = 1 bytes
>> attrData = 04
>>
>> compression magic = .
>> compression type = 44000000 (?)
>> uncompressed size = 1224979098644812920 bytes
>>
>>
>> ####################
>>
>> I can't put this all together yet but if anyone can see what is
>> going on I would much appreciate it. What throws me though it that
>> the damage only happens when copying the whole /usr/bin/ directory…
>>
>> Again - this is rsync 3.0.6 patched with fileflags, crtimes and
>> Mike's hfs patch.
>>
>> Thanks, Rob
>>
>>
>>
>>
>>
>> On Jun 25, 2010, at 4:03 PM, Mike Bombich wrote:
>>
>>> Can you share the steps required to reproduce this? I've seen two
>>> reports of this but have not been able to reproduce it myself, and
>>> the users that reported it also could not reproduce it.
>>>
>>> Mike
>>>
>>> On Jun 25, 2010, at 5:16 AM, Robert DuToit wrote:
>>>
>>>> Hi All,
>>>>
>>>> I have been using rysnc 3.0.6 with Mike's rsync_3.0.6-hfs-
>>>> compression_20091027.diff patch (as well as the standard osx
>>>> patches) with good results. I discovered that clones done on 10.6
>>>> using this rsync cause some executable files ( the ones that are
>>>> mostly compressed this way on osx ) to be damaged.
>>>>
>>>> I did a test twice and indeed a handful of them are damaged. They
>>>> all appear to be in /Volume/Snowy/usr/bin and/or /Volume/Snowy/
>>>> usr/share.
>>>>
>>>> I ran xattr on it and
>>>>
>>>> robert-dutoits-macbook-pro:~ astrid$ xattr -l /Volume/Snowy/usr/
>>>> bin/file
>>>> com.apple.ResourceFork:
>>>> com.apple.decmpfs:
>>>>
>>>> These attributes are supposed to be hidden when on 10.6. The
>>>> files have 0 bytes to appear to be compressed but can not be un-
>>>> compresed. I tried file on the clone and it doesn't work. Someone
>>>> else reported du not working so there are some pretty basic files
>>>> damaged. This appears to be the same issue that happens with some
>>>> people when installing Snow Leopard fresh - some execs are
>>>> damaged in this way.
>>>>
>>>> There are some threads about this, specifically bugs in the Snow
>>>> Leopard causing this very thing to happen.
>>>>
>>>> There is a tool someone created for scanning for these damaged
>>>> files at:
>>>>
>>>> http://www.charlessoft.com/find_sl_damaged_files.zip
>>>>
>>>>
>>>> After running rsync 3.0.6 , with patches for hfs+, it reports
>>>>
>>>> /Volume/Snowy/usr/bin/chudRemoteCtrl appears to be damaged
>>>> /Volume/Snowy/usr/bin/chumAddRights appears to be damaged
>>>> /Volume/Snowy/usr/bin/ci appears to be damaged
>>>> /Volume/Snowy/usr/bin/bzip2recover appears to be damaged
>>>> /Volume/Snowy/usr/bin/file appears to be damaged
>>>> /Volume/Snowy/usr/bin/xed appears to be damaged
>>>>
>>>> These same files are fine on the source volume.
>>>>
>>>> Anyone know why this is happening? The hfs+ compressed files are
>>>> otherwise handled perfectly using Mike's patch.
>>>>
>>>> Thanks Rob
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Please use reply-all for most replies to avoid omitting the
>>>> mailing list.
>>>> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
>>>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>>>
>>
>> --
>> Please use reply-all for most replies to avoid omitting the mailing
>> list.
>> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
More information about the rsync
mailing list