rsync and hfs+ compressed files damaged
Mike Bombich
mike at bombich.com
Thu Jul 1 16:13:31 MDT 2010
Well, it looks like the hfs_compression patch that I posted was the problem, and I've solved it since then. I can reproduce the issue with the patch, but not with my subsequent changes (my changes are posted in their entirety, vs. breaking out the hfs_compression-specific changes). Here is an updated patch, and I have verified that it resolves this issue.
As an aside, the patch contains a couple bug fixes to the xattr code as well. I'll break those out and submit them separately in the near future.
Mike
On Jul 1, 2010, at 6:16 AM, Robert DuToit wrote:
> Hi Mike and All,
>
> I have been doing some research on the damaged HFS compressed files and found some interesting clues. Looking at mike's patch so far I can't find any reason for the damage (I have to learn more C language ) but what still seems odd is that the damage only happens when copying the whole /usr directory or /usr/bin /usr/share. Individually copying one of the original files does not create the damaged file.
>
> Pertaining to the use of Mike's patch on OSX 10.6.4 copying /usr/bin and resulting damaged files
>
> In /usr/bin/ I found that the originals of the damaged files are different from the other "normal" exec files. In this case I am looking at the "find" exec.
>
> If I copy "find" from /usr/bin/ to the desktop via finder copy, it is changed to resemble the "normal" execs. And indeed if I replace the original with this copy in /usr/bin and copy /usr/bin with patched rsync, the copied "find" is no longer damaged!
>
> So it appears that when the patched rsync encounters one of these "odd" files ( and only when copying the whole /usr/bin/ directory containing them ) it causes the problem.
>
> Here are results using hfsdebug (sorry about the amount of reading!)
>
>
> ####################
> original "file" exec - works ok but different from other execs in /usr/bin. ( It shows the com.apple.decmpfs flag and data is stored in Resource Fork)
>
> flags = 0000000000000110
> . File has extended attributes.
>
> # Data Fork
> logicalSize = 0 bytes
> # Resource Fork
> logicalSize = 101755 bytes
> totalBlocks = 25
>
> # Attributes
> <Attributes B-Tree node = 6754 (sector 0x294b0)>
> # Attribute Key
> keyLength = 46
> pad = 0
> fileID = 3904426
> startBlock = 0
> attrNameLen = 17
> attrName = com.apple.decmpfs
> # Inline Data
> recordType = 0x10
> reserved[0] = 0
> reserved[1] = 0
> attrSize = 16 bytes
> attrData = 66 70 6d 63 04 00 00 00 b0 a0 03 00 00 00 00 00
> f p m c
>
> compression magic = cmpf
> compression type = 4 (resource fork has compressed data)
> uncompressed size = 237744 bytes
>
> ####################
> A normal exec "find" in /usr/bin/ ( data stored in # Data Fork so it is showing as uncompressed in finder)
>
> flags = 0000000000000010
>
> # BSD Info
> ownerID = 0 (root)
> groupID = 0 (wheel)
> = 0
> # Data Fork
> logicalSize = 147680 bytes
> totalBlocks = 37
> clumpSize = 2445
>
> # Resource Fork
> logicalSize = 0 bytes
>
>
> ####################
> the original "file" exec now copied via patched rsync within the whole /usr/bin directory (Note: no data in Data Fork or Resource Fork)
>
> First note that the xattrs show up ( not supposed to when seen from 10.6)
>
> robert-dutoits-macbook-pro:~ astrid$ xattr -l /Users/astrid/Desktop/copied_bin_test/bin/file
> com.apple.ResourceFork:
> com.apple.decmpfs:
>
> and the hfsdebug report:
>
> flags = 0000000000000110
> . File has extended attributes.
>
> # Data Fork
> logicalSize = 0 bytes
> # Resource Fork
> logicalSize = 1 bytes
> totalBlocks = 1
>
> # Attributes
> <Attributes B-Tree node = 5258 (sector 0x23730)>
> # Attribute Key
> keyLength = 46
> startBlock = 0
> attrNameLen = 17
> attrName = com.apple.decmpfs
> # Inline Data
> recordType = 0x10
> reserved[0] = 0
> reserved[1] = 0
> attrSize = 1 bytes
> attrData = 04
>
> compression magic = .
> compression type = 44000000 (?)
> uncompressed size = 1224979098644812920 bytes
>
>
> ####################
>
> I can't put this all together yet but if anyone can see what is going on I would much appreciate it. What throws me though it that the damage only happens when copying the whole /usr/bin/ directory…
>
> Again - this is rsync 3.0.6 patched with fileflags, crtimes and Mike's hfs patch.
>
> Thanks, Rob
>
>
>
>
>
> On Jun 25, 2010, at 4:03 PM, Mike Bombich wrote:
>
>> Can you share the steps required to reproduce this? I've seen two reports of this but have not been able to reproduce it myself, and the users that reported it also could not reproduce it.
>>
>> Mike
>>
>> On Jun 25, 2010, at 5:16 AM, Robert DuToit wrote:
>>
>>> Hi All,
>>>
>>> I have been using rysnc 3.0.6 with Mike's rsync_3.0.6-hfs-compression_20091027.diff patch (as well as the standard osx patches) with good results. I discovered that clones done on 10.6 using this rsync cause some executable files ( the ones that are mostly compressed this way on osx ) to be damaged.
>>>
>>> I did a test twice and indeed a handful of them are damaged. They all appear to be in /Volume/Snowy/usr/bin and/or /Volume/Snowy/usr/share.
>>>
>>> I ran xattr on it and
>>>
>>> robert-dutoits-macbook-pro:~ astrid$ xattr -l /Volume/Snowy/usr/bin/file
>>> com.apple.ResourceFork:
>>> com.apple.decmpfs:
>>>
>>> These attributes are supposed to be hidden when on 10.6. The files have 0 bytes to appear to be compressed but can not be un-compresed. I tried file on the clone and it doesn't work. Someone else reported du not working so there are some pretty basic files damaged. This appears to be the same issue that happens with some people when installing Snow Leopard fresh - some execs are damaged in this way.
>>>
>>> There are some threads about this, specifically bugs in the Snow Leopard causing this very thing to happen.
>>>
>>> There is a tool someone created for scanning for these damaged files at:
>>>
>>> http://www.charlessoft.com/find_sl_damaged_files.zip
>>>
>>>
>>> After running rsync 3.0.6 , with patches for hfs+, it reports
>>>
>>> /Volume/Snowy/usr/bin/chudRemoteCtrl appears to be damaged
>>> /Volume/Snowy/usr/bin/chumAddRights appears to be damaged
>>> /Volume/Snowy/usr/bin/ci appears to be damaged
>>> /Volume/Snowy/usr/bin/bzip2recover appears to be damaged
>>> /Volume/Snowy/usr/bin/file appears to be damaged
>>> /Volume/Snowy/usr/bin/xed appears to be damaged
>>>
>>> These same files are fine on the source volume.
>>>
>>> Anyone know why this is happening? The hfs+ compressed files are otherwise handled perfectly using Mike's patch.
>>>
>>> Thanks Rob
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Please use reply-all for most replies to avoid omitting the mailing list.
>>> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
>>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20100701/65603c53/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rsync_3.0.7-hfs_compression_20100701.diff
Type: application/octet-stream
Size: 58826 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/rsync/attachments/20100701/65603c53/attachment.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20100701/65603c53/attachment-0001.html>
More information about the rsync
mailing list