Feature Concept: enable iCloud Drive with rsync

Brian "bex" Exelbierd bex at bexelbie.com
Tue Sep 12 09:22:00 UTC 2023


Hi,

I have also posted this on GitHub but it isn’t clear that was the right place: https://github.com/WayneD/rsync/issues/522

iCloud Drive will evict files that are unused or when additional space is needed on the local drive. The evicted files are replace by "bookmark" files that allow MacOS to continue to report the files in the file system as though they were actually present. The files are downloaded again either on request or when needed.

rsync, like all similar tools I can find, doesn't have any way of handling these evicted files. I have been thinking about this and I think I know how to make it work.

The short explanation is that iCloud Drive preserves access to the required metadata. Here is an abbreviated output from `mdls` of an evicted file:

```
% mdls .1998-tax-return.pdf.icloud
kMDItemContentCreationDate = 2018-08-28 18:32:24 +0000
kMDItemContentCreationDate_Ranking = 2021-11-05 00:00:00 +0000
kMDItemContentModificationDate = 2018-08-28 18:32:24 +0000
kMDItemDateAdded = 2021-11-05 15:42:15 +0000
kMDItemDisplayName = "1998-tax-return.pdf"
kMDItemFSContentChangeDate = 2018-08-28 18:32:24 +0000
kMDItemFSCreationDate = 2018-08-28 18:32:24 +0000
kMDItemFSSize = 1929932
kMDItemInterestingDate_Ranking = 2018-08-28 00:00:00 +0000
kMDItemLogicalSize = 1929932
kMDItemPhysicalSize = 1929932
```

This metadata, I think, is enough to pass the rsync quick check as described in the man page. Therefore, I suspect what needs to happen, from a code perspective, is that rsync needs to be modified to do the following when it finds an evicted file. All evicted files are named consistently, `.<filename>.iCloud` so they are easy to spot.

1. Perform the system call equivalent of the mdls above to obtain the appropriate dates and sizes.
2. If the file cannot be skipped because it either has changed or a checksum is required, perform the system call equivalent of `brctl download <filename>` to get the file downloaded.
3. Rsync the file
4. If the file was downloaded by rsync, perform the system call equivalent of `brctl evict <filename>` to remove the file to leave the system in the same state.

This simplistic algorithm would leave some open issues/caveats:
- It is likely that the rsync can only be run one-way from iCloud Drive to non-iCloud Drive data storage. While it is possible it could be run two-way research is needed on whether you’d have to download the old file before you replace it.
- Timeouts may happen. iCloud Drive still gets stuck sometimes and there will be non-zero time during downloads that don’t get stuck.
- There is no guarantee there is ever enough room on disk to hold a specific file from iCloud Drive. This is likely resolved by MacOS directly, however, this may take excessive time.
- If you thought running with checksums was slow before, strap in, because those downloads are going to add up.

All that said, I think this would accomplish a high percentage of what users are looking for.

Is this kind of a patch welcome? Is the plan sound? I haven't written C since college (in the 90s!) and would be open to advice/mentoring.

Thank you.

regards,

bex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20230912/5dc50a2c/attachment.htm>


More information about the rsync mailing list