Feature Concept: enable iCloud Drive with rsync
kmk at sanitarium.net
Tue Sep 12 17:35:42 UTC 2023
Is this being accessed via a fuse mount? If so it seems like that is
where this kind of feature should be implemented (like a mount option to
decide how to handle such files). Rsync shouldn't need special features
to deal with every kind of file storage.
On 9/12/23 05:22, Brian "bex" Exelbierd via rsync wrote:
> I have also posted this on GitHub but it isn’t clear that was the right
> place: https://github.com/WayneD/rsync/issues/522
> iCloud Drive will evict files that are unused or when additional space
> is needed on the local drive. The evicted files are replace by
> "bookmark" files that allow MacOS to continue to report the files in the
> file system as though they were actually present. The files are
> downloaded again either on request or when needed.
> rsync, like all similar tools I can find, doesn't have any way of
> handling these evicted files. I have been thinking about this and I
> think I know how to make it work.
> The short explanation is that iCloud Drive preserves access to the
> required metadata. Here is an abbreviated output from `mdls` of an
> evicted file:
> % mdls .1998-tax-return.pdf.icloud
> kMDItemContentCreationDate = 2018-08-28 18:32:24 +0000
> kMDItemContentCreationDate_Ranking = 2021-11-05 00:00:00 +0000
> kMDItemContentModificationDate = 2018-08-28 18:32:24 +0000
> kMDItemDateAdded = 2021-11-05 15:42:15 +0000
> kMDItemDisplayName = "1998-tax-return.pdf"
> kMDItemFSContentChangeDate = 2018-08-28 18:32:24 +0000
> kMDItemFSCreationDate = 2018-08-28 18:32:24 +0000
> kMDItemFSSize = 1929932
> kMDItemInterestingDate_Ranking = 2018-08-28 00:00:00 +0000
> kMDItemLogicalSize = 1929932
> kMDItemPhysicalSize = 1929932
> This metadata, I think, is enough to pass the rsync quick check as
> described in the man page. Therefore, I suspect what needs to happen,
> from a code perspective, is that rsync needs to be modified to do the
> following when it finds an evicted file. All evicted files are named
> consistently, `.<filename>.iCloud` so they are easy to spot.
> 1. Perform the system call equivalent of the mdls above to obtain the
> appropriate dates and sizes.
> 2. If the file cannot be skipped because it either has changed or a
> checksum is required, perform the system call equivalent of `brctl
> download <filename>` to get the file downloaded.
> 3. Rsync the file
> 4. If the file was downloaded by rsync, perform the system call
> equivalent of `brctl evict <filename>` to remove the file to leave the
> system in the same state.
> This simplistic algorithm would leave some open issues/caveats:
> - It is likely that the rsync can only be run one-way from iCloud Drive
> to non-iCloud Drive data storage. While it is possible it could be run
> two-way research is needed on whether you’d have to download the old
> file before you replace it.
> - Timeouts may happen. iCloud Drive still gets stuck sometimes and there
> will be non-zero time during downloads that don’t get stuck.
> - There is no guarantee there is ever enough room on disk to hold a
> specific file from iCloud Drive. This is likely resolved by MacOS
> directly, however, this may take excessive time.
> - If you thought running with checksums was slow before, strap in,
> because those downloads are going to add up.
> All that said, I think this would accomplish a high percentage of what
> users are looking for.
> Is this kind of a patch welcome? Is the plan sound? I haven't written C
> since college (in the 90s!) and would be open to advice/mentoring.
> Thank you.
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. Kevin at FutureQuest.net (work)
Orlando, Florida kmk at sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
More information about the rsync