[clug] SparkleShare vs. Dropbox

[Carlo Hamalainen]

On 15/04/13 12:33, Jim Croft wrote:
> Did somebody mention this? A mate in the US did it :
> http://fak3r.com/2009/09/14/howto-build-your-own-open-source-dropbox-clone/
> Tech stuff here:
> https://github.com/philcryer/lipsync

It's not quite Dropbox, but I knocked up a little program to help me 
manage my digital camera photos and movies (currently around 100Gb):

https://github.com/carlohamalainen/checker

I treat a directory on my laptop as a one-way archive. The program uses 
md5sum to compute checksums.

Compute missing checksums (using md5sum) and check existing checksums:

checker --computemissing <dir>
checker --checkall <dir>

List files that are missing a checksum, or checksums that don't 
correspond to a file:

checker --show-locals-missing-checksum   <local dir>
checker --show-local-orphaned-checksums  <local dir>

I use s3cmd to sync the collection to Amazon S3 (just the files, not the 
checksums, since S3 stores its own checksum info). So then one can find 
files that are stored both locally and in S3, in the local directory but 
not in S3, in S3 but not locally, and finally compare checksums:

checker --show-in-both                    <s3 url> <local path>
checker --show-in-local-but-not-s3        <s3 url> <local path>
checker --show-in-s3-but-not-local        <s3 url> <local path>
checker --check-checksums-local-vs-s3     <s3 url> <local path>

I figure this will cover me against bitrot in metadata and data in my 
master copy, S3, and external USB drives.

-- 
Carlo Hamalainen
http://carlo-hamalainen.net