[Samba] Am I nuts to attempt a VFS local cache module?

[Jeremy Allison]

On Wed, Apr 10, 2019 at 01:26:59PM -0700, Mark Winslow via samba wrote:
> Sorry if this isn't the right place to ask this question.
> 
> I'm thinking of writing a module that intercepts Samba requests to keep only
> a subset of a file store on a local server with the rest being stored on a
> cloud.  It would be an open source module similar to these products:
> 
> https://aws.amazon.com/storagegateway/
> 
> https://docs.microsoft.com/en-us/azure/storage/files/storage-sync-files-deployment-guide?tabs=azure-portal
> 
> I'm thinking that I would have to do the following.
> 
> 1. Keep my own directory lists, probably in Mongodb
> 2. Intercept File Open,Dir,and Stat requests from Samba
> 3. When a request to open is received for a file that doesn't exist locally,
> create a thread to retrieve the file from the cloud, blocking the request
> until the file is retrieved.
> 4. Have a background process that archives files to the cloud when necessary
> 5. Have a background process that retrieves files from the cloud when the
> local store needs to be rebuilt.
> 
> If Samba does any file access operations outside of interceptable VFS
> requests, then I guess I'm out of luck.

No it doesn't. *Everything* goes through the VFS for
just this reason. You're in luck :-).

> The reasons I'm interested in doing this are:
> 
> 1. Won't have to incessantly worry about disk size on  servers
> 2. Can rebuild servers seemlessly from cloud when local server is
> compromised.
> 
> Is this idea wrong?

No, it's a great idea ! See here:

https://www.snia.org/sites/default/files/SDC15_presentations/smb/JeremyAllison_The_Future_is_Cloudy.pdf

for a possible design (from 2015 no less :-).

> Has it already been done?  Is there a better
> alternative?

Had the design, no time to write it. I'm happy
to help with anything you want to discuss :-).

Loughborough University in the UK wrote this code
but never released it back to the community (I
asked, they ignored me :-( ).

Jeremy.