DFS-R client implementation status update

[Samuel Cabrero]

Hi,

this is an update about the status of the DFS-R client implementation I
am working on. I would like to receive some comments to validate the
architecture and ensure I am in the right track before continue working on this, in particular about spawning a new smbd process to install the updates (see below). 

This is a summary of what is done so far:

Establish connections and sessions to partners
------------------------------------------------
- When the client starts for the first time, it initializes its own
DFSR-LocalSettings.

- It automatically joins to the sysvol replication group and enables
sysvol folder replication if 'dfsrsrv:sysvol_join = yes'.

- It scans the replication group subscriptions, establishes DFS-R
sessions for all folders in the group and creates connections to all
partners following the configured group topology (except for sysvol
group, which follows the NTDS settings topology).

- It loads the stored known version vectors for each folder on startup
to resume the replication.

- The subscriptions and topology is refreshed periodically and the
connections or sessions restarted if closed by previous errors.


Retrieving the updates
------------------------
- The updates are pulled when the server notifies a new version vector
is available following the request updates state machine. The updates
are queued in a version vector queue. 

- When all updates pertaining to a version vector are retrieved, the
version vector is queued for processing. Each version vector has its
own updates queue.


Processing updates
------------------
- The process loop picks an install candidate. The updates have to be
installed in ancestral order, parents before children to avoid
conflicts. This is done checking in the database if the parent update
is installed. Also checks for superseded updates.

- The selected update is downloaded and staged into a file. If the
update does not fit in the  buffer provided to
InitializeFileTransferAsync, it is fully downloaded using
RawGetFileData.

- When the update is staged, an IRPC message is sent to the meet module to install the staged update and wait for the result. If the
update can not be installed (the file is opened for example) it is
reinserted into the queue to try again later.

- When all updates in the version vector queue have been installed, the
stored known version vectors are updated.


Installing updates
------------------
- Updates are installed by a smbd process (which I call the meet module).
- It creates a connection_struct to install the updates through the VFS layer.
- The staged update is uncompressed into a new file in the Installing directory.
- The uncompressed update is unmarshalled. Currently only metadata,
security and flat data streams are handled. The MS-BKUP header of the
flat data stream is just skipped by now.
- If the update is properly installed it is stored in the database.
- The installation result is reported to the dfsr service


TODO
----
- Management tools. It is imposible to set the msDFSR-RootPath using
the DFS console in windows, ADSI edit has to be used.
- Implement a credit system to throttle update retrieval and install.
- Use the hash field in the updates to determine if it is necessary to
download the full update.
- Handle additional marshaled streams (reparse, compression)
- Handle the flat data stream as an MS-BKUP blob
- Implement the Slow-Sync protocol.
- Test against a server with real usage pattern to detect problems
installing updates. I have generated tens of thousands of updates using
the python test, but it always perform the same set of basic file
operations.


The code can be found at:

https://github.com/kernevil/samba/tree/dfs-r

Cheers,

Samuel