Greyhole VFS module(s)

[Guillaume Boudreau]

On Thu, Nov 25, 2010 at 21:29, Guillaume Boudreau <samba at pommepause.com> wrote:
> On Thu, Nov 25, 2010 at 17:25, Holger Hetterich <hhetter at novell.com> wrote:
>> Am Donnerstag, den 25.11.2010, 17:13 -0500 schrieb Guillaume Boudreau:
>>> On Thu, Nov 25, 2010 at 16:56, Jeremy Allison <jra at samba.org> wrote:
>>> > On Thu, Nov 25, 2010 at 03:18:32PM -0500, Guillaume Boudreau wrote:
>>> >>
>>> >> Logging to a flat file allows Samba to work even if the actual
>>> >> Greyhole daemon is offline, for whatever reason.
>>> >> I didn't want to 'loose' file operations because the daemon was not
>>> >> running, or worse, prevent users from modifying files when that
>>> >> happened.
>>> >> The daemon, when it's running, can then parse those logs at whatever
>>> >> speed it can, and act on them accordingly.
>>> >
>>> > Why go through syslogd then ? Why not have Samba
>>> > write into a sqllite database, or maybe a transactional tdb
>>> > that the daemon can access simultaneously if it's running ?
>>> >
>>> > Right now you're using syslogd as an intermediate daemon
>>> > already to linearize the logs. Might be possible to build
>>> > this into the vfs module.
>>>
>>> Greyhole, when it read the syslog, do some processing and then inserts
>>> the rows in MySQL (or SQLite; configured in greyhole.conf). So yes,
>>> cutting the middle-man and inserting directly in the database from the
>>> VFS module would be very nice.
>>> Sadly, I'm not proficient enough in C to code this in the VFS module.
>>> Thus why my module writes in syslog (the simplest C code possible!),
>>> to be inserted in the DB by the Greyhole daemon.
>>
>> The sqlite3 C interface is relativly simple, and perfect for the module,
>> as it could do anything (create the database... etc) on its own. You
>> would use sqlite3 >= 3.7.0 which allows WAL (Write Ahead Logging), then
>> you can have multiple readers while a writer (the module) is accessing
>> the database.
>
> Greyhole, as it is, reads the syslog, and insert 'tasks' in a MySQL or
> SQLite database (per greyhole.conf).
> Since the greyhole daemon isn't multi-threaded, this parsing happens
> between each task processing.
> So it's something like:
> - parse syslog > insert new tasks in tasks table
> - get first task from tasks table > execute it > delete it from tasks table
> - parse syslog > insert new tasks in tasks table
> - get first task from tasks table > execute it > delete it from tasks table
> and so on.
>
> There are many optimizations path possible here.
>
> Option 1. Have the VFS module insert directly in the MySQL or SQLite
> tasks table.
> This would be the most complicated one to code, but would simplify the
> daemon code the most (no more logs parsing).
> This would also move some of the business logic from the Greyhole
> daemon into the VFS module... Not sure how 'clean' that is...
>
> Question: Can the VFS module access the greyhole.conf file easily, to
> parse it and extract the DB-connection settings from ?
> I don't think this parsing should happen each time the VFS module is
> called, so maybe there's a way to parse it once, and use the result
> when file operations needs to be logged ? I know one can provide
> parameters to the VFS module in the smb.conf, but I doubt forcing the
> user to repeat DB connection settings there, and for every share,
> would make sense...
>
> Option 2. Replace syslogd by a 'raw' sqlite or tdb, but continue to
> have the Greyhole daemon consumes those logs as it consumes syslog
> currently, i.e. read from raw db, and insert in own tasks table.
>
> Question: If writing to sqlite3: I don't think sqlite3 supports
> multiple threads writing in the same database simultaneously (does
> it?) As such, if a client (or multiple clients) are writing files
> simultaneously, one of the log operation would fail, unless the
> sqlite3 C interface takes care of that (does it?)
>
> Maybe all this would be better discussed on IRC ?
> Are you guys on there often during weekdays ? I can be there between
> 9:30am and 15pm (GMT-5) every weekday for the next couple of weeks.
>

I've document how Greyhole works, since it might be relevant to this discussion.

http://code.google.com/p/greyhole/wiki/HowGreyholeWorks

I'm on IRC as gboudreau is anyone wants to discuss further.