rsync-ing IMAP mbox-format mailboxes to NetApp

[Chris Wakelin]

Hi all,

Sorry for the longish post!

I've been looking at using rsync to mirror our mailstore (BSD/mbox
format, i.e. flat files consisting of concatenated plaintext messages)
to a NetApp Nearstore (basically a filer with SATA rather than FC disk)
mounted over NFS.

I want to do this in such a way that the NetApp automatic snapshots are
kept as small as possible, so hopefully several versions of the
mailstore can be stored "live" (the NetApp WAFL filesystem "stores" disk
blocks as they are changed).

After some experimenting, I discovered that I needed "--no-whole-file"
as well as "--inplace" otherwise rsync copies the entire file by default
for "local" copying, including over NFS (and so the whole previous
version ends up in the snapshot).

This, of course, is quite CPU-hungry, but is probably worth it!

I've done some playing with "--block-size" for a 80MB mailbox and found
that ~16384 uses least CPU at the cost of extra blocks copied (I'm using
multiples of 4096 as that's the WAFL disk-block size, I think). However
4096 is closer to the "average" message size.

I'd like to use the "--append" option instead, as mailboxes frequently
have messages appended and (our) users delete/move messages
comparatively rarely.

However, our IMAP software (Dovecot, but UW-IMAP does this too) stores
important, *changing* information (UIDVALIDITY and UIDNEXT) in a header
in the *first* mail message in the mailbox, so the first block (only!)
fails the checksum test.

I guess this means some sort of patch to add an option to do a "nearly
append except the first 'n' blocks", but I'm wondering (before I
reinvent a possibly square wheel) whether anybody else has tried this?

Best Wishes,
Chris

-- 
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
Christopher Wakelin,                           c.d.wakelin at reading.ac.uk
IT Services Centre, The University of Reading,  Tel: +44 (0)118 378 8439
Whiteknights, Reading, RG6 2AF, UK              Fax: +44 (0)118 975 3094