samba-tool domain backup with lmdb on btrfs fails

[Andreas Schneider]

On Tuesday, 14 February 2023 20:26:52 CET Andrew Bartlett wrote:
> On Tue, 2023-02-14 at 18:12 +0100, Andreas Schneider via samba-
> 
> technical wrote:
> > On Tuesday, 14 February 2023 16:39:59 CET Andreas Schneider via
> > samba-
> > 
> > technical wrote:
> > > Hi,
> > > 
> > > I've discovered on my Fedora 37 system that all tests related to
> > > domain
> > > backup fail. The strange thing is that I can't reliably reproduce
> > > it, I see
> > > reliably two different errors:
> > > 
> > > lib/ldb/ldb_key_value/ldb_kv_search.c:162: Invalid data for index
> > > DN=@INDEXLIST
> > > 
> > > or
> > > 
> > > lib/ldb/ldb_mdb/ldb_mdb.c:419: MDB corrupted for key [ DN=@OPTIONS]
> > > 
> > > ldb: Improve debug outout
> > > https://gitlab.com/samba-team/samba/-/merge_requests/2936
> > > 
> > > 
> > > It the tests worked just fine on my other system with openSUSE
> > > Tumbleweed.
> > > So I've tried to track it down. My home directory on openSUSE
> > > Tumbleweed is
> > > ext4 and on Fedora 37 my home is btrfs.
> > > 
> > > So I created a btrfs filesystem on Tumbleweed and could reproduce
> > > the issue.
> > > The next thing I did is replace the compression algorithm for the
> > > domain
> > > backup tarball to gzip and it also failed.
> > > 
> > > Now the question is if lmdb or btrfs needs a fix and how to create
> > > a minimal
> > > reproducer. As I never worked on this codepath, pointers and ideas
> > > would be
> > > really helpful!
> > 
> > I've did a quick hack and created the backup tarball (offlinebackupdc
> > env) on
> > ext4 and restored it on btrfs.
> > 
> > @@ -3426,11 +3426,12 @@ sub setup_offlinebackupdc
> > 
> >                                                     $dnsupdate_option
> > 
> > s);
> > 
> >         # create an offline backup of the 'backupfromdc' target
> > 
> > -       my $backupdir = File::Temp->newdir();
> > -       my $cmd = "offline --configfile $dcvars->{SERVERCONFFILE}";
> > -       my $backup_file = $self->create_backup($env, $dcvars,
> > -                                              $backupdir, $cmd);
> > +       my $backupdir = '/tmp/samba';
> > +       #my $cmd = "offline --configfile $dcvars->{SERVERCONFFILE}";
> > +       # my $backup_file = $self->create_backup($env, $dcvars,
> > +       #                                      $backupdir, $cmd);
> > 
> > +       my $backup_file = '/tmp/samba/samba-
> > backup-2023-02-14T16-14-17.289039.tar.bz2';
> > 
> >         unless($backup_file) {
> >         
> >                 return undef;
> >         
> >         }
> > 
> > Backup file successfully restored to /var/tmp/samba-test/samba/st/
> > offlinebackupdc
> > 
> > So the corruption could also happen when creating the tarball. I will
> > try to
> > checksum the file before backup put it into a file and store it into
> > the
> > tarball, then I can unpack it and check the sums.
> 
> Yikes, this certainly is a concern.  Having each file (or a rolling
> checksum over all the files as they are added) checksummed or such
> would be a useful feature.

I created a checkum before they added to the tarball now. After unpacking the 
checksums still match.

Now I wonder if mdb_copy might be the problem.

> All the best chasing this down.

Thanks


	Andreas

-- 
Andreas Schneider                      asn at samba.org
Samba Team                             www.samba.org
GPG-ID:     8DFF53E18F2ABC8D8F3C92237EE0FC4DCC014E3D