[Samba] AD Backup Best Practice

[Tim Beale]

Hi,

Thanks for the feedback. I appreciate the whole backup/restore process
is more complicated than it needs to be (especially when you're trying
to restore in a panic).

I've explained below the technical reasons why the backup/restore has
these current limitations. The short answer is:
- We spent a huge amount of time implementing the current backup/restore
tool. Unfortunately, we just ran out of time to make the backup nicer to
use. We figured further improvements could be made in future, as needed.
- We're primarily samba developers, rather than samba users, so there's
some guesses we made as to how it'd be used/what's acceptable.

I think updating the restore tool to avoid these annoyances is not that
hard. Part of the hard part is just testing it manually to make sure we
get things right. On that note, would any samba users out there be
willing to volunteer to help test this? i.e. clone a development git
branch, build the source, and try out the backup/restore tool?

>> you *must not* specify a DC that has previously existed

The restore process currently works like this:
1. The backed up database (which still contains all the old DCs) is
restored to disk.
2. The new DC is added to the database.
3. All the previous DCs are removed from the database.

Note that:
- #3 has to happen to ensure the restored domain doesn't try to
communicate with the old/failed domain (just in case you forgot to
switch off a DC).
- The DC added in #2 will always be a "new" DC (even if it has the same
hostname, it will have different a invocation-ID, etc).
- You can't have zero DCs in the database, i.e. the new DC needs to be
added before all the old ones are removed.

Currently you can't add a new DC with the same name because it already
exists in the database. Yes, we could avoid this problem by checking the
database, and removing the old DC first. It was just more work than we
had time to implement.

>> restore the domain database into a different targetdir,
>> and then use the '-s' option when running samba

A domain join allows you to overwrite the existing samba database, which
is the behaviour you're asking for here. We wanted to avoid that in the
restore case because:
- If the restore didn't work successfully, you've just obliterated your
database.
- We want to be really careful that remnants from the old database
didn't get left behind, so wanted the restore to always be done into an
empty directory.

The simple solution would be to rename /usr/local/samba/ to
/usr/local/samba-old/ first, then run the restore. The problem here is
the samba-tool program that you'll use to do the restore is also in the
directory you just renamed.

Probably a better solution would be for the restore tool to move the
sub-directories it's going to restore first. i.e. automatically move
/usr/local/samba/private to /usr/local/samba/private-old.

>> Are you saying that restoring to a "new DC" is as simple as changing the
>> hostname of "DC1" to "DC1_1"? DNS, GPO, smb.conf would all
>> automatically refer to the new hostname after the restore?

So, a restore should be similar in functionality to a 'domain join',
i.e. the hostname specified in the command should get setup in smb.conf
and DNS.

Most of the sysvol/GPO backup/restore was implemented by another
developer, so I don't know that much about it. Basically all the sysvol
files and their NTACLs should be preserved as part of the backup
process. So you shouldn't need to do anything to the restored DC. As you
rejoin the DCs after the restore, you'd need to rsync the sysvol
directory (same as you would if you joined a new DC).

>> Let's assume the DC is in a filesystem that allows snapshots, do I
>> assume correctly that stopping samba would not be required in that case?

If you're foregoing the backup tools and just taking backing up the raw
files on disk, then you'd need to stop samba. Otherwise, I think you're
at risk of corruption from not holding appropriate DB locks.

The offline backup (in 4.10) is essentially just taking a snapshot of
the files on disk. However, it does it in a way that's safe WRT DB
locks. Samba doesn't need to be stopped in this case.

Also note that part of the point of the online backup is to try to
protect you from latent/benign database corruptions that might exist on
disk. i.e. the raw database on disk has a corruption but samba is still
running OK, but then that corrupt object gets modified again and
everything dies. If your backup has the same latent corruption, then
restoring it won't be much help long-term.

Regards,
Tim

On 11/02/19 5:40 AM, Rowland Penny via samba wrote:
> I am cc'ing Tim Beale on this, he wrote the wikipage and, I think, most
> of the backup & restore code.
>
> On Sun, 10 Feb 2019 17:07:55 +0100
> Viktor Trojanovic <viktor at troja.ch> wrote:
>
>> See comments inline.
>>
>> On Sun, 10 Feb 2019 at 16:33, Rowland Penny via samba
>> <samba at lists.samba.org> wrote:
>>
>>> On Sun, 10 Feb 2019 14:13:27 +0100
>>> Viktor Trojanovic via samba <samba at lists.samba.org> wrote:
>>>
>>>> I'm currently reviewing my own backup strategy for Samba and I
>>>> realize it is not in line with best practices provided in the
>>>> Wiki.
>>>> ( https://wiki.samba.org/index.php/Back_up_and_Restoring_a_Samba_AD_DC)
>>>> Said best practices, however, seem a bit like a nightmare to me.
>>>>
>>>> Assuming the AD is gone and you want to restore just one DC, and
>>>> you want things to look just as they did before the crash, the
>>>> process according to the Wiki looks as follows:
>>>>
>>>> 1. Install a Samba DC on a new (!) temporary host and provision
>>>> the domain, just like you would when doing a new install from
>>>> scratch. That task alone is tremendous.
>>>> 2. Stop Samba and restore the AD from backup to this domain not
>>>> (!) into the default Samba folder, advise Samba accordingly when
>>>> starting it. 3. On the original host, set up a Samba DC and join
>>>> the domain. 4. If GPO or scripts exist on sysvol, manually set up
>>>> sysvol replication to get them to the original DC.
>>>> 5. Remove the temporary host.
>>>>
>>>> Just... wow. :)
>>> Tend to agree with you, the wiki page asks this question 'So which
>>> backup should I use?' It then goes on to enumerate 5 different
>>> reasons why you would need a backup and seems to totally miss the
>>> point. Your domain has gone down and it is headless chicken time ;-)
>>> All you would want to do is to get your domain back up again as
>>> quickly as possible.
>>>
>> Yes. So I'm really glad I haven't encountered that page in a moment
>> of true need! :-)
>>
>>
>>> I think you would only do '1' if you wanted to rename the domain.
>>>
>>> Not sure where you got restoring into a different folder from, I
>>> thought the restore put everything back to where it came from.
>>>
>>>
>> I got both of this from the section "Restoring the backup-file" in the
>> wiki. It says that if you're going to use the command "samba-tool
>> domain backup restore", you *must not* specify a DC that has
>> previously existed. Further, with regards to the files itself, it
>> says that the Samba team recommends "that you restore the domain
>> database into a different targetdir, and then use the '-s' option
>> when running samba".
> OK, re-read the wiki-page and I do not understand why you have to do
> this ?
>
> Tim, Why do you have to do this ? I would image that, unless the DC
> is kaput, Samba will be the only thing that needs fixing. Most
> sysadmins would just clear out the Samba dirs and then carry out the
> restore. I can understand renaming the DC, but why does it have to be
> a new install ?
>   
>>
>>> You shouldn't have to do '4', the backup contains a copy of sysvol
>>> and smb.conf, so you should be able to restore to the DC it came
>>> from, it would just have to be the only DC and all DC's would have
>>> to be stopped, it would probably be better to rename the old DC
>>> before carrying out the restore.
>>>
>>>
>> As mentioned, at least to my understanding of the the wiki a restore
>> of the original DC is not possible using the backup made from it. Are
>> you saying that restoring to a "new DC" is as simple as changing the
>> hostname of "DC1" to "DC1_1"? DNS, GPO, smb.conf would all
>> automatically refer to the new hostname after the restore?
> That is my understanding and if not, Tim should be able to explain why
> not ;-)
>
>>
>>>> Isn't there a simpler way of doing this? Namely, if all the
>>>> restore operations are done offline anyway, why is it frowned
>>>> upon to simply do everything on the original DC, i.e. forgo the
>>>> temporary host, overwrite the configuration files (/etc/samba)
>>>> and the local Samba folder (e.g. /var/lib/samba) with what's in
>>>> the backup and be done with it? What's the difference between
>>>> doing this and just restoring the whole machine running the DC
>>>> bit for bit (dd backup and restore)?
>>> If you are talking about stopping the DC and copying it (somehow),
>>> then this should work, but you would have to be aware that you
>>> would have to stop your DC regularly and that your backup would
>>> only be valid for the time you took it, anything between that
>>> backup and the next would be lost.
>>>
>> Let's assume the DC is in a filesystem that allows snapshots, do I
>> assume correctly that stopping samba would not be required in that
>> case? 
> The problem is that a Samba AD DC is constantly in flux, that is, it
> changes constantly, if your 'snapshot' can guarantee it is correct,
> then I see no problem, but you would only really know when you tried
> to restore it. 
>
>> With regards to information between 2 backups being lost, how
>> is that different with other backup strategies, for example using
>> samba-tool online backup?
> That is the problem with any AD DC backup method, the backups can
> quickly become out of date.
>
> Rowland
>
>
>