[Samba] Proper sysvol replication solution...

Achim Gottinger achim at ag-web.biz
Fri Aug 22 15:48:19 MDT 2014

Am 22.08.2014 23:26, schrieb Rowland Penny:
> On 22/08/14 22:12, Achim Gottinger wrote:
>> Am 22.08.2014 22:58, schrieb Rowland Penny:
>>> On 22/08/14 21:46, Achim Gottinger wrote:
>>>> Am 22.08.2014 22:28, schrieb Rowland Penny:
>>>>> On 22/08/14 20:57, Achim Gottinger wrote:
>>>>>> Am 22.08.2014 21:16, schrieb Rowland Penny:
>>>>>>> On 22/08/14 19:51, Achim Gottinger wrote:
>>>>>>>> Am 22.08.2014 20:31, schrieb Rowland Penny:
>>>>>>>>> On 22/08/14 18:40, Achim Gottinger wrote:
>>>>>>>>>> Am 22.08.2014 17:09, schrieb Rowland Penny:
>>>>>>>>>>> On 22/08/14 16:03, L.P.H. van Belle wrote:
>>>>>>>>>>>> when i look on my sysvol and subfolders
>>>>>>>>>>>> and
>>>>>>>>>>>> getfacl /home/samba/sysvol/internal.domain.tld/
>>>>>>>>>>>> the on both servers is the same.
>>>>>>>>>>>> and when i look from within windows both shares are exacly 
>>>>>>>>>>>> the same with rights.
>>>>>>>>>>>> share files and folders.
>>>>>>>>>>>> maybe im lucky ;-)
>>>>>>>>>>> Lucky ?? that's nearly impossible, I bet you win everything 
>>>>>>>>>>> you enter ;-)
>>>>>>>>>>>> no, i dont need these mappings imo since i dont have any 
>>>>>>>>>>>> linux/unix user to the sysvol share.
>>>>>>>>>>>> makes life lot easier..
>>>>>>>>>>> Yes but it doesn't just affect sysvol, it can affect any share.
>>>>>>>>>>> Rowland
>>>>>>>>>>>> Louis
>>>>>>>>>>>>> -----Oorspronkelijk bericht-----
>>>>>>>>>>>>> Van: ryana at reachtechfp.com
>>>>>>>>>>>>> [mailto:samba-bounces at lists.samba.org] Namens Ryan Ashley
>>>>>>>>>>>>> Verzonden: vrijdag 22 augustus 2014 16:47
>>>>>>>>>>>>> Aan: samba at lists.samba.org
>>>>>>>>>>>>> Onderwerp: Re: [Samba] Proper sysvol replication solution...
>>>>>>>>>>>>> Domain users don't enter the equation here though. We're 
>>>>>>>>>>>>> talking the
>>>>>>>>>>>>> "well-known" RID's like Rowland just mentioned. In other 
>>>>>>>>>>>>> words, if my
>>>>>>>>>>>>> "Authenticated Users" is XID 30001 on DC1 but it is 30009 
>>>>>>>>>>>>> on DC2, I
>>>>>>>>>>>>> would not want to clone the UID/GID info from DC1 to DC2, 
>>>>>>>>>>>>> only
>>>>>>>>>>>>> the files
>>>>>>>>>>>>> and directories. That where the "idmap.ldb" file comes 
>>>>>>>>>>>>> into play.
>>>>>>>>>>>>> Apparently it is created with random XID's when Samba is
>>>>>>>>>>>>> configured and
>>>>>>>>>>>>> ran for the first time on each machine. Does your script 
>>>>>>>>>>>>> copy those
>>>>>>>>>>>>> UID/GID numbers over? If so, I was under the impression 
>>>>>>>>>>>>> that such a
>>>>>>>>>>>>> thing was bad.
>>>>>>>>>>>>> I am currently setting up two virtual machines 
>>>>>>>>>>>>> (VirtualBox) to
>>>>>>>>>>>>> play with
>>>>>>>>>>>>> your script. I want to understand how it works and this 
>>>>>>>>>>>>> way I am not
>>>>>>>>>>>>> worried about destroying a client's domain while I am 
>>>>>>>>>>>>> toying around.
>>>>>>>>>>>>> On 08/22/2014 10:34 AM, L.P.H. van Belle wrote:
>>>>>>>>>>>>>>> -----Oorspronkelijk bericht-----
>>>>>>>>>>>>>>> Van: ryana at reachtechfp.com
>>>>>>>>>>>>>>> [mailto:samba-bounces at lists.samba.org] Namens Ryan Ashley
>>>>>>>>>>>>>>> Verzonden: vrijdag 22 augustus 2014 15:30
>>>>>>>>>>>>>>> Aan: samba at lists.samba.org
>>>>>>>>>>>>>>> Onderwerp: Re: [Samba] Proper sysvol replication 
>>>>>>>>>>>>>>> solution...
>>>>>>>>>>>>>>> I know it is, on purpose. Copying the ACLs from one DC to
>>>>>>>>>>>>>>> another is bad
>>>>>>>>>>>>>>> because for some reason each DC may have a different ID for
>>>>>>>>>>>>>>> the built-in
>>>>>>>>>>>>>>> accounts. In fact I believe it was Rowland who was 
>>>>>>>>>>>>>>> saying something
>>>>>>>>>>>>>>> about copying the "idmap.tdb" file over if you sync the 
>>>>>>>>>>>>>>> ACLs.
>>>>>>>>>>>>>>> My method
>>>>>>>>>>>>>>> only copies over the files and directories so no ACLs get
>>>>>>>>>>>>>>> screwy. Maybe
>>>>>>>>>>>>>>> Rowland can reiterate what he said on the other thread.
>>>>>>>>>>>>>> Yes, your right, it can be different if you dont keep de the
>>>>>>>>>>>>> default settings.
>>>>>>>>>>>>>> Which should be ok. I dont have any reason to change the
>>>>>>>>>>>>> acls on sysvol.
>>>>>>>>>>>>>> And i only use my ADDC as DC and not as file server so i
>>>>>>>>>>>>> dont have any idmap problems.
>>>>>>>>>>>>>> And i only use windows users to access these shares.
>>>>>>>>>>>>>> No linux users or what have access to these machines. only
>>>>>>>>>>>>> me as linux user and windows user.
>>>>>>>>>>>>>> and that are 2 different accounts. my linux user is only use
>>>>>>>>>>>>> to maintain the system.
>>>>>>>>>>>>>>> I actually did use rsync for a while and then realized I 
>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>> messing up
>>>>>>>>>>>>>>> after several machines stopped being able to access the 
>>>>>>>>>>>>>>> sysvol
>>>>>>>>>>>>>>> share on
>>>>>>>>>>>>>>> the second DC. When I ran "samba-tool ntacl sysvolcheck" it
>>>>>>>>>>>>>>> would crash.
>>>>>>>>>>>>>>> I ran a reset and all was good, until the next rsync. I
>>>>>>>>>>>>>>> finally figured
>>>>>>>>>>>>>>> it out and removed rsync and the rsync cron job.
>>>>>>>>>>>>>>> So tell me, why are you using rsync and unison? I am 
>>>>>>>>>>>>>>> also trying to
>>>>>>>>>>>>>>> figure out how your script figures out which way to use 
>>>>>>>>>>>>>>> rsync
>>>>>>>>>>>>>>> (DC1->DC2
>>>>>>>>>>>>>>> or DC2->DC1) based on what changes are on each DC. In other
>>>>>>>>>>>>>>> words, would
>>>>>>>>>>>>>>> your script work if a directory and file was created on 
>>>>>>>>>>>>>>> DC1 AND a
>>>>>>>>>>>>>>> directory was removed on DC2? Would the new stuff get 
>>>>>>>>>>>>>>> copied
>>>>>>>>>>>>>>> to DC2 and
>>>>>>>>>>>>>>> would the directory be removed on DC1? Forgive me, but I am
>>>>>>>>>>>>> not a bash
>>>>>>>>>>>>>>> scripting god. I know the basics and can get my work 
>>>>>>>>>>>>>>> done, but
>>>>>>>>>>>>>>> I am not
>>>>>>>>>>>>>>> understanding how your script does two-way stuff via the
>>>>>>>>>>>>> rsync portion.
>>>>>>>>>>>>>> Well normal questions.. so
>>>>>>>>>>>>>> If a file is remove from sysvol on DC1 it removes it also 
>>>>>>>>>>>>>> from DC2.
>>>>>>>>>>>>>> If at the same time a folder or file was remove from DC2
>>>>>>>>>>>>> then it does the same with it on DC1.
>>>>>>>>>>>>>> just test it.. you see.
>>>>>>>>>>>>>> And best tip of the day.. make a backup of your sysvol
>>>>>>>>>>>>> before you start,
>>>>>>>>>>>>>> when you rest things and you forget to remove the unison
>>>>>>>>>>>>> hash files you can delete you sysvol..
>>>>>>>>>>>>>> we need rsync to create the directory structure with
>>>>>>>>>>>>> extended attributes,
>>>>>>>>>>>>>> the (now) unison setup copies only the extened attributes 
>>>>>>>>>>>>>> on files.
>>>>>>>>>>>>>> cron does the sync every 5 min for me and you can also use
>>>>>>>>>>>>> inotify for that.
>>>>>>>>>>>>>> You can search for that one on the list.
>>>>>>>>>>>>>> Hope answere this helps you.
>>>>>>>>>>>>>>> On 08/22/2014 02:21 AM, L.P.H. van Belle wrote:
>>>>>>>>>>>>>>>> ah you didnt see my script...
>>>>>>>>>>>>>>>> https://secure.bazuin.nl/scripts/3-setup-sysvol-bidirectional.sh 
>>>>>>>>>>>>>>>> You way is missing the ACL.. you need rsync with unison.
>>>>>>>>>>>>>>>> its all in my script.
>>>>>>>>>>>>>>>> Greetz,
>>>>>>>>>>>>>>>> Louis
>>>>>>>>>>>>>>>>> -----Oorspronkelijk bericht-----
>>>>>>>>>>>>>>>>> Van: ryana at reachtechfp.com
>>>>>>>>>>>>>>>>> [mailto:samba-bounces at lists.samba.org] Namens Ryan Ashley
>>>>>>>>>>>>>>>>> Verzonden: vrijdag 22 augustus 2014 0:14
>>>>>>>>>>>>>>>>> Aan: samba at lists.samba.org
>>>>>>>>>>>>>>>>> Onderwerp: [Samba] Proper sysvol replication solution...
>>>>>>>>>>>>>>>>> I see the Samba guide suggests using rsync to keep 
>>>>>>>>>>>>>>>>> sysvols in
>>>>>>>>>>>>>>>>> sync, but
>>>>>>>>>>>>>>>>> this poses a problem with ID's and it is only one-way. 
>>>>>>>>>>>>>>>>> I have been
>>>>>>>>>>>>>>>>> hesitant to suggest anything because of the flak I have
>>>>>>>>>>>>>>> been getting,
>>>>>>>>>>>>>>>>> but I do believe I have a much better solution that 
>>>>>>>>>>>>>>>>> transfers
>>>>>>>>>>>>>>>>> files via
>>>>>>>>>>>>>>>>> SSH, is bi-directional (no more only editing group 
>>>>>>>>>>>>>>>>> policy on one
>>>>>>>>>>>>>>>>> server), and does NOT set UID/GID information. In other
>>>>>>>>>>>>> words, it is
>>>>>>>>>>>>>>>>> PERFECT for sysvol replication, and has been working on
>>>>>>>>>>>>>>> several of my
>>>>>>>>>>>>>>>>> domains for around a year and a half without a hitch. The
>>>>>>>>>>>>>>>>> solution I am
>>>>>>>>>>>>>>>>> proposing is to use unison, which also works on 
>>>>>>>>>>>>>>>>> Windows and (I
>>>>>>>>>>>>>>>>> think) Mac.
>>>>>>>>>>>>>>>>> The way I have unison working on my systems is to install
>>>>>>>>>>>>>>>>> unison on all
>>>>>>>>>>>>>>>>> DC's, which is required. You also need an SSH server and
>>>>>>>>>>>>>>> client on all
>>>>>>>>>>>>>>>>> DC's, but I assume most of you do anyway. Once they're
>>>>>>>>>>>>>>>>> installed, it is
>>>>>>>>>>>>>>>>> as simple as the command below. This will synchronizes 
>>>>>>>>>>>>>>>>> changes
>>>>>>>>>>>>>>>>> BOTH WAYS
>>>>>>>>>>>>>>>>> without touching your UID/GID setup. If you're 
>>>>>>>>>>>>>>>>> paranoid, you could
>>>>>>>>>>>>>>>>> always do a sysvolreset when done though.
>>>>>>>>>>>>>>>>> unison -batch "/path/to/sysvol"
>>>>>>>>>>>>>>> "ssh://dc02.domain.lan//path/to/sysvol"
>>>>>>>>>>>>>>>>> If you do this at a command-line, it will prompt you for
>>>>>>>>>>>>>>> your password
>>>>>>>>>>>>>>>>> on the remote machine. This would prevent a cron job, but
>>>>>>>>>>>>> I overcame
>>>>>>>>>>>>>>>>> that as well. You can create an SSH key that does not 
>>>>>>>>>>>>>>>>> require
>>>>>>>>>>>>>>>>> a password
>>>>>>>>>>>>>>>>> for the systems to use. This means you can now create a
>>>>>>>>>>>>> cron job to
>>>>>>>>>>>>>>>>> handle the replication every fifteen minutes or so. You
>>>>>>>>>>>>>>> could also use
>>>>>>>>>>>>>>>>> something like "incrond" to monitor for changes in the 
>>>>>>>>>>>>>>>>> sysvol
>>>>>>>>>>>>>>>>> and launch
>>>>>>>>>>>>>>>>> unison as well, but I don't personally modify the 
>>>>>>>>>>>>>>>>> sysvol often, so
>>>>>>>>>>>>>>>>> replication every fifteen minutes works for me.
>>>>>>>>>>>>>>>>> To create an SSH key to allow password-less 
>>>>>>>>>>>>>>>>> replication via
>>>>>>>>>>>>>>> unison, do
>>>>>>>>>>>>>>>>> the following.
>>>>>>>>>>>>>>>>> ssh-keygen -t dsa
>>>>>>>>>>>>>>>>> When it prompts for a file to save the key in, it 
>>>>>>>>>>>>>>>>> should be
>>>>>>>>>>>>>>> your home
>>>>>>>>>>>>>>>>> directory in a ".ssh" directory. I run as root, so 
>>>>>>>>>>>>>>>>> this is
>>>>>>>>>>>>>>>>> "/root/.ssh/id_dsa" for me. It will then prompt for a
>>>>>>>>>>>>>>> password. Ignore
>>>>>>>>>>>>>>>>> this and just press enter. It will ask you to verify the
>>>>>>>>>>>>>>>>> password. Press
>>>>>>>>>>>>>>>>> enter again. If you enter a password here, it cannot run
>>>>>>>>>>>>>>>>> without user input!
>>>>>>>>>>>>>>>>> Next, you need to copy the key to your other domain
>>>>>>>>>>>>>>>>> controllers. You can
>>>>>>>>>>>>>>>>> do so as follows. Note that my example is run as root.
>>>>>>>>>>>>>>> Substitute your
>>>>>>>>>>>>>>>>> user's path if needed.
>>>>>>>>>>>>>>>>> ssh-copy-id -i /root/.ssh/id_dsa.pub root at dc02.domain.lan
>>>>>>>>>>>>>>>>> Once that is done, login to the domain controller you 
>>>>>>>>>>>>>>>>> copied
>>>>>>>>>>>>>>>>> the key to
>>>>>>>>>>>>>>>>> (in the example, dc02) and check 
>>>>>>>>>>>>>>>>> "/root/.ssh/authorized_keys"
>>>>>>>>>>>>>>>>> to verify
>>>>>>>>>>>>>>>>> that the key was added and nothing unexpected is 
>>>>>>>>>>>>>>>>> there. You
>>>>>>>>>>>>>>>>> can do this
>>>>>>>>>>>>>>>>> with "cat /root/.ssh/authorized_keys". You should see 
>>>>>>>>>>>>>>>>> a key on
>>>>>>>>>>>>>>>>> a single
>>>>>>>>>>>>>>>>> line followed by the hostname of your primary domain
>>>>>>>>>>>>>>> controller. If it
>>>>>>>>>>>>>>>>> is there, they may now connect via SSH without a 
>>>>>>>>>>>>>>>>> password!
>>>>>>>>>>>>>>>>> You may now copy the key to any other domain 
>>>>>>>>>>>>>>>>> controllers in
>>>>>>>>>>>>>>>>> your domain
>>>>>>>>>>>>>>>>> so they trust the primary DC as well. After that, all 
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> is left is
>>>>>>>>>>>>>>>>> the synchronization. I urge you to run the first 
>>>>>>>>>>>>>>>>> synchronization
>>>>>>>>>>>>>>>>> manually, like below.
>>>>>>>>>>>>>>>>> unison "/path/to/sysvol" 
>>>>>>>>>>>>>>>>> "ssh://dc02.domain.lan//path/to/sysvol"
>>>>>>>>>>>>>>>>> Make sure everything looks good, synchronize it, then 
>>>>>>>>>>>>>>>>> repeat
>>>>>>>>>>>>>>>>> for each DC
>>>>>>>>>>>>>>>>> on your domain. Once done, you can create cron jobs to 
>>>>>>>>>>>>>>>>> sync
>>>>>>>>>>>>>>>>> each server,
>>>>>>>>>>>>>>>>> or use a script like mine below. This script is on my
>>>>>>>>>>>>> primary DC. I
>>>>>>>>>>>>>>>>> actually only have two DC's, but I added more as an 
>>>>>>>>>>>>>>>>> example here.
>>>>>>>>>>>>>>>>> #!/bin/bash
>>>>>>>>>>>>>>>>> SERVERLIST="dc02.domain.lan dc03.domain.lan 
>>>>>>>>>>>>>>>>> dc04.domain.lan"
>>>>>>>>>>>>>>>>> SVPATH="/path/to/sysvol"
>>>>>>>>>>>>>>>>> # Synchronize all of the domain controllers
>>>>>>>>>>>>>>>>> for sLoop in ${SERVERLIST}
>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>      unison -batch "${SVPATH}" "ssh://${sLoop}/${SVPATH}"
>>>>>>>>>>>>>>>>> done
>>>>>>>>>>>>>>>>> exit 0
>>>>>>>>>>>>>>>>> Now set that script to run in a cron job and you're 
>>>>>>>>>>>>>>>>> golden.
>>>>>>>>>>>>>>> You could
>>>>>>>>>>>>>>>>> also setup "incrond" on all of your DC's and have it call
>>>>>>>>>>>>>>>>> unison to sync
>>>>>>>>>>>>>>>>> the other DC's whenever a write happens in your sysvol,
>>>>>>>>>>>>> but I do not
>>>>>>>>>>>>>>>>> need such a thing and have not personally tried it,
>>>>>>>>>>>>> though I have a
>>>>>>>>>>>>>>>>> fellow IT lead who has and likes it. My crontab job 
>>>>>>>>>>>>>>>>> entry is
>>>>>>>>>>>>>>>>> listed below.
>>>>>>>>>>>>>>>>> 15 * * * * /root/sysvolsync.sh &> /dev/null
>>>>>>>>>>>>>>>>> I hope this helps somebody and if you see something 
>>>>>>>>>>>>>>>>> wrong,
>>>>>>>>>>>>>>>>> feel free to
>>>>>>>>>>>>>>>>> let me know.
>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>> To unsubscribe from this list go to the following URL 
>>>>>>>>>>>>>>>>> and read the
>>>>>>>>>>>>>>>>> instructions: 
>>>>>>>>>>>>>>>>> https://lists.samba.org/mailman/options/samba
>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>> To unsubscribe from this list go to the following URL 
>>>>>>>>>>>>>>> and read the
>>>>>>>>>>>>>>> instructions: https://lists.samba.org/mailman/options/samba
>>>>>>>>>>>>> -- 
>>>>>>>>>>>>> To unsubscribe from this list go to the following URL and 
>>>>>>>>>>>>> read the
>>>>>>>>>>>>> instructions: https://lists.samba.org/mailman/options/samba
>>>>>>>>>> Rowland do you remeber Michael Adam's patch here from april? 
>>>>>>>>>> Seems it did not yet make it into an stable release.
>>>>>>>>> Yes, I vaguely remember it, but it did not go far enough (no 
>>>>>>>>> disrespect meant Michael, I couldn't have got that far),  I 
>>>>>>>>> 'think' it only dealt with 'BUILTIN' objects, but they are a 
>>>>>>>>> small part of the well known RID's .
>>>>>>>>> Rowland
>>>>>>>> Good point,
>>>>>>>> An nss_idmap module or extention to nss_winbind which resolves 
>>>>>>>> xidNumbers to objectSID's might do the trick. It is not 
>>>>>>>> neccesarry to translate the SID into an name to solve the 
>>>>>>>> mapping problem.
>>>>>>>> achim~
>>>>>>> You mean like the reverse of what ever happens to create the 
>>>>>>> entries in idmap.ldb ?
>>>>>>> Rowland
>>>>>> Yes,
>>>>>> For your example in the other mail "getent group" would return
>>>>>> S-1-5-32-544:*:3000000
>>>>>> on the dc and
>>>>>> S-1-5-32-544:*:2000
>>>>> That would be good, but it would be better if was:
>>>>> administrators:*:3000000
>>>>> on the dc and
>>>>> administrators:*:2000
>>>>> Rowland
>>>> Odd thing is the sid to name mapping is already there 
>>>> (librpc/idl/security.idl) for well know sid's and rid's. Makes it 
>>>> even more strange that only BUILDIN groups had been implemented by 
>>>> the patch.
>>>> @steve you would force the users to reserve an predefined range of 
>>>> numbers in their usermanagement for windows standard groups und users.
>>> And your problem with this is ???
>>> If people could get their heads around the fact that they do not 
>>> need any local unix users and just have domain users instead, this 
>>> would not really be a problem.
>>> Rowland
>> Not my problem but one. Well-know-rid's for two different domains. 
>> The sid's differ in the domain part but would get the same fixed gid 
>> because they have an identical well-known rid part.
> I cannot see that this is a problem, if it is a problem then microsoft 
> has it as well. We are talking about the well know RID's here and 
> apart from the few that use the domain SID, every Domain uses the same 
> SID i.e. administrators is always  'S-1-5-32-544'
> Rowland
Well you talked about well known rid's earlier. The well known sid's are 
the same on all domains, rid's are always prefixed with the domain sid. 
To prove myself wrong, these do resolve well and cause no problems.
As for the sid's (builtin/security) the only problem on the linux side 
is that they do not resolve at all to an gid. It is not necessary that 
they resolve to the same gid on every machine, they just must resolve to 
an number. If an group resolves to different gid's at two systems rsync 
will take care of the number replacement if not the gid will be the same.


More information about the samba mailing list