[Samba] Proper sysvol replication solution...

[steve]

On Fri, 2014-08-22 at 20:45 +0100, Rowland Penny wrote:
> On 22/08/14 20:17, steve wrote:
> > On Fri, 2014-08-22 at 20:51 +0200, Achim Gottinger wrote:
> >> Am 22.08.2014 20:31, schrieb Rowland Penny:
> >>> On 22/08/14 18:40, Achim Gottinger wrote:
> >>>> Am 22.08.2014 17:09, schrieb Rowland Penny:
> >>>>> On 22/08/14 16:03, L.P.H. van Belle wrote:
> >>>>>> when i look on my sysvol and subfolders
> >>>>>> and
> >>>>>> getfacl /home/samba/sysvol/internal.domain.tld/
> >>>>>> the on both servers is the same.
> >>>>>> and when i look from within windows both shares are exacly the same
> >>>>>> with rights.
> >>>>>> share files and folders.
> >>>>>>
> >>>>>> maybe im lucky ;-)
> >>>>> Lucky ?? that's nearly impossible, I bet you win everything you
> >>>>> enter ;-)
> >>>>>
> >>>>>> no, i dont need these mappings imo since i dont have any linux/unix
> >>>>>> user to the sysvol share.
> >>>>>> makes life lot easier..
> >>>>> Yes but it doesn't just affect sysvol, it can affect any share.
> >>>>>
> >>>>> Rowland
> >>>>>> Louis
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> -----Oorspronkelijk bericht-----
> >>>>>>> Van: ryana at reachtechfp.com
> >>>>>>> [mailto:samba-bounces at lists.samba.org] Namens Ryan Ashley
> >>>>>>> Verzonden: vrijdag 22 augustus 2014 16:47
> >>>>>>> Aan: samba at lists.samba.org
> >>>>>>> Onderwerp: Re: [Samba] Proper sysvol replication solution...
> >>>>>>>
> >>>>>>> Domain users don't enter the equation here though. We're talking the
> >>>>>>> "well-known" RID's like Rowland just mentioned. In other words, if my
> >>>>>>> "Authenticated Users" is XID 30001 on DC1 but it is 30009 on DC2, I
> >>>>>>> would not want to clone the UID/GID info from DC1 to DC2, only
> >>>>>>> the files
> >>>>>>> and directories. That where the "idmap.ldb" file comes into play.
> >>>>>>> Apparently it is created with random XID's when Samba is
> >>>>>>> configured and
> >>>>>>> ran for the first time on each machine. Does your script copy those
> >>>>>>> UID/GID numbers over? If so, I was under the impression that such a
> >>>>>>> thing was bad.
> >>>>>>>
> >>>>>>> I am currently setting up two virtual machines (VirtualBox) to
> >>>>>>> play with
> >>>>>>> your script. I want to understand how it works and this way I am not
> >>>>>>> worried about destroying a client's domain while I am toying around.
> >>>>>>>
> >>>>>>> On 08/22/2014 10:34 AM, L.P.H. van Belle wrote:
> >>>>>>>>> -----Oorspronkelijk bericht-----
> >>>>>>>>> Van: ryana at reachtechfp.com
> >>>>>>>>> [mailto:samba-bounces at lists.samba.org] Namens Ryan Ashley
> >>>>>>>>> Verzonden: vrijdag 22 augustus 2014 15:30
> >>>>>>>>> Aan: samba at lists.samba.org
> >>>>>>>>> Onderwerp: Re: [Samba] Proper sysvol replication solution...
> >>>>>>>>>
> >>>>>>>>> I know it is, on purpose. Copying the ACLs from one DC to
> >>>>>>>>> another is bad
> >>>>>>>>> because for some reason each DC may have a different ID for
> >>>>>>>>> the built-in
> >>>>>>>>> accounts. In fact I believe it was Rowland who was saying something
> >>>>>>>>> about copying the "idmap.tdb" file over if you sync the ACLs.
> >>>>>>>>> My method
> >>>>>>>>> only copies over the files and directories so no ACLs get
> >>>>>>>>> screwy. Maybe
> >>>>>>>>> Rowland can reiterate what he said on the other thread.
> >>>>>>>>>
> >>>>>>>> Yes, your right, it can be different if you dont keep de the
> >>>>>>> default settings.
> >>>>>>>> Which should be ok. I dont have any reason to change the
> >>>>>>> acls on sysvol.
> >>>>>>>> And i only use my ADDC as DC and not as file server so i
> >>>>>>> dont have any idmap problems.
> >>>>>>>> And i only use windows users to access these shares.
> >>>>>>>> No linux users or what have access to these machines. only
> >>>>>>> me as linux user and windows user.
> >>>>>>>> and that are 2 different accounts. my linux user is only use
> >>>>>>> to maintain the system.
> >>>>>>>>> I actually did use rsync for a while and then realized I was
> >>>>>>>>> messing up
> >>>>>>>>> after several machines stopped being able to access the sysvol
> >>>>>>>>> share on
> >>>>>>>>> the second DC. When I ran "samba-tool ntacl sysvolcheck" it
> >>>>>>>>> would crash.
> >>>>>>>>> I ran a reset and all was good, until the next rsync. I
> >>>>>>>>> finally figured
> >>>>>>>>> it out and removed rsync and the rsync cron job.
> >>>>>>>>>
> >>>>>>>>> So tell me, why are you using rsync and unison? I am also trying to
> >>>>>>>>> figure out how your script figures out which way to use rsync
> >>>>>>>>> (DC1->DC2
> >>>>>>>>> or DC2->DC1) based on what changes are on each DC. In other
> >>>>>>>>> words, would
> >>>>>>>>> your script work if a directory and file was created on DC1 AND a
> >>>>>>>>> directory was removed on DC2? Would the new stuff get copied
> >>>>>>>>> to DC2 and
> >>>>>>>>> would the directory be removed on DC1? Forgive me, but I am
> >>>>>>> not a bash
> >>>>>>>>> scripting god. I know the basics and can get my work done, but
> >>>>>>>>> I am not
> >>>>>>>>> understanding how your script does two-way stuff via the
> >>>>>>> rsync portion.
> >>>>>>>> Well normal questions.. so
> >>>>>>>> If a file is remove from sysvol on DC1 it removes it also from DC2.
> >>>>>>>> If at the same time a folder or file was remove from DC2
> >>>>>>> then it does the same with it on DC1.
> >>>>>>>> just test it.. you see.
> >>>>>>>> And best tip of the day.. make a backup of your sysvol
> >>>>>>> before you start,
> >>>>>>>> when you rest things and you forget to remove the unison
> >>>>>>> hash files you can delete you sysvol..
> >>>>>>>> we need rsync to create the directory structure with
> >>>>>>> extended attributes,
> >>>>>>>> the (now) unison setup copies only the extened attributes on files.
> >>>>>>>>
> >>>>>>>> cron does the sync every 5 min for me and you can also use
> >>>>>>> inotify for that.
> >>>>>>>> You can search for that one on the list.
> >>>>>>>>
> >>>>>>>> Hope answere this helps you.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On 08/22/2014 02:21 AM, L.P.H. van Belle wrote:
> >>>>>>>>>> ah you didnt see my script...
> >>>>>>>>>>
> >>>>>>>>>> https://secure.bazuin.nl/scripts/3-setup-sysvol-bidirectional.sh
> >>>>>>>>>>
> >>>>>>>>>> You way is missing the ACL.. you need rsync with unison.
> >>>>>>>>>> its all in my script.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Greetz,
> >>>>>>>>>>
> >>>>>>>>>> Louis
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> -----Oorspronkelijk bericht-----
> >>>>>>>>>>> Van: ryana at reachtechfp.com
> >>>>>>>>>>> [mailto:samba-bounces at lists.samba.org] Namens Ryan Ashley
> >>>>>>>>>>> Verzonden: vrijdag 22 augustus 2014 0:14
> >>>>>>>>>>> Aan: samba at lists.samba.org
> >>>>>>>>>>> Onderwerp: [Samba] Proper sysvol replication solution...
> >>>>>>>>>>>
> >>>>>>>>>>> I see the Samba guide suggests using rsync to keep sysvols in
> >>>>>>>>>>> sync, but
> >>>>>>>>>>> this poses a problem with ID's and it is only one-way. I have
> >>>>>>>>>>> been
> >>>>>>>>>>> hesitant to suggest anything because of the flak I have
> >>>>>>>>> been getting,
> >>>>>>>>>>> but I do believe I have a much better solution that transfers
> >>>>>>>>>>> files via
> >>>>>>>>>>> SSH, is bi-directional (no more only editing group policy on one
> >>>>>>>>>>> server), and does NOT set UID/GID information. In other
> >>>>>>> words, it is
> >>>>>>>>>>> PERFECT for sysvol replication, and has been working on
> >>>>>>>>> several of my
> >>>>>>>>>>> domains for around a year and a half without a hitch. The
> >>>>>>>>>>> solution I am
> >>>>>>>>>>> proposing is to use unison, which also works on Windows and (I
> >>>>>>>>>>> think) Mac.
> >>>>>>>>>>>
> >>>>>>>>>>> The way I have unison working on my systems is to install
> >>>>>>>>>>> unison on all
> >>>>>>>>>>> DC's, which is required. You also need an SSH server and
> >>>>>>>>> client on all
> >>>>>>>>>>> DC's, but I assume most of you do anyway. Once they're
> >>>>>>>>>>> installed, it is
> >>>>>>>>>>> as simple as the command below. This will synchronizes changes
> >>>>>>>>>>> BOTH WAYS
> >>>>>>>>>>> without touching your UID/GID setup. If you're paranoid, you
> >>>>>>>>>>> could
> >>>>>>>>>>> always do a sysvolreset when done though.
> >>>>>>>>>>>
> >>>>>>>>>>> unison -batch "/path/to/sysvol"
> >>>>>>>>> "ssh://dc02.domain.lan//path/to/sysvol"
> >>>>>>>>>>> If you do this at a command-line, it will prompt you for
> >>>>>>>>> your password
> >>>>>>>>>>> on the remote machine. This would prevent a cron job, but
> >>>>>>> I overcame
> >>>>>>>>>>> that as well. You can create an SSH key that does not require
> >>>>>>>>>>> a password
> >>>>>>>>>>> for the systems to use. This means you can now create a
> >>>>>>> cron job to
> >>>>>>>>>>> handle the replication every fifteen minutes or so. You
> >>>>>>>>> could also use
> >>>>>>>>>>> something like "incrond" to monitor for changes in the sysvol
> >>>>>>>>>>> and launch
> >>>>>>>>>>> unison as well, but I don't personally modify the sysvol
> >>>>>>>>>>> often, so
> >>>>>>>>>>> replication every fifteen minutes works for me.
> >>>>>>>>>>>
> >>>>>>>>>>> To create an SSH key to allow password-less replication via
> >>>>>>>>> unison, do
> >>>>>>>>>>> the following.
> >>>>>>>>>>>
> >>>>>>>>>>> ssh-keygen -t dsa
> >>>>>>>>>>>
> >>>>>>>>>>> When it prompts for a file to save the key in, it should be
> >>>>>>>>> your home
> >>>>>>>>>>> directory in a ".ssh" directory. I run as root, so this is
> >>>>>>>>>>> "/root/.ssh/id_dsa" for me. It will then prompt for a
> >>>>>>>>> password. Ignore
> >>>>>>>>>>> this and just press enter. It will ask you to verify the
> >>>>>>>>>>> password. Press
> >>>>>>>>>>> enter again. If you enter a password here, it cannot run
> >>>>>>>>>>> without user input!
> >>>>>>>>>>>
> >>>>>>>>>>> Next, you need to copy the key to your other domain
> >>>>>>>>>>> controllers. You can
> >>>>>>>>>>> do so as follows. Note that my example is run as root.
> >>>>>>>>> Substitute your
> >>>>>>>>>>> user's path if needed.
> >>>>>>>>>>>
> >>>>>>>>>>> ssh-copy-id -i /root/.ssh/id_dsa.pub root at dc02.domain.lan
> >>>>>>>>>>>
> >>>>>>>>>>> Once that is done, login to the domain controller you copied
> >>>>>>>>>>> the key to
> >>>>>>>>>>> (in the example, dc02) and check "/root/.ssh/authorized_keys"
> >>>>>>>>>>> to verify
> >>>>>>>>>>> that the key was added and nothing unexpected is there. You
> >>>>>>>>>>> can do this
> >>>>>>>>>>> with "cat /root/.ssh/authorized_keys". You should see a key on
> >>>>>>>>>>> a single
> >>>>>>>>>>> line followed by the hostname of your primary domain
> >>>>>>>>> controller. If it
> >>>>>>>>>>> is there, they may now connect via SSH without a password!
> >>>>>>>>>>>
> >>>>>>>>>>> You may now copy the key to any other domain controllers in
> >>>>>>>>>>> your domain
> >>>>>>>>>>> so they trust the primary DC as well. After that, all that
> >>>>>>>>> is left is
> >>>>>>>>>>> the synchronization. I urge you to run the first synchronization
> >>>>>>>>>>> manually, like below.
> >>>>>>>>>>>
> >>>>>>>>>>> unison "/path/to/sysvol" "ssh://dc02.domain.lan//path/to/sysvol"
> >>>>>>>>>>>
> >>>>>>>>>>> Make sure everything looks good, synchronize it, then repeat
> >>>>>>>>>>> for each DC
> >>>>>>>>>>> on your domain. Once done, you can create cron jobs to sync
> >>>>>>>>>>> each server,
> >>>>>>>>>>> or use a script like mine below. This script is on my
> >>>>>>> primary DC. I
> >>>>>>>>>>> actually only have two DC's, but I added more as an example here.
> >>>>>>>>>>>
> >>>>>>>>>>> #!/bin/bash
> >>>>>>>>>>> SERVERLIST="dc02.domain.lan dc03.domain.lan dc04.domain.lan"
> >>>>>>>>>>> SVPATH="/path/to/sysvol"
> >>>>>>>>>>>
> >>>>>>>>>>> # Synchronize all of the domain controllers
> >>>>>>>>>>> for sLoop in ${SERVERLIST}
> >>>>>>>>>>> do
> >>>>>>>>>>>       unison -batch "${SVPATH}" "ssh://${sLoop}/${SVPATH}"
> >>>>>>>>>>> done
> >>>>>>>>>>>
> >>>>>>>>>>> exit 0
> >>>>>>>>>>>
> >>>>>>>>>>> Now set that script to run in a cron job and you're golden.
> >>>>>>>>> You could
> >>>>>>>>>>> also setup "incrond" on all of your DC's and have it call
> >>>>>>>>>>> unison to sync
> >>>>>>>>>>> the other DC's whenever a write happens in your sysvol,
> >>>>>>> but I do not
> >>>>>>>>>>> need such a thing and have not personally tried it,
> >>>>>>> though I have a
> >>>>>>>>>>> fellow IT lead who has and likes it. My crontab job entry is
> >>>>>>>>>>> listed below.
> >>>>>>>>>>>
> >>>>>>>>>>> 15 * * * * /root/sysvolsync.sh &> /dev/null
> >>>>>>>>>>>
> >>>>>>>>>>> I hope this helps somebody and if you see something wrong,
> >>>>>>>>>>> feel free to
> >>>>>>>>>>> let me know.
> >>>>>>>>>>> -- 
> >>>>>>>>>>> To unsubscribe from this list go to the following URL and read
> >>>>>>>>>>> the
> >>>>>>>>>>> instructions: https://lists.samba.org/mailman/options/samba
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>> -- 
> >>>>>>>>> To unsubscribe from this list go to the following URL and read the
> >>>>>>>>> instructions: https://lists.samba.org/mailman/options/samba
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>> -- 
> >>>>>>> To unsubscribe from this list go to the following URL and read the
> >>>>>>> instructions: https://lists.samba.org/mailman/options/samba
> >>>>>>>
> >>>>>>>
> >>>> Rowland do you remeber Michael Adam's patch here from april? Seems it
> >>>> did not yet make it into an stable release.
> >>> Yes, I vaguely remember it, but it did not go far enough (no
> >>> disrespect meant Michael, I couldn't have got that far),  I 'think' it
> >>> only dealt with 'BUILTIN' objects, but they are a small part of the
> >>> well known RID's .
> >>>
> >>> Rowland
> >> Good point,
> >>
> >> An nss_idmap module or extention to nss_winbind which resolves
> >> xidNumbers to objectSID's might do the trick. It is not neccesarry to
> >> translate the SID into an name to solve the mapping problem.
> >>
> >> achim~
> >>
> > As the same idmap.ldb is always produced upon first domain provision, I
> > can see no reason not to simply make a copy of that first idmap
> > database, remove the step that produces it on subsequent DCs and sync it
> > to subsequent DCs. Would the problem be the sync? If not, have  blank
> > copy of the database with the xidNumbers and builtin SIDs already
> > available after make install. Or something else tat the devs will code.
> >
> > This issue causes so many problems, I wonder if we bugzilla it it may
> > get to the devs? Development seems to be moving away from what is
> > causing problems. And why not? If I were a coder, I'd want to work on
> > new stuff too.
> > J & S
> >
> >
> In my opinion, it's a bit more complex than that, if on my DC I run:
> 
> getfacl /var/lib/samba/sysvol/
> 
> I get this:
> 
> getfacl: Removing leading '/' from absolute path names
> # file: var/lib/samba/sysvol/
> # owner: root
> # group: 3000000
> user::rwx
> user:root:rwx
> group::rwx
> group:3000000:rwx
> group:3000001:r-x
> group:3000002:rwx
> group:3000003:r-x
> mask::rwx
> other::---
> default:user::rwx
> default:user:root:rwx
> default:group::---
> default:group:3000000:rwx
> default:group:3000001:r-x
> default:group:3000002:rwx
> default:group:3000003:r-x
> default:mask::rwx
> default:other::---
> 
> As you can see, the group is given as '3000000', so what is this number ?
> 
> If I look in idmap.ldb, I find this:
> 
> dn: CN=S-1-5-32-544
> cn: S-1-5-32-544
> objectClass: sidMap
> objectSid: S-1-5-32-544
> type: ID_TYPE_BOTH
> xidNumber: 3000000
> distinguishedName: CN=S-1-5-32-544
> 
> So the SID for '3000000' is 'S-1-5-32-544'
> 
> If we do a search on the internet it easy to found out that this is 
> 'BUILTIN\administrators'
> 
> To prove this:
> 
> wbinfo --name-to-sid=BUILTIN\\administrators
> S-1-5-32-544 SID_ALIAS (4)
> 
> But, if we try to reverse this on a client, we get this:
> 
> wbinfo --sid-to-gid=S-1-5-32-544
> 2000
Because wbinfo must adjust to whatever range you have set for the non
domain range in smb.conf.
> 
> Same command on the DC returns '3000000'

Exactly. Because it is reading the value from the idmap database. 
> 
> Hopefully when 4.2 gets released with Andrew Bartlett's multitude of 
> patches to use the seperate winbind daemon, we will be able to get the 
> same numbers anywhere
Unless he has a schema extension to read the gidNumber of a builtin from
AD, I can't see how winbind is going to do this. Hence, the
once-talked-about-by-the-coders idea of hard coded ids for builtins.
Cheers,
J & S