[Samba] Is it possible to clone an NT ACL from one file or dir to a totally different file or dir ?
Ken McDonald
ken at generation.tech
Sat Apr 7 20:02:52 UTC 2018
On 04/05/2018 08:32 PM, Andrew Bartlett wrote:
> On Sat, 2018-03-24 at 20:38 -0400, Ken McDonald via samba wrote:
>>> By default, step 4 takes forever to run on large datasets because it
>>> uses named entities. My ultimate plan was to use the numeric
>>> user/group id's in that step instead of named ones so the Winbind cost
>>> is not incurred. Seems for the whole process, the calls to Winbind to
>>> resolve the named entity to it's numeric ID are the reason for the
>>> slowdown. That's why, even when using the normal Windows security tab
>>> or samba-tool, it takes days to update large datasets. I'm exploring
>>> options around that issue.
> If this is on the AD DC, then I fixed part of the slowness here a
> couple of week ago with
>
> commit d418d0ca33afb41a793a2fff19ca68871aa5e9ef
> Author: Andrew Bartlett <abartlet at samba.org>
> Date: Wed Mar 21 20:43:10 2018 +1300
>
> winbindd: Add a cache of the samr and lsa handles for the passdb
> domain
>
> Otherwise, I'm not sure folks have looked into this terribly much, you
> are already working at a pretty low level.
>
> I hope this helps a little.
>
> Andrew Bartlett
I have found a working solution using parallel processing on multiple
cores that greatly reduces the time required to clone permissions. On a
user directory with > 1TB data of 100,000s of dirs/files with all kinds
of exotic naming styles, the process in the script below completes in
about 60 minutes instead of the 5 days it took with either samba-tool or
Windows Explorer.
This kind of performance was needed to move a bunch of user home
directories over to the Samba file server, and I also have some cron
jobs that periodically run to ensure a few shared file directories
always have a standard set of permissions. In prior email list replies,
I mentioned the perms were not properly cloning because the cloned copy
had "extras" in them. This turned was because the top-level POSIX user &
group were not the same in the cloned target. While I was correctly
cloning POSIX extended ACL & attributes from the source, I did not
realize that would not include the top-level permissions entries. Once
that setting was added to the process, cloning Samba/POSIX permissions &
attributes worked perfectly and ended up with exactly the same
permission structure as would have been created by using Windows Explorer.
While I haven't looked at Andrew's code update helping Samba cache
winbind lookups, I will mention that the process in the script below
does not make use of any Samba/Winbind username/group lookup calls
UNLESS the Windows ACL dir/file ownership is being set with a Windows
username or group. For my case, all the dirs/files are owned by
Administrator (root) and user access rights are delegated through
Windows ACL entries, so no winbind lookups were needed when the script
runs. If a Windows user/group account is configured for dir/file
ownership, there will be slight delays during script execution, but
since it's only for dir/file ownership, not actual Windows ACL perms,
the delay is relatively tiny.
This process is extremely fast compared to samba-tool and I hope it's
helpful to anyone that finds this thread. If you need to configure a
standard suite of Windows ACL permissions on a large amount of
directories & files this is currently the fastest way I've found to do it.
It's possible "sneak in" pure POSIX ACL entries manually using
Linux-side only users/groups and the full cloning still works. In those
case, once the POSIX ACL entry is created outside of Samba/Windows, you
have to go back into the Windows Explorer Security tab for the perms
template source dir/file and clean up some unneeded entries like
"Everyone" and "CREATER OWNER." In the end, I was able to clone a source
template dir/file with both POSIX/Windows ACL entries from both
Linux-side & Samba-winbind sourced usernames/groups!
I'd appreciate any helpful comments on making my script more efficient
or better simplified. The one portion I wished could have been compacted
is the shell'ing-out of xargs to the separate tiny helper script
"setperm.sh". I couldn't get that helper snippet to run as a shell
function while still allowing xargs to call it, especially because of
the variables being passed. Also, as dir/file names can be complex and
contain all kinds of crazy special characters, there were a lot of
challenges making the script and command/program calls handle these
issues without running into escape'ing problems. I think all of these
leaky problems have been corrected but I'd certainly appreciate any
feedback there.
By using deep extended attribute cloning, only getfattr is needed
because it includes the ACL entries normally configured by using
getfacl. This helped ...a lot... in reducing winbind username/group
lookup calls. The key line that clones POSIX/Windows ACL entries is this.
getfattr -d -m - $1 | sed 1d | sed "1 i\# file: $ENCODED" | setfattr
--restore=-
THE FULL SCRIPTS FOLLOW:
####################################### cloneperms.sh
#######################################
#!/bin/bash
# Clone permissions & attributes from a template source to a target
destination,
# possibly recursing all subdirectories & files. Also supports parallelism.
#
# USAGE: cloneperms.sh [-r] [-p numcores] template target
#
# -r : recurse all subdirectories. pointless if template & target
are files
# -p : number of cores for parallel processing, defaults to 1.
pointless if template & target are files
# template : source directory or file with permissions to clone
# target : destination directory or file for cloning perms to
#
# version 1.0
USAGE="Usage: $0 [-r] [-p numcores] template target"
# determine parameter type
test_type()
{
TYPE_TEST=$1
if [ -d "${TYPE_TEST}" ]
then return 1; # directory
elif [ -f "${TYPE_TEST}" ]
then return 2; # file
else
echo "${TYPE_TEST} is not found!"; echo $USAGE; exit 1;
fi
}
# check for options
CORES=1
RECURSE=0
while getopts rp: o
do case "$o" in
r) RECURSE=1;;
p) CORES=$OPTARG
if [ -z "${CORES##*[!0-9]*}" ] || [ $CORES -le 0 ]
then
echo "numcores must be number greater than 0"; echo
$USAGE; exit 1;
fi;;
[?]) echo $USAGE; exit 1;;
esac
done
shift $(($OPTIND - 1))
# enough arguments?
if [ "$#" -ne 2 ]
then
echo "missing template or target"; echo $USAGE; exit 1;
fi
TEMPLATE=$1
TARGET=$2
# no self targets
if [ $TEMPLATE == $TARGET ]
then
echo "template & target cannot be the same!"; echo $USAGE; exit 1;
fi
# test template & target types
test_type $TEMPLATE
TEMPLATE_TYPE=$?
test_type $TARGET
TARGET_TYPE=$?
# ensure template, target, recurse, and cores combinations are valid
if [ $TEMPLATE_TYPE == 1 ]
then
FIND_FLAG="d";
if [ $TARGET_TYPE == 2 ]
then
echo "template cannot be directory and target a file!"; echo
$USAGE; exit 1;
fi
else
FIND_FLAG="f";
if [ $TARGET_TYPE == 1 ]
then
if [ $RECURSE == 0 ]
then
echo "template cannot be a file and target a directory
without recurse option"; echo $USAGE; exit 1;
fi
else
if [ $CORES -gt 1 ]
then
echo "template & target are files, ignoring numcores option";
CORES=1;
fi
if [ $RECURSE == 1 ]
then
echo "template & target are files, ignoring recurse option";
RECURSE=0;
fi fi fi
TEMPLATE_OWN=$(stat -c %U $TEMPLATE) # get the template top-level
POSIX owner
TEMPLATE_GROUP=$(stat -c %G $TEMPLATE) # get the template top-level
POSIX group
TEMPLATE_PERMS=$(stat -c %a $TEMPLATE) # get the template top-level
POSIX permissions
current_dir=$(dirname $(readlink -f $0)) # script can run from another
directory
if [ $RECURSE == 1 ]
then
find $TARGET -type $FIND_FLAG -print0 | xargs -0 -n1 -P$CORES
chown $TEMPLATE_OWN;
find $TARGET -type $FIND_FLAG -print0 | xargs -0 -n1 -P$CORES
chgrp $TEMPLATE_GROUP;
find $TARGET -type $FIND_FLAG -print0 | xargs -0 -n1 -P$CORES
chmod $TEMPLATE_PERMS;
# get the template POSIX extended attributes, which include
extended ACL
find $TARGET -type $FIND_FLAG -print0 | xargs -0 -n1 -P$CORES
-I {} $current_dir/setperm.sh $TEMPLATE {};
else
chown $TEMPLATE_OWN $TARGET;
chgrp $TEMPLATE_GROUP $TARGET;
chmod $TEMPLATE_PERMS $TARGET;
# get the template POSIX extended attributes, which include
extended ACL
$current_dir/setperm.sh $TEMPLATE $TARGET;
fi
######################################################################################
####################################### setperm.sh
#######################################
#!/bin/bash
# Sets a single permissions & attributes entry on a target based on a
template source example
#
# There's no parameter checking in this script, so don't run this script
directly.
# It's meant to be run from cloneperms.sh.
#
# USAGE: setperm.sh template target
#
# template : source directory or file with permissions to clone
# target : destination directory or file for cloning perms to
#
# version 1.0
ENCODED=$(printf '%s\n' "$2" | sed 's|\\|\\\\134|g') # setfattr syntax
for target name containing forward slashes
getfattr -d -m - $1 | sed 1d | sed "1 i\# file: $ENCODED" | setfattr
--restore=-
######################################################################################
I also have a script for recursively removing all extended ACL &
attribute entries if anyone is interested. Strangely there is a way to
remove the ACL "setfacl -R -b" recursively but I found no way to
similarly remove ALL attributes. I made a script to do it regardless of
what the attributes are named...
TLDR?
-Ken
More information about the samba
mailing list