[Samba] Slow directory listing after adding new trusted domain to current forest

[Jeff Hodge]

We have an ubuntu 14.04 server running samba 4.1.6 that is a member of our
OLDDOMAIN.  We recently added a new trusted domain to our forest.  We
noticed any new directories created by users in that new domain take a long
time for the directory to list in the command line.

This does not seem to happen when listing the directories with only
user/group ID:

    root at d101:/home/OLDDOMAIN/test/data/Production# time ls -lan
    total 36
    drwxrwxrwx  6   0    0 4096 Jul 29 12:50 .
    drwxrwxrwx  3   0    0 4096 Jul 28 10:24 ..
    drwxrwxr-x+ 2 590 1319 4096 Jul 28 16:16 NEWDOMAIN-Jeff
    drwxrwxr-x+ 2 500  504 4096 Jul 28 14:46 NEWDOMAIN-Jeff-acl
    drwxrwxr-x+ 2 500  504 4096 Jul 29 12:50 Jeff

    real    0m0.002s
    user    0m0.002s
    sys     0m0.000s

When trying to list the directory with the user/group mappings, it is much
slower:

    root at d101:/home/OLDDOMAIN/test/data/Production# time ls -la
    total 36
    drwxrwxrwx  6 root            root                  4096 Jul 29 12:50 .
    drwxrwxrwx  3 root            root                  4096 Jul 28 10:24 ..
    drwxrwxr-x+ 2 NEWDOMAIN\jhodge NEWDOMAIN\domain users 4096 Jul 28 16:16
NEWDOMAIN-Jeff
    drwxrwxr-x+ 2 OLDDOMAIN\jhodge     OLDDOMAIN\domain users     4096 Jul
28 14:46 NEWDOMAIN-Jeff-acl
    drwxrwxr-x+ 2 OLDDOMAIN\jhodge     OLDDOMAIN\domain users     4096 Jul
29 12:50 Jeff

    real    0m19.727s
    user    0m0.005s
    sys     0m0.000s

It does seem to cache the information, because another listing a few
moments later is normal.  However if you wait another 10 minutes, it will
take 10-20 seconds to list the directory.

    root at d101:/home/OLDDOMAIN/test/data/Production# time ls -la
    total 36
    drwxrwxrwx  6 root            root                  4096 Jul 29 12:50 .
    drwxrwxrwx  3 root            root                  4096 Jul 28 10:24 ..
    drwxrwxr-x+ 2 NEWDOMAIN\jhodge NEWDOMAIN\domain users 4096 Jul 28 16:16
NEWDOMAIN-Jeff
    drwxrwxr-x+ 2 OLDDOMAIN\jhodge     OLDDOMAIN\domain users     4096 Jul
28 14:46 NEWDOMAIN-Jeff-acl
    drwxrwxr-x+ 2 OLDDOMAIN\jhodge     OLDDOMAIN\domain users     4096 Jul
29 12:50 Jeff

    real    0m0.010s
    user    0m0.000s
    sys     0m0.006s

It would seem this is a problem with the mapping of the new domain
user/group ID's.  This behavior was never seen before we added the new
domain.  I noticed the group mapping/account polices have not been updated
since the server was added to the domain, does this need to be updated?
The winbindd_cache does seem to update and idmap table have updated
recently.

    root at d101:/var/lib/samba# ll
    total 7656
    drwxr-xr-x  6 root root          4096 Apr 13 15:26 .
    drwxr-xr-x 43 root root          4096 Feb  4  2015 ..
    -rw-------  1 root root        421888 Apr 29  2014 account_policy.tdb
    -rw-------  1 root root        425984 Apr 29  2014 group_mapping.tdb
    drwxr-xr-x 10 root root          4096 Apr 29  2014 printers
    drwxr-xr-x  3 root root          4096 Mar  4  2015 private
    -rw-------  1 root root        528384 Apr 29  2014 registry.tdb
    -rw-------  1 root root        421888 Jul 29 13:04 share_info.tdb
    drwxrwx--T  2 root sambashare    4096 Jul 28 09:56 usershares
    -rw-------  1 root root       5353472 Aug  1 10:36 winbindd_cache.tdb
    -rw-r--r--  1 root root        663552 Jul 29 12:53 winbindd_idmap.tdb
    drwxr-x---  2 root root          4096 Apr 13 15:26 winbindd_privileged

Any idea why this slowness would happen and how it can be resolved?
Rejoining the domain?  Can you force samba to recreate the group
mappings/idmaps?

Thank you for your time.