[Samba] 5~Re: DFS redirection to sub-folders beyond share

Fri Mar 3 04:44:01 GMT 2006

William Law <law at tc.cornell.edu> wrote:
> resolve_hosts: Attempting host lookup for name ctcfsrv9.tc.cornell.edu\tc_q\users<0x20>
> Connection to ctcfsrv9.tc.cornell.edu\tc_q\users failed
> Unable to follow dfs referral [//ctcfsrv9.tc.cornell.edu\tc_q\users/law]
> ... NT_STATUS_PATH_NOT_COVERED

Hello,

Early February mail from William Law and Jeremy Allison caught my
eye, and I'm trying to follow up.  The mail describes a problem
where smbclient is unable to follow DFS referrals.  We have the
same problem.  I'm pretty sure I know what's going on.

Samba is expecting a DFS referral answer of "\server\share", but
is confused to get a DFS referral answer of "\server\share\path".
In its confusion, Samba tries to connect to a server which is named
"server\share".  This fails, because it is an invalid server name.
(Trivially: backslashes are not allowed in DNS hostnames.)

I don't know Samba's architecture or design well enough to be
able to suggest a patch, but I can provide more details, in case
someone knowledgeable could use them to work towards a fix:

1) I can describe the problem.
2) I can provide smbclient output, both normal and with -d 3.
3) I can run smbclient under a debugger and set a breakpoint in
   cli_cm_open(), which shows Samba trying to open an invalid hostname.
4) I can provide a network packet dump, which shows the SMB Trans2
   packets, and shows that Samba is doing DNS lookup for an
   invalid hostname.
5) I could even provide a unix shell account, if someone
   knowledgeable about Samba internals needs to look at this
   directly before knowing how to fix it.

I apologize about the length of thie message, but hope that the
information may be useful if any developers can use it to improve
Samba's ability to handle DFS.

Thank you for your time,
 Jacob Morzinski <morzinski at mit.edu>

1) Describing the problem
=========================

We have a Windows 2003 server with DFS shares.
It works fine when accessed from Windows XP clients.

Samba (smbclient) can connect to the root of our MS-DFS server,
and can connect to the DFS shares if I explicitly tell it to
connect to a particular server/share.  However, if I try to
follow a path that takes me through DFS referrals, Samba fails,
with: NT_STATUS_PATH_NOT_COVERED.

Investigation shows the problem: Samba queries the DFS referral,
but isn't prepared for the answer it gets from the server.
It only expects to get a response of "\A\B", but the actual
response is "\A\B\C" (or "\A\B\C\D").  Samba mis-parses this,
tries to connect to a share that is named "\C" on a server that
is named "A\B", and fails to find the server.

You can see this in William Law's case: Samba tries to find a
host named "ctcfsrv9.tc.cornell.edu\tc_q\users"... but that isn't
even a valid internet hostname.

The same thing happens in our case.  Samba queries the referral:
   GET_DFS_REFERRAL query:
      File: \w92dfs1\DFS\profiles\j\jmorzins

Samba gets an answer:
   GET_DFS_REFERRAL answer:
      Path: \w92dfs1\DFS\profiles\j
      Node: \profiler3.mit.edu\homes3$\j

Samba is confused by the answer, and tries to connect to the wrong host:
   DNS query:
      Query: profiler3.mit.eduhome3$
   DNS answer:
      Host profiler3.mit.eduhome3$ not found: 3(NXDOMAIN)

NOTE: this referral answer is only three levels deep: \A\B\C.
Deeper levels are possible: \A\B\C\D, etc.

2) smbclient output
===================

I'll test two UNC paths.  The second test path is replicated on
two DFS sites.  I'll show normal smbclient output first, and -d 3
debugging output second.

2.1) Normal smbclient output
----------------------------

The UNC paths I am trying to reach are:

  \\win.mit.edu\dfs\profiles\j\jmorzins
  \\win.mit.edu\dfs\profiles\ops\scripts\common

Note that the second path is a more complicated DFS referral,
multiple levels deep, replicated on two sites.

$ smbclient -k //win.mit.edu/dfs
OS=[Windows Server 2003 3790 Service Pack 1] Server=[Windows Server 2003 5.2]
smb: \> cd profiles
smb: \profiles\> cd j
Connection to profiler3.mit.edu\homes3$ failed
Unable to follow dfs referral [//profiler3.mit.edu\homes3$/j]
cd \profiles\j\: NT_STATUS_PATH_NOT_COVERED
smb: \profiles\> quit

$ smbclient -k //profiler3.mit.edu/homes3$
OS=[Windows Server 2003 3790 Service Pack 1] Server=[Windows Server 2003 5.2]
smb: \> cd j
smb: \j\> cd jmorzins
smb: \j\jmorzins\> dir
[...]

(((To make this faster, I'm going to start using "-D /path")))

$ smbclient -k //win.mit.edu/dfs -D /ops/scripts/common
OS=[Windows Server 2003 3790 Service Pack 1] Server=[Windows Server 2003 5.2]
Connection to 24dfs1.mit.edu\auto\operational failed
Unable to follow dfs referral [//24dfs1.mit.edu\auto\operational/scripts]
cd \ops\scripts\common\: NT_STATUS_PATH_NOT_COVERED

$ smbclient -k //24dfs1.mit.edu/auto -D /operational/scripts/common
OS=[Windows Server 2003 3790 Service Pack 1] Server=[Windows Server 2003 5.2]
smb: \operational\scripts\common\> dir
[...]

2.2) smbclient debugging output
-------------------------------

Same as above: demonstrate a failed connect, then demonstrate a
successful connect, for each of the two paths:

  \\win.mit.edu\dfs\profiles\j\jmorzins
  \\win.mit.edu\dfs\profiles\ops\scripts\common

Note that the second path is a more complicated DFS referral,
multiple levels deep, replicated on two sites.

$ smbclient -d 3 -k //win.mit.edu/dfs -D /profiles/j/jmorzins
lp_load: refreshing parameters
Initialising global parameters
params.c:pm_process() - Processing configuration file "/mit/samba/arch/i386_rhel4/lib/smb.conf"
added interface ip=18.152.3.97 bcast=18.152.255.255 nmask=255.255.0.0
Client started (version 3.0.21a).
resolve_lmhosts: Attempting lmhosts lookup for name win.mit.edu<0x20>
resolve_wins: Attempting wins lookup for name win.mit.edu<0x20>
resolve_wins: WINS server resolution selected and no WINS servers listed.
resolve_hosts: Attempting host lookup for name win.mit.edu<0x20>
Connecting to 18.7.14.183 at port 445
Doing spnego session setup (blob length=107)
got OID=1 2 840 48018 1 2 2
got OID=1 2 840 113554 1 2 2
got OID=1 2 840 113554 1 2 2 3
got OID=1 3 6 1 4 1 311 2 2 10
got principal=w92dcr$@WIN.MIT.EDU
Doing kerberos session setup
Ticket in ccache[/tmp/krb5cc_p3089] expiration Fri, 03 Mar 2006 08:46:20 EST
OS=[Windows Server 2003 3790 Service Pack 1] Server=[Windows Server 2003 5.2]
resolve_lmhosts: Attempting lmhosts lookup for name win.mit.edu<0x20>
resolve_wins: Attempting wins lookup for name win.mit.edu<0x20>
resolve_wins: WINS server resolution selected and no WINS servers listed.
resolve_hosts: Attempting host lookup for name win.mit.edu<0x20>
Connecting to 18.7.7.88 at port 445
Doing spnego session setup (blob length=107)
got OID=1 2 840 48018 1 2 2
got OID=1 2 840 113554 1 2 2
got OID=1 2 840 113554 1 2 2 3
got OID=1 3 6 1 4 1 311 2 2 10
got principal=w20dc1$@WIN.MIT.EDU
Doing kerberos session setup
Ticket in ccache[/tmp/krb5cc_p3089] expiration Fri, 03 Mar 2006 08:46:16 EST
dos_clean_name []
resolve_lmhosts: Attempting lmhosts lookup for name 24DFS1<0x20>
resolve_wins: Attempting wins lookup for name 24DFS1<0x20>
resolve_wins: WINS server resolution selected and no WINS servers listed.
resolve_hosts: Attempting host lookup for name 24DFS1<0x20>
Connecting to 18.7.7.87 at port 445
Doing spnego session setup (blob length=107)
got OID=1 2 840 48018 1 2 2
got OID=1 2 840 113554 1 2 2
got OID=1 2 840 113554 1 2 2 3
got OID=1 3 6 1 4 1 311 2 2 10
got principal=24dfs1$@WIN.MIT.EDU
Doing kerberos session setup
Ticket in ccache[/tmp/krb5cc_p3089] expiration Fri, 03 Mar 2006 08:46:16 EST
resolve_lmhosts: Attempting lmhosts lookup for name 24DFS1<0x20>
resolve_wins: Attempting wins lookup for name 24DFS1<0x20>
resolve_wins: WINS server resolution selected and no WINS servers listed.
resolve_hosts: Attempting host lookup for name 24DFS1<0x20>
Connecting to 18.7.7.87 at port 445
Doing spnego session setup (blob length=107)
got OID=1 2 840 48018 1 2 2
got OID=1 2 840 113554 1 2 2
got OID=1 2 840 113554 1 2 2 3
got OID=1 3 6 1 4 1 311 2 2 10
got principal=24dfs1$@WIN.MIT.EDU
Doing kerberos session setup
Ticket in ccache[/tmp/krb5cc_p3089] expiration Fri, 03 Mar 2006 08:46:16 EST
dos_clean_name []
dos_clean_name []
dos_clean_name [\profiles\j\jmorzins\]
dos_clean_name [\profiles\j\jmorzins\\]
resolve_lmhosts: Attempting lmhosts lookup for name profiler3.mit.edu\homes3$<0x20>
resolve_wins: Attempting wins lookup for name profiler3.mit.edu\homes3$<0x20>
resolve_wins: WINS server resolution selected and no WINS servers listed.
resolve_hosts: Attempting host lookup for name profiler3.mit.edu\homes3$<0x20>
name_resolve_bcast: Attempting broadcast lookup for name profiler3.mit.edu\homes3$<0x20>
Connection to profiler3.mit.edu\homes3$ failed
Unable to follow dfs referral [//profiler3.mit.edu\homes3$/j]
cd \profiles\j\jmorzins\: NT_STATUS_PATH_NOT_COVERED

$ smbclient -d 3 -k //profiler3.mit.edu/homes3$ -D /j/jmorzins
lp_load: refreshing parameters
Initialising global parameters
params.c:pm_process() - Processing configuration file "/mit/samba/arch/i386_rhel4/lib/smb.conf"
added interface ip=18.152.3.97 bcast=18.152.255.255 nmask=255.255.0.0
Client started (version 3.0.21a).
resolve_lmhosts: Attempting lmhosts lookup for name profiler3.mit.edu<0x20>
resolve_wins: Attempting wins lookup for name profiler3.mit.edu<0x20>
resolve_wins: WINS server resolution selected and no WINS servers listed.
resolve_hosts: Attempting host lookup for name profiler3.mit.edu<0x20>
Connecting to 18.7.14.213 at port 445
Doing spnego session setup (blob length=110)
got OID=1 2 840 48018 1 2 2
got OID=1 2 840 113554 1 2 2
got OID=1 2 840 113554 1 2 2 3
got OID=1 3 6 1 4 1 311 2 2 10
got principal=profiler3$@WIN.MIT.EDU
Doing kerberos session setup
Ticket in ccache[/tmp/krb5cc_p3089] expiration Fri, 03 Mar 2006 08:46:35 EST
OS=[Windows Server 2003 3790 Service Pack 1] Server=[Windows Server 2003 5.2]
resolve_lmhosts: Attempting lmhosts lookup for name profiler3.mit.edu<0x20>
resolve_wins: Attempting wins lookup for name profiler3.mit.edu<0x20>
resolve_wins: WINS server resolution selected and no WINS servers listed.
resolve_hosts: Attempting host lookup for name profiler3.mit.edu<0x20>
Connecting to 18.7.14.213 at port 445
Doing spnego session setup (blob length=110)
got OID=1 2 840 48018 1 2 2
got OID=1 2 840 113554 1 2 2
got OID=1 2 840 113554 1 2 2 3
got OID=1 3 6 1 4 1 311 2 2 10
got principal=profiler3$@WIN.MIT.EDU
Doing kerberos session setup
Ticket in ccache[/tmp/krb5cc_p3089] expiration Fri, 03 Mar 2006 08:46:35 EST
dos_clean_name []
dos_clean_name []
dos_clean_name [\j\jmorzins\]
dos_clean_name [\j\jmorzins\\]
smb: \j\jmorzins\> quit

$ smbclient -d 3 -k //win.mit.edu/dfs -D /ops/scripts/common
lp_load: refreshing parameters
Initialising global parameters
params.c:pm_process() - Processing configuration file "/mit/samba/arch/i386_rhel4/lib/smb.conf"
added interface ip=18.152.3.97 bcast=18.152.255.255 nmask=255.255.0.0
Client started (version 3.0.21a).
resolve_lmhosts: Attempting lmhosts lookup for name win.mit.edu<0x20>
resolve_wins: Attempting wins lookup for name win.mit.edu<0x20>
resolve_wins: WINS server resolution selected and no WINS servers listed.
resolve_hosts: Attempting host lookup for name win.mit.edu<0x20>
Connecting to 18.7.14.184 at port 445
Doing spnego session setup (blob length=105)
got OID=1 2 840 48018 1 2 2
got OID=1 2 840 113554 1 2 2
got OID=1 2 840 113554 1 2 2 3
got OID=1 3 6 1 4 1 311 2 2 10
got principal=edc1$@WIN.MIT.EDU
Doing kerberos session setup
Ticket in ccache[/tmp/krb5cc_p3089] expiration Fri, 03 Mar 2006 08:46:16 EST
OS=[Windows Server 2003 3790 Service Pack 1] Server=[Windows Server 2003 5.2]
resolve_lmhosts: Attempting lmhosts lookup for name win.mit.edu<0x20>
resolve_wins: Attempting wins lookup for name win.mit.edu<0x20>
resolve_wins: WINS server resolution selected and no WINS servers listed.
resolve_hosts: Attempting host lookup for name win.mit.edu<0x20>
Connecting to 18.7.14.183 at port 445
Doing spnego session setup (blob length=107)
got OID=1 2 840 48018 1 2 2
got OID=1 2 840 113554 1 2 2
got OID=1 2 840 113554 1 2 2 3
got OID=1 3 6 1 4 1 311 2 2 10
got principal=w92dcr$@WIN.MIT.EDU
Doing kerberos session setup
Ticket in ccache[/tmp/krb5cc_p3089] expiration Fri, 03 Mar 2006 08:46:20 EST
dos_clean_name []
resolve_lmhosts: Attempting lmhosts lookup for name 24DFS1<0x20>
resolve_wins: Attempting wins lookup for name 24DFS1<0x20>
resolve_wins: WINS server resolution selected and no WINS servers listed.
resolve_hosts: Attempting host lookup for name 24DFS1<0x20>
Connecting to 18.7.7.87 at port 445
Doing spnego session setup (blob length=107)
got OID=1 2 840 48018 1 2 2
got OID=1 2 840 113554 1 2 2
got OID=1 2 840 113554 1 2 2 3
got OID=1 3 6 1 4 1 311 2 2 10
got principal=24dfs1$@WIN.MIT.EDU
Doing kerberos session setup
Ticket in ccache[/tmp/krb5cc_p3089] expiration Fri, 03 Mar 2006 08:46:16 EST
resolve_lmhosts: Attempting lmhosts lookup for name 24DFS1<0x20>
resolve_wins: Attempting wins lookup for name 24DFS1<0x20>
resolve_wins: WINS server resolution selected and no WINS servers listed.
resolve_hosts: Attempting host lookup for name 24DFS1<0x20>
Connecting to 18.7.7.87 at port 445
Doing spnego session setup (blob length=107)
got OID=1 2 840 48018 1 2 2
got OID=1 2 840 113554 1 2 2
got OID=1 2 840 113554 1 2 2 3
got OID=1 3 6 1 4 1 311 2 2 10
got principal=24dfs1$@WIN.MIT.EDU
Doing kerberos session setup
Ticket in ccache[/tmp/krb5cc_p3089] expiration Fri, 03 Mar 2006 08:46:16 EST
dos_clean_name []
dos_clean_name []
dos_clean_name [\ops\scripts\common\]
dos_clean_name [\ops\scripts\common\\]
resolve_lmhosts: Attempting lmhosts lookup for name 24dfs1.mit.edu\auto\operational<0x20>
resolve_wins: Attempting wins lookup for name 24dfs1.mit.edu\auto\operational<0x20>
resolve_wins: WINS server resolution selected and no WINS servers listed.
resolve_hosts: Attempting host lookup for name 24dfs1.mit.edu\auto\operational<0x20>
name_resolve_bcast: Attempting broadcast lookup for name 24dfs1.mit.edu\auto\operational<0x20>
Connection to 24dfs1.mit.edu\auto\operational failed
Unable to follow dfs referral [//24dfs1.mit.edu\auto\operational/scripts]
cd \ops\scripts\common\: NT_STATUS_PATH_NOT_COVERED

$ smbclient -d 3 -k //24dfs1.mit.edu/auto -D /operational/scripts/common
lp_load: refreshing parameters
Initialising global parameters
params.c:pm_process() - Processing configuration file "/mit/samba/arch/i386_rhel4/lib/smb.conf"
added interface ip=18.152.3.97 bcast=18.152.255.255 nmask=255.255.0.0
Client started (version 3.0.21a).
resolve_lmhosts: Attempting lmhosts lookup for name 24dfs1.mit.edu<0x20>
resolve_wins: Attempting wins lookup for name 24dfs1.mit.edu<0x20>
resolve_wins: WINS server resolution selected and no WINS servers listed.
resolve_hosts: Attempting host lookup for name 24dfs1.mit.edu<0x20>
Connecting to 18.7.7.87 at port 445
Doing spnego session setup (blob length=107)
got OID=1 2 840 48018 1 2 2
got OID=1 2 840 113554 1 2 2
got OID=1 2 840 113554 1 2 2 3
got OID=1 3 6 1 4 1 311 2 2 10
got principal=24dfs1$@WIN.MIT.EDU
Doing kerberos session setup
Ticket in ccache[/tmp/krb5cc_p3089] expiration Fri, 03 Mar 2006 08:46:16 EST
OS=[Windows Server 2003 3790 Service Pack 1] Server=[Windows Server 2003 5.2]
resolve_lmhosts: Attempting lmhosts lookup for name 24dfs1.mit.edu<0x20>
resolve_wins: Attempting wins lookup for name 24dfs1.mit.edu<0x20>
resolve_wins: WINS server resolution selected and no WINS servers listed.
resolve_hosts: Attempting host lookup for name 24dfs1.mit.edu<0x20>
Connecting to 18.7.7.87 at port 445
Doing spnego session setup (blob length=107)
got OID=1 2 840 48018 1 2 2
got OID=1 2 840 113554 1 2 2
got OID=1 2 840 113554 1 2 2 3
got OID=1 3 6 1 4 1 311 2 2 10
got principal=24dfs1$@WIN.MIT.EDU
Doing kerberos session setup
Ticket in ccache[/tmp/krb5cc_p3089] expiration Fri, 03 Mar 2006 08:46:16 EST
dos_clean_name []
dos_clean_name []
dos_clean_name [\operational\scripts\common\]
dos_clean_name [\operational\scripts\common\\]
smb: \operational\scripts\common\> quit

3) smbclient run in gdb, with a breakpoint in cli_cm_open
=========================================================

To make life simpler, I only attempt a single failed connect,
  \\win.mit.edu\dfs\ops\scripts\common

Notice that the final call to cli_cm_open is with parameters of:
  server="24dfs1.mit.edu\\auto\\operational"
  share="scripts"

(gdb) break cli_cm_open
Breakpoint 1 at 0x54432: file libsmb/clidfs.c, line 271.
(gdb) run -k //win.mit.edu/dfs -D /ops/scripts/common
Starting program: /afs/sipb.mit.edu/project/sandbox/jmorzins/samba/linux/source/bin/smbclient -k //win.mit.edu/dfs -D /ops/scripts/common
Breakpoint 1 at 0xf07432: file libsmb/clidfs.c, line 271.
params.c:OpenConfFile() - Unable to open configuration file "/tmp/samba/arch/i386_rhel4/lib/smb.conf":
	No such file or directory
creating lame upcase table
creating lame lowcase table
/afs/sipb.mit.edu/project/sandbox/jmorzins/samba/linux/source/bin/smbclient: Can't load /tmp/samba/arch/i386_rhel4/lib/smb.conf - run testparm to debug it

Breakpoint 1, cli_cm_open (server=0xf9aea0 "", 
    share=0xf9aaa0 "\\\\win.mit.edu\\dfs", show_hdr=1) at libsmb/clidfs.c:271
(gdb) cont
Continuing.
OS=[Windows Server 2003 3790 Service Pack 1] Server=[Windows Server 2003 5.2]

Breakpoint 1, cli_cm_open (server=0x891d69c "win.mit.edu", 
    share=0xf78e3a "IPC$", show_hdr=0) at libsmb/clidfs.c:271
(gdb) cont
Continuing.

Breakpoint 1, cli_cm_open (server=0x8940d54 "24DFS1", share=0xf78e3a "IPC$", 
    show_hdr=0) at libsmb/clidfs.c:271
(gdb) cont
Continuing.

Breakpoint 1, cli_cm_open (server=0x8940d54 "w92dfs1", share=0xf78e3a "IPC$", 
    show_hdr=0) at libsmb/clidfs.c:271
(gdb) cont
Continuing.

Breakpoint 1, cli_cm_open (server=0x8940d54 "w92dfs1", share=0xf78e3a "IPC$", 
    show_hdr=0) at libsmb/clidfs.c:271
(gdb) cont
Continuing.

Breakpoint 1, cli_cm_open (
    server=0xbfed7e30 "24dfs1.mit.edu\\auto\\operational", 
    share=0xbfed7d30 "scripts", show_hdr=0) at libsmb/clidfs.c:271
(gdb) cont
Continuing.
Connection to 24dfs1.mit.edu\auto\operational failed
Unable to follow dfs referral [//24dfs1.mit.edu\auto\operational/scripts]
cd \ops\scripts\common\: NT_STATUS_PATH_NOT_COVERED

4) network packets
==================

Ethereal makes a great packet viewer; you can load this file into
Ethereal, and step directly into the SMB packets, and observe
what DFS referral requests and responses are going back and forth.

Download my packet dump.  Get either of:
  http://web.mit.edu/jmorzins/www/samba-bug/smbclient.pcap.zip
  http://web.mit.edu/jmorzins/www/samba-bug/smbclient.pcap

I ran two smbclient commands:

  smbclient -k //win.mit.edu/dfs -D /profiles/j/jmorzins
  smbclient -k //win.mit.edu/dfs -D /ops/scripts/common

For \\win.mit.edu\dfs\profiles\j\jmorzins, 
see packets 69-76 (SMB Trans2 query, plus DNS query)

For \\win.mit.edu\dfs\ops\scripts\common,
see packets 234-242 (SMB Trans2 query, plus DNS query)

Note that the second path is a more complicated DFS referral,
multiple levels deep, replicated on two sites.

Thanks for reading this far!
Let me know if #(5) would be a useful aid for fixing this.

Regards,
 Jacob Morzinski                                jmorzins at mit.edu