issue -- cifs automounts stopped working
Steve French
smfrench at gmail.com
Wed Jul 22 04:03:10 UTC 2020
There are two common Kerberos libraries, MIT and Heimdal (Heimdal is
also packaged as an optional part of Samba as an alternative to MIT
Kerberos) ... would be interesting if there is a way to work around
this by switching to Heimdal krb5 libraries.
Adding samba-technical as some of the developers on samba-technical
know the Kerberos maintainers and may have suggestions on how to track
down this possible regression in MIT kerberos.
On Tue, Jul 21, 2020 at 10:48 PM Michael Keane, K1MK <mkeane at arrl.org> wrote:
>
> Tried setting KRB5_CCNAME to KEYRING:persistent:1000 with no change
> (Also tried KRB5CCNAME as well, as that's the environment variable
> which seems to work with kinit & klist)
>
> Also tried setting default_ccahe_name in /etc/krb5.conf to a legacy
> value of
>
> default_ccache_name = /tmp/krb5_%{uid}
>
> in order to force it to use the filesystem rather than the kernel space
> for the credentials cache. And it still fails in the same way. I think
> that tends to makes this more likely a regression in krb5 than the kernel.
>
> I believe that I may have verified this by reverting from
> krb5-*/-/1.18.2-10 to krb5-*-1.17.1-5 which resolves the issue
>
>
>
> On 07/21/2020 3:59 PM, Steve French wrote:
> > Would setting KRB5_CCNAME (as would be expected in the past) for that
> > process fix the issue?
> >
> > On Tue, Jul 21, 2020 at 2:52 PM Michael Keane, K1MK <mkeane at arrl.org> wrote:
> >> KRB5_CCNAME is not set in the environment of the user for whom the
> >> automount is failing.
> >>
> >> In /etc/krb5.conf the default is set to:
> >>
> >> default_ccache_name = KEYRING:persistent:%{uid}
> >>
> >> The calls to keyctl() in the strace for cifs.upcall indicate that it's
> >> successfully locating the user's keyring in the kernel and that it is
> >> able to read key data from that keying
> >>
> >> I haven't gone through and tried unpacking and decoding the contents of
> >> those reads, but the reads aren't failing outright like it couldn't find
> >> the credentials cache or the contents that it's looking for
> >>
> >>
> >> On 07/21/2020 1:36 PM, Steve French wrote:
> >>> That is plausible but I also wonder about other whether other parts of
> >>> krb5 configuration are changed/broken eg KRB5_CCNAME environment variable
> >>>
> >>> See https://web.mit.edu/kerberos/krb5-1.12/doc/basic/ccache_def.html
> >>>
> >>> On Tue, Jul 21, 2020, 12:24 Michael Keane, K1MK <mkeane at arrl.org
> >>> <mailto:mkeane at arrl.org>> wrote:
> >>>
> >>> Thanks
> >>>
> >>> In trying to further debug this problem, I've stood up a separate
> >>> Fedora 31 instance for testing with the result that the same
> >>> configuration that is failing on Fedora 32 is working under Fedora 31
> >>>
> >>> Using strace it would appear the deviation occurs at the point of
> >>> "handle_krb5_mech" where the Fedora 31 instance proceeds though
> >>> several keyctl() calls to a successful return while the Fedora 32
> >>> instance goes through a similar sequence of keyctl() calls but
> >>> rather than finishing up with a final call to keyctl() to
> >>> instantiate the key it appears that the krb5 library and/or sssd
> >>> is going through a series of steps to locate the KDC (and failing)
> >>> even though the KDCs have always been explicitly configured in
> >>> /etc/krb5.conf
> >>>
> >>> So the root cause of this issue may not be anything in cifs.upcall
> >>> but rather something in the sssd or kerberos that got changed as a
> >>> result of moving from Fedora 31 to Fedora 32
> >>>
> >>> On 07/18/2020 8:40 PM, Steve French wrote:
> >>>> Looks like error obtaining the key (either keyutils package not
> >>>> installed or no Kerberos ticket found).
> >>>>
> >>>> This page has debug instructions for cifs.upcall. see of that
> >>>> information is helpful.
> >>>>
> >>>> http://sprabhu.blogspot.com/2014/12/debugging-calls-to-cifsupcall.html?m=1
> >>>>
> >>>> Also try kinit as a local user and then mount with cruid mount
> >>>> option pointing to their uid to see if that helps.
> >>>>
> >>>> On Fri, Jul 17, 2020, 09:04 Michael Keane, K1MK <mkeane at arrl.org
> >>>> <mailto:mkeane at arrl.org>> wrote:
> >>>>
> >>>> CIFS automounts to local NAS devices have stopped working
> >>>> recently
> >>>>
> >>>> Fedora 32
> >>>>
> >>>> kernel 5.7.8-200.fc32.x86_64
> >>>>
> >>>> mount.cifs version: 6.9
> >>>>
> >>>> auto.misc: it-share
> >>>> -fstype=cifs,multiuser,cruid=${UID},rw,vers=default,sec=krb5
> >>>> ://filer5/IT_Share
> >>>>
> >>>> dmesg:
> >>>>
> >>>> [ 3428.883661] fs/cifs/cifsfs.c: Devname:
> >>>> //filer5.arrlhq.org/IT_Share
> >>>> <http://filer5.arrlhq.org/IT_Share> flags: 0
> >>>> [ 3428.883702] fs/cifs/connect.c: Username: root
> >>>> [ 3428.883706] fs/cifs/connect.c: file mode: 0755 dir
> >>>> mode: 0755
> >>>> [ 3428.883709] fs/cifs/connect.c: CIFS VFS: in
> >>>> mount_get_conns as
> >>>> Xid: 12 with uid: 0
> >>>> [ 3428.883710] fs/cifs/connect.c: UNC:
> >>>> \\filer5.arrlhq.org <http://filer5.arrlhq.org>\IT_Share
> >>>> [ 3428.883721] fs/cifs/connect.c: Socket created
> >>>> [ 3428.883723] fs/cifs/connect.c: sndbuf 16384 rcvbuf 131072
> >>>> rcvtimeo 0x1b58
> >>>> [ 3428.884126] fs/cifs/fscache.c:
> >>>> cifs_fscache_get_client_cookie:
> >>>> (0x000000006d2b3226/0x0000000090edec3a)
> >>>> [ 3428.884130] fs/cifs/connect.c: CIFS VFS: in
> >>>> cifs_get_smb_ses as
> >>>> Xid: 13 with uid: 0
> >>>> [ 3428.884131] fs/cifs/connect.c: Existing smb sess not found
> >>>> [ 3428.884135] fs/cifs/smb2pdu.c: Negotiate protocol
> >>>> [ 3428.884141] fs/cifs/connect.c: Demultiplex PID: 5778
> >>>> [ 3428.884156] fs/cifs/transport.c: Sending smb: smb_len=252
> >>>> [ 3429.010818] fs/cifs/connect.c: RFC1002 header 0x11c
> >>>> [ 3429.010830] fs/cifs/smb2misc.c: SMB2 data length 96
> >>>> offset 128
> >>>> [ 3429.010832] fs/cifs/smb2misc.c: SMB2 len 224
> >>>> [ 3429.010835] fs/cifs/smb2misc.c: length of negcontexts
> >>>> 60 pad 0
> >>>> [ 3429.010871] fs/cifs/transport.c: cifs_sync_mid_result:
> >>>> cmd=0
> >>>> mid=0 state=4
> >>>> [ 3429.010885] fs/cifs/misc.c: Null buffer passed to
> >>>> cifs_small_buf_release
> >>>> [ 3429.010891] fs/cifs/smb2pdu.c: mode 0x1
> >>>> [ 3429.010893] fs/cifs/smb2pdu.c: negotiated smb3.1.1 dialect
> >>>> [ 3429.010903] fs/cifs/asn1.c: OID len = 7 oid = 0x1 0x2
> >>>> 0x348 0xbb92
> >>>> [ 3429.010907] fs/cifs/asn1.c: OID len = 7 oid = 0x1 0x2
> >>>> 0x348 0x1bb92
> >>>> [ 3429.010910] fs/cifs/asn1.c: OID len = 10 oid = 0x1 0x3
> >>>> 0x6 0x1
> >>>> [ 3429.010912] fs/cifs/smb2pdu.c: decoding 2 negotiate
> >>>> contexts
> >>>> [ 3429.010914] fs/cifs/smb2pdu.c: decode SMB3.11
> >>>> encryption neg
> >>>> context of len 4
> >>>> [ 3429.010916] fs/cifs/smb2pdu.c: SMB311 cipher type:1
> >>>> [ 3429.010921] fs/cifs/connect.c: Security Mode: 0x1
> >>>> Capabilities:
> >>>> 0x300046 TimeAdjust: 0
> >>>> [ 3429.010923] fs/cifs/smb2pdu.c: Session Setup
> >>>> [ 3429.010926] fs/cifs/smb2pdu.c: sess setup type 5
> >>>> [ 3429.010949] fs/cifs/cifs_spnego.c: key description =
> >>>> ver=0x2;host=filer5.arrlhq.org
> >>>> <http://filer5.arrlhq.org>;ip4=10.1.123.38;sec=krb5;uid=0x0;creduid=0x3e8;user=root;pid=0x1690
> >>>> [ 3429.025053] CIFS VFS: \\filer5.arrlhq.org
> >>>> <http://filer5.arrlhq.org> Send error in SessSetup
> >>>> = -126
> >>>> [ 3429.025056] fs/cifs/connect.c: CIFS VFS: leaving
> >>>> cifs_get_smb_ses
> >>>> (xid = 13) rc = -126
> >>>> [ 3429.025059] fs/cifs/connect.c: build_unc_path_to_root:
> >>>> full_path=\\filer5.arrlhq.org
> >>>> <http://filer5.arrlhq.org>\IT_Share
> >>>> [ 3429.025059] fs/cifs/connect.c: build_unc_path_to_root:
> >>>> full_path=\\filer5.arrlhq.org
> >>>> <http://filer5.arrlhq.org>\IT_Share
> >>>> [ 3429.025060] fs/cifs/connect.c: build_unc_path_to_root:
> >>>> full_path=\\filer5.arrlhq.org
> >>>> <http://filer5.arrlhq.org>\IT_Share
> >>>> [ 3429.025062] fs/cifs/dfs_cache.c: __dfs_cache_find:
> >>>> search path:
> >>>> \filer5.arrlhq.org <http://filer5.arrlhq.org>\IT_Share
> >>>> [ 3429.025063] fs/cifs/dfs_cache.c: get_dfs_referral: get
> >>>> an DFS
> >>>> referral for \filer5.arrlhq.org
> >>>> <http://filer5.arrlhq.org>\IT_Share
> >>>> [ 3429.025065] fs/cifs/dfs_cache.c: dfs_cache_noreq_find:
> >>>> path:
> >>>> \filer5.arrlhq.org <http://filer5.arrlhq.org>\IT_Share
> >>>> [ 3429.025071] fs/cifs/fscache.c:
> >>>> cifs_fscache_release_client_cookie:
> >>>> (0x000000006d2b3226/0x0000000090edec3a)
> >>>> [ 3429.025076] fs/cifs/connect.c: CIFS VFS: leaving
> >>>> mount_put_conns
> >>>> (xid = 12) rc = 0
> >>>> [ 3429.025077] CIFS VFS: cifs_mount failed w/return code = -2
> >>>>
> >>>> journalctl:
> >>>>
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: key
> >>>> description:
> >>>> cifs.spnego;0;0;39010000;ver=0x2;host=filer5.arrlhq.org
> >>>> <http://filer5.arrlhq.org>;ip4=10.1.123.38;sec=krb5;uid=0x0;creduid=0x3e8;user=root;pid=0x1690
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: ver=2
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]:
> >>>> host=filer5.arrlhq.org <http://filer5.arrlhq.org>
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: ip=10.1.123.38
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: sec=1
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: uid=0
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: creduid=1000
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: user=root
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: pid=5776
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]:
> >>>> get_cachename_from_process_env: pathname=/proc/5776/environ
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]:
> >>>> get_existing_cc: default ccache is
> >>>> KEYRING:persistent:1000:1000
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]:
> >>>> handle_krb5_mech: getting service ticket for
> >>>> filer5.arrlhq.org <http://filer5.arrlhq.org>
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]:
> >>>> cifs_krb5_get_req: unable to get credentials for
> >>>> filer5.arrlhq.org <http://filer5.arrlhq.org>
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]:
> >>>> handle_krb5_mech: failed to obtain service ticket
> >>>> (-1765328370)
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: Unable to
> >>>> obtain service ticket
> >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG
> >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: Exit status
> >>>> -1765328370
> >>>>
> >>>> $ klist
> >>>> Ticket cache: KEYRING:persistent:1000:1000
> >>>> Default principal: mkeane at ARRLHQ.ORG <mailto:mkeane at ARRLHQ.ORG>
> >>>>
> >>>> Valid starting Expires Service principal
> >>>> 07/17/2020 09:43:06 07/17/2020 19:43:06
> >>>> krbtgt/ARRLHQ.ORG at ARRLHQ.ORG <mailto:ARRLHQ.ORG at ARRLHQ.ORG>
> >>>> renew until 07/24/2020 09:42:57
> >>>>
> >>>> Filer5 is a QNAP TS-870U-RP Version 4.3.6.1070 (2019/09/10)
> >>>> NAS device
> >>>> but having this issue with other NAS device on LAN
> >>>>
> >>>> --
> >>>> Michael Keane, K1MK
> >>>> IT Manager
> >>>> ARRL, The National Association for Amateur Radio™
> >>>> 225 Main Street, Newington, CT 06111-1494 USA
> >>>> Telephone: (860) 594-0285
> >>>> email: mkeane at arrl.org <mailto:mkeane at arrl.org>
> >>>>
> >>> --
> >>> Michael Keane, K1MK
> >>> IT Manager
> >>> ARRL, The National Association for Amateur Radio™
> >>> 225 Main Street, Newington, CT 06111-1494 USA
> >>> Telephone: (860) 594-0285
> >>> email:mkeane at arrl.org <mailto:mkeane at arrl.org>
> >>>
> >> --
> >> Michael Keane, K1MK
> >> IT Manager
> >> ARRL, The National Association for Amateur Radio™
> >> 225 Main Street, Newington, CT 06111-1494 USA
> >> Telephone: (860) 594-0285
> >> email: mkeane at arrl.org
> >>
> >
>
> --
> Michael Keane, K1MK
> IT Manager
> ARRL, The National Association for Amateur Radio™
> 225 Main Street, Newington, CT 06111-1494 USA
> Telephone: (860) 594-0285
> email: mkeane at arrl.org
>
--
Thanks,
Steve
More information about the samba-technical
mailing list