[Samba] Crash in 4.12.10 in (also 4.13.4)

Peter Eriksson pen at lysator.liu.se
Wed Jan 27 12:28:07 UTC 2021 

Ah! Found it! The problem occurs when:

[2021/01/27 12:43:28.106396,  0] ../../source3/lib/idmap_cache.c:60(idmap_cache_find_sid2unixid)
  Parsing value for key [IDMAP/SID2XID/S-1-18-1]: value=[-1:N]
[2021/01/27 12:43:28.106428,  1] ../../source3/passdb/lookup_sid.c:1283(sids_to_unixids)
  ID_TYPE_NOT_SPECIFIED and NOT expired: i=17, rid=-1)

Ie, when we have a non-expired negative idmap cache entry!

Looking at the code in idmap_cache.c:idmap_cache_find_sid2unixid() it’s not so surprising. When it finds a negative cache entry(-1:N) that hasn’t expired it sets the type to ID_TYPE_NOT_SPECIFIED and returns success - which then triggers the bug in lookup_sid.c:sids_to_unixids()…

Possibly one should use some other key to distinguish between non-expired negative cache entries and not-yet-looked-up entries, but one easy fix to avoid the buffer overrun is:

-- source3/passdb/lookup_sid.c.ORIG    2021-01-26 16:25:59.235435000 +0100
+++ source3/passdb/lookup_sid.c 2021-01-27 13:18:20.264864000 +0100
@@ -1277,7 +1278,8 @@
                if (idmap_cache_find_sid2unixid(&sids[i], &ids[i], &expired)
                    && !expired)
                {
-                       continue;
+                       if (ids[i].type != ID_TYPE_NOT_SPECIFIED)
+                               continue;
                }
                ids[i].type = ID_TYPE_NOT_SPECIFIED;
                memcpy(&wbc_sids[num_not_cached], &sids[i],

This kinds of defeats the purpose of the negative cache entry - but fixing that probably requires adding a new “ID_TYPE_NEGATIVE_MATCH” type or something (so one can avoid the non-expired Sid being looked up again with wbcSidsToUnixIds().

- Peter




> On 26 Jan 2021, at 20:16, Peter Eriksson via samba <samba at lists.samba.org> wrote:
> 
> One way to force this to happen (for me atleast) is to simply kill winbindd and restart it (without restarting smbd).
> 
> A wild guess is that the pipe between smbd & winbindd being used for the communication gets closed prematurely and then some buffer that was supposed to contain some result from winbindd contains just NUL’s - which matches ID_TYPE_NOT_SPECIFIED (0)…
> 
> Anyway, I’ve uploaded a small patch that makes this function a bit more robust and so far my sambas behave better..
> It’s available in the bugzilla:
> 
>  https://bugzilla.samba.org/show_bug.cgi?id=14571 <https://bugzilla.samba.org/show_bug.cgi?id=14571> <https://bugzilla.samba.org/show_bug.cgi?id=14571 <https://bugzilla.samba.org/show_bug.cgi?id=14571>>
> 
> - Peter
> 
> 
>> On 26 Jan 2021, at 14:32, Peter Eriksson via samba <samba at lists.samba.org <mailto:samba at lists.samba.org> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org>>> wrote:
>> 
>> Added an assert that checks if we over-run the wbc_sids[] array and now 4.13.4 dies immediately when I try it.
>> 
>> Looking at the code, the suspicious code is around line 1277 in source3/passwd/lookup_sid.c:
>> 
>> 
>>               if (sid_peek_check_rid(&global_sid_Unix_Groups,
>>                                      &sids[i], &rid)) {
>>                       ids[i].type = ID_TYPE_GID;
>>                       ids[i].id = rid;
>>                       continue;
>>               }
>> —>          if (idmap_cache_find_sid2unixid(&sids[I], &ids[i], &expired)
>>                   && !expired)
>>               {
>>                       continue;
>>               }
>>               ids[i].type = ID_TYPE_NOT_SPECIFIED;
>>               memcpy(&wbc_sids[num_not_cached], &sids[i],
>>                      ndr_size_dom_sid(&sids[i], 0));
>>               num_not_cached += 1;
>> 
>> If that idmap_cache_find() call, or !expired is true - and ids[I].type happens to already be ID_TYPE_NOT_SPECIFIED then num_not_cached will not be incremented, but the loop further down will detect it and then the buffer overrun will happen…
>> 
>> - Peter
>> 
>> 
>> 
>>> On 26 Jan 2021, at 13:04, Peter Eriksson via samba <samba at lists.samba.org <mailto:samba at lists.samba.org> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org>> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org>>>> wrote:
>>> 
>>> It seems to be crashing in source3/passwd/lookup_sids.c line 1307:
>>> 
>>>> switch (wbc_ids[num_not_cached].type) {
>>> 
>>> Due it trying to access outside the allocated data area…
>>> 
>>> (gdb) print num_not_cached
>>> $23 = 68
>>> (gdb) print num_sids
>>> $24 = 245
>>> (gdb) print wbc_ids[66]
>>> $21 = {type = WBC_ID_TYPE_NOT_SPECIFIED, id = {uid = 0, gid = 0}}
>>> (gdb) print wbc_ids[67]
>>> $22 = {type = WBC_ID_TYPE_NOT_SPECIFIED, id = {uid = 48, gid = 48}}
>>> (gdb) print wbc_ids[68]
>>> Cannot access memory at address 0x81b339000
>>> 
>>> wbc_ids is talloc_array():n at line 1290, but unfortunately the num_not_cached variable is reused so I can’t in the core dump see how many entries actually was allocated there. I’ll recompile add a debugging assertion check to see what’s happening there.
>>> 
>>> The user that was connected to the smbd at the time of the crash is a member of some 90 AD groups, of which 24 has gidNumber set. Dunno if that’s relevant for this case but anyway.
>>> 
>>> I’ll try to do some more debugging.
>>> 
>>> https://bugzilla.samba.org/show_bug.cgi?id=14571 <https://bugzilla.samba.org/show_bug.cgi?id=14571> <https://bugzilla.samba.org/show_bug.cgi?id=14571 <https://bugzilla.samba.org/show_bug.cgi?id=14571>> <https://bugzilla.samba.org/show_bug.cgi?id=14571 <https://bugzilla.samba.org/show_bug.cgi?id=14571> <https://bugzilla.samba.org/show_bug.cgi?id=14571 <https://bugzilla.samba.org/show_bug.cgi?id=14571>>> <https://bugzilla.samba.org/show_bug.cgi?id=14571 <https://bugzilla.samba.org/show_bug.cgi?id=14571> <https://bugzilla.samba.org/show_bug.cgi?id=14571 <https://bugzilla.samba.org/show_bug.cgi?id=14571>> <https://bugzilla.samba.org/show_bug.cgi?id=14571 <https://bugzilla.samba.org/show_bug.cgi?id=14571> <https://bugzilla.samba.org/show_bug.cgi?id=14571 <https://bugzilla.samba.org/show_bug.cgi?id=14571>>>>
>>> 
>>> - Peter
>>> 
>>> 
>>> 
>>>> On 11 Nov 2020, at 12:43, Andrew Walker via samba <samba at lists.samba.org <mailto:samba at lists.samba.org> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org>> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org>>> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org>> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org>>>>> wrote:
>>> 
>>>> On Tue, Nov 10, 2020 at 5:01 PM Peter Eriksson via samba <
>>>> samba at lists.samba.org <mailto:samba at lists.samba.org> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org>> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org>>> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org>> <mailto:samba at lists.samba.org <mailto:samba at lists.samba.org><mailto:samba at lists.samba.org <mailto:samba at lists.samba.org>>>>> wrote:
>>>> 
>>>>> I just got an INTERNAL ERROR: Signal 11 in smbd (4.12.10) in something
>>>>> that sids_to_unixids() in source3/winbindd/idmap_hash/idmap_has.c calls and
>>>>> 3 levels down - unfortunately the stack trace doesn’t say what it is -
>>>>> probably optimised into inline code or something.
>>>>> 
>>>>> Recently upgraded from Samba 4.12.5 to 4.12.10 (self-compiled). FreeBSD
>>>>> 12.2
>>>>> 
>>>>> It happened right after 10 hours since that smbd processes started so the
>>>>> 10 hours Kerberos ticket lifetime is probably involved somehow…
>>>>> 
>>>>> Nov 10 21:39:11 runur01 smbd_audit[23768]:    #3 sig_fault + 0x6c
>>>>> [ip=0x80129a7a9] [sp=0x7fffffffcbb0]
>>>>> Nov 10 21:39:11 runur01 smbd_audit[23768]:    #4 <unknown symbol>
>>>>> [ip=0x801517b70] [sp=0x7fffffffcbc0]
>>>>> Nov 10 21:39:12 runur01 smbd_audit[23768]:    #5 <unknown symbol>
>>>>> [ip=0x80151713f] [sp=0x7fffffffcf80]
>>>>> Nov 10 21:39:12 runur01 smbd_audit[23768]:    #6 <unknown symbol>
>>>>> [ip=0x7ffffffff003] [sp=0x7fffffffcff0]
>>>>> Nov 10 21:39:12 runur01 smbd_audit[23768]:    #7 sids_to_unixids + 0x25d
>>>>> 
>>>>> Unfortunately no core dump :-(
>>>>> 
>>>> You may need to run the command "sysctl kern.sugid_coredump=1" and also set
>>>> kern.corefile to an appropriate path (for example /tmp/%N_%P.core -- this
>>>> ensures program name and pid are in corefile name).  If possible compile
>>>> without optimizations and with debugging symbols (this will improve
>>>> visibility of error).
>>>> -- 
>>>> To unsubscribe from this list go to the following URL and read the
>>>> instructions:  https://lists.samba.org/mailman/options/samba <https://lists.samba.org/mailman/options/samba> <https://lists.samba.org/mailman/options/samba <https://lists.samba.org/mailman/options/samba>> <https://lists.samba.org/mailman/options/samba <https://lists.samba.org/mailman/options/samba> <https://lists.samba.org/mailman/options/samba <https://lists.samba.org/mailman/options/samba>>> <https://lists.samba.org/mailman/options/samba <https://lists.samba.org/mailman/options/samba> <https://lists.samba.org/mailman/options/samba <https://lists.samba.org/mailman/options/samba>> <https://lists.samba.org/mailman/options/samba <https://lists.samba.org/mailman/options/samba> <https://lists.samba.org/mailman/options/samba <https://lists.samba.org/mailman/options/samba>>>>
>>> -- 
>>> To unsubscribe from this list go to the following URL and read the
>>> instructions:  https://lists.samba.org/mailman/options/samba <https://lists.samba.org/mailman/options/samba> <https://lists.samba.org/mailman/options/samba <https://lists.samba.org/mailman/options/samba>> <https://lists.samba.org/mailman/options/samba <https://lists.samba.org/mailman/options/samba> <https://lists.samba.org/mailman/options/samba <https://lists.samba.org/mailman/options/samba>>>
>> -- 
>> To unsubscribe from this list go to the following URL and read the
>> instructions:  https://lists.samba.org/mailman/options/samba <https://lists.samba.org/mailman/options/samba> <https://lists.samba.org/mailman/options/samba <https://lists.samba.org/mailman/options/samba>>
> -- 
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba <https://lists.samba.org/mailman/options/samba>

More information about the samba mailing list