[Bug 12769] error allocating core memory buffers (code 22) depending on source file system

Dave Gordon dg32768 at zoho.eu
Fri Oct 18 22:57:54 UTC 2019


On 11/10/2019 13:53, just subscribed for rsync-qa from bugzilla via
rsync wrote:
> https://bugzilla.samba.org/show_bug.cgi?id=12769
>
> --- Comment #5 from Simon Matter <simon.matter at invoca.ch> ---
> I'm suffering the same problem and was wondering if anyone found a solution or
> work around or other tool to do the job?
>
> First I thought maybe it's a bug which is fixed already and tried with the
> latest release 3.1.3. Unfortunately no joy and I still get the same issue.
>
> Any help would be much appreciated!
The hash table doubles each time it reaches 75% full.
A hash table for 32m items @ 16 bytes each (8 byte key, 8 byte void *
data) needs 512MB of memory.
At the next doubling (up to 64m items) it hits the array allocator limit
in utils2.c:

#define MALLOC_MAX 0x40000000

void *_new_array(unsigned long num, unsigned int size, int use_calloc)
{
        if (num >= MALLOC_MAX/size)
                return NULL;
        return use_calloc ? calloc(num, size) : malloc(num * size);
}
void *_realloc_array(void *ptr, unsigned int size, size_t num)
{
        if (num >= MALLOC_MAX/size)
                return NULL;
        if (!ptr)
                return malloc(size * num);
        return realloc(ptr, size * num);
}

No single array allocation or reallocation is allowed to exceed
MALLOC_MAX (1GB).
Hence rsync can only handle up to 32m items per invocation if a hash
table is required (e.g. for tracking hardlinks when -H is specified).

HTH,
Dave





More information about the rsync mailing list