error allocating core memory buffers (code 22) at util2.c(106) [sender=3.1.2]

John Hawkinson jhawk at MIT.EDU
Tue Apr 11 12:33:50 UTC 2017

> I guess I can turn on core dumps and increase (unlimit completely) the 
> stack size...
> Although it doesn't seem to have segfaulted, so I'm not sure having
> core dumps enabled would have helped?

Indeed. I reran it on the 50 remaining individual directories,
it seems to have made it through 10 before failing again overnight:

66596  20:38:03 rsync -vP -iaHJAX  `cat /tmp/unback`  /Volumes/platinum-barratry/x/Backups.backupdb/pb3/

rsync: unpack_smb_acl: sys_acl_get_info(): Undefined error: 0 (0)
rsync: unpack_smb_acl: sys_acl_get_info(): Undefined error: 0 (0)
.f....og.a.. 2015-06-19-072520/platinum-bar2/Users/jhawk/.emacs.d/auto-save-list/.saves-1541-platinum-bar2.local~
ERROR: out of memory in expand_item_list [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(106) [sender=3.1.2]

No segfault, no coredump, no syslogs.

Do I need to run this under lldb and set a breakpoint in expand_item_list()?
Quick inspection suggests running with -vvvv might give some useful output:

1680         if (DEBUG_GTE(FLIST, 3)) {
1681                 rprintf(FINFO, "[%s] expand %s to %s bytes, did%s move\n",
1682                         who_am_i(), desc, big_num(new_size * item_size),
1683                         new_ptr == lp->items ? " not" : "");
1684         }
1685         if (!new_ptr)
1686                 out_of_memory("expand_item_list");

Although I imagine that output might be voluminous [but maybe not]?

Again, I don't have time to build test cases and reproduce this
carefully, every run is painful and long and slow. But I'd like to do the
responsible thing if someone can tell me what that is.

> p.s.: If I had to start over, I would have spent less time just deleting
> the data and recopying it, rather than trying to fixup the metadata and

Indeed, it's looking like fixing the metadata with rsync is an order
of magnitude slower, even as far as I've gotten. So maybe it's time to find
another method. I don't think fts(3) is optimized any better for large
hardlink farms, so I think maybe I need a homegrown solution? Ughhhh.

--jhawk at
  John Hawkinson

More information about the rsync mailing list