[distcc] What's this about dlopen?

[Martin Pool]

On 27 Apr 2003, Aaron Lehmann <aaronl at vitelus.com> wrote:
> > static linking
> >
> >     cachegrind shows that a large fraction of client runtime is spent in the
> >     dynamic linker, which is kind of a waste.  In principle using dietlibc
> >     might reduce the fixed overhead of the client.  However, the nsswitch
> >     functions are always dynamically linked: even if we try to produce a
> >     static client it will include dlopen and eventually indirectly get libc,
> >     so it's probably not practical.
> 
> I don't understand why this would be the case. I was curious, so I
> linked distcc against dietlibc (I also had to link popt against it). I
> got a binary that was about 1/8 the size of a statically linked distcc
> using glibc. I did not see any dlopen's in the strace output, and it
> appeared to run very fast (although so did the dynamically linked
> version ;) ).

I must have made a mistake before.  When I did a build now (by just
setting CC='diet gcc'), then dietlibc seems to do the most efficient
thing: reading /etc/hosts and if that misses then reading
/etc/resolv.conf.

open("/tmp/distcc_000003e9/lock_cpu_tcp_nevada.frungy.org_3632_0", O_WRONLY|O_CREAT, 0600) = 3
open("/tmp/distcc_000003e9/lock_xmit_tcp_nevada.frungy.org_3632_0", O_WRONLY|O_CREAT, 0600) = 4
open("/etc/hosts", O_RDONLY)            = 6
open("/etc/resolv.conf", O_RDONLY)      = 6
open("/tmp/distcc_000003e9/cppout_0000018856.i", O_RDONLY) = 7
open("hello.o", O_WRONLY|O_CREAT|O_TRUNC|O_EXCL, 0666) = 4

By contrast, distcc linked to glibc needs to pull in a lot more files:

open("/etc/ld.so.preload", O_RDONLY)    = 3
open("/etc/ld.so.cache", O_RDONLY)      = 3
open("/lib/libpopt.so.0", O_RDONLY)     = 3
open("/usr/lib/debug/libc.so.6", O_RDONLY) = 3
open("/tmp/distcc_000003e9/lock_cpu_tcp_nevada.frungy.org_3632_0", O_WRONLY|O_CREAT, 0600) = 3
open("/tmp/distcc_000003e9/lock_xmit_tcp_nevada.frungy.org_3632_0", O_WRONLY|O_CREAT, 0600) = 4
open("/etc/resolv.conf", O_RDONLY)      = 6
open("/etc/nsswitch.conf", O_RDONLY)    = 6
open("/etc/ld.so.cache", O_RDONLY)      = 6
open("/usr/lib/debug/libnss_files.so.2", O_RDONLY) = 6
open("/etc/host.conf", O_RDONLY)        = 6
open("/etc/hosts", O_RDONLY)            = 6
open("/etc/ld.so.cache", O_RDONLY)      = 6
open("/usr/lib/debug/libnss_dns.so.2", O_RDONLY) = 6
open("/usr/lib/debug/libresolv.so.2", O_RDONLY) = 6
open("/tmp/distcc_000003e9/cppout_0000021506.i", O_RDONLY) = 6
open("hello.o", O_WRONLY|O_CREAT|O_TRUNC|O_EXCL, 0666) = 4

Of course NSS exists for a good reason, but all that indirection must
come at a price.

It might be interesting to measure it using the i386 performance
monitoring counters, but I don't know of a tool that easily exposes
them.  I suppose there is one.

The static dietlibc client is about 70k and the server 85k on my
machine.

-- 
Martin