Samba3 memory usage, iconv

[Dmitry Shatrov]

Profiling smbd memory usage with valgrind --tool=massif gives 
interesting results.
It turns out that quite a lot of memory (~300 Kbytes) is allocated right 
at the start of each smbd process by init_iconv() (lib/charcnv.c).

I think that this must be something like conversion tables, which could 
be shared between smbds giving ~300KB per smbd.

Here's the output from massif for current samba3 from svn (here, I ran 
smbd and performed mount.cifs //localhost/... after a while):
http://img30.picoodle.com/img/img30/8/7/2/f_massifm_95d3e94.png

massif output for a clean smbd start:
http://img37.picoodle.com/img/img37/8/7/2/f_smbdm_c99cc1c.png
...and the most relevant backtraces:

Context accounted for 43.2% of measured spacetime
  0x81AD8B: __gconv_open (in /lib/tls/libc-2.3.4.so)
  0x81A8C4: iconv_open (in /lib/tls/libc-2.3.4.so)
  0x2A0C8E: smb_iconv_open (iconv.c:245)
  0x2763D7: init_iconv (charcnv.c:156)
  0x774C9: lp_load (loadparm.c:5410)
  0x49723C: reload_services (server.c:644)
  0x497ED0: main (server.c:964)
  0x819DE2: (below main) (in /lib/tls/libc-2.3.4.so)

Context accounted for 43.1% of measured spacetime
  0x81AD8B: __gconv_open (in /lib/tls/libc-2.3.4.so)
  0x81A8C4: iconv_open (in /lib/tls/libc-2.3.4.so)
  0x2A0C31: smb_iconv_open (iconv.c:237)
  0x2763D7: init_iconv (charcnv.c:156)
  0x774C9: lp_load (loadparm.c:5410)
  0x49723C: reload_services (server.c:644)
  0x497ED0: main (server.c:964)
  0x819DE2: (below main) (in /lib/tls/libc-2.3.4.so)

For comparison, here are massif profiling results for a simple program 
which calls iconv_open ("CP866", "UTF8"), sleeps for a second and then 
exits:
http://img29.picoodle.com/img/img29/8/7/2/f_hellom_46acf01.png
according to massif, iconv_open() costs about 30 Kbytes of memory in 
this case.

Putting fprintf before smb_iconv_open() at charcnv.c:156 shows that 
iconv gets initialized for the following conversion pairs (looks like 
quite a lot):
UTF-16LE, UTF-16LE
UTF-8, UTF-16LE
UTF-8, UTF-16LE
CP850, UTF-16LE
UTF8, UTF-16LE
UTF-16BE, UTF-16LE
UTF-16LE, UTF-8
UTF-8, UTF-8
UTF-8, UTF-8
CP850, UTF-8
UTF8, UTF-8
UTF-16BE, UTF-8
UTF-16LE, UTF-8
UTF-8, UTF-8
UTF-8, UTF-8
CP850, UTF-8
UTF8, UTF-8
UTF-16BE, UTF-8
UTF-16LE, CP850
UTF-8, CP850
UTF-8, CP850
CP850, CP850
UTF8, CP850
UTF-16BE, CP850
UTF-16LE, UTF8
UTF-8, UTF8
UTF-8, UTF8
CP850, UTF8
UTF8, UTF8
UTF-16BE, UTF8
UTF-16LE, UTF-16BE
UTF-8, UTF-16BE
UTF-8, UTF-16BE
CP850, UTF-16BE
UTF8, UTF-16BE
UTF-16BE, UTF-16BE

I would deal with this by trying to share as much of common data between 
smbds as possible by shared memory or file mmap'ing.
Your suggestions?

Btw, some useful information on memory usage reduction can be found at 
live.gnome.org:
http://live.gnome.org/MemoryReduction
There's a link to a compact overview of using massif, too:
http://developer.gnome.org/doc/guides/optimisation/Massif.html

Best regards,
Dmitry Shatrov