memory usage and performance during schema modifications in samba4

[Anatoliy Atanasov]

Hi List, 

Quick update: Andrew Bartlett found a way to unlink dsdb_schema objects in the ldb context so now the memory consumption is back to normal*.
There are still lots of ldb_msg objects hanging in the ldb that should probably be unlinked too or shouldn't be in that context.

Regards,
Anatoliy
> -----Original Message-----
> From: samba-technical-bounces at lists.samba.org [mailto:samba-technical-
> bounces at lists.samba.org] On Behalf Of Anatoliy Atanasov
> Sent: Tuesday, June 29, 2010 8:11 PM
> To: samba-technical at samba.org
> Subject: memory usage and performance during schema modifications in
> samba4
> 
> Hi List,
> 
> I've stumbled upon an performance issue with samba4 during schema
> modifications so i've been busy lately looking into it. I couldn't figure out
> and solve the issue by myself i'll be happy if someone has time to help me
> resolve it.
> 
> I attached some useful scripts and patches that i did working on this.
> 1. I prepared a script, which is in the called flood_schema.py run it against
> samba process:
> python lib/ldb/tests/python/flood_schema.py ldap://localhost:389 -
> UAdministrator%pass
> 
> you can compare this against windows virtual box, the diff in the speed is
> 	win2k8		samba
> real	0m0.536s	0m7.469s
> 
> also running the script will cause huge increase in the memory usage: 250
> adds are ~ 1G of memory
> 
> 2. valgrind log after running the script - valgrind_flood.txt kamen and i did
> some fixes(0001-mem-fixes.patch) to reduce mem usage but this didn't fix
> the bigger problem
> 
> 3. i did run this script profiling with google perf tools
> 	* first patch to modify the waf build to link google perf tools - 0001-
> modify-waf....(this patch needs code review)
> 	* cpu.gif and memcheck.gif are the profiles i did on my machine
> 
> 
> Looking at valgrind logs kamen thinks we load schemas in the context that
> we never free because of too much references so this might be it but i don't
> understand the area enough to debug and fix it.
> 
> Regards,
> Anatoliy