memory usage and performance during schema modifications in samba4

[Matthias Dieter Wallnöfer]

Hi Anatoliy,

I push parts of the memory fixes patch ("auth/session.c" and 
"schema/schema_set.c") since especially the latter one is a really 
serious problem.

The "ldb_dn.c" one is unnecessary since "dn->components" is a child of 
the "dn" context (and also "dn->ext_components" names are allocated on 
"dn->ext_components" which is also a child of "dn").

Also the "ldb_msg.c" patch doesn't bring benefits as far as I can see 
and freeing "msg2" on the end is also completely wrong (who says that we 
don't want to have it available after this call?).

Matthias

Anatoliy Atanasov wrote:
> Hi List,
>
> I've stumbled upon an performance issue with samba4 during schema modifications so i've been busy lately looking into it. I couldn't figure out and solve the issue by myself i'll be happy if someone has time to help me resolve it.
>
> I attached some useful scripts and patches that i did working on this.
> 1. I prepared a script, which is in the called flood_schema.py
> run it against samba process:
> python lib/ldb/tests/python/flood_schema.py ldap://localhost:389 -UAdministrator%pass
>
> you can compare this against windows virtual box, the diff in the speed is
> 	win2k8		samba
> real	0m0.536s	0m7.469s
>
> also running the script will cause huge increase in the memory usage: 250 adds are ~ 1G of memory
>
> 2. valgrind log after running the script - valgrind_flood.txt
> kamen and i did some fixes(0001-mem-fixes.patch) to reduce mem usage but this didn't fix the bigger problem
>
> 3. i did run this script profiling with google perf tools
> 	* first patch to modify the waf build to link google perf tools - 0001-modify-waf....(this patch needs code review)
> 	* cpu.gif and memcheck.gif are the profiles i did on my machine
>
>
> Looking at valgrind logs kamen thinks we load schemas in the context that we never free because of too much references so this might be it but i don't understand the area enough to debug and fix it.
>
> Regards,
> Anatoliy