Samba write performance anomaly

[J. Robert von Behren]

Greetings, all - 

I've actually just started doing some profiling of Samba performance
under Linux.  I haven't gathered any interesting results yet, but I have
put together a patch to make turning on profiling easier.  In all, the
patch does the following:

   * registers a function with atexit() to twidle filenames, and prevent
the various smbd processes from overwriting each other's gmon.out files.
   * Adjusts the configure.in and Makefile.in files to include profiling
   * adds a --with-gprof-support option to configure, to allow this to
be turned on and off.

I'd like to submit the patch for inclusion in the main source tree (it
is pretty unobtrusive ;-).  Is there a standard way that the Samba
community usually does this?

Best regards,

-Rob von Behren


P.S.  I'm planning on instrumenting the Linux kernel with SGI's kernel
profiling patches, so I can monitor kernel activity under Samba.  I'll
post any interesting findings to the list.  



Kenichi Okuyama wrote:
> 
> >>>>> "DC" == David Collier-Brown <David.Collier-Brown at canada.sun.com> writes:
> >> Every few months I get sick of the miserable write performance on my
> >> Samba servers and vow to fix it.  It's that time again.
> DC>     And he was kind enough to provide enough information
> DC>     to expose some rather **odd** symptoms....
> 
> I'm running Samba v2.0.7 on Linux kernel 2.3.99-pre6 on ThinkPad535E.
> 
> I've created samba with
>  "-pg + forcing static link + gprof library + patch of my own"
> and looked at result of "access with one client" result.
> 
> # I had to force to link libc_p.a to make gprof enabled for libc.
> # -pg linking option did not seems to work. Can anybody give me a
> #clue about correct way of creating "gprof" version of samba?
> 
> I've placed results on
> 
> http://www.dd.iij4u.or.jp/~okuyamak/FreeSoft/samba/
> 
> There are four files there.
> 
> SimplyOpen.orig.gprof.bz2
>         for only this profile, I run it on samba without "patch of
>         my own." So, this is result of original samba-2.0.7.
>         What I did was, run smbd on server, then at windows NT
>         workstation, I've clicked an icon descibing the server.
>         NT showed the first window listing netlogin directory,
>         then I've closed the window and waited for 1 min., which
>         I setted at "deadtime".
>         Then by using gmon.out being created, I run gprof smbd.
> 
> SimplyOpen.new.gprof.bz2
>         What I did was exactly the same as what I did with
>         "SimplyOpen.orig.gprof.bz2", except this one is
>         "patched" smbd.
> 
> Move-NetNews-Linux2NT.new.gprof.bz2
>         I had files of
>           24987kbytes total
>           554 files
>           30 directories
>         in a single directory. I've moved this entire directory from
>         samba-server to local NT directory.
>         samba-server was one with my patch.
> 
> Move-NetNews-NT2Linux.new.gprof.bz2
>         I've copied the same directory as mentioned on
>         "Move-NetNews-Linux2NT.new.gprof.bz2", only this time
>         from NT local disk to Linux samba-server.
> 
> As you can see, even though total run time is quite small in samba,
> I found several keypoints.
> 
> 1) __mcount_internal is being called extreamely frequently.
>   This function itself is created by compiler with -pg options.
>   So, this does not effect performance. But, very fact that this
>   function being called many times, means that you are doing
>   function calls extreamely many times. We should check for
>   function call overheads, and reduce them by re-checking the
>   algorithms as well as inlining them.
> #Ying Chen @ IBM's work with inlining functions do seems
> #to work great, but there are extreamely more places to do.
> 
> 2) skip_kanji_multibyte_char is being called many times.
>   And as much, string manipulation functions are being called
>   many, many, many, times. Information of strlen() are being thrown
>   away, strings are being copied to many places. getc() calling
>   overhead (this includes flockfile/funlockfile calls inside
>   libc).
> 
>   This bad habits exists in both "smb.conf parsing" unit and
>   "directory/filename" treatings. They will come up as "overhead"
>   of very first connections( I mean when smbd fork, for child
>   process of smbd will re-scan smb.conf again ).
> 
> I beleave there must be more and more characteristic cases hidden in
> these profile results, though I have not found them yet.
> 
> You can get my patch from
> 
> http://www.dd.iij4u.or.jp/~okuyamak/FreeSoft/samba-2.0.7.patch.bz2
> 
> DC>     Anyone recognize a cause for slow write-to-server performance?
> 
> I've heard about one thing on FreeBSD.
> 
> "If you set
>   net.inet.tcp.delayed_ack to 0 using sysctl,
>   file transfer of windows->samba will become faster."
> 
> I've not found the reasons yet.
> 
> Usually this will loose performance with usual tcp/ip
> communications. 5 to 10% of communication speeds becomes slower
> between FreeBSD/Linux-es. But as described on
> 
> http://www.dd.iij4u.or.jp/~okuyamak/Documents/tuning.english.html
> 
> winsock2 is not "usual."
> 
> The bad news about this, is that there are no such thing as
> "delayd_ack" on Linux. There must be something quite similar to
> this, but I could not find any.
> 
> Wishing that these information might be of any help to you.
> regards,
> ----
> Kenichi Okuyama