Samba write performance anomaly

[Kenichi Okuyama]

>>>>> "DC" == David Collier-Brown <David.Collier-Brown at canada.sun.com> writes:
>> Every few months I get sick of the miserable write performance on my
>> Samba servers and vow to fix it.  It's that time again.
DC> 	And he was kind enough to provide enough information
DC> 	to expose some rather **odd** symptoms....

I'm running Samba v2.0.7 on Linux kernel 2.3.99-pre6 on ThinkPad535E.

I've created samba with 
 "-pg + forcing static link + gprof library + patch of my own"
and looked at result of "access with one client" result.

# I had to force to link libc_p.a to make gprof enabled for libc.
# -pg linking option did not seems to work. Can anybody give me a
#clue about correct way of creating "gprof" version of samba?


I've placed results on 

http://www.dd.iij4u.or.jp/~okuyamak/FreeSoft/samba/

There are four files there.

SimplyOpen.orig.gprof.bz2
	for only this profile, I run it on samba without "patch of
	my own." So, this is result of original samba-2.0.7.
	What I did was, run smbd on server, then at windows NT
	workstation, I've clicked an icon descibing the server.
	NT showed the first window listing netlogin directory,
	then I've closed the window and waited for 1 min., which
	I setted at "deadtime".
	Then by using gmon.out being created, I run gprof smbd.

SimplyOpen.new.gprof.bz2
	What I did was exactly the same as what I did with
	"SimplyOpen.orig.gprof.bz2", except this one is 
	"patched" smbd.

Move-NetNews-Linux2NT.new.gprof.bz2
	I had files of
	  24987kbytes total
	  554 files
	  30 directories
        in a single directory. I've moved this entire directory from
        samba-server to local NT directory.
	samba-server was one with my patch.

Move-NetNews-NT2Linux.new.gprof.bz2
	I've copied the same directory as mentioned on
	"Move-NetNews-Linux2NT.new.gprof.bz2", only this time
	from NT local disk to Linux samba-server.


As you can see, even though total run time is quite small in samba,
I found several keypoints.

1) __mcount_internal is being called extreamely frequently.
  This function itself is created by compiler with -pg options.
  So, this does not effect performance. But, very fact that this
  function being called many times, means that you are doing
  function calls extreamely many times. We should check for
  function call overheads, and reduce them by re-checking the
  algorithms as well as inlining them.
#Ying Chen @ IBM's work with inlining functions do seems
#to work great, but there are extreamely more places to do.

2) skip_kanji_multibyte_char is being called many times.
  And as much, string manipulation functions are being called
  many, many, many, times. Information of strlen() are being thrown
  away, strings are being copied to many places. getc() calling
  overhead (this includes flockfile/funlockfile calls inside
  libc).

  This bad habits exists in both "smb.conf parsing" unit and
  "directory/filename" treatings. They will come up as "overhead"
  of very first connections( I mean when smbd fork, for child
  process of smbd will re-scan smb.conf again ).


I beleave there must be more and more characteristic cases hidden in
these profile results, though I have not found them yet.


You can get my patch from 

http://www.dd.iij4u.or.jp/~okuyamak/FreeSoft/samba-2.0.7.patch.bz2



DC> 	Anyone recognize a cause for slow write-to-server performance?

I've heard about one thing on FreeBSD. 

"If you set
  net.inet.tcp.delayed_ack to 0 using sysctl,
  file transfer of windows->samba will become faster."

I've not found the reasons yet.

Usually this will loose performance with usual tcp/ip
communications. 5 to 10% of communication speeds becomes slower
between FreeBSD/Linux-es. But as described on

http://www.dd.iij4u.or.jp/~okuyamak/Documents/tuning.english.html

winsock2 is not "usual."

The bad news about this, is that there are no such thing as
"delayd_ack" on Linux. There must be something quite similar to
this, but I could not find any.


Wishing that these information might be of any help to you.
regards,
---- 
Kenichi Okuyama