[Samba] high load averages on large file transfers

[Dexter Santucci]

Hello,

  I work for a production studio in Canada. I have a small problem I thought
I’d submit as it is now affecting our work. First, some background: Our
production server is a Dell Poweredge 2800 (dual-xeon, dual-core) with 3 GB of
ram running Fedora Core 3 (default install, stock Samba: 3.0.8pre1-0.pre1.3).
 We have about 60 clients, and often running 200+ smbd processes. The server
is connected to a SUN StorEdge fibre channel array (3 TB, RAID5) where our
database volumes are located. These volumes are shared with Samba, and nothing
else (two shares total).

  I am observing increasingly dangerous load averages which is now affecting
our database process. Every time somebody does any kind of ‘heavy’ operation
on the share (search, flip through 100’s of bitmaps with ACDSee, transfer
large Photoshop .PSD files from one share to another), the load averages go to
the roof (15+) and our database process (tbdbserver, one process only, not
multithreaded) starts slowing down, freezing access to the database. When
idle, our load averages are close to 0. Under load, all 4 reported cores do
rise somewhat (~20% each) in top.

  I attempted to upgrade both OS and Samba, with terrifying results: Fedora
Core 5, which comes with a newer version of Samba, was not only worse, but
made problems with some of our network rendering software. We had to get back
to FC3. I have reduced the number of shares to a minimum, have attempted to
isolate the database server process on one core as well as lowering its nice
value, to no effect. The amount of RAM actually used is around 1.5 GB. When
under heavy load, I always trace the guilty samba process back to a
workstation running some heavy file transfer.

  Is this Samba behavior normal? If not, is there a version of Samba you would
recommend?
Here is our smb.conf:

[global]
    socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192
    workgroup = FILMSTUDIO
    server string = prodserver
    load printers = no
    security = SHARE
    logon path =
    logon drive =
    logon home =
    encrypt passwords = yes
    blocking locks = false
    level2 oplocks = false
    oplocks = false

[usa]
    comment = USAnimation binaries
    browsable = yes
    read only = no
    guest ok = yes
    create mask = 0777
    directory mask = 0777
    path = /usa
    follow symlinks = yes

[prod]
    comment = Fibre channel array (production)
    browsable = yes
    read only = no
    guest ok = yes
    create mask = 0777
    directory mask = 0777
    path = /mnt/production
    follow symlinks = yes

[fcarray]
    comment = Fibre channel array (database)
    browsable = yes
    read only = no
    guest ok = yes
    create mask = 0777
    directory mask = 0777
    path = /mnt/array
    follow symlinks = yes


Here is a top at idle:
top - 09:10:41 up 91 days, 20:10,  2 users,  load average: 1.16, 1.14, 1.04
Tasks: 259 total,   1 running, 258 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.8% us,  6.6% sy,  0.0% ni, 63.7% id, 23.5% wa,  0.0% hi,  1.4% si
Mem:   3112960k total,  3080224k used,    32736k free,  1014392k buffers
Swap:  2031608k total,     7712k used,  2023896k free,   991960k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1205 nobody    15   0 11856 3408 9480 S  5.5  0.1   0:00.63 smbd
24083 nobody    15   0 33544 6412 9484 S  4.1  0.2   1:01.38 smbd
24161 nobody    15   0 69548  13m 9484 S  2.8  0.4  16:43.94 smbd
 2778 usabatch  16   0 23620  11m  17m S  2.8  0.4   2:19.90 tbdbserver	
database server process
27480 nobody    15   0 19504 9568 9480 D  2.8  0.3   4:35.55 smbd
11141 root      15   0  3216 1064 1664 R  2.8  0.0   0:00.64 top
 2605 root      15   0  5200 1568 3524 S  1.4  0.1   6:20.13 sshd
20485 nobody    15   0 20460 5520 9484 S  1.4  0.2   0:53.68 smbd
23137 nobody    16   0 12576 3696 9484 S  1.4  0.1   0:08.76 smbd


Dexter