[Samba] SMBD CPU climbs sky high when writing DPX files

[Andy Liebman]

Hello,

I am experiencing a strange problem when writing (capturing) DPX video 
files to a Linux/Samba share. Basically, I'm seeing seeing a single smbd 
process go from 9 percent CPU utilization to 100 percent CPU utilization 
over the course of about 40 minutes.  When smbd reaches 100 percent, the 
capture stops (drops frames). I have tested this with three different 
filesystem formats (XFS, ext3 and ext4) with similar -- although 
slightly different -- results.

My test systems have been running the following:

2.6.31.4 kernel with Mandriva 2009.1 with Samba 3.3.2
2.6.28-11 Ubuntu 9.04 with Samba 3.3.2
2.6.27.38 kernel with Samba 3.0.23d
2.6.20.15 kernel with Samba 3.0.23d

For those of you who may not be familiar with the DPX format, unlike 
QuickTime or other "streaming" media formats, DPX is "one file per 
frame". So, a 20 minute DPX movie in the US "NTSC TV" format would 
consist of 30 fps x 60 secs x 20 minutes = 36000 individual files all 
written into one directory.

I have a very capable networked storage system. The underlying hardware 
RAID is capable of reading and writing at least 700 MB/sec sustained. 
The storage system is running on a 3.2 Ghz Intel 5482 Quad Core CPU with 
4 or 6 GBs of RAM (depending on the system). I don't have any problem 
writing uncompressed high-definition QuickTime files to Samba shares at 
160 MB/sec over 10 Gigabit Ethernet.  However, trying to write standard 
definition DPX files over Samba -- only 40 MB/sec -- fails consistently 
after about 60,000 files have been written, plus or minus about 5000 files.

In the screenshots shown in the link below, you can this happening when 
I was writing to a 6 TB ext4 filesystem. This was a real shocker.

  http://sites.google.com/site/andrewl733info/ext4_and_samba_2


On the above system (which had 6 GB of RAM), it seems that most cpu 
activity was restricted to a single core of a Quad Core CPU.  SMBD cpu 
utilization climbed steadily throughout the capture.  CPU began at 7 
percent for one core and by the time I got to 66000 frames, I was at 100 
percent cpu for one core. I started taking screen shots a third of the 
way through. Then I went back and started again.

I also tested XFS as the filesystem.  In fact, that's my preferred 
filesystem.  I formatted the same 6 TB volume using the suggestions of 
experienced XFS developers (specifying "lazy-count=1" for the "log" and 
properly specifying the "sunit" and "swidth" for the data portion) and I 
mounted it with the important flags "-o,noatime,nobarrier,inode64".  
Here's a record of what happened under these circumstances.

  http://sites.google.com/site/andrewl733info/xfs_and_dpx-1

Depending on the kernel tested (and in some cases whether an irqbalance 
daemon was running), either most activity was concentrated on a single 
core, or it was spread out evenly among all 4 cores.  But in either 
case, smbd cpu utilization would climb steadily from under 10 percent at 
the beginning to 10 times that at the end. And when "top" showed that my 
single active smbd was at 100 percent, the capture would fail. (To 
clarify this last point, when cpu activity is spread evenly among all 
cores, top still shows the total activity for smbd. When it says 100 
percent, that would be about 25 percent on EACH core, or 100 percent out 
of 400 percent total.)

My questions are:

1)  why is smbd cpu utilization climbing so high in just 40 minutes?
2)  could the cause be related to creating, opening and closing so many 
files so quickly?  After all, we are creating, opening and closing 30 
files each second.
3)  is there anything I can do about it (besides give up)? Samba doesn't 
seem to be running out of memory. There is still 3-cores worth of CPU 
being unused at the time the capture fails.

I have collected Level 10 logs of the capture to the XFS volume.  Please 
feel free to look at them. I'm sorry if the format isn't totally 
convenient.  The actual log file turned out to be 1.6 GB.  I made a 
single bz2 file out of this -- 27 MB, but Google Sites only lets me 
upload in 11 MB pieces, so I split up the bz2 file into 3 pieces with 
the "split" program. I believe you will have to download all of them, 
and then put them back together with something like:

   "cat sambalogpart_a* sambalog.bz2"

        http://sites.google.com/site/andrewl733info/files/sambalogpart_aa?attredirects=0&d=1

        http://sites.google.com/site/andrewl733info/files/sambalogpart_ab?attredirects=0&d=1

        http://sites.google.com/site/andrewl733info/files/sambalogpart_ac?attredirects=0&d=1


Andy