[Samba] SMBD CPU climbs sky high when writing DPX files

[Volker Lendecke]

On Thu, Nov 05, 2009 at 01:43:59PM -0500, Andy Liebman wrote:
> >The problem is the following: When creating a file we have
> >to prove that the file does not exist in a different
> >upper/lower case combination. Under Unix, the only way to
> >prove this is to list the whole directory and do a case
> >insensitive comparison on each existing file.
> >
> >  
> 
> Thank you SO much. Your suggestion totally solved the problem. It makes 
> sense when you think about it.
> 
> Results:  When capturing standard definition DPX files, CPU utilization 
> is now down at 7 percent (out of 400 percent) and holds steady forever 
> basically.  I was able to capture 150,000 frames (83 minutes) before I 
> got bored and stopped my test. 
> 
> In addition, I'm now capturing HD-444 DPX files which run at about 260 
> MB/sec and that's doing fine as well -- running at 37 percent total CPU 
> (out of 400 percent).

If you want to read them out at wire speed, you might want
to look at the vfs_preopen module, part of Samba 3.4. In
case you have latency problems, this module forks helper
processes that open and read the next n files in sequence.
It was written for exactly your application, out of a
cluster file system with pretty high open latency we got
almost 400MBytes/second with that module well tuned,
regardless of the broken access pattern :-)

Volker
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba/attachments/20091105/5e2950e1/attachment.pgp>