[Samba] Possible Issue with Samba Blocking I/O and CPU

[Jeremy Allison]

On Thu, Jun 10, 2010 at 10:21:13AM -0400, Andy Liebman wrote:
> Hi,
>
> I am experiencing a puzzling problem that may or may not be related to  
> recent versions of Samba.  I'm posting on this list, however, because it  
> seems that setting "write cache = 262144" (256K) in smb.conf resolves  
> the issue and so I have reason to believe the problem might somehow be  
> related to Samba.
>
> The problem I am seeing is this.  I have a Linux Samba server. When  
> simultaneously reading and writing from multiple client workstations --  
> let's say I'm writing a total of 80 MB/sec from 4 Windows clients and  
> reading 220 MB/sec from 12 other Windows clients -- every half hour or  
> so, all I/O seems to grind to a halt for 1 or 2 seconds. You can see  
> this graphically using a program such as "gkrellm".  You just see a  
> dropout of all reading and writing from/to the disks. SMB traffic  
> continues to come in from the network, but SMB traffic stops going out  
> TO the network.  This same pattern has been observed on multiple  
> servers, so the problem isn't caused by some bad RAID card or other  
> piece of defective hardware.
>
> Looking at data from /proc  (this is easy using a program like  
> "collectl"), at the moment of the dropout, you can see that "idle" time  
> goes to 0 on any CPU core running smbd processes, and wait goes very  
> high.  On the core handling the interrupts for the RAID card driver,  
> "idle" time goes to 100 percent.  This is running the 2.6.32.11 Linux  
> kernel with Samba 3.5.3 (the latest as of today) or Samba 3.4.2 (the  
> version that comes with my distro).
>
> Curiously, the 1-to-2 second drops don't occur with the exact same  
> hardware and workload when running the 2.6.20 Linux kernel with Samba   
> 3.0.23d.  Since 2.6.20, there have been huge changes made in the vm  
> layer of the Linux kernel (specifically related to pdflush and the per  
> device "bdi" mechanism).  There have also been many changes to the  
> deadline i/o scheduler.  For several weeks, my colleagues and I have  
> been thinking that changes to one of those code areas might account for  
> the difference.  However, after much study and experimentation in  
> tweaking those subsystems, we could not make this occasional "drop out"  
> behavior go away.
>
> By the way, this dropout behavior does not occur with a "pure write" or  
> "pure read" workflow. Only when there is a mixture of read and write.  
> Our theory is that we might occasionally be getting a bunch of  
> small-sized blocks of data to write,  causing our RAID-5 configured RAID  
> card to do a large number of "partial stripe writes", which would result  
> in reading useless data from the drives in order to calculate parity for  
> a number of stripes, and interfering with reading data that has actually  
> been requested by clients.
>
> Two days ago -- grasping a bit for straws -- we tried messing around  
> with some smb.conf settings.  It turns out that setting "write cache =  
> 262144" made this specific problem go away.  We have repeated our tests  
> for over 8 hours since making this change and we have not seen a single  
> dropout like before. Presumably, with this setting, Samba will only  
> write out in 256K blocks (which happens to be the stripe size on our 
> RAIDS)
>
> My question is, does anyone have a clue why setting "write cache" like  
> this would have such an effect?  Is setting "write cache" just covering  
> up some other problem? Is there any downside to using "write cache"?   
> And why didn't we have this issue with the 2.6.20 kernel + Samba 3.0.23d?

This looks like a kernel issue to me. Setting "write cache = 262144"
will massively change the read and write patterns that smbd does to
your disks. You didn't see the problem with your earlier setup because
it's a different kernel, without the issue.

With this setting of write cache, if the apps have a good locality
of data reference you'll almost never hit the disk, as everything
will be served out of that memory cache.

Sorry I can't be too helpful here, but this really doesn't look like
a Samba problem.

Jeremy.