[Samba] Possible Issue with Samba Blocking I/O and CPU
jra at samba.org
Thu Jun 10 10:33:00 MDT 2010
On Thu, Jun 10, 2010 at 10:21:13AM -0400, Andy Liebman wrote:
> I am experiencing a puzzling problem that may or may not be related to
> recent versions of Samba. I'm posting on this list, however, because it
> seems that setting "write cache = 262144" (256K) in smb.conf resolves
> the issue and so I have reason to believe the problem might somehow be
> related to Samba.
> The problem I am seeing is this. I have a Linux Samba server. When
> simultaneously reading and writing from multiple client workstations --
> let's say I'm writing a total of 80 MB/sec from 4 Windows clients and
> reading 220 MB/sec from 12 other Windows clients -- every half hour or
> so, all I/O seems to grind to a halt for 1 or 2 seconds. You can see
> this graphically using a program such as "gkrellm". You just see a
> dropout of all reading and writing from/to the disks. SMB traffic
> continues to come in from the network, but SMB traffic stops going out
> TO the network. This same pattern has been observed on multiple
> servers, so the problem isn't caused by some bad RAID card or other
> piece of defective hardware.
> Looking at data from /proc (this is easy using a program like
> "collectl"), at the moment of the dropout, you can see that "idle" time
> goes to 0 on any CPU core running smbd processes, and wait goes very
> high. On the core handling the interrupts for the RAID card driver,
> "idle" time goes to 100 percent. This is running the 184.108.40.206 Linux
> kernel with Samba 3.5.3 (the latest as of today) or Samba 3.4.2 (the
> version that comes with my distro).
> Curiously, the 1-to-2 second drops don't occur with the exact same
> hardware and workload when running the 2.6.20 Linux kernel with Samba
> 3.0.23d. Since 2.6.20, there have been huge changes made in the vm
> layer of the Linux kernel (specifically related to pdflush and the per
> device "bdi" mechanism). There have also been many changes to the
> deadline i/o scheduler. For several weeks, my colleagues and I have
> been thinking that changes to one of those code areas might account for
> the difference. However, after much study and experimentation in
> tweaking those subsystems, we could not make this occasional "drop out"
> behavior go away.
> By the way, this dropout behavior does not occur with a "pure write" or
> "pure read" workflow. Only when there is a mixture of read and write.
> Our theory is that we might occasionally be getting a bunch of
> small-sized blocks of data to write, causing our RAID-5 configured RAID
> card to do a large number of "partial stripe writes", which would result
> in reading useless data from the drives in order to calculate parity for
> a number of stripes, and interfering with reading data that has actually
> been requested by clients.
> Two days ago -- grasping a bit for straws -- we tried messing around
> with some smb.conf settings. It turns out that setting "write cache =
> 262144" made this specific problem go away. We have repeated our tests
> for over 8 hours since making this change and we have not seen a single
> dropout like before. Presumably, with this setting, Samba will only
> write out in 256K blocks (which happens to be the stripe size on our
> My question is, does anyone have a clue why setting "write cache" like
> this would have such an effect? Is setting "write cache" just covering
> up some other problem? Is there any downside to using "write cache"?
> And why didn't we have this issue with the 2.6.20 kernel + Samba 3.0.23d?
This looks like a kernel issue to me. Setting "write cache = 262144"
will massively change the read and write patterns that smbd does to
your disks. You didn't see the problem with your earlier setup because
it's a different kernel, without the issue.
With this setting of write cache, if the apps have a good locality
of data reference you'll almost never hit the disk, as everything
will be served out of that memory cache.
Sorry I can't be too helpful here, but this really doesn't look like
a Samba problem.
More information about the samba