Win2K/XP, oplocks, and readahead.

[Peter Godman]

Hi!

I'm forwarding this message, which I orignally posted to the samba list,
in the hopes of reaching a wider audience for my question.  Thanks for
considering my problem.

I'm working with Samba backed by a high performance filesystem.  From a
Windows 2K and Windows XP client I'm trying to achieve very high speed
single file throughput over GigE from the Windows client using either
open/write or CreateFile/ReadFile APIs.  I'd rather not venture into
overlapped IO there so that we don't have to recommend that all our
customers rewrite their applications!

I'm seeing a problem where it appears that windows is not reading data far
enough ahead (or maybe at all) to keep the pipeline full.  From assessing
the load on samba, it is a apparent that much of its time is spent idle.  
The characteristics of the load suggest that WinXP or Win2000 is not
requesting readahead far enough to be useful.  My link roundtrip latency
is around .4 ms beyond the data transmission time, the samba servicing
time for 32K of data (32K is the blocksize I'm using) is about .27 ms, and
the wire time for 32K of data should be around .27ms on GigE.

I understand that it is necessary that oplocking be functional to have the
windows client read ahead.  However, I have verified that oplocks are
being established, yet still the readahead seems either nonexistant or
minimal.  I don't know how to establish which.  FreeBSD is the host OS.  
I have verified that samba is not sleeping on socket buffer space, and
this is borne out by the fact that changing TCP window sizes on client and
server improves performance very little.  In addition, samba isn't
sleeping on the filesystem reads, as readahead on the filesystem ensures
data is always available when the windows client requests it.

I'd be grateful for answers to any of the following questions:

1.  How do Windows clients determine appropriate levels of readahead?  
    Are there any caps on this algorithm that I might be hitting? 
2.  Is the client or server responsible for producing readahead data (I'm
    assuming this is the client)? 
3.  Any other tips on how to make this work or anecdotal evidence of
    single file performance in the half-gigabit/sec ballpark in the read
    and write of a single file? 
4.  Is there any way for samba to send unsolicited readahead data to the
    Windows client when an oplock has been established?
5.  Any samba tricks for debugging this?

In terms of registry keys I've already changed those for window sizes,
MTU, UseOpportunisticLocking, SizReqBuf (to 64K), EnablePMTUDiscovery,
MaxCmds.

Here's my samba configuration:

[global]
  encrypt passwords = yes
  log file = /var/log/samba.log
  max log size = 100
  local master = no
  read size = 8192
  # below socket sizes have been varied without effect
  socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE SO_RCVBUF=65536
SO_SNDBUF=65536
  dns proxy = no
  change notify timeout = 3000
  disable spoolss = yes
  smb passwd file = /usr/local/private/smbpasswd
  password server = *
  winbind uid = 10000-20000
  winbind gid = 10000-20000
  winbind enum users = yes
  winbind enum groups = yes
  workgroup = FOO
  server string = A Samba Server
  hosts allow = 
  security = SHARE
  oplocks = true

[myfs]
  printable = no
  level2 oplocks = true
  guest ok = yes
  path = /myfs
  comment = myfs
  read only = no
  hide dot files = no
  share modes = no