[Samba] PDB files and "Delayed Write Failed"

[James Chamberlain]

Hello Samba Community,

I have what is probably a very unique problem.  Allow me to explain:

Background:
We build software for Windows, among other things.  Most of our developers 
are not on Windows, but they need to do Windows builds.  To facilitate 
this, we've set up a complex build system where calling "make" 
automatically connects (rsh/ssh) to a cmd shell on the Windows build 
server, translates our Makefile into something more suitable for Windows, 
and executes the build.  The source code is not on the build server's local 
disks, but is instead sitting on a file server which the build server 
accesses through Samba.  This leads to the problem.


The Problem(s):
We're seeing mysterious and unpredictable problems in this environment. 
Looking through the Event Viewer, we've seen 2658 "Delayed Write Failed" 
messages since October.  Only 19 of them did not relate to ".pdb" files. 
The Samba logs don't indicate a problem.

We're getting messages from the compiler that it can't find header files 
which definitely exist and are definitely in the include path.  We're also 
getting the occasional "gmake: *** Makefile: Permission denied.  Stop." 
message.  Simply starting the "make" again without changing any permissions 
allows the build to continue.


Build Server:
* Windows Server 2003 SP2
* 4x 3 GHz Xeon (5160)
* 4 GB RAM
* 2x 10k RPM SAS drives, hardware RAID 1

File Server:
* CentOS 5.2
* 8x 3 GHz Xeon (5450)
* 4 GB RAM
* 14x 15k RPM SAS drives, hardware RAID 6
* Samba 3.0.25b-1.el5_1.4
* Authenticates against Windows domain controller(s)


What I've tried already (not necessarily in this order):
* Rebooted the Build Server.
* Swapped OSs on the Build Server.  We started with NT, then moved to XP
   and are now on Server 2003.
* Swapped Ethernet cable on the Build Server.
* Swapped Ethernet switch port for the Build Server.
* Swapped Ethernet switch for the Build Server.
* Swapped Ethernet NIC on the Build Server.
* Swapped the Build Server hardware itself.
* Switched from explicitly mapping drives at the start of each remote cmd
   session to using UNC paths.

* Swapped OSs on the File Server.  We started with Red Hat Linux 8 for i386
   and have moved up through several iterations to Centos 5.2 for x86_64.
* Swapped Ethernet cable on the File Server.
* Swapped Ethernet switch port for the File Server.
* Swapped Ethernet switch for the File Server.
* Swapped Ethernet NIC on the File Server.
* Swapped the File Server hardware itself.
* Upgraded to the latest version of Samba available from the CentOS team.
   This broke domain authentication for us, so we rolled back to 3.0.25.

* Added a backup domain controller.  (NT4 domain environment still.  Yes,
   I know, I'm working on it)

* Changed the Samba socket options from "TCP_NODELAY SO_RCVBUF=8192
   SO_SNDBUF=8192" to "TCP_NODELAY IPTOS_LOWDELAY".
* Set "large readwrite = no"
* Set "write raw = no"
* Explicitly turned on oplocks and level2 oplocks, though I believe they
   are on by default.
* "dos filetimes = yes"
* "fake directory create times = yes"
* "dos filetime resolution = yes"
* "allocation roundup size = 0"

Thusfar, any time we've managed to improve performance back to the expected 
level, it has been unclear what did the trick... and it didn't last.  If 
anyone has any thoughts on other things I can try, I would certainly 
appreciate it.  If there's any further information that would help in 
making an assessment, I'd be happy to post what I can.

Thanks,

James