How to debug 100x variance in writing the same file?

[John Gerth]

I'm writing out of some desperation and not so much looking for a direct
answer as some clues about what directions to look in.

The problem is occuring on a 1.3 GHz Apple G4 Xserve 10.3.6 which is Samba 3.0.5
I've been working to reduce the number of variables and so all clients
and the servers are now on the same gigabit VLAN (Cisco switches). I have
mapped drives from the Xserve and from a 2.8GHz Win2k3 server
The share on the Xserve is in /tmp on a local disk. At Apple's advice:
     socket options = TCP_NODELAY SO_RCVBUF=64240

The client is 3.2GHz WinXPpro also on gig-e and my test cases are of the form:
     timethis xcopy c:\temp\nnnM   f:nnnM
where nnn is the size of the file in megabytes, 10,20,40,100, etc.
and chose at time

Just to get a baseline I also did "scp" of the files from a Linux gig-e
box to the Xserve and it runs at a very nice 50MB/s as do the xcopys
of 100M to the Win2K3 box.  This leads me to believe that I don't have
any network problems and that the Xserve is fully capable of absorbing
the data at network speeds.

However, writing to the Xserve's Samba is just all over the map.
xcopys of even 10M files, vary from .2 to 55 seconds (files of
other sizes up to 100M show similar ratios). I even went
so far as to symlink a target filename to /dev/null to try and
remove any vagaries with respect to actually writing to disk,
but the wild variations continue (usually very fast or very slow).

I thought perhaps it might be scheduling anomalies in OS-X, but
I can't prove it. Sometimes I can do 10 xcopys in a row at the
fast speed and then it tanks (although larger files do seem
to tank more often).  You can sit on the client and watch the
task manager's network line graph and see only three sorts of
shapes. Very spikey for fast, spikes and idles for slow, and the
occasional square wave for the also occasional middle speeds.

I am submitting this question to Apple as well, but I don't have
much hope of getting help on something this subtle.

Should I be looking at the socket options or asking Apple about
their TCP stack? Are there WinXP tweaks to consider?

I'm just stuck at the moment and not sure that going down to the
ethereal level is going to help.

Any advice on how to proceed would be appreciated,
  /John