[Samba] Re: Win2k - samba-2.2.8a - can't get above 4MB/sec

[Andrew Kohlsmith]

Sorry for the late reply.  Do you have any idea why these messages aren't 
getting into the mailing list?

> I can't imagine 64k TCP packets. You probably mean the
> window size.

Nope, I mean TCP packets -- TCP packets can be pretty damned large, they 
just get fragmented to fit into the ethernet frames, which is exactly what 
I'm trying to do.  The window sizing can get much larger too but that's 
only advantageous for "long thin" networks instead of fast LANs.

>   # time tar cbf 64 - . | dd of=/dev/null obs=64k
>   4141517+0 records out  # that's 271,418,458,112 bytes
>   real    46m49.755s     # that's 2,809.755 seconds
>
> only 96.6 megabytes per second.

That is impressive, but I find it hard to believe that ATA133 can sustain 
this in a real test (multiple users accessing different parts of the 
filesystem, most likely with many small file accesses).  I agree though, 
your setup blows mine out of the water for straight raw throughput!

> There's very little scope for tuning CIFS and NBT,
> other than having a faster server and disks. Using

But that's the issue!  it's _not_ a disk bottleneck, and it isn't even a bus 
bottleneck.

> special programs to measure transfer rates by
> creating and then copying files with windows API
> calls, I do get about 9,5 MB/s from most clients but
> the actual real-life average is closer to 3-5 MB/s in
> my experience. One of my samba servers is backup

Sounds like my experience.

> server collecting data from a dozen Windows clients so
> I have lots of data showing that Windows clients are
> unable to exceed about 4 MB/s sending data to the
> backup server, most often going at 2.2-3.5 MB/s. All
> of them W2K with 2-3 GHz processors, new IDE disks and
> plenty of memory. You know what? I always start all
> the backups simultaneously to save time. It takes just
> a little longer than backing up any one of them alone.

If all they can do is 4MB/sec each then naturally doing the backups 
concurrently will be advantageous, as the server is just sitting there idle 
most of the time, even during network transfers.

> You can't have 64k NBT packets just the same as you
> can't have TCP packets (in which NetBEUI packets are
> encapsulated for TCP/IP transport) longer than about
> 1514 bytes at most. 1460 sounds good to me.

I'm trying to get win2k to fragment NBT packets by making them larger than 
what a single ethernet frame can hold.  I have to check again but I believe 
that samba <--> samba transfers use large packet sizes and have 
significantly higher throughputs.  Samba's set up to allow up to 64k 
packets, but win2k isn't letting it get that high.

I can simulate this with NFS: if I set rsize/wsize to 1460 bytes I get 
almost identical throughput, but if I set the rsize/wsize up to 8k can 
almost hit 10MB/sec, and tcpdump is showing that the packet size is 
changing (and being fragmented to fit into ethernet frames).  I'm trying to 
do the same for NBT packets.

I'd also tried "raw tcp" -- port 445, no NetBIOS.  Same speed, same packet 
size.

> You are mixing up the NFS mount options read/write
> block size with packets. These are different things.

If they are different things, then why is tcpdump showing the NFS packets to 
be 8k packets, fragmented down to ethernet frame sizes?  I understand that 
no ethernet frame can be greater than 1500 bytes (in my network anyway) and 
thus TCP is fragmenting the larger blocks in order to fit, but the 
throughput is significantly higher.

> optimal to set the block size of NFS shares to 1500,
> so go figure.

My benchmarks disagree with this: my optimal throughput is with 8k 
rsize/wsize, even though it's being fragmented.  

> IMO the best way to speed up your smb traffic is to
> get rid of those lame scuzzies. Try a solid Opteron
> board with 64-bit/66 MHz PCI slots if you're so keen
> on speed. (I am too, but it's still a bit expensive:-)

Again, my SCSI system punks out at 40MB/sec, not 4MB/sec.  Going to faster 
disk I/O isn't going to help the situation since I'm nowhere near the limit 
as is.

> On the other hand, try to think out of box. Perhaps
> reducing the packet size to something like 776 or 93
> bytes could do the trick. Who knows, perhaps there's a
> magic number somewhere in the range 41-1460. I've
> never tried but it's sure a lot easier to do than
> increasing the ethernet packet size to 64k.

I'm not trying to increase the ethernet frame size; I'm trying to get Samba 
and Win2k to agree to transfer large packets (which get fragmented and 
reassembled), just like I can do with NFS.  

I will try the smaller NBT packet sizes but I don't think it'll yield much 
more.  I am under the impression that the bottleneck lies in the read() 
call -- reading 1460 bytes at a time is not as effecient as reading 8k or 
even 64k at a time and then spoon-feeding it to the ethernet card.  I will 
try it though and see.

Thanks for your reply.  I'm not sure why I'm not seeing these posts in the 
mailing list but at least someone got it.  :-)

Regards,
Andrew