[Samba] suggestions for a "fast" fileserver - 1G / 10G

[Linda W]

Sabuj Pattanayek wrote:
>> E.g. from my Mac Pro I get smb r/w +- 40MB/s, with ftp I get 90MB/s on a
>> 1Gbit Server.
>>
>> So I try to eliminate some bottle necks. But where are they?
>>
>> I know there are some protocol overheads etc. comparing smb and e.g. ftp.
>>
>>     
>
> What samba version are you running ? I'm using the latest (or close to
> latest) samba sernet 4.1.x . I think you're running into protocol
> overheads. I got the same max 40MB/s on reads and writes over a gig
> connection to our server which has 2x10gb LACP bond from a linux client
> running RHEL6/CENTOS6 the kernel for which only supports smb protocol ver=1
> mounts.
---
Win7 can get close to line speeds on 1Gb HW in benchmarking, but latency 
due to
seeking and small r/w sizes of files will bring that down to 10MB (+ or -)
real quick.

Even with 10Gb, (which is really 8Gb, BTW -- the protocol uses only 8 bits
out of 10 like old modems).  So max theoretical in MB is about 768MB/s.

The issue of packet sizes came up in a Tbird bug report...I did
a bench testing effects of TCP I/O size (underlying ethernet is up to
9000 bytes) in SMB ... was a quick & dirty test, but you can see the
trend really fast.   I never did bench the 4k size as it would have taken
about near 15 minutes to run by itself....

Speeds are MB/s R, then W(rite) for a given packet size.  Same amount is 
transferred in
each test... only variation is the packet size. 

These are speeds on a 10Gb dedicated line, but background processes on 
both systems
were still running:

size      Read                   Write

128M  10.04s, 428 MB/s,       6.58s, 652 MB/s
 64M   9.74s, 441 MB/s,       6.68s, 642 MB/s
 32M   9.50s, 452 MB/s,       6.79s, 632 MB/s
 16M   9.79s, 439 MB/s,       7.11s, 604 MB/s
  8M   9.44s, 455 MB/s,       7.36s, 583 MB/s
  4M  10.77s, 399 MB/s,       7.57s, 568 MB/s
  2M  11.68s, 368 MB/s,      10.20s, 421 MB/s
  1M  11.91s, 361 MB/s,       9.45s, 455 MB/s
128K  19.02s, 226 MB/s,      20.75s, 207 MB/s
64K   29.043, 148 MB/s,      33.01s, 130 MB/s
32K   56.46s, 76.1 MB/s,     60.91s, 70.5 MB/s
16K  109.92s, 39.1 MB/s,    110.15s, 39.0 MB/s
 8K  198.86s, 21.6 MB/s,    234.77s, 18.3 MB/s

---

4K would have been maybe around 800 seconds or greater looking at the 
degradation.

Limiting factor is CPU -- smbd is usually peg'ed @ 100% during these tests.
Unfortunately with 1 TCP connection/server connect, the protocol is a 
bit of a
bottleneck (not easily distributed to multiple cores).