[Samba] Gigabit Throughput too low

Fri May 13 19:26:49 GMT 2005

512 Megs of ram in server
Pentium 4 2.0 Ghz
2 Ultra 133 controllers - 8 drives total - all 300 gig drives running at
max UDMA support with write back cache turned off on each drive.
(Clients connected are all at 1 Gig full duplex)

FC2, with Samba 3.0.10-1

Any tweaks I try I test before and after and have only left in place
what tweaks seemed to improve performance.

I am just running into a wall with Linux's manner that it handles
caching of the files I think before it writes them to disk.  I have seen
my transfers start out as high as 50 Megabytes per second, but then they
slowly go down (seen it go as low as 1 Megabyte per sec) My guess is if
I added more memory to the server that time for it to slow down would be
increased a bit. (I was going to confirm that this weekend)

I just verified since this entire time I have been ONLY focused on
copying to the Linux server. My numbers all level off to almost exactly
10 MegaBytes per sec if I am copying FROM the linux box down to a
workstation. (Test copy of 4 gig file, starts off at 30-40 - till XP I
assume fills its buffers and starts spooling to disk, then drops and
holds at 10)

Also in your case if you NEVER see your rates change. (They just stay at
around 10Mbytes per sec I recall?) Do you have the ability to try jumbo
frames? >1500 MTU on all devices?  My current switch does not support
jumbo frames so I can't.

I believe it's tied to this pdflush. (My issue)

I found this regarding bdflush, and I think it would help me if I could
tweak pdflush:

Hi all,
    I recently switched to FC2 with the CCRMA 2.6.8 kernel. I was
wondering if there is a way to tune pdflush in the 2.6 kernel like you
could bdflush in the 2.4 kernel? I have seen advice on this list to set
bdflush by adding *"echo* "10 64 0 0 2000 15000 90 5 0" >
/proc/sys/vm/*bdflush" to the rc.local file, thus causing bdflush to
write smaller chunks more often to disk instead of waiting for the
"dirty buffer" to accumulate alot of stuff to write to disk all at once,

possibly causing dropouts. Is there a way to do this with pdflush or is
it even necessary with the 2.6 kernel? Thanks in advance.

-----Original Message-----
From: Paul Griffith [mailto:paulg at cs.yorku.ca]
Sent: Friday, May 13, 2005 12:23 PM
To: Duncan, Brian M.
Cc: samba at lists.samba.org
Subject: Re: [Samba] Gigabit Throughput too low

Greetings,

Can you post more info on your setup, i.e - configuration of machine,
etc....

I am also looking into Linux network stack tweaks. I have seen my tweaks
posted for GigE, but you have to be careful. You may actually get a
lower throughput.

For example:
 sysctl -w net.ipv4.tcp_mem='8388608 8388608 8388608'

This tell your kernel it can use

8388608 pages ( a page is 4K on the Intel platform) for all your TCP
currently in use.

so (8388608 * 4096)/1048576 = 32768MB

Take a look at:

http://ipsysctl-tutorial.frozentux.net/ipsysctl-tutorial.html

On Fri, May 13, 2005 at 12:06:29PM -0500, Duncan, Brian M. wrote:
>
> Hi I was wondering if you ever got better performance out of your
> Gigabit/IDE/Fc2?
>
> I am facing a similar situation. I am running FC2 with Samba 3.x
>
> My problem lies in not that I am limited to 10 MBytes per second
> sustained.  I think it's related to this pdflush and how it's buffers
> are setup. (I have been doing some research and before 2.6 kernels
> bdflush was the method that was used and it was tweakable.  Have yet
> to find anything on HOW to tweak pdflush.
>
> My issue is that I can copy over the network at 15+ MBytes per second
> but then after a few minutes it will drop to 4-8 MBytes per second. 
> Yet a drive to drive copy on the linux box itself can sustain 14+
> MBytes per second on the same size file. (4+ gigs) While I watch a Top

> when the network copying is going on PDFlush is bouncing around with
> Samba, not sure what is going on.
>
> Was curious if you had improved your situation, and if you did what
> you did.
>
> BTW here are some tweaks for network stack related stuff for Gig.
>
> sysctl -w net.core.rmem_max=8388608
> sysctl -w net.core.wmem_max=8388608
> sysctl -w net.core.rmem_default=65536
> sysctl -w net.core.wmem_default=65536
> sysctl -w net.ipv4.tcp_rmem='4096 87380 8388608'
> sysctl -w net.ipv4.tcp_wmem='4096 65536 8388608'
> sysctl -w net.ipv4.tcp_mem='8388608 8388608 8388608'
> sysctl -w net.ipv4.route.flush=1
>
>
>
> Brian M. Duncan
> Katten Muchin Rosenman LLP
> 525 West Monroe Street
> Chicago IL 60661-3693
> 312-577-8045
>
> brian.duncan at kattenlaw.com
>
>
>
>
>
> ===========================================================
>
> Important:
> This electronic mail message and any attached files contain
> information intended for the exclusive use of the individual or entity

> to whom it is addressed and may contain information that is
> proprietary, privileged, confidential and/or exempt from disclosure
> under applicable law.  If you are not the intended recipient, you are
> hereby notified that any viewing, copying, disclosure or distribution
> of this information may be subject to legal restriction or sanction. 
> Please notify the sender, by electronic mail or telephone, of any
> unintended recipients and delete the original message without making
> any copies.
>
> NOTIFICATION:  Katten Muchin Rosenman LLP is an Illinois limited
> liability partnership that has elected to be governed by the Illinois
> Uniform Partnership Act (1997).
>
> ===========================================================
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/listinfo/samba

===========================================================

Important:
This electronic mail message and any attached files contain information
intended for the exclusive use of the individual or entity to whom it is
addressed and may contain information that is proprietary, privileged,
confidential and/or exempt from disclosure under applicable law.  If you
are not the intended recipient, you are hereby notified that any viewing,
copying, disclosure or distribution of this information may be subject to
legal restriction or sanction.  Please notify the sender, by electronic
mail or telephone, of any unintended recipients and delete the original
message without making any copies.

NOTIFICATION:  Katten Muchin Rosenman LLP is an Illinois limited liability
partnership that has elected to be governed by the Illinois Uniform
Partnership Act (1997).

===========================================================