[Samba] massive performance problems if transferring many sma ll files

[Simon.Harris at nationwide.co.uk]

-----BEGIN PGP SIGNED MESSAGE-----

********************************************************************
The contents of this email are intended exclusively for the
addressee. If you are not the addressee you must not read
use or disclose the email contents ; you should notify us
immediately [ by clicking "Reply" ] and delete this email.

Nationwide monitors e-mails to ensure its systems operate
effectively and to minimise the risk of viruses.  Whilst it has
taken reasonable steps to scan this email, it does not
accept liability for any virus that may be contained in it.
********************************************************************

What does mangle method hash2 do differently?

My case involves similar amount of files in a single directory in the
format:
five-digits[0-9].{ABC}
(where I can have three extensions for same five-digits)
This is managed by an application so will not get duplicates.
Is there anything I can tune to speed-up the findfirst that my application
is doing for every file? 

Am using the defaults in samba.conf!

- -----Original Message-----
From: abartlet at samba.org [mailto:abartlet at samba.org] 
Sent: 05 February 2004 20:48
To: steffen.k at gmx.at
Cc: samba at lists.samba.org; abartlet at samba.org
Subject: Re: [Samba] massive performance problems if transferring many small
files

Security warning. Details in WARNING.TXT about the possible problem.
- --------------------------------------------------------------------
On Thu, 2004-02-05 at 21:34, Steffen Kauka wrote:
> > On Thu, 2004-02-05 at 20:12, steffen.k at gmx.at wrote:
> > > Hi,
> > >
> > > If I try to transfer about 27.000 small HTML-Files from my Windows-PC
to
> > our
> > > central SAMBA-Server I'll get  massive performance problems. First the
> > > performance looks ok but after a time period of 3 minutes it goes slow
> > do
> > wn -
> > > appr. one file per second will be transferred. Has anyone an idea what
I
> > can tune
> > > to avoid this annoying behaviour.
> >
> > > - samba version 2.2.8a (64bit compiled)
> >
> > Are all/many of these files in the same directory?
>
> Yes, it's a flat directory hierarchy.

Because unix is case sensitive, and windows is not, this is almost a
pathalogical case for Samba.

Samba must scan the *entire* directory, to see if there is a matching
file (of potentially different case), so it can say 'sorry, file by that
name already'.  Naturally, this isn't exactly fast as you approach 27000
files...

> > I would suggest that Samba 3.0 might handle this situation better, or in
> > 2.2 set 'mangle method = hash2'.
>
> I'll try it....

The hash2 method is not only much faster, it has a much lower collision
rate.  This helps if for some reason, the 8.3 names are being used.

If you can put the files into a hierarchy, then things will be *much*
better.  Or, for this copy only, you might want to turn 'case sensitive
= yes' on in your smb.conf - however the implications of that are nasty
for normal windows operations (and perhaps even the copy, depending on
what you use).

Another quick hack might simply be to zip the files up, and run 'unzip'
on the server.

Andrew Bartlett

- --
Andrew Bartlett                                 abartlet at pcug.org.au
Manager, Authentication Subsystems, Samba Team  abartlet at samba.org
Student Network Administrator, Hawker College   abartlet at hawkerc.net
http://samba.org     http://build.samba.org     http://hawkerc.net
- -- To unsubscribe from this list go to the following URL and read the
instructions:  http://lists.samba.org/mailman/listinfo/samba



-----BEGIN PGP SIGNATURE-----
Version: PGPsdk version 1.1.1 (C) 1997 Pretty Good Privacy, Inc.

iQEVAwUBQCtSBG4wJNdvwuyFAQE0Ogf/ejDWK4i+kj4lPnlkA4diWY3DRzWgkMwo
qNBS06B/2eQt5YL0uEL7k1kNNkCfxaIQvQ5ivtOb3rL9Tb/NQ2x7VIIVnsQVV8gl
cLkv2wFHyhatnvagEySRaBvGoizGjOkDdWTKLU1VjLHsd82dNLmQqPKSFNAtkAiZ
cNh9E7wQwhgKrAd6wWXBJZMljXKY/eSizdgeSx7kOPL0UtHz1ozD518PZdU+XZXJ
lDsYXdUckdvlzMsNwqpcY42DCU8OUEjmqlDmrazVaLLVpHy4FLSKzNyggePhUXjg
z8wMEm9ggquwhRg/hLCVI5Nqyd2+qrs8tsgZ+MV33ftQsNO0HQywjQ==
=L9/L
-----END PGP SIGNATURE-----