[jcifs] Performance loss with listFiles

[Michael B Allen]

Hi René,

It sounds like the problem could just be your algorithm and that it
would not matter how you list files. However, the regular
SmbFile.listFiles builds an array of objects in memory which of course
is not good for very large listings. You could try to use the
SmbFile.listFiles(SmbFileFilter filter) method and do the analysis in
SmbFileFilter.accept() and then just return false so that the SmbFile
is not saved in the internal ArrayList used by listFiles. See
examples/LargeListFiles.java. Although beware that if accept() takes
too long you will need to reduce jcifs.smb.client.listCount = 20 to
spread out the file listing over more network messages or you may get
response timeouts.

Mike

On Tue, Jan 12, 2010 at 5:11 AM, René Liebmann
<rene_liebmann at datamatics.de> wrote:
> Hi everybody,
>
>
>
> I have implemented a multithreaded File analyser, which compares files
> within network shares to find duplicates. That works fine so far, but there
> is a continuously loss of performance. If I put for example 400.000 times
> the same SmbFile into my logic, then everything is fine, but as soon as I
> use the listFiles method of SmbFile and I’m going recursively down that loss
> of performance is there. Right after the start I’m able to analyse around
> 250 files per second, but after 100.000 files I arrive at 50 files per
> second and it goes still down. In huge file systems with several terra bytes
> this is not so good.
>
>
>
> I had used several versions of jCifs so far, but there was no change.  Does
> anyone know a reason for this behavior or do I use jCifs in a way it is not
> intend for?
>
>
>
> Thanks
>
>
>
> René
>
>



-- 
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/