[jcifs] too much timeout when multithreaded

[Benoit Daviaud]

Hej,

Thank you for your answer.
I've been trying your ThreadedSmbCrowler. But I could just find the version
using list in the exemple directory. Your approch is more to have one thread
for every directory. My crowler is design to affect one thread for each
host.This thread performs a recursive crowling of the host. It is not
possible that more than one thread would work on the same host. When a
thread is finished with a host. It dies and a new one is created to crowl
the next host on the list. I check the number of living threads using
activeCount() in ThreadGroup.
About the BCAST option it was in the list but after WINS and we are actually
using a wins server. But I followed your advise and removed it even if it
didn't changed the problem.
In fact all this thread stuff is to improve the speed of the crowling (we
are working on a quite big network gathering approximatively 2500 hosts and
the normal crowling can long as much as 4 or 5 hours...) but I don't want to
sacrify exostivity for speed. It's not a real problem if the crowling long
several hours. (even with my current threaded crowling, it can still long 2
hours ...).
It is very clear that with 100 threads working in parallele on 100 different
hosts, I get almost no output, it seems that almost all call to listFiles()
on the directories result in a timeout exception. Even when the directory is
accessible (I check it using internet explorer...). On the other hand with
only 20 threads I don't have this problem and I get lots of output. The
thing is that I'd like to know where is the limit between those two cases.
Because I'd like to know if I miss some files even when using only 20
threads. And it is difficult to make a comparison because the number of
accessible files on the network vary from one second to another...
After some experimentations, it looks like the more threads I use, the more
files I lose. And there is no clear border.
I also have been trying to use the threadedSmbCrawler in the same situation
as mine. I still see the same behaviour when I specify 100 threads: I get
almost nothing as out, only errors. But againt it is the version using
list() and not listFiles() ...
In the worth case I will use a just a small number of threads and things
should be alright. But I'm curious to know why I have this problem ...

Benoit



----- Original Message -----
From: "Michael B.Allen" <mba2000 at ioplex.com>
To: "Benoit Daviaud" <benda472 at student.liu.se>
Cc: <jcifs at lists.samba.org>
Sent: Sunday, April 06, 2003 8:43 PM
Subject: Re: [jcifs] too much timeout when multithreaded


> On Sun, 6 Apr 2003 18:55:04 +0200
> "Benoit Daviaud" <benda472 at student.liu.se> wrote:
>
> > Hello,
> >
> > I have a program that is harvesting on a local network all the files
> <snip>
> > Then I've been experimenting on what happend for different maximum
> number of threads. It seems to work well for 10 or 20 threads. But if
> I try with 100 threads, I have a very high number of timeout. In fact
>
> Try the ThreadedSmbCrawler. Make sure you use the one from the latest
> release that uses listFiles instead of just list. Use that with the
> same parameters as your crawler and look at the difference. Make sure
> your resolveOrder property does not have BCAST in it. I don't know off
> hand what the problem could be. Is it possible that all 10-20 threads
> are crawling over the same server at any one time? That's going to be
> significantly slower than having one thread crawling over each host. Also,
> make sure your algorithm does not build up a big list of stale SmbFiles;
> keep the working-list small. The ThreadedSmbCrawler does this. It is
> normal to receive many exceptions when doing this sort of thing. But if
> you investigate any one host the issue should be reproducable because the
> host is down or in a bad state or the IP returned by WINS is not routable
> ...etc. Finally, you have to take care when writing a crawler. It's
> not trivial. A simple mistake can make it *very* slow. The algorithm is
> critical to performance due to the latency of querying each node.
>
> Mike
>
> --
> A  program should be written to model the concepts of the task it
> performs rather than the physical world or a process because this
> maximizes  the  potential  for it to be applied to tasks that are
> conceptually  similar and, more important, to tasks that have not
> yet been conceived.