[jcifs] SmbFile.listFiles returning too low number of files

[Michael B Allen]

Hi Volker,

It is not unexpected that changing
jcifs.smb.client.{listSize,listCount} changes the result list size
because the various buffer sizes will be quantized differently. But if
there isn't some combination that yields the correct results, then
it's not crystal clear that the code responsible for the various
levels of buffering has a problem. It could simply be something else
entirely like some obscure issue with the particular version of
Windows you're using.

Can JCIFS correctly list files of another directory on that PC with
regular sized / named files? Create a shared folder with a few hundred
files in it and see if JCIFS can list it correctly.

Or maybe this is an artifact of an obscure network issue. Meaning some
packets are being fragmented in an unsual way that JCIFS is not used
to handling. That would explain why the results are different each
time. Do you have another PC from which to run JCIFS so that you can
try it over another interface other than loopback? If yes, can JCIFS
list the share correctly from the other PC? As you may have noticed,
the loopback interface on Windows is a bit odd.

Mike

On Thu, Apr 28, 2011 at 12:11 PM, Volker Schmelich <volker at quadnova.com> wrote:
> Hi Mike,
>
> When setting jcifs.smb.client.listCount = 400, I receive a count of 647
> files. This is still lower than the expected 664. The default setting of
> this parameter produces lower results and results may differ
> occasionally when running the test in a loop with a 1s delay (most times
> 606, few times in the beginning 644).
>
> Using jcifs.smb.client.listSize=32000 and the default for listCount
> produces a count of 658.
> Using the above listSize with listCount=400 produces a count of 657.
>
> In conclusion, yes, these params influence the count but I could not
> tweak them so that I get the correct count.
>
> The packet capture - did it look strange to you pointing to some
> specific issues with my current system?
> I just can't imagine what could be so different in my system that this
> happens on just this one system but that it works on any other system
> I've tried so far.
>
> If you need anything else from me just let me know.
>
> Regards,
> Volker
>
> -----Original Message-----
> From: Michael B Allen [mailto:ioplex at gmail.com]
> Sent: Wednesday, April 27, 2011 9:05 PM
> To: Volker Schmelich
> Cc: jcifs at lists.samba.org
> Subject: Re: [jcifs] SmbFile.listFiles returning too low number of files
>
> [CC jcifs list again now that I have the capture]
>
> Hi Volker,
>
> Yeah, capturing loopback on Windows is an oddity. Maybe netcap.exe from
> the Support Tools package would be easier. I don't know. I use Linux for
> development.
>
> But your capture was successful albeit somewhat fragmented as you said.
>
> Regarding your capture and listing error, my best guess at this point is
> that the particularly orderly list of long filenames and multiple layers
> of buffering used by the TRANS2_FIND_FIRST2/NEXT2 commands have
> conspired to generate incorrect results. More specifically I think that
> even though jcifs.smb.client.listSize = 65535, the
> jcifs.smb.client.listCount = 200 actually does not fit in the 65535
> buffer in just such a way to cause errant results. Try setting
> jcifs.smb.client.listCount = 400 and see if you get different behavior.
> If yes, that would be very interesting. You could also trying playing
> around with those two properties a little to see if they have any
> effect. They shouldn't (unless you set them really low).
> So if they do, it means there's a bug somewhere that would warrant
> further pursuit.
>
> Mike
>
> --
> Michael B Allen
> Java Active Directory Integration
> http://www.ioplex.com/
>
> On Mon, Apr 18, 2011 at 11:06 PM, Volker Schmelich <volker at quadnova.com>
> wrote:
>> Hi Mike,
>>
>> Please find attached a ZIP archive containing a Wireshark pcap file.
>>
>> The problem is that capturing traffic to localhost seems to be
>> impossible. I had to add special route according to
>> http://wiki.wireshark.org/CaptureSetup/Loopback.
>>
>> route add 192.168.0.100 mask 255.255.255.255 192.168.0.1 metric 1
>>
>> They say: "Doing so, every network traffic from your machine to itself
>
>> will use the physical network interface, it will then go to the
>> gateway, back to you. Therefor, you will see each packet twice, but it
>
>> can be filtered on the view."
>>
>> I hope the capture does help anyway. I saved everything matching this
>> display filter:
>> udp.port eq 137 or udp.port eq 138 or udp.port eq 139 or udp.port eq
>> 445 or tcp.port eq 137 or tcp.port eq 138 or tcp.port eq 139 or
>> tcp.port eq
>> 445
>>
>> Regards,
>> Volker
>>
>>
>> -----Original Message-----
>> From: Michael B Allen [mailto:ioplex at gmail.com]
>> Sent: Saturday, April 16, 2011 7:15 AM
>> To: Volker Schmelich
>> Subject: Re: [jcifs] SmbFile.listFiles returning too low number of
>> files
>>
>> Hi Volker,
>>
>> I really will need a packet capture to figure out what this problem
> is.
>> Logs are not that useful really. Another good tool for getting packet
>> captures that is not described on the page I previously cited is the
>> Microsoft netcap.exe utility. It is from the "Support Tools"
>> package.
>>
>> It is quite strange that you should run into this now. We have not had
>
>> a bug like this for many years. And it's not like it's something
>> subtle that could go unnoticed for so long. So I'm slightly skeptical.
>> I cannot help but think there is some outside force that is
> responsible.
>>
>> Mike
>>
>> On Thu, Apr 14, 2011 at 9:11 PM, Volker Schmelich
>> <volker at quadnova.com>
>> wrote:
>>> Hi Mike,
>>>
>>> I tried it with your suggestions and the outcome was the same.
>>>
>>> String sourceDir =
>>> "file://tempshare:tempshare@192.168.0.100/dceinshare/test/";
>>> SmbFile src = new SmbFile(sourceDir);
>>>
>>> String sourceDir2 = "file:// 192.168.0.100/dceinshare/test/"; SmbFile
>
>>> src = new SmbFile(sourceDir2, new NtlmPasswordAuthentication(null,
>>> "tempshare", "tempshare"));
>>>
>>> Not sure what the fully qualified DNS hostname would be as it is my
>>> local development box. So what I do is I connect to my own local PC
>>> via jCIFS. But that should not make any difference as we also do this
>
>>> on another system where it works fine.
>>>
>>> This time I tested with 664 files and different numbers are reported
>>> by listFiles, e.g. 625 or 606.
>>>
>>> The result is returned very fast. When I use a Windows 2000 virtual
>>> machine on the same subnet, it is much slower but it returns the
>>> correct values.
>>>
>>> There also seems to be a reproducible pattern. E.g. when I use the
>>> listFiles or copyTo method in our application on the same set of
>>> files, the same files always remain uncopied in the source folder
>>> (for
>>
>>> example
>>> 1 to 219 is copied, 220 to 2xx is not copied).
>>>
>>> My system where this happens has the latest patches and SP1, some
>>> antivirus software, backup software but I think this should not
>> matter.
>>> If it would be some kind of timeout or cache issue I think I should
>>> not be able to reproduce exactly the same file leftovers during copy.
>>> They should be different each time I try.
>>>
>>> I have attached a ZIP archive with three logs produced with log level
>
>>> 10. That's all I can get right now in terms of packet capture.
>>>
>>> W2k_correct.txt contains the log of the correct listing from my local
>
>>> PC accessing a share on a virtual machine on the same subnet.
>>>
>>> W7_incorrect_1.txt contains the log of the wrong listing from my
>>> local
>>
>>> PC accessing a share on my local PC. It returned 606 instead of 664.
>>>
>>> W7_incorrect_2.txt contains the log of the wrong listing from my
>>> local
>>
>>> PC accessing a share on my local PC. It returned 625 instead of 664.
>>> This was taken just some seconds after the first one. Nothing changed
>
>>> in source code or file system, just a new click on run.
>>>
>>>
>>> Thanks for your help. It would be good to know if we just have to
>>> stay
>>
>>> away from certain operating systems or if this issue could hit us
>>> sooner or later on all of them.
>>>
>>> Regards,
>>> Volker
>>>
>>> -----Original Message-----
>>> From: Michael B Allen [mailto:ioplex at gmail.com]
>>> Sent: Thursday, April 14, 2011 8:24 PM
>>> To: Volker Schmelich
>>> Cc: jcifs at lists.samba.org
>>> Subject: Re: [jcifs] SmbFile.listFiles returning too low number of
>>> files
>>>
>>> On Tue, Apr 12, 2011 at 3:29 PM, Volker Schmelich
>>> <volker at quadnova.com>
>>> wrote:
>>>> Hi there,
>>>>
>>>> We experience a strange issue on one of our Windows PCs with jCIFS
>>>> 1.3.14 and 1.3.15. The Windows PC is running Windows 7 Home Premium
>>>> x64 with SP1. We use JDK 1.6.0_24 both 32 bit and 64 bit.
>>>>
>>>> When we call SmbFile.listFiles(), the method does not return all
>>>> files
>>>
>>>> available in the directory. E.g. we have 1146 files in there, but
>>>> the
>>
>>>> method returns 1094 and sometimes 1127 (when we let it run in a loop
>
>>>> with 1s delay).
>>>>
>>>> The bug is reproducible on this one PC every time we use more than
>>>> about
>>>> 200 files. It works fine on one of our Windows 2000 and one of our
>>>> 2003 servers and other Windows 7 PCs, too.
>>>>
>>>>
>>>> This Java snippet is as simple as it gets to reproduce it:
>>>>
>>>> String sourceDir =
>>>> "file://tempshare:tempshare@vs-cyberpower/dceinshare/test2/";
>>>> SmbFile src = new SmbFile(sourceDir); SmbFile[] list =
>>>> src.listFiles(); int cnt1 = list.length;
>>>>
>>>> As copyTo depends on listFiles, copying all the files in the
>>>> directory
>>>
>>>> does not copy all of them, only those returned by listFiles.
>>>>
>>>> There are no exceptions, no stack traces. The example above uses the
>
>>>> plain JAR file, no extra jCFIS configurations.
>>>>
>>>> We also tried jcifs.smb.client.attrExpirationPeriod = 0 with no
>>>> difference.
>>>>
>>>> Is there any direction you could point us - we tried to figure it
>>>> out
>>
>>>> for hours but we are lost.
>>>
>>> Hi Volker,
>>>
>>> Try the fully qualified DNS hostname. And for testing it might be
>>> better to use the IP address. Also, supply the credentials to SmbFile
>
>>> using an NtlmPasswordAuthentication object. If that changes the
>>> result, we will have a better idea of what the problem is. If that
>>> does not change the behavior, can you get a packet capture?
>>>
>>> Mike
>>>
>>> --
>>> Michael B Allen
>>> Java Active Directory Integration
>>> http://www.ioplex.com/
>>>
>>
>>
>>
>> --
>> Michael B Allen
>> Java Active Directory Integration
>> http://www.ioplex.com/
>>
>



-- 
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/