[jcifs] SmbFile.listFiles returning too low number of files

[Michael B Allen]

Hi Volker,

Well that is odd. Maybe it was some kind of filesystem corruption.

Anyway, thanks for the follow through.

Mike

On Tue, May 10, 2011 at 6:40 PM, Volker Schmelich <volker at quadnova.com> wrote:
> Hi Mike,
>
> Unfortunately I had to re-install my Windows before I could get a chance
> to test share access from the outside.
>
> Guess what... re-install solved my listSize problem...
>
> Regards,
> Volker
>
> -----Original Message-----
> From: Michael B Allen [mailto:ioplex at gmail.com]
> Sent: Friday, April 29, 2011 7:54 PM
> To: Volker Schmelich
> Cc: jcifs at lists.samba.org
> Subject: Re: [jcifs] SmbFile.listFiles returning too low number of files
>
> Hi Volker,
>
> It is not unexpected that changing
> jcifs.smb.client.{listSize,listCount} changes the result list size
> because the various buffer sizes will be quantized differently. But if
> there isn't some combination that yields the correct results, then it's
> not crystal clear that the code responsible for the various levels of
> buffering has a problem. It could simply be something else entirely like
> some obscure issue with the particular version of Windows you're using.
>
> Can JCIFS correctly list files of another directory on that PC with
> regular sized / named files? Create a shared folder with a few hundred
> files in it and see if JCIFS can list it correctly.
>
> Or maybe this is an artifact of an obscure network issue. Meaning some
> packets are being fragmented in an unsual way that JCIFS is not used to
> handling. That would explain why the results are different each time. Do
> you have another PC from which to run JCIFS so that you can try it over
> another interface other than loopback? If yes, can JCIFS list the share
> correctly from the other PC? As you may have noticed, the loopback
> interface on Windows is a bit odd.
>
> Mike
>
> On Thu, Apr 28, 2011 at 12:11 PM, Volker Schmelich <volker at quadnova.com>
> wrote:
>> Hi Mike,
>>
>> When setting jcifs.smb.client.listCount = 400, I receive a count of
>> 647 files. This is still lower than the expected 664. The default
>> setting of this parameter produces lower results and results may
>> differ occasionally when running the test in a loop with a 1s delay
>> (most times 606, few times in the beginning 644).
>>
>> Using jcifs.smb.client.listSize=32000 and the default for listCount
>> produces a count of 658.
>> Using the above listSize with listCount=400 produces a count of 657.
>>
>> In conclusion, yes, these params influence the count but I could not
>> tweak them so that I get the correct count.
>>
>> The packet capture - did it look strange to you pointing to some
>> specific issues with my current system?
>> I just can't imagine what could be so different in my system that this
>
>> happens on just this one system but that it works on any other system
>> I've tried so far.
>>
>> If you need anything else from me just let me know.
>>
>> Regards,
>> Volker
>>
>> -----Original Message-----
>> From: Michael B Allen [mailto:ioplex at gmail.com]
>> Sent: Wednesday, April 27, 2011 9:05 PM
>> To: Volker Schmelich
>> Cc: jcifs at lists.samba.org
>> Subject: Re: [jcifs] SmbFile.listFiles returning too low number of
>> files
>>
>> [CC jcifs list again now that I have the capture]
>>
>> Hi Volker,
>>
>> Yeah, capturing loopback on Windows is an oddity. Maybe netcap.exe
>> from the Support Tools package would be easier. I don't know. I use
>> Linux for development.
>>
>> But your capture was successful albeit somewhat fragmented as you
> said.
>>
>> Regarding your capture and listing error, my best guess at this point
>> is that the particularly orderly list of long filenames and multiple
>> layers of buffering used by the TRANS2_FIND_FIRST2/NEXT2 commands have
>
>> conspired to generate incorrect results. More specifically I think
>> that even though jcifs.smb.client.listSize = 65535, the
>> jcifs.smb.client.listCount = 200 actually does not fit in the 65535
>> buffer in just such a way to cause errant results. Try setting
>> jcifs.smb.client.listCount = 400 and see if you get different
> behavior.
>> If yes, that would be very interesting. You could also trying playing
>> around with those two properties a little to see if they have any
>> effect. They shouldn't (unless you set them really low).
>> So if they do, it means there's a bug somewhere that would warrant
>> further pursuit.
>>
>> Mike
>>
>> --
>> Michael B Allen
>> Java Active Directory Integration
>> http://www.ioplex.com/
>>
>> On Mon, Apr 18, 2011 at 11:06 PM, Volker Schmelich
>> <volker at quadnova.com>
>> wrote:
>>> Hi Mike,
>>>
>>> Please find attached a ZIP archive containing a Wireshark pcap file.
>>>
>>> The problem is that capturing traffic to localhost seems to be
>>> impossible. I had to add special route according to
>>> http://wiki.wireshark.org/CaptureSetup/Loopback.
>>>
>>> route add 192.168.0.100 mask 255.255.255.255 192.168.0.1 metric 1
>>>
>>> They say: "Doing so, every network traffic from your machine to
>>> itself
>>
>>> will use the physical network interface, it will then go to the
>>> gateway, back to you. Therefor, you will see each packet twice, but
>>> it
>>
>>> can be filtered on the view."
>>>
>>> I hope the capture does help anyway. I saved everything matching this
>
>>> display filter:
>>> udp.port eq 137 or udp.port eq 138 or udp.port eq 139 or udp.port eq
>>> 445 or tcp.port eq 137 or tcp.port eq 138 or tcp.port eq 139 or
>>> tcp.port eq
>>> 445
>>>
>>> Regards,
>>> Volker
>>>
>>>
>>> -----Original Message-----
>>> From: Michael B Allen [mailto:ioplex at gmail.com]
>>> Sent: Saturday, April 16, 2011 7:15 AM
>>> To: Volker Schmelich
>>> Subject: Re: [jcifs] SmbFile.listFiles returning too low number of
>>> files
>>>
>>> Hi Volker,
>>>
>>> I really will need a packet capture to figure out what this problem
>> is.
>>> Logs are not that useful really. Another good tool for getting packet
>
>>> captures that is not described on the page I previously cited is the
>>> Microsoft netcap.exe utility. It is from the "Support Tools"
>>> package.
>>>
>>> It is quite strange that you should run into this now. We have not
>>> had
>>
>>> a bug like this for many years. And it's not like it's something
>>> subtle that could go unnoticed for so long. So I'm slightly
> skeptical.
>>> I cannot help but think there is some outside force that is
>> responsible.
>>>
>>> Mike
>>>
>>> On Thu, Apr 14, 2011 at 9:11 PM, Volker Schmelich
>>> <volker at quadnova.com>
>>> wrote:
>>>> Hi Mike,
>>>>
>>>> I tried it with your suggestions and the outcome was the same.
>>>>
>>>> String sourceDir =
>>>> "file://tempshare:tempshare@192.168.0.100/dceinshare/test/";
>>>> SmbFile src = new SmbFile(sourceDir);
>>>>
>>>> String sourceDir2 = "file:// 192.168.0.100/dceinshare/test/";
>>>> SmbFile
>>
>>>> src = new SmbFile(sourceDir2, new NtlmPasswordAuthentication(null,
>>>> "tempshare", "tempshare"));
>>>>
>>>> Not sure what the fully qualified DNS hostname would be as it is my
>>>> local development box. So what I do is I connect to my own local PC
>>>> via jCIFS. But that should not make any difference as we also do
>>>> this
>>
>>>> on another system where it works fine.
>>>>
>>>> This time I tested with 664 files and different numbers are reported
>
>>>> by listFiles, e.g. 625 or 606.
>>>>
>>>> The result is returned very fast. When I use a Windows 2000 virtual
>>>> machine on the same subnet, it is much slower but it returns the
>>>> correct values.
>>>>
>>>> There also seems to be a reproducible pattern. E.g. when I use the
>>>> listFiles or copyTo method in our application on the same set of
>>>> files, the same files always remain uncopied in the source folder
>>>> (for
>>>
>>>> example
>>>> 1 to 219 is copied, 220 to 2xx is not copied).
>>>>
>>>> My system where this happens has the latest patches and SP1, some
>>>> antivirus software, backup software but I think this should not
>>> matter.
>>>> If it would be some kind of timeout or cache issue I think I should
>>>> not be able to reproduce exactly the same file leftovers during
> copy.
>>>> They should be different each time I try.
>>>>
>>>> I have attached a ZIP archive with three logs produced with log
>>>> level
>>
>>>> 10. That's all I can get right now in terms of packet capture.
>>>>
>>>> W2k_correct.txt contains the log of the correct listing from my
>>>> local
>>
>>>> PC accessing a share on a virtual machine on the same subnet.
>>>>
>>>> W7_incorrect_1.txt contains the log of the wrong listing from my
>>>> local
>>>
>>>> PC accessing a share on my local PC. It returned 606 instead of 664.
>>>>
>>>> W7_incorrect_2.txt contains the log of the wrong listing from my
>>>> local
>>>
>>>> PC accessing a share on my local PC. It returned 625 instead of 664.
>>>> This was taken just some seconds after the first one. Nothing
>>>> changed
>>
>>>> in source code or file system, just a new click on run.
>>>>
>>>>
>>>> Thanks for your help. It would be good to know if we just have to
>>>> stay
>>>
>>>> away from certain operating systems or if this issue could hit us
>>>> sooner or later on all of them.
>>>>
>>>> Regards,
>>>> Volker
>>>>
>>>> -----Original Message-----
>>>> From: Michael B Allen [mailto:ioplex at gmail.com]
>>>> Sent: Thursday, April 14, 2011 8:24 PM
>>>> To: Volker Schmelich
>>>> Cc: jcifs at lists.samba.org
>>>> Subject: Re: [jcifs] SmbFile.listFiles returning too low number of
>>>> files
>>>>
>>>> On Tue, Apr 12, 2011 at 3:29 PM, Volker Schmelich
>>>> <volker at quadnova.com>
>>>> wrote:
>>>>> Hi there,
>>>>>
>>>>> We experience a strange issue on one of our Windows PCs with jCIFS
>>>>> 1.3.14 and 1.3.15. The Windows PC is running Windows 7 Home Premium
>>>>> x64 with SP1. We use JDK 1.6.0_24 both 32 bit and 64 bit.
>>>>>
>>>>> When we call SmbFile.listFiles(), the method does not return all
>>>>> files
>>>>
>>>>> available in the directory. E.g. we have 1146 files in there, but
>>>>> the
>>>
>>>>> method returns 1094 and sometimes 1127 (when we let it run in a
>>>>> loop
>>
>>>>> with 1s delay).
>>>>>
>>>>> The bug is reproducible on this one PC every time we use more than
>>>>> about
>>>>> 200 files. It works fine on one of our Windows 2000 and one of our
>>>>> 2003 servers and other Windows 7 PCs, too.
>>>>>
>>>>>
>>>>> This Java snippet is as simple as it gets to reproduce it:
>>>>>
>>>>> String sourceDir =
>>>>> "file://tempshare:tempshare@vs-cyberpower/dceinshare/test2/";
>>>>> SmbFile src = new SmbFile(sourceDir); SmbFile[] list =
>>>>> src.listFiles(); int cnt1 = list.length;
>>>>>
>>>>> As copyTo depends on listFiles, copying all the files in the
>>>>> directory
>>>>
>>>>> does not copy all of them, only those returned by listFiles.
>>>>>
>>>>> There are no exceptions, no stack traces. The example above uses
>>>>> the
>>
>>>>> plain JAR file, no extra jCFIS configurations.
>>>>>
>>>>> We also tried jcifs.smb.client.attrExpirationPeriod = 0 with no
>>>>> difference.
>>>>>
>>>>> Is there any direction you could point us - we tried to figure it
>>>>> out
>>>
>>>>> for hours but we are lost.
>>>>
>>>> Hi Volker,
>>>>
>>>> Try the fully qualified DNS hostname. And for testing it might be
>>>> better to use the IP address. Also, supply the credentials to
>>>> SmbFile
>>
>>>> using an NtlmPasswordAuthentication object. If that changes the
>>>> result, we will have a better idea of what the problem is. If that
>>>> does not change the behavior, can you get a packet capture?
>>>>
>>>> Mike
>>>>
>>>> --
>>>> Michael B Allen
>>>> Java Active Directory Integration
>>>> http://www.ioplex.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Michael B Allen
>>> Java Active Directory Integration
>>> http://www.ioplex.com/
>>>
>>
>
>
>
> --
> Michael B Allen
> Java Active Directory Integration
> http://www.ioplex.com/
>



-- 
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/