[jcifs] listFiles produce multiple entrys for some files..

[Clapham, Paul]

Hi Mike,

I sent you a log from that program off-list.

PC2 

-----Original Message-----
From: Michael B Allen [mailto:ioplex at gmail.com] 
Sent: October 26, 2009 08:55
To: Clapham, Paul
Cc: michael.studer at sisa.ch; jcifs at lists.samba.org
Subject: Re: [jcifs] listFiles produce multiple entrys for some files..

Hi Paul and Michael,

Better still, everyone should be using examples/ListFiles.java to start so that we are all on the same page.

Thanks for helping out.

Mike

On Mon, Oct 26, 2009 at 11:29 AM, Clapham, Paul <pclapham at core-mark.com> wrote:
> I have access to AS400 and I'm willing to work on this problem. I asked michael.studer (accidentally off-list) to send me a short compilable program which I could modify and test. So far I just got a large lot of code which was way too much to deal with. Seems to me that a program to test this should be less than 50 lines of code.
>
> PC2
>
> -----Original Message-----
> From: jcifs-bounces at lists.samba.org 
> [mailto:jcifs-bounces at lists.samba.org] On Behalf Of Michael B Allen
> Sent: October 26, 2009 08:25
> To: michael.studer at sisa.ch
> Cc: jcifs at lists.samba.org
> Subject: Re: [jcifs] listFiles produce multiple entrys for some files..
>
> You could send me a packet capture. But I don't have access to AS400 and I don't really have "Free" time so unless it's an obvious fix this could sit for a while. But I know JCIFS has worked with mainframes before so I have to wonder why this is showing up now. Is AS400 really old or really new or unmaintained?
>
> Mike
>
> On Mon, Oct 12, 2009 at 3:17 AM,  <michael.studer at sisa.ch> wrote:
>>
>> Hi again
>>
>> I figured out, the problem seems to occur only on AS400 shares. I did 
>> not face it on windows servers.
>>
>> Again, i try to explain the problem. When i use the "smbFile.listFiles()"
>> method on a file, which is located inside a AS400 share, the amount 
>> of returned files is multiple times the actual amount it should return.
>> For example: 2 Files in the share returns me 3 Files, where one File 
>> is twice in the collection. This would not be such a big deal, its 
>> just getting nasty when you have folders with a lot of Files inside.
>> While i tested, i faced some folders with arround 5000+ Files. This 
>> results in arround 250'000 returned SMBFile Objects, with a total 
>> used amount of like 500+MB of memory and a runtime out of memory.
>>
>> Heres the full code for my directory reading method(s):
>>
>>         public HashSet<String> getFilesList(String relativePath, 
>> boolean recursive, boolean includeFolders) throws 
>> MalformedURLException, SmbException, ExsException {
>>
>>                 HashSet<String> collection = new HashSet<String>();
>>                 HashSet<String> unreadable = new HashSet<String>();
>>
>>
>> getFilesList(relativePath,recursive,includeFolders,false,collection,
>> unreadable);
>>
>>                 return collection;
>>
>>         }
>>
>>         private void getFilesList(String path, boolean recursive, 
>> boolean includeFolders, boolean isFullPath, HashSet<String> 
>> collection, HashSet<String> unreadable) throws MalformedURLException, 
>> SmbException, ExsException {
>>
>>                 if(path == null) {
>>                         path = "";
>>                 }
>>
>>                 if(path.trim().length() > 0 &&
>> (!path.trim().endsWith("/") && !path.trim().endsWith("\\"))) {
>>                         path = path + "/";
>>                 }
>>
>>                 String url = "";
>>
>>                 if(isFullPath) {
>>                         url = path;
>>                 } else {
>>                         url = generateURLString(path);
>>                 }
>>
>>                 SmbFile smbFolder = new SmbFile(url);
>>
>>                 if(!smbFolder.exists() || !smbFolder.isDirectory()) {
>>                         throw new ExsException("Error: Folder cannot 
>> be read from smb share: " + smbFolder.getPath());
>>                 }
>>
>>                 if(!smbFolder.canRead()) {
>>                         unreadable.add(smbFolder.getPath());
>>                         return;
>>                 }
>>
>>                 // this helps preventing heap overflows but it wont 
>> prevent it when to many files at once are opend
>>                 HashSet<SmbFile> uniqueList = new HashSet<SmbFile>();
>>                 for(SmbFile currentSmbFile : smbFolder.listFiles()) {
>>                         uniqueList.add(currentSmbFile);
>>                 }
>>
>>                 for(SmbFile currentSmbFile : uniqueList) {
>>
>>                         String currentPath = 
>> currentSmbFile.getPath();
>>
>>                         if(currentSmbFile.isFile()) {
>>                                 currentSmbFile = null;
>>                                 collection.add(currentPath);
>>
>>                         } else if(currentSmbFile.isDirectory()) {
>>                                 int size = 0;
>>                                 try {
>>                                         String[] list = 
>> currentSmbFile.list();
>>                                         size = list.length;
>>                                         list = null;
>>                                 } catch(Throwable t) {
>>                                         continue;
>>                                 }
>>
>>                                 currentSmbFile = null;
>>                                 if(includeFolders) {
>>                                         collection.add(currentPath);
>>                                 }
>>                                 if(recursive) {
>>                                         if(size > 0) {
>>
>> getFilesList(currentPath, recursive, includeFolders, true, 
>> collection, unreadable);
>>                                         }
>>                                 }
>>                         }
>>                 }
>>
>>                 uniqueList.clear();
>>                 uniqueList = null;
>>         }
>>
>>
>>
>
>
>
> --
> Michael B Allen
> Java Active Directory Integration
> http://www.ioplex.com/
>



--
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/