[jcifs] listFiles produce multiple entrys for some files..
Michael B Allen
ioplex at gmail.com
Mon Oct 26 11:09:11 MDT 2009
Oh yeah. Wow. That's pretty messed up. Can you send me a packet capture?
On Mon, Oct 26, 2009 at 12:49 PM, Clapham, Paul <pclapham at core-mark.com> wrote:
> Hi Mike,
>
> I sent you a log from that program off-list.
>
> PC2
>
> -----Original Message-----
> From: Michael B Allen [mailto:ioplex at gmail.com]
> Sent: October 26, 2009 08:55
> To: Clapham, Paul
> Cc: michael.studer at sisa.ch; jcifs at lists.samba.org
> Subject: Re: [jcifs] listFiles produce multiple entrys for some files..
>
> Hi Paul and Michael,
>
> Better still, everyone should be using examples/ListFiles.java to start so that we are all on the same page.
>
> Thanks for helping out.
>
> Mike
>
> On Mon, Oct 26, 2009 at 11:29 AM, Clapham, Paul <pclapham at core-mark.com> wrote:
>> I have access to AS400 and I'm willing to work on this problem. I asked michael.studer (accidentally off-list) to send me a short compilable program which I could modify and test. So far I just got a large lot of code which was way too much to deal with. Seems to me that a program to test this should be less than 50 lines of code.
>>
>> PC2
>>
>> -----Original Message-----
>> From: jcifs-bounces at lists.samba.org
>> [mailto:jcifs-bounces at lists.samba.org] On Behalf Of Michael B Allen
>> Sent: October 26, 2009 08:25
>> To: michael.studer at sisa.ch
>> Cc: jcifs at lists.samba.org
>> Subject: Re: [jcifs] listFiles produce multiple entrys for some files..
>>
>> You could send me a packet capture. But I don't have access to AS400 and I don't really have "Free" time so unless it's an obvious fix this could sit for a while. But I know JCIFS has worked with mainframes before so I have to wonder why this is showing up now. Is AS400 really old or really new or unmaintained?
>>
>> Mike
>>
>> On Mon, Oct 12, 2009 at 3:17 AM, <michael.studer at sisa.ch> wrote:
>>>
>>> Hi again
>>>
>>> I figured out, the problem seems to occur only on AS400 shares. I did
>>> not face it on windows servers.
>>>
>>> Again, i try to explain the problem. When i use the "smbFile.listFiles()"
>>> method on a file, which is located inside a AS400 share, the amount
>>> of returned files is multiple times the actual amount it should return.
>>> For example: 2 Files in the share returns me 3 Files, where one File
>>> is twice in the collection. This would not be such a big deal, its
>>> just getting nasty when you have folders with a lot of Files inside.
>>> While i tested, i faced some folders with arround 5000+ Files. This
>>> results in arround 250'000 returned SMBFile Objects, with a total
>>> used amount of like 500+MB of memory and a runtime out of memory.
>>>
>>> Heres the full code for my directory reading method(s):
>>>
>>> public HashSet<String> getFilesList(String relativePath,
>>> boolean recursive, boolean includeFolders) throws
>>> MalformedURLException, SmbException, ExsException {
>>>
>>> HashSet<String> collection = new HashSet<String>();
>>> HashSet<String> unreadable = new HashSet<String>();
>>>
>>>
>>> getFilesList(relativePath,recursive,includeFolders,false,collection,
>>> unreadable);
>>>
>>> return collection;
>>>
>>> }
>>>
>>> private void getFilesList(String path, boolean recursive,
>>> boolean includeFolders, boolean isFullPath, HashSet<String>
>>> collection, HashSet<String> unreadable) throws MalformedURLException,
>>> SmbException, ExsException {
>>>
>>> if(path == null) {
>>> path = "";
>>> }
>>>
>>> if(path.trim().length() > 0 &&
>>> (!path.trim().endsWith("/") && !path.trim().endsWith("\\"))) {
>>> path = path + "/";
>>> }
>>>
>>> String url = "";
>>>
>>> if(isFullPath) {
>>> url = path;
>>> } else {
>>> url = generateURLString(path);
>>> }
>>>
>>> SmbFile smbFolder = new SmbFile(url);
>>>
>>> if(!smbFolder.exists() || !smbFolder.isDirectory()) {
>>> throw new ExsException("Error: Folder cannot
>>> be read from smb share: " + smbFolder.getPath());
>>> }
>>>
>>> if(!smbFolder.canRead()) {
>>> unreadable.add(smbFolder.getPath());
>>> return;
>>> }
>>>
>>> // this helps preventing heap overflows but it wont
>>> prevent it when to many files at once are opend
>>> HashSet<SmbFile> uniqueList = new HashSet<SmbFile>();
>>> for(SmbFile currentSmbFile : smbFolder.listFiles()) {
>>> uniqueList.add(currentSmbFile);
>>> }
>>>
>>> for(SmbFile currentSmbFile : uniqueList) {
>>>
>>> String currentPath =
>>> currentSmbFile.getPath();
>>>
>>> if(currentSmbFile.isFile()) {
>>> currentSmbFile = null;
>>> collection.add(currentPath);
>>>
>>> } else if(currentSmbFile.isDirectory()) {
>>> int size = 0;
>>> try {
>>> String[] list =
>>> currentSmbFile.list();
>>> size = list.length;
>>> list = null;
>>> } catch(Throwable t) {
>>> continue;
>>> }
>>>
>>> currentSmbFile = null;
>>> if(includeFolders) {
>>> collection.add(currentPath);
>>> }
>>> if(recursive) {
>>> if(size > 0) {
>>>
>>> getFilesList(currentPath, recursive, includeFolders, true,
>>> collection, unreadable);
>>> }
>>> }
>>> }
>>> }
>>>
>>> uniqueList.clear();
>>> uniqueList = null;
>>> }
>>>
>>>
>>>
>>
>>
>>
>> --
>> Michael B Allen
>> Java Active Directory Integration
>> http://www.ioplex.com/
>>
>
>
>
> --
> Michael B Allen
> Java Active Directory Integration
> http://www.ioplex.com/
>
--
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/
More information about the jCIFS
mailing list