[jcifs] SmbFile.copyTo - how reliable under heavy load

[Michael B Allen]

On Tue, Jun 7, 2011 at 11:00 PM, Volker Schmelich <volker at quadnova.com> wrote:
> Hi Mike,
>
> Thank you for the clarification on copyTo.
>
> When switching to stream copy we receive an SmbException 0xC0000205
> (NT_STATUS_INSUFF_SERVER_RESOURCES - "Not enough server storage is
> available to process this command.").
> The description is a bit misleading because storage space is available
> but it seems to indicate that there are not enough system resources
> available to process the request.
>
> May I suggest to add a note to the API docs regarding the reliability of
> copyTo, i.e. one cannot rely on exception on copy error?

This has is already on the TODO list as per list dialog with
cje at atp.dk Mar 3, 2009.

Mike

> Just from
> looking at the API docs I would never have guessed that there could be
> such a problem. As it is not documented when a SmbException is thrown, I
> have assumed it will be thrown on error (which a failed copy is in my
> opinion).
>
> What would you suggest for our use case scenario - copy one file at a
> time but many different files in parallel (reliability is top priority,
> disastrous if a file does not get copied):
>
> 1) Use copyTo() and check for exists() within in a loop until exists()
> == true or number of retries reached.
>
> Pro: easy to code, do not have to mess with manual create times,
> attributes, stream handling
> Con: maybe too many jCIFS threads created (overhead, thread deadlocks?);
> additional call to exists() necessary which costs performance
>
> 2) Use I/O streams to copy file byte by byte.
>
> Pro: I assume we can rely on SmbException thrown on error and if none is
> thrown, the destination file exists. We do not have to call exists()
> afterwards to verify.
> Con: More manual code needed, handling of existing files / overwriting,
> exceptions possible at multiple places.
>
> My preference would be (1) because it is easier to use and in my tests I
> got a successful copy after 2 retries max. Do you think this is a good
> approach given our usage scenario or should copyTo() only be used if we
> intend to copy multiple files/directories at once because the "side
> effects" might outweigh the convenience if we deal with single files
> only?
>
> Regards,
> Volker
>
>
>
> -----Original Message-----
> From: Michael B Allen [mailto:ioplex at gmail.com]
> Sent: Monday, June 06, 2011 9:11 PM
> To: Volker Schmelich
> Cc: jcifs at lists.samba.org
> Subject: Re: [jcifs] SmbFile.copyTo - how reliable under heavy load
>
> Hi Volker,
>
> SmbFile.copyTo is designed to copy entire directories in an efficient
> way (it can read and write data at the same time), set attributes and
> timestamps and so on. But if an error occurs on one file, rather than
> aborting the entire copy, it will just log the exception and move on.
> So it is very possible that some file was temporarily inaccessible such
> as because it was locked or the network stalled and that one file was
> never successfully copied.
>
> If you want to explicitly validate each step of a copy you will just
> have to write that code yourself. Meaning open the source file with
> SmbFileInputStream and the destination file with SmbFileOutputStream,
> read / write in a loop, close, etc. That will give you the opportunity
> to handle any errors along the way.
>
> Also, make sure that you do not have more than one thread operating on
> the same SmbFile object at the same time. SmbFile methods are not
> thread-safe. If you want to operate on the same file at the same time
> using multiple threads that is ok but you will have to create separate
> SmbFile objects. If your threads are trying to operate on the same
> object at the same time, that certainly could yield odd results.
>
> Mike
>
> On Thu, Jun 2, 2011 at 1:17 PM, Volker Schmelich <volker at quadnova.com>
> wrote:
>> Hi there,
>>
>> We are using the latest jCIFS on Windows within two applications. Both
>
>> applications are multithreaded and do work which includes a lot of
>> file copy operations with jCIFS.
>>
>> So for example, app 1 has 5 worker threads which in turn copy let's
>> say
>> 50 files on average from one share to another share on the same box.
>> App2 also has 5 worker threads and is doing the same, basically
>> waiting for app1 to signal done, then pick up the files and copy from
>> a share to another. This is a development scenario - in production
>> both apps will run on different servers.
>>
>> Under this heavy load, some copied files do not appear in the target.
>> There is no SmbException thrown. It seems like the copyTo is called,
>> does not work, throws no Exception to indicate it did not work.
>>
>> When I wrap the copyTo call in a retry loop and check the target file
>> existence, the copyTo succeeds after the first or second retry.
>>
>> So my question is: Is it possible, that jCIFS (or specifically the
>> copy
>> thread) dies but no SmbException is thrown? Maybe because the thread
>> died?
>>
>> Regards,
>> Volker
>>
>
>
>
> --
> Michael B Allen
> Java Active Directory Integration
> http://www.ioplex.com/
>



-- 
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/