[jcifs] SmbFile.copyTo - how reliable under heavy load

[Volker Schmelich]

Hi Mike,

Thank you for the clarification on copyTo.

When switching to stream copy we receive an SmbException 0xC0000205
(NT_STATUS_INSUFF_SERVER_RESOURCES - "Not enough server storage is
available to process this command.").
The description is a bit misleading because storage space is available
but it seems to indicate that there are not enough system resources
available to process the request.

May I suggest to add a note to the API docs regarding the reliability of
copyTo, i.e. one cannot rely on exception on copy error? Just from
looking at the API docs I would never have guessed that there could be
such a problem. As it is not documented when a SmbException is thrown, I
have assumed it will be thrown on error (which a failed copy is in my
opinion).

What would you suggest for our use case scenario - copy one file at a
time but many different files in parallel (reliability is top priority,
disastrous if a file does not get copied):

1) Use copyTo() and check for exists() within in a loop until exists()
== true or number of retries reached.

Pro: easy to code, do not have to mess with manual create times,
attributes, stream handling
Con: maybe too many jCIFS threads created (overhead, thread deadlocks?);
additional call to exists() necessary which costs performance

2) Use I/O streams to copy file byte by byte.

Pro: I assume we can rely on SmbException thrown on error and if none is
thrown, the destination file exists. We do not have to call exists()
afterwards to verify.
Con: More manual code needed, handling of existing files / overwriting,
exceptions possible at multiple places.

My preference would be (1) because it is easier to use and in my tests I
got a successful copy after 2 retries max. Do you think this is a good
approach given our usage scenario or should copyTo() only be used if we
intend to copy multiple files/directories at once because the "side
effects" might outweigh the convenience if we deal with single files
only?

Regards,
Volker



-----Original Message-----
From: Michael B Allen [mailto:ioplex at gmail.com] 
Sent: Monday, June 06, 2011 9:11 PM
To: Volker Schmelich
Cc: jcifs at lists.samba.org
Subject: Re: [jcifs] SmbFile.copyTo - how reliable under heavy load

Hi Volker,

SmbFile.copyTo is designed to copy entire directories in an efficient
way (it can read and write data at the same time), set attributes and
timestamps and so on. But if an error occurs on one file, rather than
aborting the entire copy, it will just log the exception and move on.
So it is very possible that some file was temporarily inaccessible such
as because it was locked or the network stalled and that one file was
never successfully copied.

If you want to explicitly validate each step of a copy you will just
have to write that code yourself. Meaning open the source file with
SmbFileInputStream and the destination file with SmbFileOutputStream,
read / write in a loop, close, etc. That will give you the opportunity
to handle any errors along the way.

Also, make sure that you do not have more than one thread operating on
the same SmbFile object at the same time. SmbFile methods are not
thread-safe. If you want to operate on the same file at the same time
using multiple threads that is ok but you will have to create separate
SmbFile objects. If your threads are trying to operate on the same
object at the same time, that certainly could yield odd results.

Mike

On Thu, Jun 2, 2011 at 1:17 PM, Volker Schmelich <volker at quadnova.com>
wrote:
> Hi there,
>
> We are using the latest jCIFS on Windows within two applications. Both

> applications are multithreaded and do work which includes a lot of 
> file copy operations with jCIFS.
>
> So for example, app 1 has 5 worker threads which in turn copy let's 
> say
> 50 files on average from one share to another share on the same box.
> App2 also has 5 worker threads and is doing the same, basically 
> waiting for app1 to signal done, then pick up the files and copy from 
> a share to another. This is a development scenario - in production 
> both apps will run on different servers.
>
> Under this heavy load, some copied files do not appear in the target.
> There is no SmbException thrown. It seems like the copyTo is called, 
> does not work, throws no Exception to indicate it did not work.
>
> When I wrap the copyTo call in a retry loop and check the target file 
> existence, the copyTo succeeds after the first or second retry.
>
> So my question is: Is it possible, that jCIFS (or specifically the 
> copy
> thread) dies but no SmbException is thrown? Maybe because the thread 
> died?
>
> Regards,
> Volker
>



--
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/