issue with fcntl in smbsh on redhat linux 7.2

[Stieglitz, Eric J. (DCSA)]

Hi.

I'm trying to debug a problem with smbsh running on Linux
with samba 2.2.4. When `smbsh` is run from the command line,
it asks for a username/password combination, and then dies
with the error

	ERROR: can't get smbw shared lock (Invalid argument)

A compilation of smbsh with the same options passed to `configure`
on a Solaris 2.8 system worked fine without any problems, so this
seems to be specific to Linux.

(Note that I'm logged into the machine as userid 500 and su'd to root.)

As far as I can tell, the program runs in two separate steps.
In the first, `smbsh` asks for my username and password, and sets
the variable shared_fd to a file descriptor to which it writes the
domain, user, and password combination. In the second, it runs
execve on /bin/bash, and the same functions which worked before
now fail miserably.

During the first part, all this works fine. The function fcntl_lock
is called multiple times to work on the file descriptor shared_fd
(set previously to a non-existent temp file) with the following arguments:

	fd = 8191, op =  13, offset =  0, count =  1, type =  2
	fd = 8191, op =  14, offset =  0, count =  1, type =  1

Note that for the variable "op", 13 == F_SETLK64 and 14 == F_SETLKW64.
An fstat on shared_fd gives the following:

	device: 2055    inode: 215332   mode: 33152     links: 0       
	uid: 0  gid: 0	device: 0       size: 0

"Size" changes, as domain, user, and pass information gets written.

The function calls leading up to each of these are:

	smbw_getshared -> lockit -> fcntl_lock -> sys_fcntl_ptr -> fcntl

All these calls complete successfully as far as I can tell.
While in the function fcntl_lock, the SMB_STRUCT_FLOCK structure
passed to sys_fcntl_ptr and fcntl is usually described by one of
the following settings:

	l_type: 2, l_whence: 0, l_start: 0, l_len: 1, l_pid: 0
	l_type: 1, l_whence: 0, l_start: 0, l_len: 1, l_pid: 0

Up until this point, everything works fine. Note that all of the
function calls above seem to be linked directly into the `smbsh`
binary.

Then, the software seems to run an "execve("/bin/bash", ["smbsh"], ..."
and begins using /usr/bin/smbwrapper.so for all of these functions
calls. Here, things begin going wrong.

The specific problem seems to occur during the following set of
function calls

	open64 -> smbw_path -> smbw_init -> smbw_getshared ->
	lockit -> fcntl_lock -> sys_fcntl_ptr -> fcntl

FYI, the command in smbw_init where the failure occurs is at
	if((p=smbw_getshared("LOGFILE"))) { ... }

fcntl() returns with value "-1" and sets errno to "22" ("EINVAL").

Until the lockit() function, shared_fd is set to my tty (/dev/pts/5)
with the following from an fstat():

	device: 5       inode: 7        mode: 8592      links: 1
	uid: 500	gid: 5  device: 34821   size: 0

Note that the int value of shared_fd here is '0'.

During the following snippet of code from lockit()...

        if (shared_fd == 0) {
                char *p = getenv("SMBW_HANDLE");
                if (!p) {
                        DEBUG(0,("ERROR: can't get smbw shared handle\n"));
                        exit(1);
                }
                shared_fd = atoi(p);
        }

...shared_fd is reset to 8191 -- the file descriptor to the temp file
above -- because it is '0'. When 8191 gets passed down, fcntl() dies
with EINVAL.

I've tried removing the snippet above and allowing shared_fd to remain '0'
and pointing at /dev/pts/5. This also results in an EINVAL error from
fcntl().

So, I'm wondering

1) Is '0' a valid value for shared_fd? If not, what could have caused this?
2) Should lockit() be working on /dev/pts/5 or whatever might have been in fd 8191?
3) What *should* be passed through these functions at this point?
4) How can I correct this? Is this a known bug?


Any help would be appreciated.

/EJS