[linux-cifs-client] files disappearing randomly - kernel 2.6.27 cifs.ko 1.54 - VFS: No response for cmd 50

Mon Apr 20 15:26:55 GMT 2009

Hello,

We recently noticed that one of our smb mounts sometimes only shows a small amount of files.
We e.g. do a "ls -1 | wc -l" and noticed that sometimes only 148 files are shown but 
the directory contains 2074 files. Just a few seconds later "ls -1 | wc -l" will again show 2074 files.
We also monitored the share with a Windows client but never saw files disappearing.

Kernel is Linux vam-uti3 2.6.27-gentoo-r10 #1 SMP Tue Apr 14 14:04:02 CEST 2009 i686 Intel(R) Xeon(TM) CPU 2.40GHz GenuineIntel GNU/Linux

# modinfo /lib/modules/2.6.27-gentoo-r10/kernel/fs/cifs/cifs.ko
filename:       /lib/modules/2.6.27-gentoo-r10/kernel/fs/cifs/cifs.ko
version:        1.54
description:    VFS to access servers complying with the SNIA CIFS Specification e.g. Samba and Windows
license:        GPL
author:         Steve French <sfrench at us.ibm.com>
srcversion:     C4B33C4846085F49140276C
depends:        
vermagic:       2.6.27-gentoo-r10 SMP mod_unload PENTIUM4 
parm:           CIFSMaxBufSize:Network buffer size (not including header). Default: 16384 Range: 8192 to 130048 (int)
parm:           cifs_min_rcv:Network buffers in pool. Default: 4 Range: 1 to 64 (int)
parm:           cifs_min_small:Small network buffers in pool. Default: 30 Range: 2 to 256 (int)
parm:           cifs_max_pending:Simultaneous requests to server. Default: 50 Range: 2 to 256 (int)

This is the share that causes problems:

# cat /proc/fs/cifs/DebugData 
...
4) Name: 10.2.13.177  Domain: ASCHENDORFF Uses: 1 OS: Windows Server 2003 R2 3790 Service Pack 2
        NOS: Windows Server 2003 R2 5.2 Capability: 0x1f3fd
        SMB session status: 1   TCP status: 1
        Local Users To Server: 2 SecMode: 0x3 Req On Wire: 0
        Shares:
        1) \\prod-fs3\Dispatch Mounts: 1 Type: NTFS DevInfo: 0x20 Attributes: 0x700ff
PathComponentMax: 255 Status: 0x1 type: DISK 

Therefore we started to have a look into the /var/log/messages 
Which contains a lot of these messages:

Apr 20 12:11:43 vam-uti3 CIFS VFS: Unexpected lookup error -5
Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 64280
Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55802
Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 117 mid 55801
Apr 20 12:11:43 vam-uti3 CIFS VFS: Unexpected lookup error -11
Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55803
Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 64286
Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55808
Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55809
Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 64294
Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55816
Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55817
Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55825
Apr 20 12:11:43 vam-uti3 CIFS VFS: No response for cmd 50 mid 55828
Apr 20 12:12:04 vam-uti3 CIFS VFS: server not responding
Apr 20 12:12:04 vam-uti3 CIFS VFS: No response for cmd 114 mid 55830
Apr 20 12:12:04 vam-uti3 CIFS VFS: Unexpected lookup error -112
Apr 20 12:12:04 vam-uti3 CIFS VFS: No response to cmd 115 mid 55831
Apr 20 12:12:04 vam-uti3 CIFS VFS: Send error in SessSetup = -11
Apr 20 12:12:04 vam-uti3 CIFS VFS: Unexpected lookup error -11

I did a "echo > 4 /proc/fs/cifs/cifsFYI" at around the same timepoint when
we were missing most of the files (saw only about 143 files out of 1901) in ls output.
I attached the /var/log/messages 

I also have a tcpdump capture avilable but I lag to post it 
since it contains smb passwords and probably other sensitive information.

Also we sometimes can't access the mount point at all:
vam-uti3:/data/dispatch/incoming/prod-fs3-ISDN_EINGANG # ls
ls: cannot open directory .: Resource temporarily unavailable

Apr 20 12:24:04 vam-uti3 CIFS VFS: server not responding
Apr 20 12:24:04 vam-uti3 CIFS VFS: server not responding
Apr 20 12:24:04 vam-uti3 CIFS VFS: No response to cmd 115 mid 25210
Apr 20 12:24:04 vam-uti3 CIFS VFS: Send error in SessSetup = -11
Apr 20 12:24:04 vam-uti3 CIFS VFS: No response for cmd 114 mid 25209
Apr 20 12:24:04 vam-uti3 CIFS VFS: Unexpected lookup error -11
Apr 20 12:24:04 vam-uti3 CIFS VFS: Unexpected lookup error -112
Apr 20 12:24:04 vam-uti3 CIFS VFS: Send error in SessSetup = -11
Apr 20 12:24:34 vam-uti3 CIFS VFS: server not responding
Apr 20 12:24:34 vam-uti3 CIFS VFS: No response to cmd 115 mid 25215
Apr 20 12:24:34 vam-uti3 CIFS VFS: Send error in SessSetup = -11
Apr 20 12:24:34 vam-uti3 CIFS VFS: Unexpected lookup error -11
Apr 20 12:24:34 vam-uti3 CIFS VFS: No response for cmd 114 mid 25214
Apr 20 12:24:34 vam-uti3 CIFS VFS: Unexpected lookup error -112
Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 50 mid 26581
Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 50 mid 34408
Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 50 mid 26594
Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 50 mid 34432
Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 162 mid 34439
Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 50 mid 26606
Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 162 mid 34443
Apr 20 12:24:39 vam-uti3 CIFS VFS: No response for cmd 50 mid 26619
Apr 20 12:24:39 vam-uti3 CIFS VFS: No response to cmd 46 mid 34459
Apr 20 12:24:39 vam-uti3 CIFS VFS: Send error in read = -11

Help and hints how to proceed/debug further are very much appreciated!

TIA Jodok Muellers

-------------- next part --------------
A non-text attachment was scrubbed...
Name: files_disappearing.log.gz
Type: application/x-gzip
Size: 7851 bytes
Desc: not available
Url : http://lists.samba.org/archive/linux-cifs-client/attachments/20090420/93872388/files_disappearing.log.bin