[Samba] Possible Filesystem Corruption with Samba 3.0.25a (with XFS and LVM)

[Andri]

Glad to finally see a reply not 100% about my hardware.

Andrew Morgan wrote:
> Samba uses standard system calls to create, modify, and delete files. 
> It does not write to random bits of /dev/hda.  If you have filesystem
> corruption, then the problem lies elsewhere.

It's not FS corruption per se, it's just that something wrote over the FS.

> Maybe the data you found came from Samba (indirectly through files your
> Bittorrent client was saving to a Samba share), but that does not imply
> that Samba was the cause of the problem.  When Samba used the system
> call write() - or whatever optimized system call it uses - some other
> piece of software (XFS, LVM, Linux kernel IDE driver) placed that data
> in the wrong place on the disk.

Possible, of course, but I or some more knowledgeable person could start tracing
if the origin of the data is found :)

> In my experience (which only counts as anecdotal evidence anyways), disk
> hardware failures are usually easily detected as ever-increasing bad
> block counts reported by the disk's S.M.A.R.T. firmware.  If the disk
> still works normally and is not reporting any SMART errors, then you can
> probably rule out hardware.

I have, and hoped my first hint of 'hardware is healthy' in my first posting
would satisfy others :) As I've explained, this issue has little to do with the
hardware, and more with seeming structured data being placed over my root partition.

This would be a good time to mention that while checking the other blocks with
similar structure, I discovered a similarity -- they seem to start with 0x47
0x71 as can be seen from the example below that I took from offset ~38MB.

00000140  00 00 00 00 00 00 00 00  47 71 00 00 00 00 01 00  |........Gq......|
00000150  89 00 12 00 07 00 00 00  40 00 00 00 b6 08 67 46  |........ at .....gF|
00000160  17 59 0c 00 00 fd 00 00  00 00 00 00 80 47 79 c0  |.Y...........Gy.|
00000170  00 00 00 00 57 31 00 00  e8 03 00 00 00 00 00 00  |....W1..........|
00000180  2f 73 74 6f 72 61 67 65  00 4d 65 64 69 61 2f 41  |/storage.Media/A|
00000190  75 64 69 6f 2f 4d 75 73  69 63 2f 41 6c 62 75 6d  |udio/Music/Album|
000001a0  73 2f 42 6c 69 6e 6b 20  31 38 32 2f 45 6e 65 6d  |s/Blink 182/Enem|
000001b0  61 20 6f 66 20 74 68 65  20 53 74 61 74 65 2f 30  |a of the State/0|
000001c0  31 20 2d 20 42 6c 69 6e  6b 20 31 38 32 20 2d 20  |1 - Blink 182 - |
000001d0  44 75 6d 70 77 65 65 64  2e 6d 70 33 00 20 56 65  |Dumpweed.mp3. Ve|
000001e0  73 6b 69 20 2d 20 4c 6f  68 75 74 75 73 65 6b 73  |ski - Lohutuseks|
000001f0  20 c3 9c 6d 62 65 72 20  4d 61 61 69 6c 6d 61 2e  | ..mber Maailma.|
00000200  6d 70 33 00 20 01 00 00  00 00 00 00 10 01 00 00  |mp3. ...........|
00000210  10 00 00 00 c9 00 00 00  41 92 47 41 ad de e1 fe  |........A.GA....|
00000220  00 fd 00 00 00 00 00 00  91 78 62 b8 00 00 00 00  |.........xb.....|
00000230  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000240  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000260  00 00 00 00 00 00 00 00  47 71 00 00 00 00 01 00  |........Gq......|
00000270  89 00 12 00 07 00 00 00  40 00 00 00 69 09 67 46  |........ at ...i.gF|
00000280  bb 20 06 00 00 fd 00 00  00 00 00 00 91 78 62 b8  |. ...........xb.|
00000290  00 00 00 00 09 39 00 00  e8 03 00 00 00 00 00 00  |.....9..........|
000002a0  2f 73 74 6f 72 61 67 65  00 4d 65 64 69 61 2f 41  |/storage.Media/A|
000002b0  75 64 69 6f 2f 4d 75 73  69 63 2f 41 6c 62 75 6d  |udio/Music/Album|
000002c0  73 2f 4a 61 6d 65 73 20  42 6c 75 6e 74 2f 42 61  |s/James Blunt/Ba|
000002d0  63 6b 20 74 6f 20 42 65  64 6c 61 6d 2f 30 31 20  |ck to Bedlam/01 |
000002e0  2d 20 4a 61 6d 65 73 20  42 6c 75 6e 74 20 2d 20  |- James Blunt - |
000002f0  48 69 67 68 2e 6d 70 33  00 41 6d 61 6e 64 61 20  |High.mp3.Amanda |
00000300  4c 65 61 72 20 2d 20 45  6e 69 67 6d 61 20 28 47  |Lear - Enigma (G|
00000310  69 76 65 20 41 20 42 69  74 20 6f 66 20 4d 6d 68  |ive A Bit of Mmh|
00000320  20 74 6f 20 4d 65 29 2e  6d 70 33 00 28 01 00 00  | to Me).mp3.(...|
00000330  00 00 00 00 f8 00 00 00  10 00 00 00 a9 00 00 00  |................|
00000340  b1 83 9c 3f ad de e1 fe  00 fd 00 00 00 00 00 00  |...?............|
00000350  22 36 36 5a 00 00 00 00  01 00 00 00 00 00 00 00  |"66Z............|
00000360  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

I myself cannot identify if these look like some filesystem journal entries or
packets Samba made up when trying to reply to a CIFS dirlist, but perhaps
someone else can.
I can see that the music file paths listed above look combined -- as if the
James Blunt entry overwrote the entry that was there before. My wild guess is
that this was not the first time something out of the ordinary was written to
these blocks. I had my torrent client download overnight, so parts of this so
called corruption could've just occured throughout the night. Seemed that the
kernel discovered the corruption after I started my second download in the
morning. The music files listed above were not related to the Bittorrent
download, but are just files that reside in the LVM mounted volume.

But then again, the 0x47 0x71 might be a coincidence, although when grepping for
\x47\x71\0 I get an awful lot of matches, and a lot of them have file paths
close to them. More and more it looks like a recursive dirlist ended up in the
wrong place, conveniently during my first time downloading something >5MB with
Deluge Client to my network share :)