Is the sky actually falling? (NTFS streams) - some additional notes

[Guenter Kukkukk]

In September 2005, Andrew Bartlett was talking about some possible xattr anomalies:
http://lists.samba.org/archive/samba-technical/2005-September/043022.html
Some interestings notes by Henrik Nordstrom
http://lists.samba.org/archive/samba-technical/2005-September/043031.html

At least, anchient OS/2 is already knocking at the *nix xattr door!

For a long time now - OS/2 users have sent bug reports - that samba is not able
to handle OS/2s EAs correctly.

Beside the fact, that latest samba still has some glitches regarding EAs, the
_main_ question to ask OS/2 users first is:
  "Does the *nix file system, which samba is using, support EAs of 64 KB size?".
-----------------
IMPORTANT:
Many OS/2 users mount ext2 or ext3 (as all do) on the *nix side.
(Both support EAs up to about 4 KB!).
On the OS/2 side, you won't get an error, if _some_ EAs are smaller than 4 KB,
_but_ the next copy could hit an EA size of 50KB - which would suddenly fail!
CONCLUSION:
OS/2 users have to realize, that they _definitely must_ mount samba
shares on file systems, which support at least 64 KB EAs!
All others talks (bug reports) seem to be pointless  ... !
I think - this behaviour should be placed somewhere onto the samba website!
Btw - if I copy a file with 50KB EAs from reiserfs to ext3 (using samba3), the
OS/2 commandline displays "Access denied!" :-(
-------------------

I guess, when talking about (current) *nix file systems meta data capabilities,
some _real_ numbers could help a bit. (See table below)
The following is only addressing 'usual' xattrs - not ACLs.

Background:
  - OS/2 is using xattr heavily (anchient stuff - but still in use)
  - MS has started using "alternate data streams" (ads) in some of their latest
    servicepaks - to flag "some possibly un-secure internet downloads"...
    Virus scanning applications (partly) use ads to store some info about the
    files they have already scanned...
 ... OS/2 xattr stuff is (usually) limited to 64 KB
 ... MS alternate data streams are only limited by the file system (TERABYTES)

-----------------------------------------------------------------------------
Obviously - the following is testing only the "user." xattr name space - as
seen by a samba client:
---
Testing for OS/2 maximum extended attributes (EAs) size
with an OS/2 C-applet (tstmaxeas.exe) running against:

  - OS/2 Warpserver5.0
  - Windows 2000 server FP4
  - samba3 svn rev. 11509
    with smb.conf entry "store dos attributes = yes" and "ea support = yes",
    so that some EA space is reserved by samba3
  - samba4 svn rev. 11509

All samba3/4 tests were done on SuSE 9.1 using the latest
samba svn revisions. (5th Nov 2005)

The "Max. EA size" is the _raw_ usable EA data size.
The "Max. EA size with overhead" includes the storage
overhead - this value is usually _displayed_ as "the
file/dir EA size on OS/2".

(table best viewed with a fixed width font)
+***********+***********+**************+**************+
| Operating | filesys   | Max. EA size | Max. EA size |
|  System   |           |  (in bytes)  | w. overhead  |
+***********+***********+**************+**************+
|   OS/2    | HPFS386   |   65522      |   65535      |
| WarpSrv5  |           |              |              |
+===========+===========+==============+==============+
| Win2000   |   NTFS    |   65522      |   65535      |
| Srv FP4   |           |              |              |
+===========+===========+==============+==============+
| samba3    | reiserfs  |   lots of    |   lots of    | see Note below
| Rev 11509 | 3.6.13-24 |  Megabytes   |  Megabytes   |
+-----------+-----------+--------------+--------------+
| samba3    |   ext2    |    4040      |    4052      |
| Rev 11509 |           |              |              |
+-----------+-----------+--------------+--------------+
| samba3    |   ext3    |    4040      |    4052      |
| Rev 11509 |           |              |              |
+-----------+-----------+--------------+--------------+
| samba3    |   jfs     |  131042      |  131062      |
| Rev 11509 |           |              |              |
+-----------+-----------+--------------+--------------+
| samba3    |   xfs     |  lots of     |  lots of     |
| Rev 11509 | 2.6.3-29  | Megabytes    | Megabytes    | see Note below
+===========+===========+==============+==============+
| samba4    | reiserfs  |   65500      |   65512      |
| Rev 11509 | 3.6.13-24 |              |              |
+-----------+-----------+--------------+--------------+
| samba4    |   ext2    | Error 112    |    N/A       | see Note2
| Rev 11509 |           |   (3924)     |   (3937)     |
+-----------+-----------+--------------+--------------+
| samba4    |   ext3    | Error 112    |    N/A       | see Note2
| Rev 11509 |           |   (3924)     |   (3937)     |
+-----------+-----------+--------------+--------------+
| samba4    |   jfs     |  Error 8     |    N/A       | see Note2
| Rev 11509 |           |  (65511)     |  (65524)     |
+-----------+-----------+--------------+--------------+
| samba4    |   xfs     |   65500      |   65512      |
| Rev 11509 | 2.6.3-29  |              |              |
+-----------+-----------+--------------+--------------+

Note:
=====
Samba3 and reiserfs 3.6.13-24 and xfs 2.6.3-29:
Samba3 is storing each EA into a separate 'EA stream' - thus
allowing very huge total EA sizes on filesystems, which do allow that!
On reiserfs3.6.13-24 (and xfs) a single EA is restricted to 64KB, but
'thousands' of those EAs can be applied.
I did run a test over-night some months ago, which i stopped
in the morning, when about 36000 EAs - each 64KB size - had
been applied to a file. (It gets painfully slow after some
minutes...). (samba4s smbclient 'allinfo' did report this
huge EA size properly - running against samba3).

OS/2 itself - and windows and samba4, too - limit
the total OS/2 EA size assigned to a single file/dir to about 64KB.
Samba4 reserves some more space for internal use than samba3!
I use an OS/2 EA-Browser applet, which sometimes gets 'a bit confused
by the samba3 way', but in the real world it should not matter
too much - cause most OS/2 apps assume a total max. size of 64KB.

As often - that is only half of the story:
==========================================
My own simple OS/2 "setea" and "getea" test applets work great
with samba3s 'enhancements' - setting and getting _many_ 64KB EAs
on the OS/2 API level(!) seems to be no problem! (some tests are leftover...)
So samba3 (with the right FS) did enhance anchient OS/2! ... :-)
... but broke strict compatibility!
------------------------------------------

Note2:
======
I have _never_ seen the todays samba4 errors in the past!
The values in '( )' are the latest ones I have taken some days ago!
I guess - that errors result from some latest code changes in samba4!
(Error 112: (os/2) ERROR_DISK_FULL)
(Error 8:   (os/2) ERROR_NOT_ENOUGH_MEMORY)
Se better take the ones in '( )'.

------------------

At least - anchient stuff is hitting *nix file systems capabilities, regarding meta data.

... so that thoughs are not getting lost...

Best wishes - Guenter Kukkukk