From Volker.Lendecke at sernet.de Thu Dec 1 07:11:44 2022 From: Volker.Lendecke at sernet.de (Volker Lendecke) Date: Thu, 1 Dec 2022 08:11:44 +0100 Subject: [cifs-protocol] "Flags" field in [MS-SMB2] 2.2.2.2.1 Symbolic Link Error Response? - TrackingID#2211090040006372 In-Reply-To: References: Message-ID: Am Wed, Nov 30, 2022 at 07:30:07PM +0000 schrieb Sreekanth Nadendla: > Hello Volker, It seems the following flags were added > recently. There is no processing step from filesharing for these > now. Hope this helps. > > #define SYMLINK_ADMIN 0x20000000 // The symlink creator is an admin > #define SYMLINK_UNTRUSTED 0x10000000 // The symlink creator is untrusted > #define SYMLINK_TRUST_UNKNOWN 0x00000000 // The symlink creator is unknown/legacy > > #define SYMLINK_TRUST_MASK 0x30000000 // Encodes the redirection trust level (maps to REDIRECTION_TRUST_LEVEL) > #define SYMLINK_TRUST_SHIFT 28 // Bits to shift to convert to/from REDIRECTION_TRUST_LEVEL Oh, that helps of course! Will those flags get added to the documentation? Thanks, Volker From Volker.Lendecke at sernet.de Thu Dec 1 07:21:37 2022 From: Volker.Lendecke at sernet.de (Volker Lendecke) Date: Thu, 1 Dec 2022 08:21:37 +0100 Subject: [cifs-protocol] [EXTERNAL] Re: SMB2_CREATE error code IO_REPARSE_TAG_NOT_HANDLED? - TrackingID#2211180040005873 In-Reply-To: References: Message-ID: Am Wed, Nov 30, 2022 at 09:35:15PM +0000 schrieb Jeff McCashland (He/him): > Ha, I must have missed something when I first investigated this > issue. While I was working on the request to update the > documentation, I found where we have already documented this error: > > [MS-SMB2] > 3.3.5.9 Receiving an SMB2 CREATE Request > The status code returned by this operation MUST be one of those defined in [MS-ERREF]. > > [MS-ERREF] > 2.3.1 NTSTATUS Values > 0xC0000279 > STATUS_IO_REPARSE_TAG_NOT_HANDLED The layered file system driver for this I/O tag did not handle it when needed. This answers my initial question "Is it somewhere else defined?" of course :-) Thanks for looking!! Volker From srenaden at microsoft.com Thu Dec 1 17:20:38 2022 From: srenaden at microsoft.com (Sreekanth Nadendla) Date: Thu, 1 Dec 2022 17:20:38 +0000 Subject: [cifs-protocol] [EXTERNAL] Re: "Flags" field in [MS-SMB2] 2.2.2.2.1 Symbolic Link Error Response? - TrackingID#2211090040006372 In-Reply-To: References: Message-ID: Yes Volker. We will have them updated in our documents. Regards, Sreekanth Nadendla Microsoft Windows Open Specifications -----Original Message----- From: Volker Lendecke Sent: Thursday, December 1, 2022 2:12 AM To: Sreekanth Nadendla Cc: cifs-protocol at lists.samba.org; Microsoft Support Subject: [EXTERNAL] Re: [cifs-protocol] "Flags" field in [MS-SMB2] 2.2.2.2.1 Symbolic Link Error Response? - TrackingID#2211090040006372 Am Wed, Nov 30, 2022 at 07:30:07PM +0000 schrieb Sreekanth Nadendla: > Hello Volker, It seems the following flags were added recently. There > is no processing step from filesharing for these now. Hope this helps. > > #define SYMLINK_ADMIN 0x20000000 // The symlink creator is an admin > #define SYMLINK_UNTRUSTED 0x10000000 // The symlink creator is untrusted > #define SYMLINK_TRUST_UNKNOWN 0x00000000 // The symlink creator is unknown/legacy > > #define SYMLINK_TRUST_MASK 0x30000000 // Encodes the redirection trust level (maps to REDIRECTION_TRUST_LEVEL) > #define SYMLINK_TRUST_SHIFT 28 // Bits to shift to convert to/from REDIRECTION_TRUST_LEVEL Oh, that helps of course! Will those flags get added to the documentation? Thanks, Volker From obaidf at microsoft.com Sat Dec 3 00:10:42 2022 From: obaidf at microsoft.com (Obaid Farooqi) Date: Sat, 3 Dec 2022 00:10:42 +0000 Subject: [cifs-protocol] [EXTERNAL] Is MS-XCA LZ-77 plain compression the same as MS-DRSR DRS_COMP_ALG_WIN2K3 compression? - TrackingID#2211280040006230 In-Reply-To: References: Message-ID: Hi Douglas: The MS-DRSR uses a different API than what MS-XCA uses. Please let me know if this does not answer your question. Regards, Obaid Farooqi Escalation Engineer | Microsoft -----Original Message----- From: Tom Jebo Sent: Monday, November 28, 2022 11:02 AM To: Douglas Bagnall ; cifs-protocol at lists.samba.org Cc: Microsoft Support Subject: RE: [EXTERNAL] Is MS-XCA LZ-77 plain compression the same as MS-DRSR DRS_COMP_ALG_WIN2K3 compression? - TrackingID#2211280040006230 [dochelp to bcc] [support mail to cc] Hi Douglas, Thanks for you request regarding MS-XCA. One of the Open Specifications team members will respond to assist you. In the meantime, we?ve created case 2211280040006230 to track this request. Please leave the case number in the subject when communicating with our team about this request. Best regards, Tom Jebo Microsoft Open Specifications Support -----Original Message----- From: Douglas Bagnall Sent: Sunday, November 27, 2022 7:30 PM To: Interoperability Documentation Help ; cifs-protocol at lists.samba.org Subject: [EXTERNAL] Is MS-XCA LZ-77 plain compression the same as MS-DRSR DRS_COMP_ALG_WIN2K3 compression? hi Dochelp, Hopefully this will be simple to answer. The algorithm in MS-XCA 2.3 and 2.4 "Plain LZ77" looks quite a lot like the one in MS-DRSR 4.1.10.5.21 "CompressOrDecompressWin2k3" or "LZ77 + DIRECT2", although they are described using entirely different words. (MS-WUSP and MS-OXCRPC also have something similar). If this is *not* easy to answer, I am happy to accept a "maybe". At this stage I am just curious. cheers, Douglas From douglas.bagnall at catalyst.net.nz Sat Dec 3 00:36:43 2022 From: douglas.bagnall at catalyst.net.nz (Douglas Bagnall) Date: Sat, 3 Dec 2022 13:36:43 +1300 Subject: [cifs-protocol] [EXTERNAL] Is MS-XCA LZ-77 plain compression the same as MS-DRSR DRS_COMP_ALG_WIN2K3 compression? - TrackingID#2211280040006230 In-Reply-To: References: Message-ID: <26645e4d-5ee5-45df-04c8-84b52c4b10b4@catalyst.net.nz> hi Obaid, Yes, that answers the question, at least to "easy to answer" threshold. It looks to me like they may once have been intended to be the same but have diverged, which doesn't matter in practice because they don't talk to each other. I don't need to know any more than this now. thank you, Douglas On 3/12/22 13:10, Obaid Farooqi wrote: > Hi Douglas: > The MS-DRSR uses a different API than what MS-XCA uses. > > Please let me know if this does not answer your question. > > Regards, > Obaid Farooqi > Escalation Engineer | Microsoft > > -----Original Message----- > From: Tom Jebo > Sent: Monday, November 28, 2022 11:02 AM > To: Douglas Bagnall ; cifs-protocol at lists.samba.org > Cc: Microsoft Support > Subject: RE: [EXTERNAL] Is MS-XCA LZ-77 plain compression the same as MS-DRSR DRS_COMP_ALG_WIN2K3 compression? - TrackingID#2211280040006230 > > [dochelp to bcc] > [support mail to cc] > > Hi Douglas, > > Thanks for you request regarding MS-XCA. One of the Open Specifications team members will respond to assist you. In the meantime, we?ve created case 2211280040006230 to track this request. Please leave the case number in the subject when communicating with our team about this request. > > Best regards, > Tom Jebo > Microsoft Open Specifications Support > > -----Original Message----- > From: Douglas Bagnall > Sent: Sunday, November 27, 2022 7:30 PM > To: Interoperability Documentation Help ; cifs-protocol at lists.samba.org > Subject: [EXTERNAL] Is MS-XCA LZ-77 plain compression the same as MS-DRSR DRS_COMP_ALG_WIN2K3 compression? > > hi Dochelp, > > Hopefully this will be simple to answer. > > The algorithm in MS-XCA 2.3 and 2.4 "Plain LZ77" looks quite a lot like the one in MS-DRSR 4.1.10.5.21 "CompressOrDecompressWin2k3" or "LZ77 + DIRECT2", although they are described using entirely different words. > > (MS-WUSP and MS-OXCRPC also have something similar). > > If this is *not* easy to answer, I am happy to accept a "maybe". At this stage I am just curious. > > cheers, > Douglas > From douglas.bagnall at catalyst.net.nz Sat Dec 17 02:02:12 2022 From: douglas.bagnall at catalyst.net.nz (Douglas Bagnall) Date: Sat, 17 Dec 2022 15:02:12 +1300 Subject: [cifs-protocol] [MS-DTYP] conditional ACE SDDL sid arrays Message-ID: <42f46784-86f1-fd08-49be-244d50bff7ae@catalyst.net.nz> hi Dochelp, I am working on conditional ACES for Samba. The documentation is mostly very clear, but I have one question prompted by example 3 in 2.4.4.19, which deals with the encoding of this SDDL snippet: > (@User.clearanceLevel>=@Resource.requiredClearance) || (Member_of{SID(BA)}) where the 'Member_of{SID(BA)}' becomes a composite token containing the single SID, followed by the Member_of operator. So far this makes sense. However, earlier, in 2.4.4.17.6 ('Relational Operator Tokens') we have > The operand type MUST be either a SID literal, or a composite, each of whose > elements is a SID literal. which is also clear. But the ABNF in 2.5.1.1 ('Syntax') look like > memberof-op = ( "Member_of" / ... ) wspace sid-array and sid-array is > sid-array = literal-SID [wspace] / "{" [wspace] literal-SID [wspace] *( "," [wspace] literal-SID [wspace]) "}" so *syntactically*, this (a literal-SID without the curly brackets) (Member_of SID(BA)) would also refer to a sid-array. Thus here's the question: would this last form be compiled as a composite value (as implied by "sid-array") or would it be a solitary SID? And if doesn't result in a solitary SID, how would such a SID be represented in SDDL, or is that not possible? The wider question is whether, for valid conditonal aces, an ACE -> SDDL -> ACE cycle should always end up at the same point as the original. As a side-note, the example omits the wspace in memberof-op. I suspect the ABNF is inexact, but it might be fiddly to fix because I don't know if '[wspace]' would work for the form without {}. cheers, Douglas From jeffm at microsoft.com Sat Dec 17 04:17:08 2022 From: jeffm at microsoft.com (Jeff McCashland (He/him)) Date: Sat, 17 Dec 2022 04:17:08 +0000 Subject: [cifs-protocol] [EXTERNAL] [MS-DTYP] conditional ACE SDDL sid arrays - TrackingID#2212170040000207 In-Reply-To: <42f46784-86f1-fd08-49be-244d50bff7ae@catalyst.net.nz> References: <42f46784-86f1-fd08-49be-244d50bff7ae@catalyst.net.nz> Message-ID: [DocHelp to BCC, support on CC, SR ID on Subject] Hi Douglas, Thank you for the question. We have created SR 2212170040000207 to track this issue. One of our engineers will respond soon to assist. Best regards, Jeff McCashland (He/him) | Senior Escalation Engineer | Microsoft Protocol Open Specifications Team Phone: +1 (425) 703-8300 x38300 | Hours: 9am-5pm | Time zone: (UTC-08:00) Pacific Time (US and Canada) Local country phone number found here: http://support.microsoft.com/globalenglish | Extension 1138300 -----Original Message----- From: Douglas Bagnall Sent: Friday, December 16, 2022 6:02 PM To: Interoperability Documentation Help ; cifs-protocol at lists.samba.org Subject: [EXTERNAL] [MS-DTYP] conditional ACE SDDL sid arrays hi Dochelp, I am working on conditional ACES for Samba. The documentation is mostly very clear, but I have one question prompted by example 3 in 2.4.4.19, which deals with the encoding of this SDDL snippet: > (@User.clearanceLevel>=@Resource.requiredClearance) || > (Member_of{SID(BA)}) where the 'Member_of{SID(BA)}' becomes a composite token containing the single SID, followed by the Member_of operator. So far this makes sense. However, earlier, in 2.4.4.17.6 ('Relational Operator Tokens') we have > The operand type MUST be either a SID literal, or a composite, each of > whose elements is a SID literal. which is also clear. But the ABNF in 2.5.1.1 ('Syntax') look like > memberof-op = ( "Member_of" / ... ) wspace sid-array and sid-array is > sid-array = literal-SID [wspace] / "{" [wspace] literal-SID [wspace] *( "," [wspace] literal-SID [wspace]) "}" so *syntactically*, this (a literal-SID without the curly brackets) (Member_of SID(BA)) would also refer to a sid-array. Thus here's the question: would this last form be compiled as a composite value (as implied by "sid-array") or would it be a solitary SID? And if doesn't result in a solitary SID, how would such a SID be represented in SDDL, or is that not possible? The wider question is whether, for valid conditonal aces, an ACE -> SDDL -> ACE cycle should always end up at the same point as the original. As a side-note, the example omits the wspace in memberof-op. I suspect the ABNF is inexact, but it might be fiddly to fix because I don't know if '[wspace]' would work for the form without {}. cheers, Douglas From douglas.bagnall at catalyst.net.nz Thu Dec 22 02:16:58 2022 From: douglas.bagnall at catalyst.net.nz (Douglas Bagnall) Date: Thu, 22 Dec 2022 15:16:58 +1300 Subject: [cifs-protocol] [MS-DTYP] meaning of sign and base and range in conditional ACE integers Message-ID: <9666fa09-842e-6367-d8da-bd54810c645e@catalyst.net.nz> hi Dochelp, In MS-DTYP 2.4.4.17.5 literal integers are encoded as a 64 bit number, followed by a byte for sign and a byte for base. The range of the integer is indicated by the token bytecode. I don't understand how the sign and base are used. In the example at the bottom of section 2.4.4.17.5 a negative number is encoded with sign 'None' and base 10. What would be different in practice if it were encoded with a different base or sign? Would it compare differently? As far as I can tell, the only use of integer literal tokens is in binary relational operators. The documentation for these operators (2.4.4.17.6) says things like > MUST evaluate to TRUE if the argument on the RHS evaluates to the exact value > (single or set value) of the argument on the LHS; otherwise, FALSE. but it doesn't define how the evaluation works with the sign, base, and range. In conventional mathematics octal 03 == decimal 3 == hex 0x03. Does this hold for conditional ACE literals? Also, in many systems, the 16 bit value '123' would equal the 32 bit values '123'. Does this hold in conditional ACEs? And the sign byte -- what is that for? Does -1 with a negative sign not equal -1 with a 'none' sign? and can -1 have a positive sign? Is the base just used for determining how the number is rendered when converted into SDDL? cheers, Douglas From Kristian.Smith at microsoft.com Thu Dec 22 17:19:34 2022 From: Kristian.Smith at microsoft.com (Kristian Smith) Date: Thu, 22 Dec 2022 17:19:34 +0000 Subject: [cifs-protocol] [EXTERNAL] [MS-DTYP] meaning of sign and base and range in conditional ACE integers - TrackingID#2212220040005997 In-Reply-To: <9666fa09-842e-6367-d8da-bd54810c645e@catalyst.net.nz> References: <9666fa09-842e-6367-d8da-bd54810c645e@catalyst.net.nz> Message-ID: [DocHelp to Bcc] Hi Douglas, Thanks for reaching out to DocHelp regarding your [MS-DTYP] questions. I have created case 2212220040005997 so that we can look into these questions. An engineer will reach out soon. Thank you, Kristian Kristian Smith Support Escalation Engineer Windows Open Spec Protocols Office: (425) 421-4442 kristian.smith at microsoft.com ________________________________ From: Douglas Bagnall Sent: Wednesday, December 21, 2022 6:16 PM To: cifs-protocol at lists.samba.org ; Interoperability Documentation Help Subject: [EXTERNAL] [MS-DTYP] meaning of sign and base and range in conditional ACE integers hi Dochelp, In MS-DTYP 2.4.4.17.5 literal integers are encoded as a 64 bit number, followed by a byte for sign and a byte for base. The range of the integer is indicated by the token bytecode. I don't understand how the sign and base are used. In the example at the bottom of section 2.4.4.17.5 a negative number is encoded with sign 'None' and base 10. What would be different in practice if it were encoded with a different base or sign? Would it compare differently? As far as I can tell, the only use of integer literal tokens is in binary relational operators. The documentation for these operators (2.4.4.17.6) says things like > MUST evaluate to TRUE if the argument on the RHS evaluates to the exact value > (single or set value) of the argument on the LHS; otherwise, FALSE. but it doesn't define how the evaluation works with the sign, base, and range. In conventional mathematics octal 03 == decimal 3 == hex 0x03. Does this hold for conditional ACE literals? Also, in many systems, the 16 bit value '123' would equal the 32 bit values '123'. Does this hold in conditional ACEs? And the sign byte -- what is that for? Does -1 with a negative sign not equal -1 with a 'none' sign? and can -1 have a positive sign? Is the base just used for determining how the number is rendered when converted into SDDL? cheers, Douglas -------------- next part -------------- An HTML attachment was scrubbed... URL: From douglas.bagnall at catalyst.net.nz Wed Dec 28 00:02:34 2022 From: douglas.bagnall at catalyst.net.nz (Douglas Bagnall) Date: Wed, 28 Dec 2022 13:02:34 +1300 Subject: [cifs-protocol] MS-XCA: RTL compression is available via Python on Windows Message-ID: <3ffdd82f-60af-0c89-0bb4-53c1625e4dc0@catalyst.net.nz> This is *not* a dochelp question, but a follow-up to a couple of unresolved issues in recent MS-XCA threads that might be of interest to cifs-protocol and perhaps Jeff and Obaid who answered the questions. In https://lists.samba.org/archive/cifs-protocol/2022-October/003821.html ("[MS-XCA] is LZ77 + Huffman the same as the Win32 compression API? - TrackingID#2210190040006868"), I wrote of the ntifs-rtl functions: > I haven't been able to compile and link a test utility using this API, which > seems to be available in kernel mode only. > > I think it is possible, but you need to compile the thing as a driver, rather > than a plain executable. > > Some of the trouble is no doubt that I have been muddling around with emacs and > gcc in various kinds of Cygwin, MinGW, MSYS2, and WSL, rather than committing to > learning Visual Studio and its ecosystem. Well, it turns out you can access them through Python on Windows, using the standard ctypes module. As attached at the end. I didn't need to compile anything, let alone learn VS. The idea came from https://github.com/coderforlife/ms-compress/blob/master/test/compressors.py. Secondly, once I had access to RTL via Python, I looked again at a file that I had constructed that didn't use LZ-77 matches, and which wouldn't decompress using the compress.h API. That's the thread ending at https://lists.samba.org/archive/cifs-protocol/2022-November/003885.html ("[MS-XCA] LZ77+ Huffman: questions about blocks - TrackingID#2211140040009096"), where Jeff says: > Please reach out to us at our DocHelp alias if you want to give us files that won't decompress, then we can trace through and tell you why. But with the Python RTL bindings, the file *does* decompress, so that bug is out of scope for us all. What follows is the Python code mentioned above. cheers, Douglas -------------8<--------------------------------------------- # Generate test vectors for Windows LZ77 Huffman compression. # # Copyright (c) 2022 Catalyst IT # # GPLv3+. # # This uses the Python ctypes module to access the lower level RTL # compression functions. import sys import argparse from ctypes import create_string_buffer, byref, windll from ctypes.wintypes import USHORT, ULONG, LONG, PULONG, LPVOID, CHAR NTSTATUS = LONG METHODS = { 'LZNT1': 2, 'XPRESS_PLAIN': 3, 'XPRESS_HUFF': 4, '2': 2, '3': 3, '4': 4 } class RtlError(Exception): pass def ntstatus_check(status, f, args): # 0x117 is STATUS_BUFFER_ALL_ZEROS status &= 0xffffffff if status in (0, 0x117): return status msg = { 0xC0000023: "buffer too small", 0xC0000242: "bad compression data", }.get(status, '') raise RtlError(f'NTSTATUS: {status:08X} {msg}') def wrap(f, result, *args): f.restype = result f.argtypes = args f.errcheck = ntstatus_check return f CompressBuffer = wrap(windll.ntdll.RtlCompressBuffer, NTSTATUS, USHORT, LPVOID, ULONG, LPVOID, ULONG, ULONG, PULONG, LPVOID) GetCompressionWorkSpaceSize = wrap(windll.ntdll.RtlGetCompressionWorkSpaceSize, NTSTATUS, USHORT, PULONG, PULONG) DecompressBufferEx = wrap(windll.ntdll.RtlDecompressBufferEx, NTSTATUS, USHORT, LPVOID, ULONG, LPVOID, ULONG, PULONG, LPVOID) def compress(data, format, effort=0): flags = USHORT(format | effort) workspace_size = ULONG(0) fragment_size = ULONG(0) comp_len = ULONG(0) GetCompressionWorkSpaceSize(flags, byref(workspace_size), byref(fragment_size)) workspace = create_string_buffer(workspace_size.value) output_len = len(data) * 9 // 8 + 260 output_buf = bytearray(output_len) CompressBuffer(flags, (CHAR * 1).from_buffer(data), len(data), (CHAR * 1).from_buffer(output_buf), output_len, 4096, byref(comp_len), workspace) return output_buf[:comp_len.value] def decompress(data, format, target_size=None): flags = USHORT(format) workspace_size = ULONG(0) fragment_size = ULONG(0) decomp_len = ULONG(0) GetCompressionWorkSpaceSize(flags, byref(workspace_size), byref(fragment_size)) workspace = create_string_buffer(workspace_size.value) if target_size is None: output_len = len(data) * 10 else: output_len = target_size output_buf = bytearray(output_len) DecompressBufferEx(format, (CHAR * 1).from_buffer(output_buf), len(output_buf), (CHAR * 1).from_buffer(data), len(data), byref(decomp_len), workspace) return output_buf[:decomp_len.value] def main(): if sys.getwindowsversion().major < 7: print("this probably won't work on your very old version of Windows\n" "but we'll try anyway!", file=sys.stderr) parser = argparse.ArgumentParser() parser.add_argument('-d', '--decompress', action='store_true', help='decompress instead of compress') parser.add_argument('-m', '--method', default='XPRESS_HUFF', choices=list(METHODS.keys()), help='use this compression method') parser.add_argument('-e', '--extra-effort', action='store_true', help='use extra effort to compress') parser.add_argument('-s', '--decompressed-size', type=int, help=('decompress to this size ' '(required for XPRESS_HUFF')) parser.add_argument('-o', '--output', help='write to this file') parser.add_argument('-i', '--input', help='read data from this file') args = parser.parse_args() method = METHODS[args.method] if all((args.decompress, args.decompressed_size is None, method == 4)): print("a size is required for XPRESS_HUFF decompression") sys.exit(1) with open(args.input, 'rb') as f: data = bytearray(f.read()) if args.decompress: output = decompress(data, method, args.decompressed_size) else: effort = 1 if args.extra_effort else 0 output = compress(data, method, effort) with open(args.output, 'wb') as f: f.write(output) main() From Kristian.Smith at microsoft.com Thu Dec 29 18:21:17 2022 From: Kristian.Smith at microsoft.com (Kristian Smith) Date: Thu, 29 Dec 2022 18:21:17 +0000 Subject: [cifs-protocol] [EXTERNAL] MS-XCA: RTL compression is available via Python on Windows In-Reply-To: <3ffdd82f-60af-0c89-0bb4-53c1625e4dc0@catalyst.net.nz> References: <3ffdd82f-60af-0c89-0bb4-53c1625e4dc0@catalyst.net.nz> Message-ID: [DocHelp to Bcc] Hi Douglas, It's good to know that this functionality is accessible via Python, thanks for reaching out and updating us with this information. Regards, Kristian Kristian Smith Support Escalation Engineer Windows Open Spec Protocols Office: (425) 421-4442 kristian.smith at microsoft.com ________________________________ From: Douglas Bagnall Sent: Tuesday, December 27, 2022 4:02 PM To: cifs-protocol at lists.samba.org ; Interoperability Documentation Help Subject: [EXTERNAL] MS-XCA: RTL compression is available via Python on Windows This is *not* a dochelp question, but a follow-up to a couple of unresolved issues in recent MS-XCA threads that might be of interest to cifs-protocol and perhaps Jeff and Obaid who answered the questions. In https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.samba.org%2Farchive%2Fcifs-protocol%2F2022-October%2F003821.html&data=05%7C01%7CKristian.Smith%40microsoft.com%7Cee7b3726f5764ac78e0208dae866da06%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638077825693030172%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=T6tW9piXLcTrL8Pnfiq9jpjdea2kHZKBvl6fU9lQtBM%3D&reserved=0 ("[MS-XCA] is LZ77 + Huffman the same as the Win32 compression API? - TrackingID#2210190040006868"), I wrote of the ntifs-rtl functions: > I haven't been able to compile and link a test utility using this API, which > seems to be available in kernel mode only. > > I think it is possible, but you need to compile the thing as a driver, rather > than a plain executable. > > Some of the trouble is no doubt that I have been muddling around with emacs and > gcc in various kinds of Cygwin, MinGW, MSYS2, and WSL, rather than committing to > learning Visual Studio and its ecosystem. Well, it turns out you can access them through Python on Windows, using the standard ctypes module. As attached at the end. I didn't need to compile anything, let alone learn VS. The idea came from https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcoderforlife%2Fms-compress%2Fblob%2Fmaster%2Ftest%2Fcompressors.py&data=05%7C01%7CKristian.Smith%40microsoft.com%7Cee7b3726f5764ac78e0208dae866da06%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638077825693030172%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SXPotHM%2Fx2wYNYevkOriGLUOlhNx5Szf9oxRU6CTCiU%3D&reserved=0. Secondly, once I had access to RTL via Python, I looked again at a file that I had constructed that didn't use LZ-77 matches, and which wouldn't decompress using the compress.h API. That's the thread ending at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.samba.org%2Farchive%2Fcifs-protocol%2F2022-November%2F003885.html&data=05%7C01%7CKristian.Smith%40microsoft.com%7Cee7b3726f5764ac78e0208dae866da06%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638077825693030172%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Wq8sIG4qVBCz2Ej%2FCa7PjDH8%2B90JvYo4ISv2JOC1I4c%3D&reserved=0 ("[MS-XCA] LZ77+ Huffman: questions about blocks - TrackingID#2211140040009096"), where Jeff says: > Please reach out to us at our DocHelp alias if you want to give us files that won't decompress, then we can trace through and tell you why. But with the Python RTL bindings, the file *does* decompress, so that bug is out of scope for us all. What follows is the Python code mentioned above. cheers, Douglas -------------8<--------------------------------------------- # Generate test vectors for Windows LZ77 Huffman compression. # # Copyright (c) 2022 Catalyst IT # # GPLv3+. # # This uses the Python ctypes module to access the lower level RTL # compression functions. import sys import argparse from ctypes import create_string_buffer, byref, windll from ctypes.wintypes import USHORT, ULONG, LONG, PULONG, LPVOID, CHAR NTSTATUS = LONG METHODS = { 'LZNT1': 2, 'XPRESS_PLAIN': 3, 'XPRESS_HUFF': 4, '2': 2, '3': 3, '4': 4 } class RtlError(Exception): pass def ntstatus_check(status, f, args): # 0x117 is STATUS_BUFFER_ALL_ZEROS status &= 0xffffffff if status in (0, 0x117): return status msg = { 0xC0000023: "buffer too small", 0xC0000242: "bad compression data", }.get(status, '') raise RtlError(f'NTSTATUS: {status:08X} {msg}') def wrap(f, result, *args): f.restype = result f.argtypes = args f.errcheck = ntstatus_check return f CompressBuffer = wrap(windll.ntdll.RtlCompressBuffer, NTSTATUS, USHORT, LPVOID, ULONG, LPVOID, ULONG, ULONG, PULONG, LPVOID) GetCompressionWorkSpaceSize = wrap(windll.ntdll.RtlGetCompressionWorkSpaceSize, NTSTATUS, USHORT, PULONG, PULONG) DecompressBufferEx = wrap(windll.ntdll.RtlDecompressBufferEx, NTSTATUS, USHORT, LPVOID, ULONG, LPVOID, ULONG, PULONG, LPVOID) def compress(data, format, effort=0): flags = USHORT(format | effort) workspace_size = ULONG(0) fragment_size = ULONG(0) comp_len = ULONG(0) GetCompressionWorkSpaceSize(flags, byref(workspace_size), byref(fragment_size)) workspace = create_string_buffer(workspace_size.value) output_len = len(data) * 9 // 8 + 260 output_buf = bytearray(output_len) CompressBuffer(flags, (CHAR * 1).from_buffer(data), len(data), (CHAR * 1).from_buffer(output_buf), output_len, 4096, byref(comp_len), workspace) return output_buf[:comp_len.value] def decompress(data, format, target_size=None): flags = USHORT(format) workspace_size = ULONG(0) fragment_size = ULONG(0) decomp_len = ULONG(0) GetCompressionWorkSpaceSize(flags, byref(workspace_size), byref(fragment_size)) workspace = create_string_buffer(workspace_size.value) if target_size is None: output_len = len(data) * 10 else: output_len = target_size output_buf = bytearray(output_len) DecompressBufferEx(format, (CHAR * 1).from_buffer(output_buf), len(output_buf), (CHAR * 1).from_buffer(data), len(data), byref(decomp_len), workspace) return output_buf[:decomp_len.value] def main(): if sys.getwindowsversion().major < 7: print("this probably won't work on your very old version of Windows\n" "but we'll try anyway!", file=sys.stderr) parser = argparse.ArgumentParser() parser.add_argument('-d', '--decompress', action='store_true', help='decompress instead of compress') parser.add_argument('-m', '--method', default='XPRESS_HUFF', choices=list(METHODS.keys()), help='use this compression method') parser.add_argument('-e', '--extra-effort', action='store_true', help='use extra effort to compress') parser.add_argument('-s', '--decompressed-size', type=int, help=('decompress to this size ' '(required for XPRESS_HUFF')) parser.add_argument('-o', '--output', help='write to this file') parser.add_argument('-i', '--input', help='read data from this file') args = parser.parse_args() method = METHODS[args.method] if all((args.decompress, args.decompressed_size is None, method == 4)): print("a size is required for XPRESS_HUFF decompression") sys.exit(1) with open(args.input, 'rb') as f: data = bytearray(f.read()) if args.decompress: output = decompress(data, method, args.decompressed_size) else: effort = 1 if args.extra_effort else 0 output = compress(data, method, effort) with open(args.output, 'wb') as f: f.write(output) main() -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kristian.Smith at microsoft.com Thu Dec 29 20:12:32 2022 From: Kristian.Smith at microsoft.com (Kristian Smith) Date: Thu, 29 Dec 2022 20:12:32 +0000 Subject: [cifs-protocol] [EXTERNAL] [MS-DTYP] conditional ACE SDDL sid arrays - TrackingID#2212170040000207 In-Reply-To: References: <42f46784-86f1-fd08-49be-244d50bff7ae@catalyst.net.nz> Message-ID: Hi Douglas, I'll be looking into this issue for you. I'll reach out when I have more information. Thanks, Kristian Kristian Smith Support Escalation Engineer Windows Open Spec Protocols Office: (425) 421-4442 kristian.smith at microsoft.com ________________________________ From: Jeff McCashland (He/him) Sent: Friday, December 16, 2022 8:17 PM To: Douglas Bagnall ; cifs-protocol at lists.samba.org Cc: Microsoft Support Subject: RE: [EXTERNAL] [MS-DTYP] conditional ACE SDDL sid arrays - TrackingID#2212170040000207 [DocHelp to BCC, support on CC, SR ID on Subject] Hi Douglas, Thank you for the question. We have created SR 2212170040000207 to track this issue. One of our engineers will respond soon to assist. Best regards, Jeff McCashland (He/him) | Senior Escalation Engineer | Microsoft Protocol Open Specifications Team Phone: +1 (425) 703-8300 x38300 | Hours: 9am-5pm | Time zone: (UTC-08:00) Pacific Time (US and Canada) Local country phone number found here: https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsupport.microsoft.com%2Fglobalenglish&data=05%7C01%7CKristian.Smith%40microsoft.com%7Cbe1d030b363846bddf0608dadfe596a5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638068474415736250%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yqLDjLspa7ij01PgRGElgxnlZXy%2FJmAHok%2FTdv%2BQxWo%3D&reserved=0 | Extension 1138300 -----Original Message----- From: Douglas Bagnall Sent: Friday, December 16, 2022 6:02 PM To: Interoperability Documentation Help ; cifs-protocol at lists.samba.org Subject: [EXTERNAL] [MS-DTYP] conditional ACE SDDL sid arrays hi Dochelp, I am working on conditional ACES for Samba. The documentation is mostly very clear, but I have one question prompted by example 3 in 2.4.4.19, which deals with the encoding of this SDDL snippet: > (@User.clearanceLevel>=@Resource.requiredClearance) || > (Member_of{SID(BA)}) where the 'Member_of{SID(BA)}' becomes a composite token containing the single SID, followed by the Member_of operator. So far this makes sense. However, earlier, in 2.4.4.17.6 ('Relational Operator Tokens') we have > The operand type MUST be either a SID literal, or a composite, each of > whose elements is a SID literal. which is also clear. But the ABNF in 2.5.1.1 ('Syntax') look like > memberof-op = ( "Member_of" / ... ) wspace sid-array and sid-array is > sid-array = literal-SID [wspace] / "{" [wspace] literal-SID [wspace] *( "," [wspace] literal-SID [wspace]) "}" so *syntactically*, this (a literal-SID without the curly brackets) (Member_of SID(BA)) would also refer to a sid-array. Thus here's the question: would this last form be compiled as a composite value (as implied by "sid-array") or would it be a solitary SID? And if doesn't result in a solitary SID, how would such a SID be represented in SDDL, or is that not possible? The wider question is whether, for valid conditonal aces, an ACE -> SDDL -> ACE cycle should always end up at the same point as the original. As a side-note, the example omits the wspace in memberof-op. I suspect the ABNF is inexact, but it might be fiddly to fix because I don't know if '[wspace]' would work for the form without {}. cheers, Douglas -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kristian.Smith at microsoft.com Thu Dec 29 20:16:35 2022 From: Kristian.Smith at microsoft.com (Kristian Smith) Date: Thu, 29 Dec 2022 20:16:35 +0000 Subject: [cifs-protocol] [EXTERNAL] [MS-DTYP] meaning of sign and base and range in conditional ACE integers - TrackingID#2212220040005997 In-Reply-To: References: <9666fa09-842e-6367-d8da-bd54810c645e@catalyst.net.nz> Message-ID: Hi Douglas, I'll be looking into this issue as well. I'll reach out when I have more information. Thanks, Kristian Kristian Smith Support Escalation Engineer Windows Open Spec Protocols Office: (425) 421-4442 kristian.smith at microsoft.com ________________________________ From: Kristian Smith Sent: Thursday, December 22, 2022 9:19 AM To: Douglas Bagnall ; cifs-protocol at lists.samba.org Cc: Interoperability Documentation Help Subject: Re: [EXTERNAL] [MS-DTYP] meaning of sign and base and range in conditional ACE integers - TrackingID#2212220040005997 [DocHelp to Bcc] Hi Douglas, Thanks for reaching out to DocHelp regarding your [MS-DTYP] questions. I have created case 2212220040005997 so that we can look into these questions. An engineer will reach out soon. Thank you, Kristian Kristian Smith Support Escalation Engineer Windows Open Spec Protocols Office: (425) 421-4442 kristian.smith at microsoft.com ________________________________ From: Douglas Bagnall Sent: Wednesday, December 21, 2022 6:16 PM To: cifs-protocol at lists.samba.org ; Interoperability Documentation Help Subject: [EXTERNAL] [MS-DTYP] meaning of sign and base and range in conditional ACE integers hi Dochelp, In MS-DTYP 2.4.4.17.5 literal integers are encoded as a 64 bit number, followed by a byte for sign and a byte for base. The range of the integer is indicated by the token bytecode. I don't understand how the sign and base are used. In the example at the bottom of section 2.4.4.17.5 a negative number is encoded with sign 'None' and base 10. What would be different in practice if it were encoded with a different base or sign? Would it compare differently? As far as I can tell, the only use of integer literal tokens is in binary relational operators. The documentation for these operators (2.4.4.17.6) says things like > MUST evaluate to TRUE if the argument on the RHS evaluates to the exact value > (single or set value) of the argument on the LHS; otherwise, FALSE. but it doesn't define how the evaluation works with the sign, base, and range. In conventional mathematics octal 03 == decimal 3 == hex 0x03. Does this hold for conditional ACE literals? Also, in many systems, the 16 bit value '123' would equal the 32 bit values '123'. Does this hold in conditional ACEs? And the sign byte -- what is that for? Does -1 with a negative sign not equal -1 with a 'none' sign? and can -1 have a positive sign? Is the base just used for determining how the number is rendered when converted into SDDL? cheers, Douglas -------------- next part -------------- An HTML attachment was scrubbed... URL: