[cifs-protocol] [EXTERNAL] MS-XCA: RTL compression is available via Python on Windows

Kristian Smith Kristian.Smith at microsoft.com
Thu Dec 29 18:21:17 UTC 2022


[DocHelp to Bcc]

Hi Douglas,

It's good to know that this functionality is accessible via Python, thanks for reaching out and updating us with this information.

Regards,
Kristian

Kristian Smith
Support Escalation Engineer
Windows Open Spec Protocols
Office: (425) 421-4442
kristian.smith at microsoft.com<mailto:kristian.smith at microsoft.com>



________________________________
From: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Sent: Tuesday, December 27, 2022 4:02 PM
To: cifs-protocol at lists.samba.org <cifs-protocol at lists.samba.org>; Interoperability Documentation Help <dochelp at microsoft.com>
Subject: [EXTERNAL] MS-XCA: RTL compression is available via Python on Windows

This is *not* a dochelp question, but a follow-up to a couple of unresolved
issues in recent MS-XCA threads that might be of interest to cifs-protocol and
perhaps Jeff and Obaid who answered the questions.

In https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.samba.org%2Farchive%2Fcifs-protocol%2F2022-October%2F003821.html&data=05%7C01%7CKristian.Smith%40microsoft.com%7Cee7b3726f5764ac78e0208dae866da06%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638077825693030172%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=T6tW9piXLcTrL8Pnfiq9jpjdea2kHZKBvl6fU9lQtBM%3D&reserved=0
("[MS-XCA] is LZ77 + Huffman the same as the Win32 compression API? -
TrackingID#2210190040006868"), I wrote of the ntifs-rtl functions:

> I haven't been able to compile and link a test utility using this API, which
> seems to be available in kernel mode only.
>
> I think it is possible, but you need to compile the thing as a driver, rather
> than a plain executable.
>
> Some of the trouble is no doubt that I have been muddling around with emacs and
> gcc in various kinds of Cygwin, MinGW, MSYS2, and WSL, rather than committing to
> learning Visual Studio and its ecosystem.

Well, it turns out you can access them through Python on Windows, using the
standard ctypes module. As attached at the end. I didn't need to compile
anything, let alone learn VS.

The idea came from
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcoderforlife%2Fms-compress%2Fblob%2Fmaster%2Ftest%2Fcompressors.py&data=05%7C01%7CKristian.Smith%40microsoft.com%7Cee7b3726f5764ac78e0208dae866da06%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638077825693030172%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SXPotHM%2Fx2wYNYevkOriGLUOlhNx5Szf9oxRU6CTCiU%3D&reserved=0.


Secondly, once I had access to RTL via Python, I looked again at a file that I
had constructed that didn't use LZ-77 matches, and which wouldn't decompress
using the compress.h API. That's the thread ending at
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.samba.org%2Farchive%2Fcifs-protocol%2F2022-November%2F003885.html&data=05%7C01%7CKristian.Smith%40microsoft.com%7Cee7b3726f5764ac78e0208dae866da06%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638077825693030172%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Wq8sIG4qVBCz2Ej%2FCa7PjDH8%2B90JvYo4ISv2JOC1I4c%3D&reserved=0
("[MS-XCA] LZ77+ Huffman: questions about blocks -
TrackingID#2211140040009096"), where Jeff says:

> Please reach out to us at our DocHelp alias if you want to give us files that won't decompress, then we can trace through and tell you why.

But with the Python RTL bindings, the file *does* decompress, so that bug is out
of scope for us all.

What follows is the Python code mentioned above.

cheers,
Douglas


-------------8<---------------------------------------------
# Generate test vectors for Windows LZ77 Huffman compression.
#
# Copyright (c) 2022 Catalyst IT
#
# GPLv3+.
#
# This uses the Python ctypes module to access the lower level RTL
# compression functions.

import sys
import argparse
from ctypes import create_string_buffer, byref, windll
from ctypes.wintypes import USHORT, ULONG, LONG, PULONG, LPVOID, CHAR
NTSTATUS = LONG


METHODS = {
     'LZNT1': 2,
     'XPRESS_PLAIN': 3,
     'XPRESS_HUFF': 4,
     '2': 2,
     '3': 3,
     '4': 4
}


class RtlError(Exception):
     pass


def ntstatus_check(status, f, args):
     # 0x117 is STATUS_BUFFER_ALL_ZEROS
     status &= 0xffffffff
     if status in (0, 0x117):
         return status
     msg = {
         0xC0000023: "buffer too small",
         0xC0000242: "bad compression data",
     }.get(status, '')

     raise RtlError(f'NTSTATUS: {status:08X} {msg}')


def wrap(f, result, *args):
     f.restype = result
     f.argtypes = args
     f.errcheck = ntstatus_check
     return f


CompressBuffer = wrap(windll.ntdll.RtlCompressBuffer, NTSTATUS,
                       USHORT, LPVOID, ULONG, LPVOID, ULONG, ULONG, PULONG,
                       LPVOID)


GetCompressionWorkSpaceSize = wrap(windll.ntdll.RtlGetCompressionWorkSpaceSize,
                                    NTSTATUS,
                                    USHORT, PULONG, PULONG)


DecompressBufferEx = wrap(windll.ntdll.RtlDecompressBufferEx,
                           NTSTATUS,
                           USHORT, LPVOID, ULONG, LPVOID, ULONG, PULONG, LPVOID)


def compress(data, format, effort=0):
     flags = USHORT(format | effort)
     workspace_size = ULONG(0)
     fragment_size = ULONG(0)
     comp_len = ULONG(0)
     GetCompressionWorkSpaceSize(flags,
                                 byref(workspace_size),
                                 byref(fragment_size))
     workspace = create_string_buffer(workspace_size.value)
     output_len = len(data) * 9 // 8 + 260
     output_buf = bytearray(output_len)
     CompressBuffer(flags,
                    (CHAR * 1).from_buffer(data), len(data),
                    (CHAR * 1).from_buffer(output_buf), output_len,
                    4096,
                    byref(comp_len),
                    workspace)
     return output_buf[:comp_len.value]


def decompress(data, format, target_size=None):
     flags = USHORT(format)
     workspace_size = ULONG(0)
     fragment_size = ULONG(0)
     decomp_len = ULONG(0)
     GetCompressionWorkSpaceSize(flags,
                                 byref(workspace_size),
                                 byref(fragment_size))
     workspace = create_string_buffer(workspace_size.value)
     if target_size is None:
         output_len = len(data) * 10
     else:
         output_len = target_size
     output_buf = bytearray(output_len)

     DecompressBufferEx(format,
                        (CHAR * 1).from_buffer(output_buf), len(output_buf),
                        (CHAR * 1).from_buffer(data), len(data),
                        byref(decomp_len),
                        workspace)
     return output_buf[:decomp_len.value]


def main():
     if sys.getwindowsversion().major < 7:
         print("this probably won't work on your very old version of Windows\n"
               "but we'll try anyway!", file=sys.stderr)

     parser = argparse.ArgumentParser()
     parser.add_argument('-d', '--decompress', action='store_true',
                         help='decompress instead of compress')
     parser.add_argument('-m', '--method', default='XPRESS_HUFF',
                         choices=list(METHODS.keys()),
                         help='use this compression method')
     parser.add_argument('-e', '--extra-effort', action='store_true',
                         help='use extra effort to compress')

     parser.add_argument('-s', '--decompressed-size', type=int,
                         help=('decompress to this size '
                               '(required for XPRESS_HUFF'))

     parser.add_argument('-o', '--output',
                         help='write to this file')
     parser.add_argument('-i', '--input',
                         help='read data from this file')

     args = parser.parse_args()

     method = METHODS[args.method]

     if all((args.decompress,
             args.decompressed_size is None,
             method == 4)):
         print("a size is required for XPRESS_HUFF decompression")
         sys.exit(1)

     with open(args.input, 'rb') as f:
         data = bytearray(f.read())

     if args.decompress:
         output = decompress(data, method, args.decompressed_size)
     else:
         effort = 1 if args.extra_effort else 0
         output = compress(data, method, effort)

     with open(args.output, 'wb') as f:
         f.write(output)


main()
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/cifs-protocol/attachments/20221229/89d36135/attachment.htm>


More information about the cifs-protocol mailing list