[clug] Programming - how to use the XZ library?

[Paul Wayper]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/11/13 00:51, jan wrote:
> Paul Wayper writes:
> 
>> I'm working on trying to add support for the XZ compression format to 
>> Cfile, and I've written a test program to try the basic process before 
>> I get too complicated.
> 
>> I'm currently trying to implement things much as the xz source code 
>> does, but it's a bit of a labyrinth.
> 
> I wrote a small program that you might be able to follow.

Hi Jan,

Thanks for that.  Your code does confirm that I'm using the right things,
and basically does what the xz source code does.

There are a couple of reasons I pursued a slightly different course.  CFile
aims to be a drop-in replacement (for certain specialised values of 'in' and
'drop') for the stdio file routines - e.g. fgets => cfgets, fprintf =>
cfprintf, fflush => cfflush, etc.  Obviously, I've been spoilt by the zlib
and bzlib libraries, because they provide most of these things as well.

Most of the reading and writing in stdio and its companions works on reading
or writing arbitrary chunks of data.  What's puzzling me is that xz seems to
only work when you feed it an entire buffer full of data at once - you don't
seem to be able to feed it chunks of data of arbitrary size, such as from
printf.  My original program tried to do this, but it produced a valid xz
file that had merrily compressed bits of the input, and some other garbage.

Now, I can do my own buffering.  I already do for reading bzip2 files.  But
that raises a bunch of issues.  How much of my own buffer is enough?  What
if someone wants to fwrite a megabyte of data and my buffer is only 4K?
What if their printf would overfill the buffer?  I really don't understand
why calling lzma_code fails, on a structure with a string that I've just
read from the file and its correct length, but succeeds if I pack it all
together in a buffer.

Aside: I'm trying to work out how to share the buffer implementation between
bzip2 and xz, but it involves thorny cross-class issues that I may need more
advice on.

Anyway, thanks for helping me see that I'm at least on the path :-)

Have fun,

Paul
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.15 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlJ6NbIACgkQu7W0U8VsXYIR0ACgvbXyuqizpqUlWCsjNNRiTRBE
dqQAoKKq7I4zABLl87Bd3v/1CRqLfxHl
=5Af4
-----END PGP SIGNATURE-----