bug? -z option and large compressed data

[Wayne Davison]

On Thu, Jul 03, 2003 at 05:25:28PM +0900, Yasuoka Masahiko wrote:
> I'm Yasuoka Masahiko from Japan.  I sent 2 messages about a bug on
> token.c.

Hi, I've been tracking your patches, but have not had much of a chance
to look into this until today.  Thanks for supplying patches with your
bug report, BTW!

> In addition to, tx_strm context keeps pending output.  It must be
> flushed here.

It seems weird to me that the code was not flushing the output, but I am
worried that changing this will make us incompatible with older rsync
versions (since this data affects the compressor on each side without
actually sending any of it over the socket).  I also haven't seen any
failures using Z_INSERT_ONLY instead of Z_SYNC_FLUSH.  Did you encounter
a failure case without flushing?

> Please check below patch.

I think I'd prefer a little simpler approach to fixing this.  Here's a
patch that expands the obuf to a larger size and just uses this larger
size in this one part of the token compression code.  This avoids the
problem in your first patch where you affected too many things (by
changing the value of MAX_DATA_COUNT) and avoids adding an extra loop
as well.  In my testing this fixed a compression failure when syncing
a large iso.

..wayne..
-------------- next part --------------
--- token.c	8 Apr 2002 08:35:30 -0000	1.22
+++ token.c	3 Jul 2003 16:57:59 -0000
@@ -137,6 +137,7 @@
 
 /* Output buffer */
 static char *obuf;
+static int obuf_size;
 
 /* Send a deflated token */
 static void
@@ -160,7 +161,10 @@
 				rprintf(FERROR, "compression init failed\n");
 				exit_cleanup(RERR_STREAMIO);
 			}
-			if ((obuf = malloc(MAX_DATA_COUNT+2)) == NULL)
+			obuf_size = CHUNK_SIZE+128;
+			if (obuf_size < MAX_DATA_COUNT+2)
+				obuf_size = MAX_DATA_COUNT+2;
+			if ((obuf = malloc(obuf_size)) == NULL)
 				out_of_memory("send_deflated_token");
 			init_done = 1;
 		} else
@@ -280,7 +284,7 @@
 		tx_strm.next_in = (Bytef *) map_ptr(buf, offset, toklen);
 		tx_strm.avail_in = toklen;
 		tx_strm.next_out = (Bytef *) obuf;
-		tx_strm.avail_out = MAX_DATA_COUNT;
+		tx_strm.avail_out = obuf_size;
 		r = deflate(&tx_strm, Z_INSERT_ONLY);
 		if (r != Z_OK || tx_strm.avail_in != 0) {
 			rprintf(FERROR, "deflate on token returned %d (%d bytes left)\n",