[distcc] Re: lzo problem under HPUX

[Martin Pool]

On 10 Feb 2004, Joe Buehler <jbuehler at hekimian.com> wrote:
> Martin Pool wrote:
> 
> >Turning off mmap works in samba, or in distcc?
> 
> Turning off mmap() in distcc makes it work.
> 
> >I think it is probably the same bug.
> 
> I agree.
> 
> >I wonder if it would be safe to munmap before we ftruncate?  That
> >might be less likely to cause any problems in interaction between mmap
> >and regular IO.
> 
> I am going to look at the code -- I assume there is some sort of
> optimization that resulted in the current code that modifies the file
> after it is mapped.

Could you try this patch please?

Do you see the problem if you don't use LZO?


Index: head/src/compress.c
===================================================================
--- head.orig/src/compress.c	2003-09-22 13:15:28.000000000 +1000
+++ head/src/compress.c	2004-02-20 15:37:30.000000000 +1100
@@ -2,7 +2,7 @@
  * 
  * distcc -- A simple distributed compiler system
  *
- * Copyright (C) 2003 by Martin Pool <mbp at samba.org>
+ * Copyright (C) 2003, 2004 by Martin Pool <mbp at sourcefrog.net>
  *
  * This program is free software; you can redistribute it and/or
  * modify it under the terms of the GNU General Public License as
@@ -76,7 +76,15 @@
  * arbitrary, but basically means that debug objects and source will be
  * mmaped, and errors and stripped output will not be.  This has the
  * incidental benefit that we won't try to mmap terminals too often.
- **/
+ *
+ * It might be nice to unify this code with that in pump.c, which deals with
+ * uncompressed files.  There are some parallels between the routines.
+ * However the details are rather different, because with compressed files we
+ * do not know ahead of time how big the expanded form will be.  This affects
+ * sending, where we need to make a large-enough temporary buffer to compress
+ * into.  It also affects receipt, where we need to allow extra space for data
+ * coming in.  So for the moment they remain separate.
+ */
 
 
 /**
@@ -283,15 +291,7 @@
              (long) in_len, (long) out_len,
              (int) (out_len ? 100*in_len / out_len : 0));
     
-    if (is_mmapped) {
-        /* we over-allocated the file before; now shrink it down */
-        if (ftruncate(out_fd, out_len) == -1) {
-            rs_log_error("ftruncate fd%d failed: %s", out_fd,
-                         strerror(errno));
-            ret = EXIT_IO_ERROR;
-            goto out;
-        }
-    } else {
+    if (!is_mmapped) {
         ret = dcc_writex(out_fd, out_buf, out_len);
     }
 
@@ -299,10 +299,20 @@
     free(in_buf);
     
     if (is_mmapped) {
+        /* NOTE: We must ftruncate *after* unmapping, or HPUX's inconsistent
+         * page/buffer cache can be corrupted. */
         if (munmap(out_buf, out_size) == -1) {
             rs_log_error("munmap (output) failed: %s",
                          strerror(errno));
-            ret = ret ? ret : EXIT_IO_ERROR;
+            ret = EXIT_IO_ERROR;
+            /* try to truncate anyhow.. */
+        }
+
+        /* we over-allocated the file before; now shrink it down */
+        if (ftruncate(out_fd, out_len) == -1) {
+            rs_log_error("ftruncate fd%d failed: %s", out_fd,
+                         strerror(errno));
+            ret = EXIT_IO_ERROR;
         }
     } else if (out_buf != NULL) {
         free(out_buf);



-- 
Martin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://lists.samba.org/archive/distcc/attachments/20040220/860daae3/attachment.bin