[Bug 1463] New: poor performance with large block size

[Wayne Davison]

On Wed, Jul 14, 2004 at 06:27:45PM -0400, Chris Shoemaker wrote:
> My initial reaction (having not actually read the code) is that it would
> be desirable make the window_size highly composite, and then ensure that
> the block size is an integer factor of the window_size.  In other words,
> avoid the memmoves altogether.

I don't believe that this is possible since the offset to map_ptr() is
often incremented by a single byte.

> Besides making window_size mod len = 0, another solution could be to use
> a circular buffer.

The callers of map_ptr() currently expect to see the entire requested
range in a single buffer, so that would be a pretty big change for some
of the callers.  We can contemplate this, though.

I'll attach my version of Craig's fix that increases the MAX_MAP_SIZE
value for large block sizes.  It also caps the block size that can be
requested or computed to 256KB (which would make the map buffer in my
patched version max out at 8MB).

..wayne..
-------------- next part --------------
--- fileio.c	16 Jul 2004 01:32:02 -0000	1.13
+++ fileio.c	16 Jul 2004 01:58:34 -0000
@@ -24,6 +24,8 @@
 
 extern int sparse_files;
 
+unsigned int max_map_size = MAX_MAP_SIZE;
+
 static char last_byte;
 static int last_sparse;
 
@@ -186,7 +188,7 @@ char *map_ptr(struct map_struct *map,OFF
 	} else {
 		window_start = 0;
 	}
-	window_size = MAX_MAP_SIZE;
+	window_size = max_map_size;
 	if (window_start + window_size > map->file_size) {
 		window_size = map->file_size - window_start;
 	}
--- generator.c	15 Jul 2004 02:20:08 -0000	1.97
+++ generator.c	16 Jul 2004 01:58:34 -0000
@@ -52,6 +52,7 @@ extern int only_existing;
 extern int orig_umask;
 extern int safe_symlinks;
 extern unsigned int block_size;
+extern unsigned int max_map_size;
 
 extern struct exclude_list_struct server_exclude_list;
 
@@ -162,7 +163,9 @@ static void sum_sizes_sqroot(struct sum_
 			c >>= 1;
 		} while (c >= 8);	/* round to multiple of 8 */
 		blength = MAX(blength, BLOCK_SIZE);
+		blength = MIN(blength, MAX_MAP_SIZE);
 	}
+	max_map_size = MIN(MAX_MAP_SIZE, blength * 32);
 
 	if (protocol_version < 27) {
 		s2length = csum_length;
--- options.c	15 Jul 2004 19:06:32 -0000	1.160
+++ options.c	16 Jul 2004 01:58:34 -0000
@@ -635,6 +635,12 @@ int parse_arguments(int *argc, const cha
 	}
 #endif
 
+	if (block_size > MAX_MAP_SIZE) {
+		rprintf(FINFO, "limiting block-size to %d bytes\n",
+			MAX_MAP_SIZE);
+		block_size = MAX_MAP_SIZE;
+	}
+
 	if (write_batch && read_batch) {
 		rprintf(FERROR,
 			"write-batch and read-batch can not be used together\n");