[Bug 1463] New: poor performance with large block size

[Chris Shoemaker]

On Sat, Jul 17, 2004 at 03:54:31AM -0700, Wayne Davison wrote:
> On Fri, Jul 16, 2004 at 08:20:51PM -0400, Chris Shoemaker wrote:
> > On Thu, Jul 15, 2004 at 07:06:28PM -0700, Wayne Davison wrote:
> > > +	max_map_size = MIN(MAX_MAP_SIZE, blength * 32);
> > 
> > This makes max_map_size a multiple (32) of blength
> > for a large range  (blength*32 < MAX_MAP_SIZE),
> 
> Oops, that was supposed to be MAX(), not MIN(), otherwise it doesn't
> help the problem of too-many memory moves for large block sizes.  I'll

Sure it does (did) - in the sense that it capped the map sizes and
therefore the memmoves.  I don't think MAX is better, because it forces
large reads even for small block sizes.  That might be ok for the case
where you will walk the whole file, but otherwise it's making a single
map_ptr() pretty expensive.

> go ahead and check that change in for now and look forward to your
> findings on improving the code.
> 
> > ISTM, that the only reason to have the slight lag in window
> > advancement is you expected to frequently service requests where the
> > offset was decreasing just a little.  I didn't see that happening
> > anywhere.  Did I miss something?
> 
> I think the only place where the calls might not advance are in the
> receiver where the map_ptr() calls can be to any block-size-multiple

Agreed, but the partial-stride window advancement in map_ptr only helps
if your subsequent calls are _only_a_little_bit_ behind the current
cursor.  I haven't wrapped my mind around hash_search() yet, so I can't
say for sure, but nothing jumps out as producing that type of access
pattern.

> in the basis file.  It seems strange to me to force a 256K read for
> every basis-file match, so maybe this is something else to look into
> optimizing.

Indeed.
-chris

> 
> ..wayne..